Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
fix(actions): an approved action reaches the vendor once, or visibly not at all (#1528)
Beenu Arora54deb0338003
diff --git a/.github/workflows/action-durability-live.yml b/.github/workflows/action-durability-live.ymlnew file mode 100644index 000000000..298f9c3c5--- /dev/null+++ b/.github/workflows/action-durability-live.yml@@ -0,0 +1,237 @@+# An approved action reaches the vendor once, or visibly not at all.+#+# Remediation phase 2. The approve path in services/actions read a record,+# compared its status in Python, called the vendor, and saved afterwards.+# Everything an approval is supposed to guarantee lived in that gap: two+# replicas both executed, a vendor that accepted a containment and dropped the+# response was recorded as "failed", an approver in any tenant could approve+# any tenant's containment, and a crash left a record indistinguishable from+# an action nobody started.+#+# Why this job and not a unit test+# --------------------------------+# `action_store` answers from a process-local dict, so one process hands two+# concurrent approvals the *same object* and the race cannot be observed at+# all. The decisive case needs two containers behind one database — which is+# the deployed shape — and a real Postgres, because the whole mechanism is a+# conditional UPDATE and a row lock.+#+# The vendor is a stub, and only the vendor. Authorization, persistence and+# retry are the things under test and are not replaced by anything: a mock+# that answers whatever it is asked cannot notice an action that ran twice.+#+# No `paths:` filter. This grades a governed response capability, and a path+# filter is how a required check reports success having examined nothing.+name: Action durability (live)++on:+ push:+ branches: [main]+ pull_request:+ workflow_dispatch:++concurrency:+ group: action-durability-live-${{ github.ref }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.sha }}+ cancel-in-progress: ${{ github.event_name == 'pull_request' }}++permissions:+ contents: read++jobs:+ action-durability-live:+ name: One approval, one vendor call, or an honest unknown+ runs-on: ubuntu-latest+ timeout-minutes: 25++ env:+ # The runtime role, not the owner. A superuser ignores every RLS policy+ # in the schema, and a live suite that runs as one is measuring a+ # posture nothing ships with.+ ACTIONS_DSN: postgresql://aisoc_app:aisoc_app_ci_only@localhost:5432/aisoc+ PGPASSWORD: aisoc_ci_only+ ENVIRONMENT: test+ OTEL_SDK_DISABLED: "true"++ steps:+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7++ - name: Start PostgreSQL with registry fallback+ uses: ./.github/actions/start-postgres+ with:+ password: aisoc_ci_only++ - uses: actions/setup-python@e9d6f990972a57673cdb72ec29e19d42ba28880f # v6+ with:+ python-version: "3.11"++ # Both services' own lockfiles, not a hand-curated pip list. The test+ # process imports services/api's actions client and approval dispatch;+ # the subprocesses it starts are services/actions. One interpreter runs+ # both, selected per step by `working-directory`, because each names its+ # top-level package `app` and importing one shadows the other.+ - name: Install api from its lockfile+ working-directory: services/api+ run: |+ set -euo pipefail+ python -m pip install --quiet --upgrade pip "poetry==2.4.1"+ poetry config virtualenvs.create false+ poetry install --no-root --no-interaction++ - name: Install actions from its lockfile+ working-directory: services/actions+ run: |+ set -euo pipefail+ poetry config virtualenvs.create false+ poetry install --no-root --no-interaction+ python -m pip install --quiet pytest pytest-asyncio greenlet++ - name: Apply every migration, in order+ run: |+ set -euo pipefail+ for f in services/api/migrations/*.sql; do+ psql -h localhost -U aisoc -d aisoc -v ON_ERROR_STOP=1 -q -f "$f"+ done++ - name: Give the runtime role its password+ run: |+ set -euo pipefail+ psql -h localhost -U aisoc -d aisoc -v ON_ERROR_STOP=1 -q -c \+ "ALTER ROLE aisoc_app WITH PASSWORD 'aisoc_app_ci_only';"+ psql -h localhost -U aisoc -d aisoc -v ON_ERROR_STOP=1 -q -c \+ "GRANT USAGE ON SCHEMA public TO aisoc_app;"++ - name: Identity, intent, the claim, and reconciliation+ working-directory: services/api+ env:+ ISOLATION_ACTIONS_DSN: ${{ env.ACTIONS_DSN }}+ PYTHONPATH: .+ run: python -m pytest ../../tests/isolation/test_action_durability_live.py -v --tb=short++ - name: The suite actually ran+ working-directory: services/api+ env:+ ISOLATION_ACTIONS_DSN: ${{ env.ACTIONS_DSN }}+ PYTHONPATH: .+ run: |+ set -euo pipefail+ python -m pytest ../../tests/isolation/test_action_durability_live.py -q 2>&1 | tee /tmp/durability-run.txt+ if grep -qE '[0-9]+ skipped' /tmp/durability-run.txt; then+ echo "FAIL: a case skipped in the job whose whole purpose is to run it live."+ exit 1+ fi+ if ! grep -qE '[0-9]+ passed' /tmp/durability-run.txt; then+ echo "FAIL: no case reported as passed."+ exit 1+ fi++ # ── negative controls ────────────────────────────────────────────────+ #+ # Each removes exactly one line of the fix and requires the suite to go+ # red. A gate nobody has watched fail is a gate nobody has tested.++ - name: The gate fails when the claim stops checking the status+ working-directory: services/api+ env:+ ISOLATION_ACTIONS_DSN: ${{ env.ACTIONS_DSN }}+ PYTHONPATH: .+ run: |+ set -euo pipefail+ store=../actions/app/services/action_store.py+ cp "$store" /tmp/action_store.bak+ python - <<'PY'+ import pathlib, sys+ p = pathlib.Path("../actions/app/services/action_store.py")+ s = p.read_text()+ needle = " AND status = $3\n"+ if needle not in s:+ print("negative control could not find the claim's status predicate; the claim has "+ "been reshaped and this control no longer proves anything")+ sys.exit(1)+ p.write_text(s.replace(needle, "", 1))+ PY+ set +e+ python -m pytest \+ "../../tests/isolation/test_action_durability_live.py::TestExecutionIsClaimedBeforeTheVendorIsContacted" \+ -q >/dev/null 2>&1+ rc=$?+ set -e+ cp /tmp/action_store.bak "$store"+ if [ "$rc" -eq 0 ]; then+ echo "FAIL: without the status predicate both replicas claimed and the suite stayed green."+ exit 1+ fi+ echo "OK: without the predicate both replicas execute and the suite goes red, as it must."++ - name: The gate fails when the upsert stops protecting a pending intent+ working-directory: services/api+ env:+ ISOLATION_ACTIONS_DSN: ${{ env.ACTIONS_DSN }}+ PYTHONPATH: .+ run: |+ set -euo pipefail+ store=../actions/app/services/action_store.py+ cp "$store" /tmp/action_store.bak+ python - <<'PY'+ import pathlib, sys+ p = pathlib.Path("../actions/app/services/action_store.py")+ s = p.read_text()+ needle = " WHERE aisoc_action_records.status NOT IN {_INTENT_LOCKED_SQL_LIST}\n"+ if needle not in s:+ print("negative control could not find the upsert's intent guard")+ sys.exit(1)+ p.write_text(s.replace(needle, " WHERE true\n", 1))+ PY+ set +e+ python -m pytest \+ "../../tests/isolation/test_action_durability_live.py::TestApprovalIsBoundToTheIntent" \+ -q >/dev/null 2>&1+ rc=$?+ set -e+ cp /tmp/action_store.bak "$store"+ if [ "$rc" -eq 0 ]; then+ echo "FAIL: a pending approval was retargeted and the suite stayed green."+ exit 1+ fi+ echo "OK: without the guard a pending approval is retargeted, and the suite goes red."++ # Tenant isolation on the approve path is two layers, so a one-line+ # removal proves nothing: take out the narrowing and the authorization+ # check refuses, take out the check and the narrowed read 404s. The+ # control removes both, which is also the honest statement of what is+ # being claimed — that the pair is load-bearing, not either alone.+ - name: The gate fails when nothing scopes an approval to a tenant+ working-directory: services/api+ env:+ ISOLATION_ACTIONS_DSN: ${{ env.ACTIONS_DSN }}+ PYTHONPATH: .+ run: |+ set -euo pipefail+ cp ../actions/app/security/authz.py /tmp/authz.bak+ cp ../actions/app/api/router.py /tmp/router.bak+ python - <<'PY'+ import pathlib, sys+ authz = pathlib.Path("../actions/app/security/authz.py")+ router = pathlib.Path("../actions/app/api/router.py")+ a, r = authz.read_text(), router.read_text()+ check = " if approver.tenant_id is not None and record_tenant and str(approver.tenant_id) != record_tenant:\n"+ narrow = " record = await _load_for_decision(action_id, _tenant_of(bound))\n"+ if check not in a or narrow not in r:+ print("negative control could not find both tenant layers; they have been reshaped "+ "and this control no longer proves anything")+ sys.exit(1)+ authz.write_text(a.replace(check, " if False:\n", 1))+ router.write_text(r.replace(narrow, " record = await _load_for_decision(action_id, None)\n"))+ PY+ set +e+ python -m pytest \+ "../../tests/isolation/test_action_durability_live.py::TestTheThreeIdentitiesStayDistinct::test_an_approver_from_another_tenant_is_refused" \+ -q >/dev/null 2>&1+ rc=$?+ set -e+ cp /tmp/authz.bak ../actions/app/security/authz.py+ cp /tmp/router.bak ../actions/app/api/router.py+ if [ "$rc" -eq 0 ]; then+ echo "FAIL: with neither layer present another tenant's containment was still refused."+ exit 1+ fi+ echo "OK: with neither layer present another tenant's host is contained, and the suite goes red."diff --git a/docs/remediation/phase-2-action-durability.md b/docs/remediation/phase-2-action-durability.mdnew file mode 100644index 000000000..974f87abf--- /dev/null+++ b/docs/remediation/phase-2-action-durability.md@@ -0,0 +1,483 @@+# Phase 2 — approval identity and durable action execution++`services/actions` is the only part of the platform that changes somebody+else's estate. Its approve path was a read, a status comparison in Python, the+vendor call, and a durable write afterwards:++```text+record = await action_store.get(action_id) # no tenant, cache first+if record["status"] != ActionStatus.AWAITING_APPROVAL: ... # checked in Python+result = await executor.execute(request) # the host is contained+await action_store.save(record) # ... and only now recorded+```++Every guarantee an approval is supposed to carry lived in the gap between the+third line and the fourth. This document records what was reproduced, what was+changed, how it was verified, and what is still open.++All seven findings below are **reproduced** — each was observed failing on the+tree as it stood, not inferred from reading it.++| # | Finding | Status | Regression test |+| --- | --- | --- | --- |+| 8 | No atomic claim; durable write after the vendor call | reproduced | `TestExecutionIsClaimedBeforeTheVendorIsContacted` |+| 9 | No unknown state; no `try/except` on the approve dispatch | reproduced | `TestAnUnknownOutcomeIsRecordedAsUnknown` |+| 12 | Decision and read paths carry no tenant | reproduced | `test_an_approver_from_another_tenant_is_refused`, `TestReadingAnActionRecordIsTenantScoped` |+| 13 | Unconditional upsert replaces a pending action's intent | reproduced | `TestApprovalIsBoundToTheIntent` |+| 15 | `requested_by` is caller-supplied free text | source-confirmed | — (see below) |+| 16 | Separation of duties passes silently with no requester | reproduced | `test_separation_of_duties_is_not_waived_by_an_absent_requester` |+| **17** | **The console approval path refuses every approval** | **reproduced (new)** | `test_an_agent_originated_request_is_approved_by_a_human`, `test_the_real_console_dispatch_path_approves` |++Everything lives in one live suite, `tests/isolation/test_action_durability_live.py`,+graded by `.github/workflows/action-durability-live.yml`.++---++## Finding 17 — every approval raised in the console was refused upstream++Not on the brief's list. Found by running the path rather than reading it, and+it is the most consequential item here.++**Affected production path.** `POST /api/v1/approvals/{id}/decide` →+`approvals._dispatch_decision` → `actions_client.submit_action` →+`POST /api/v1/actions` → `actions_client.decide_action` →+`POST /api/v1/actions/{id}/approve`.++**What happened.** `_dispatch_decision` built one principal — the deciding+console user — and sent it as the submit's `principal`. `router.submit_action`+records `principal.user_id` as `requested_by_user_id`. The same principal was+then sent as the approver, so `authorize_approver` compared the approver+against itself and raised *"approver cannot be the requester (separation of+duties)"*. The 403 became an `ActionsServiceError`, which `_dispatch_decision`+recorded as `{"state": "failed", "stage": "decide"}`, which surfaced to the+operator as a 502 reading *"Decision recorded, but the action service refused+it"*.++The code was aware of the distinction and solved it in the wrong field: it+passed `requested_by="agent"` with a comment saying *"Sending the approver here+would make them both, and separation of duties would pass by accident."*+`requested_by` is a free-text label that nothing compares; `principal` is what+the comparison reads.++Reproduced directly against the shipped app before anything was changed:++```+submit: 200 awaiting_approval requested_by_user_id= e84b8dfe-980c-4210-bda6-0741cea66aa8+approve: 403 {"detail":"approver cannot be the requester (separation of duties)"}+```++**Fix.** `ActionRequest` gains `requester`, the originating identity, separate+from `principal`, the invoking identity whose permissions gate the submit. The+three identities are now distinct by construction:++| Identity | Field | Used for |+| --- | --- | --- |+| who originally asked | `requester` | separation of duties, provenance |+| who relayed the call | `principal` | least-privilege check on the submit |+| who approved | `approver` on the approve call | permission tier, tenant, separation of duties |++`approvals.create_approval` stamps the authenticated principal that raised the+approval into `action["requester_principal"]` — server-written, always+overwritten, never read from the request body — and `_dispatch_decision` sends+it as `requester`. For an agent-raised approval that is the agents service's+own service principal, so the agent identity survives a human approving it;+for one a human raised it is that human, so self-approval is still refused.++**Compatibility.** `requester` is optional. When absent, `requested_by_user_id`+falls back to `principal.user_id`, which is exactly what shipped — so a caller+that submits on its own behalf is unchanged. Approvals created before this+change carry no stamp; their dispatch sends no `requester`, separation of+duties has nothing to compare, and the action records+`separation_of_duties: "unevaluated"` rather than claiming it passed.++---++## Finding 8 — two replicas could both contain the host++**Affected production path.** `POST /api/v1/actions/{id}/approve`, and the+auto-execute branch of `POST /api/v1/actions`.++**Reproduced.** Two approvals in flight against one action, from two replicas+behind one Postgres, produced **two** containment requests at the recording+vendor. On a single replica the race is invisible: `action_store.get` answered+from a module-global dict and handed both callers *the same object*, so the+status check and the mutation operated on one record. That is why a second+container is required to see it at all, and why this suite starts two.++**Fix.** `action_store.claim_for_execution` — one conditional `UPDATE` moving+`awaiting_approval → running`, scoped by id, tenant and expected status, which+writes the claim and the claiming worker into the record and returns the row:++```sql+UPDATE aisoc_action_records+ SET status = $4, record = jsonb_set(...), updated_at = now()+ WHERE id = $1 AND tenant_id = $2 AND status = $3+RETURNING record+```++Postgres takes a row lock, so a second claimant blocks until the first commits+and then re-evaluates the predicate against the committed row, finds `running`+and matches nothing. That re-evaluation is the whole mechanism; it is the same+one `scheduler_leases` (migration 115) uses, and it is why this is not a+`SELECT` followed by an `UPDATE` — that pair has no re-evaluation and elects+two executors.++The claim is also the fail-closed point. `save` still tolerates a store+failure, deliberately: losing a *pending* containment because the store blinked+is the failure the module was written to prevent, and an outcome write that+raised would discard the only record of a vendor already touched. A claim is+different — it *is* the authorization for an external effect — so it returns+`STORE_UNAVAILABLE` and the route answers 503 having contacted nobody.++Finally, the durable write now goes first and decision reads pass+`authoritative=True`, which skips the process-local cache when a DSN is+configured. That cache never expired: the submitting replica held an+`awaiting_approval` copy for the life of the process and would re-execute an+action another replica had already run.++**Compatibility.** `RUNNING` already existed in `ActionStatus` and nothing+wrote it; it is now written before every dispatch. With no DSN configured —+the single-process and unit-test configuration — the claim is an in-memory+compare-and-set under an `asyncio.Lock`, so the 1,237-case offline suite runs+unchanged.++---++## Finding 9 — a vendor that acted was recorded as having failed++**Affected production path.** The dispatch in both `approve_action` and the+auto-execute branch of `submit_action`.++**Reproduced, and the shape is not quite what the brief describes.** There was+indeed no `try/except` around `approve_action`'s `executor.execute`, and no+status meaning "unknown". But the executors catch their *own* vendor errors and+return `ActionResult(status=FAILED, error=str(exc))`, so the dominant symptom+was not a 500 — it was a record reading `failed` for a containment the EDR had+accepted. That is the dangerous direction: the obvious response to a failure is+to run it again.++**Fix, and why not exception classification.** Classifying the exception+answers "did the client raise", and the question is "did the estate change" —+for the case that matters, a vendor that accepted a containment and then+dropped the response, those two answers disagree. So a dispatch that does not+come back clean is resolved by *reading the vendor back*, using+`PostActionVerifier`, which already re-queries with the same clients the+forward action uses and already refuses to invent a `VERIFIED` it did not earn:++| Read-back | Recorded as |+| --- | --- |+| effect present | `completed` (the response was lost, not the action) |+| effect absent | `failed` |+| cannot be read | `reconciliation_required` |++`ActionStatus.RECONCILIATION_REQUIRED` is the new member. An action in it+cannot be approved again — the claim's `expected_status` does not match, and+the 409 names the reconcile route.++**One deliberate limit.** Below MEDIUM blast radius the executor's own verdict+stands and no probe runs. A reconciliation queue filled with retryable Slack+messages and tickets is one an operator learns to ignore, and a control nobody+reads is not a control. `_UNKNOWN_OUTCOME_FLOOR` in `router.py` is where that+line is drawn, and `test_a_refused_dispatch_is_still_a_failure` is the control+proving the rule does not collapse into "everything is unknown".++**The crash window in `approvals.py`** was real: `_dispatch_decision` ran at one+line and `db.commit()` at the next, so a crash in between left the approval+`pending` against an action that had run, with nothing in the row saying a+dispatch had been attempted. The decision is now committed first with+`dispatch.state = "dispatching"`, then dispatched, then committed again with+the outcome. That costs the property the old ordering protected — a window+where the row says approved and the action has not run — and makes that window+*visible*, which is the one that matters: a repeat decide re-enters on an+unresolved dispatch, and the actions service's claim is what stops a re-entry+from executing twice.++`actions_client` now distinguishes a refusal from silence: a `ConnectError`+wrote nothing to a socket, while a timeout or a dropped connection on a POST+may well have been processed. The second raises with `indeterminate=True`, and+`_dispatch_decision` records `{"state": "unknown"}` and answers **504** with+"whether the action ran is not yet known" rather than 502 "the action did not+run".++**Compatibility.** `reconciliation_required` is a new status string. Checked+against every consumer:++* `services/api`'s usage meter keys `EXECUTED_STATUS` on `"completed"`, so an+ unresolved action is correctly *not* counted as a vendor touch.+* `services/slack-bot`'s `commands.py` tests for+ `{"awaiting_approval", "pending_approval", "pending"}` to decide whether to+ post an approval card; anything else renders as a status line, which is+ right.+* `packages/types`' `AlertAction.status` union is a console-side concept that+ already does not mirror `ActionStatus` (it has no `awaiting_approval`,+ `approved` or `rejected`, and its `type` union contains verbs `ActionType`+ has never had), so it is not a consumer of this enum.+* The status column is written through `_column_text`, so the new member lands+ as `reconciliation_required` and not `ActionStatus.RECONCILIATION_REQUIRED` —+ the defect migration 095 had to repair.++---++## Finding 12 — an id was a capability across tenants++**Affected production path.** `approve_action`, `reject_action` and+`get_action`.++**Reproduced.** An approver holding `actions:execute:high` in a freshly-made+tenant approved another tenant's `isolate_host` from the second replica, and+the host was contained. `GET /api/v1/actions/{id}` returned the full record —+target, parameters and all — to any holder of the shared service token.++`action_store.get`'s docstring said a tenant "should always be supplied from a+request path"; no request path supplied one. Its claim that the table "carries+no RLS policy" is **imprecise**: migration 060 does create+`aisoc_action_records_tenant`. The consequence stated is still right, for a+different reason — the policy reads `app.current_tenant_id`, which nothing in+this service sets, and its unset arm is deliberately permissive so cross-tenant+workers are not starved. The policy is inert here, so the query predicate is+the only control.++**Fix.** Two layers on the decision paths:++1. The deciding identity is resolved **before** the record is read+ (`_resolve_identity`), and the read is narrowed by its tenant. Fetching+ globally and comparing in Python is the same defect one line later.+2. `authorize_approver` compares the approver's tenant to the action's, which+ `authorize_action` has done since it was written and only the approve side+ went without.++The claim carries `AND tenant_id = $2` as well, so the write cannot land+cross-tenant even if both reads were wrong.++On `get_action`: a caller declaring `X-Tenant-ID` gets the record when it owns+it and **404** when it does not — never 403, which would make the route an+existence oracle over every tenant's action ids. A caller declaring none keeps+working and is answered with status only (`_UNSCOPED_READ_FIELDS`), so the+route stays usable for polling without handing out the host an action was+aimed at. The header is read off `request.headers` rather than declared as a+`Header(...)` parameter, because one such parameter adds a parameter and a 422+response to the published OpenAPI document for every operation it guards.++**Compatibility.** No caller of `GET /api/v1/actions/{id}` exists anywhere in+the tree — `services/slack-bot` and `services/api` call only `/approve` and+`/reject` — so the redaction breaks nothing today and the full record remains+one header away. `docs/openapi.yaml` is byte-identical after the change+(`scripts/export_openapi.py --check`: *"docs/openapi.yaml is up to date"*).++---++## Finding 13 — a pending approval could be retargeted++**Affected production path.** `POST /api/v1/actions` with an id that already+exists.++**Reproduced.** Submitting `ws-4471`, then the same id with `target: "dc-01"`,+left the record naming `dc-01` while the approval card said `ws-4471`.++**Fix.** The upsert's `DO UPDATE` is now conditional, comparing the three+fields that define what an approver signed off:++```sql+ WHERE aisoc_action_records.status NOT IN ('awaiting_approval','running','reconciliation_required')+ OR ( record->>'action_type' IS NOT DISTINCT FROM EXCLUDED.record->>'action_type'+ AND record->>'target' IS NOT DISTINCT FROM EXCLUDED.record->>'target'+ AND record->'parameters' IS NOT DISTINCT FROM EXCLUDED.record->'parameters' )+RETURNING id+```++`RETURNING id` is how the refusal is detected: an `ON CONFLICT DO UPDATE` whose+`WHERE` is unsatisfied returns no row, which is the only signal Postgres gives.+`save` raises `ActionIntentConflict` and the route answers 409.++`rationale` and `confidence` are deliberately **not** part of the intent: the+first is prose shown to a human, the second is re-graded by the gate on every+submission, and neither changes what happens to the estate.++**Compatibility.** The store's standing contract — *"a replay of the same+submission is idempotent rather than a duplicate pending approval"* — is+preserved and pinned by `test_an_identical_resubmission_is_still_idempotent`.+Only a *changed* intent is refused, and only while a decision is outstanding.++---++## Finding 16 — separation of duties passed on the requests it could not evaluate++`if separation_of_duties and requester and str(requester) == str(approver.user_id)`+— the `and requester` arm meant a submission carrying no principal produced an+approval the control reported as satisfied, and the row afterwards was+indistinguishable from one where it had been checked and passed.++**Fix.** `authorize_approver` returns which state applied and the route records+it on the action as `separation_of_duties: "enforced" | "unevaluated"`.++Refusing instead was considered and rejected: `AISOC_ACTIONS_REQUIRE_PRINCIPAL`+defaults false, and a hard denial would brick every legacy system-initiated+action during an incident. The honest fix is that it must *be a fact in the+row*, not an omission.++## Finding 15 — `requested_by` is caller-supplied free text++**Source-confirmed, deliberately not "fixed" by trusting it.** It remains a+label. Separation of duties reads `requester` / `principal`, which arrive from+authenticated context, and the fix for finding 17 is what makes the originating+identity trustworthy rather than cosmetic. Using `requested_by` to *refuse* an+approval was considered and rejected: a caller controlling that string could+name an analyst and block them from approving during an incident, which trades+a non-bypass for a denial of service on the containment path.++---++## No migration++`aisoc_action_records` needs no schema change, and that is a decision rather+than an omission:++* The claim's predicates are the existing `id`, `tenant_id` and `status`+ columns, and `ix_action_records_tenant_status` already serves the operator's+ `status = 'running'` scan.+* Dispatch provenance (`dispatch.claimed_by`, `dispatch.claimed_at`,+ `verification`, `approved_at`) lives in the JSONB record, which is the+ argument migration 055 makes for storing it whole: *"mirroring it as columns+ here would make every contract change a migration"*.+* The intent comparison is a JSONB expression, so there is no second definition+ of "the intent" to drift from the Python one and no backfill to get wrong.+* Migration 061 already grants the runtime role DML on every table, and 060+ already carries the RLS policy.++## Recovery and reconciliation — what an operator runs++### 1. Find actions that were claimed and never resolved++```sql+SELECT id, tenant_id, status, updated_at,+ record->'dispatch'->>'claimed_by' AS claimed_by,+ record->'dispatch'->>'claimed_at' AS claimed_at,+ record->>'action_type' AS action_type,+ record->>'target' AS target+ FROM aisoc_action_records+ WHERE status IN ('running', 'reconciliation_required')+ AND updated_at < now() - interval '10 minutes'+ ORDER BY updated_at;+```++`running` means a worker took the action and has not written an outcome — it+either died mid-dispatch or is still inside a vendor call.+`reconciliation_required` means it finished and could not establish what+happened. `claimed_by` is `hostname/pid`, which is how you find the container.++**A stale claim is never stolen automatically.** A timeout that reassigns a+claim is exactly the duplicate execution this work exists to prevent: the+previous holder may have reached the vendor. Resolution is always the probe+below.++### 2. Reconcile by asking the vendor++```bash+curl -sS -X POST "$ACTIONS_URL/api/v1/actions/$ACTION_ID/reconcile" \+ -H "Authorization: Bearer $AISOC_ACTIONS_SERVICE_TOKEN" \+ -H "X-Tenant-ID: $TENANT_ID" \+ -H 'Content-Type: application/json' \+ -d '{"parameters": {"cs_client_id": "...", "cs_client_secret": "..."}}'+```++`parameters` is merged over the stored ones and is how you present a current+credential rather than depending on whatever was captured at submit time. The+route **never re-executes** — it runs the read-back probe only.++| Response `status` | Meaning | Next step |+| --- | --- | --- |+| `completed` | the effect is in force; the response was lost, not the action | nothing |+| `failed` | the vendor was read and the effect is absent | submit a **new action id** if it is still wanted |+| `reconciliation_required` | the vendor could not be read | fix the credential or vendor reachability and run it again; check by hand if it stays |++`verification.reason` on the returned record says which probe ran and what it+saw. A verb with no registered probe reports+`no read-back verifier for <verb>`, which stays `reconciliation_required` — the+honest answer, and never a guess in either direction.++### 3. Never do this++* Do not re-approve. The claim refuses it (409) and the refusal is the point.+* Do not hand-edit `status` to `completed` without a probe that said so — that+ is the fabricated-verdict shape the project disavows everywhere else.+* Do not resubmit the **same** action id with different parameters: the upsert+ refuses it (409) while a decision is outstanding, by design.++## Verification performed++Against PostgreSQL 16 with all 135 migrations applied, replicas connecting as+the runtime role `aisoc_app` (`NOBYPASSRLS`, not the owner), two actions+containers on loopback sockets, and a recording HTTP vendor.++| Check | Result |+| --- | --- |+| `tests/isolation/test_action_durability_live.py` **before** the fix | 20 failed, 7 passed |+| the same suite **after** | **27 passed** |+| `services/actions` offline suite | 1,237 passed (baseline 1,234 + 3 new) |+| `services/api` offline suite | 4,964 passed, 164 skipped |+| `ruff check services/ scripts/ tests/ tools/` | All checks passed |+| `ruff format --check .` (0.16.10, the pin) | 2,596 files already formatted |+| `scripts/check_mypy_baseline.py` | OK — no tree got worse |+| `scripts/export_openapi.py --check` | docs/openapi.yaml is up to date |+| `check_tenant_query_predicates`, `check_raw_sql_columns`, `check_route_authz`, `check_action_contract`, `check_attribution`, `check_competitor_names`, `check_machine_paths`, `check_gate_coverage`, `check_text_bind_casts`, `check_comment_paths`, `check_logger_kwargs` | all pass |++### Negative controls++Each removes one piece of the fix and requires the suite to go red. All three+were executed, not asserted, and are wired into the workflow.++| Control | Result |+| --- | --- |+| drop `AND status = $3` from the claim | 3 of 4 cases fail, including both replicas containing the host |+| make the upsert unconditional (`WHERE true`) | the pending approval is retargeted to `dc-01` — 409 becomes 200 |+| remove both tenant layers | tenant A's approver contains tenant B's host and is returned the full record |++The third removes two lines rather than one deliberately: tenant isolation here+is the narrowed read *plus* the authorization check, and removing either alone+leaves the other refusing, so a one-line control would prove nothing. Saying+which pair is load-bearing is the accurate claim.++### Two test expectations were corrected after seeing the implementation++Written against a weaker design, both now assert a stronger property, and both+were re-run against the pre-fix tree to confirm they still fail there (they are+in the 20).++* A lost vendor response was expected to produce `reconciliation_required`. The+ implementation reads the device back, finds it contained, and records+ `completed` — which is better, and the test now asserts that plus a separate+ case where the device API is down.+* "No probe available" was expressed by sending empty `parameters`, which the+ reconcile route merges over the stored ones, so the probe ran. It now makes+ the vendor's read-back route unavailable instead.++## Remaining limitations++* **No hosted vendor has been exercised.** The forward call and the read-back+ both go to a stub on a loopback socket. That proves the client constructs,+ authenticates, and that the two are distinct routes; it does not prove any+ vendor's real API behaves as modelled.+* **Vendor-side idempotency keys are not used**, because none of the clients in+ `services/actions/app/clients/` accepts one. The duplicate-execution+ guarantee is therefore entirely ours — the claim — rather than ours plus the+ vendor's. Adding `Idempotency-Key` per client is tracked work, not done here.+* **The unknown-outcome rule has a floor.** Below MEDIUM blast radius an+ unverifiable failure is still recorded as `failed`. Stated above with its+ reasoning; it is a judgement, not an oversight.+* **Reconciliation is operator-driven.** There is no worker sweeping stale+ claims, by choice: an automatic sweep that resolved an unreadable probe would+ be guessing, and one that re-dispatched would duplicate. The SQL above is the+ queue.+* **`chatops_callback` still mutates the record without saving it.** Reached+ only by the ChatOps verification link, pre-existing, and outside this phase's+ scope — but it means a user's reply updates the in-process copy and not the+ durable row on any other replica.+* **The API-side crash window is narrowed, not closed.** A crash between the+ first commit and the dispatch leaves `dispatch.state = "dispatching"`, which+ a repeat decide re-enters safely. Nothing automatically re-enters it; a+ stalled approval needs a human or a client retry.+* **`services/api`'s own route layer is exercised by its offline suite, not+ here.** The live suite drives `_dispatch_decision` with a stand-in ORM row+ and principal, because the question it answers is what crosses the wire to+ the actions service. The session, auth dependency and ORM behind+ `decide_approval` are covered by `services/api/tests/`.diff --git a/services/actions/app/api/router.py b/services/actions/app/api/router.pyindex b87dca05c..0c512d455 100644--- a/services/actions/app/api/router.py+++ b/services/actions/app/api/router.py@@ -4,23 +4,30 @@ Action Execution Service REST API.from __future__ import annotations+import os+import socketfrom datetime import UTC, datetimefrom typing import Anyfrom uuid import UUIDimport structlog-from fastapi import APIRouter, Depends, HTTPException, Query+from fastapi import APIRouter, Depends, HTTPException, Query, Requestfrom fastapi.responses import HTMLResponse+from pydantic import BaseModel, Fieldfrom app.core.config import get_settingsfrom app.models.action import (+ ACTION_BLAST_RADIUS,ActionPrincipal,ActionRequest,+ ActionResult,ActionStatus,ActionType,+ BlastRadius,ChatOpsApprover,)from app.security.authz import (+ SOD_UNEVALUATED,ActionAuthzError,authorize_action,authorize_approver,@@ -29,15 +36,39 @@ from app.security.authz import (from app.security.chatops_identity import resolve_approverfrom app.security.chatops_token import ChatOpsTokenError, verify_tokenfrom app.services import action_store+from app.services.action_store import ActionIntentConflict, ActionStoreUnavailable, ClaimOutcomefrom app.services.approval_gate import apply_matrixfrom app.services.blast_radius import BlastRadiusGatefrom app.services.executor_registry import EXECUTOR_REGISTRYfrom app.services.timeline_client import TimelineClientError, post_timeline_event+from app.services.verification import PostActionVerifier, VerificationOutcomelogger = structlog.get_logger()router = APIRouter()gate = BlastRadiusGate()+#: Which replica claimed an execution. A hostname and a pid: enough for an+#: operator holding a stuck `running` record to find the container that took+#: it, and nothing a caller supplies.+_WORKER_ID = f"{socket.gethostname()}/{os.getpid()}"++#: The header a service-to-service caller declares its tenant on.+#:+#: Read off the raw request rather than declared as a `Header(...)` parameter:+#: one such parameter on a route rewrites the published OpenAPI document with+#: a parameter and a 422 response, and a tenant that services exchange between+#: themselves does not belong in the public contract.+_TENANT_HEADER = "X-Tenant-ID"++#: Below this, an unverifiable failure is recorded as a failure.+#:+#: The rule above it is "unknown until the vendor says so", which is right for+#: anything that changes an estate. Applied to a Slack message or a ticket it+#: would fill the reconciliation queue with items whose duplicate costs+#: nothing — and a queue an operator learns to ignore is not a control. So+#: MINIMAL and LOW keep the executor's own verdict, and MEDIUM upward do not.+_UNKNOWN_OUTCOME_FLOOR: frozenset[BlastRadius] = frozenset({BlastRadius.MEDIUM, BlastRadius.HIGH, BlastRadius.CRITICAL})+# The module-global `_actions` dict that used to live here is gone; storage is# `app.services.action_store`, which persists to Postgres when a DSN is# configured. A restart used to lose every action awaiting approval, so an@@ -50,6 +81,127 @@ gate = BlastRadiusGate()_chatops_replied: set[str] = set()+def _requester_id(request: ActionRequest) -> str | None:+ """Who asked for this action, for separation of duties.++ The explicit originator when the caller supplied one, otherwise the+ invoking principal — which is what this has always recorded, and is still+ right for a caller that submits on its own behalf.++ Never ``request.requested_by``: that is a free-text label a caller picks,+ and a separation-of-duties comparison against it would be forgeable from+ the request body.+ """+ if request.requester is not None:+ return request.requester.user_id+ if request.principal is not None:+ return request.principal.user_id+ return None+++async def _verify_effect(request: ActionRequest) -> tuple[VerificationOutcome, dict[str, Any]]:+ """Ask the vendor whether the action's effect is actually present.++ ``PostActionVerifier`` already re-queries the vendor with the same clients+ the forward action uses, and already refuses to invent a ``VERIFIED`` it+ did not earn. It is used here rather than classifying the exception,+ because an exception type answers "did the client raise" and the question+ is "did the estate change" — and for the case that matters, a vendor that+ accepted a containment and then dropped the response, those two answers+ disagree.+ """+ verdict = await PostActionVerifier().verify(request.action_type, request.target, dict(request.parameters or {}))+ return verdict.outcome, {+ "outcome": verdict.outcome.value,+ "reason": verdict.reason,+ "vendor": verdict.vendor,+ "checked_at": datetime.now(UTC).isoformat(),+ }+++async def _apply_outcome(+ record: dict[str, Any],+ request: ActionRequest,+ result: ActionResult | None,+ error: str | None,+) -> None:+ """Turn a dispatch into a status the record can be trusted on.++ A clean success is a success. Anything else is unknown until the vendor is+ read back: the executors catch their own vendor errors and return+ ``ActionResult(status=FAILED, error=str(exc))``, so a containment the EDR+ had accepted was recorded as ``failed`` — the reading whose obvious remedy+ is to run it again.+ """+ if result is not None and result.status == ActionStatus.COMPLETED:+ record["status"] = ActionStatus.COMPLETED+ record["output"] = result.output+ record["rollback_data"] = result.rollback_data+ record.pop("error", None)+ return++ reported = error or (result.error if result is not None else None) or "the executor reported no outcome"+ if result is not None and result.output:+ record["output"] = result.output++ blast = ACTION_BLAST_RADIUS.get(request.action_type, BlastRadius.HIGH)+ if blast not in _UNKNOWN_OUTCOME_FLOOR:+ record["status"] = ActionStatus.FAILED+ record["error"] = reported+ return++ outcome, verification = await _verify_effect(request)+ record["verification"] = verification+ if outcome is VerificationOutcome.VERIFIED:+ # The dispatch reported a problem and the effect is in force anyway:+ # the response was lost, not the action.+ record["status"] = ActionStatus.COMPLETED+ record["dispatch_error"] = reported+ record.pop("error", None)+ return+ if outcome is VerificationOutcome.FAILED:+ record["status"] = ActionStatus.FAILED+ record["error"] = reported+ return+ record["status"] = ActionStatus.RECONCILIATION_REQUIRED+ record["error"] = reported+++async def _dispatch_claimed(record: dict[str, Any], request: ActionRequest) -> dict[str, Any]:+ """Run an action whose execution this process has already claimed.++ The claim is the durable authorization for the vendor call, so by the time+ control reaches here the record says ``running`` in Postgres and no other+ replica, retry or approval can start the same action. What is left is to+ run it and write down what happened — including when "what happened" is+ that nobody knows.+ """+ executor = EXECUTOR_REGISTRY.get(request.action_type)+ if executor is None:+ record["status"] = ActionStatus.FAILED+ record["error"] = f"No executor found for action type: {request.action_type}"+ else:+ try:+ result = await executor.execute(request)+ await _apply_outcome(record, request, result, None)+ except Exception as exc: # noqa: BLE001 — never leave a claimed action without an outcome+ # The message is logged in full and never echoed: it can carry+ # internal hostnames and stack detail.+ logger.error("Action execution failed", action_id=str(request.id), error=str(exc), exc_info=True)+ await _apply_outcome(record, request, None, f"execution failed ({type(exc).__name__})")++ dispatch = dict(record.get("dispatch") or {})+ dispatch["state"] = "resolved"+ dispatch["resolved_at"] = datetime.now(UTC).isoformat()+ record["dispatch"] = dispatch++ try:+ await action_store.save(record)+ except ActionIntentConflict: # pragma: no cover - the intent cannot change under a claim+ logger.error("action_store.outcome_write_refused", action_id=str(request.id))+ return record++@router.post("/actions", response_model=dict)async def submit_action(request: ActionRequest, _auth: None = Depends(require_service_auth)):"""Submit an action for execution (may require approval)."""@@ -84,8 +236,10 @@ async def submit_action(request: ActionRequest, _auth: None = Depends(require_se"tenant_id": str(request.tenant_id),"rationale": request.rationale,# W4.4 — remember who requested it so an approver can't approve their- # own action (separation of duties).- "requested_by_user_id": request.principal.user_id if request.principal else None,+ # own action (separation of duties). The originator when the caller+ # distinguishes one, because a relaying service sending its own+ # principal here makes the requester and the approver the same person.+ "requested_by_user_id": _requester_id(request),# Kept so approve can rebuild the request faithfully. Without this the# approve path reconstructed an ActionRequest with an empty# `parameters`, so an action that was gated *because* of what it@@ -93,32 +247,40 @@ async def submit_action(request: ActionRequest, _auth: None = Depends(require_se# approval that silently changes what it approved is not an approval."parameters": dict(request.parameters or {}),}- await action_store.save(record)+ try:+ await action_store.save(record)+ except ActionIntentConflict as exc:+ # A second submission of an id that is already awaiting a decision, with+ # a different target or parameters. Refused rather than applied: an+ # approval card showing one host while the record names another is an+ # approval of something nobody read.+ raise HTTPException(+ status_code=409,+ detail=("this action id is already awaiting a decision on a different target or parameters; submit a new action id"),+ ) from exc# Auto-execute if approvedif status == ActionStatus.APPROVED:- executor = EXECUTOR_REGISTRY.get(request.action_type)- if executor:- try:- result = await executor.execute(request)- record["status"] = result.status- record["output"] = result.output- record["rollback_data"] = result.rollback_data- if result.error:- record["error"] = result.error- except Exception as exc:- # Log the full detail server-side, but never echo the raw- # exception message (which can carry internal paths / stack- # detail) back to the API caller — return only the error type.- logger.error("Action execution failed", error=str(exc), exc_info=True)- record["status"] = ActionStatus.FAILED- record["error"] = f"execution failed ({type(exc).__name__})"+ claim = await action_store.claim_for_execution(+ str(request.id),+ tenant_id=str(request.tenant_id),+ expected_status=ActionStatus.APPROVED.value,+ claimed_by=_WORKER_ID,+ )+ if claim.ok and claim.record is not None:+ record = await _dispatch_claimed(claim.record, request)+ elif claim.outcome is ClaimOutcome.STORE_UNAVAILABLE:+ # The gate said this may run unattended and the store cannot record+ # that it did. Refuse: an external effect nobody can prove was+ # authorized is worse than an action an operator can resubmit.+ raise HTTPException(+ status_code=503,+ detail="the action store is unavailable, so execution was not authorized; retry when it is reachable",+ )else:- record["status"] = ActionStatus.FAILED- record["error"] = f"No executor found for action type: {request.action_type}"-- # Persist again so the executed status is durable, not just the pending one.- await action_store.save(record)+ # Another replica took it, or it is no longer in a state that can+ # run. Report what the store holds rather than executing.+ record = claim.record or recordlogger.info("Action submitted",@@ -155,6 +317,33 @@ def _bind_approver(human at all when it comes from the approval-timeout scheduler — requiringone would leave expired requests stuck in ``awaiting_approval`` forever."""+ approver = _resolve_identity(action_id, approver, assertion, require=require)+ if approver is None:+ return None++ try:+ record["separation_of_duties"] = authorize_approver(record, approver)+ except ActionAuthzError as exc:+ logger.warning("Approval denied", action_id=action_id, reason=str(exc))+ raise HTTPException(status_code=403, detail=str(exc)) from exc+ return approver+++def _resolve_identity(+ action_id: str,+ approver: ActionPrincipal | None,+ assertion: ChatOpsApprover | None,+ *,+ require: bool,+) -> ActionPrincipal | None:+ """The identity half of :func:`_bind_approver`, with no record in hand.++ Split out so the route can learn the deciding principal's tenant *before*+ it reads the action, and narrow the read by it. Fetching globally and+ comparing afterwards leaves the lookup itself unscoped, which on a table+ whose row-level policy is inert on this service's connections means the id+ is the only control.+ """if approver is None and assertion is not None:approver = resolve_approver(assertion.platform, assertion.platform_user_id)if approver is None:@@ -171,13 +360,40 @@ def _bind_approver(if require and (settings.AISOC_ACTIONS_REQUIRE_APPROVER or settings.AISOC_ACTIONS_REQUIRE_PRINCIPAL):raise HTTPException(status_code=403, detail="an approver identity is required")return None+ return approver++def _tenant_of(principal: ActionPrincipal | None) -> str | None:+ """The tenant to narrow a read by, or None when the identity carries none.++ A ChatOps approver comes from operator configuration and may legitimately+ have no tenant; the narrowing is then skipped and+ :func:`authorize_approver` is the control. The tenant is never taken from+ the request body.+ """+ if principal is None or principal.tenant_id is None:+ return None+ return str(principal.tenant_id)+++async def _load_for_decision(action_id: str, tenant_id: str | None) -> dict[str, Any]:+ """Read the record a decision will act on, from the database.++ ``authoritative`` because the process-local cache is what let an approve+ on the submitting replica resurrect an action another replica had already+ executed, and because a store that is configured and unreachable must not+ be reported as "no such action".+ """try:- authorize_approver(record, approver)- except ActionAuthzError as exc:- logger.warning("Approval denied", action_id=action_id, reason=str(exc))- raise HTTPException(status_code=403, detail=str(exc)) from exc- return approver+ record = await action_store.get(action_id, tenant_id=tenant_id, authoritative=True)+ except ActionStoreUnavailable as exc:+ raise HTTPException(+ status_code=503,+ detail="the action store is unavailable; the decision was not recorded and nothing was executed",+ ) from exc+ if not record:+ raise HTTPException(status_code=404, detail="Action not found")+ return record@router.post("/actions/{action_id}/approve")@@ -194,47 +410,103 @@ async def approve_action(no resolvable identity is refused by default(``AISOC_ACTIONS_REQUIRE_APPROVER``), because separation of duties cannotbe evaluated against nobody. ChatOps callers send ``chatops_approver``- (verified platform identity) and the mapping supplies the permissions."""- record = await action_store.get(action_id)- if not record:- raise HTTPException(status_code=404, detail="Action not found")- if record["status"] != ActionStatus.AWAITING_APPROVAL:- raise HTTPException(status_code=400, detail=f"Action is not awaiting approval (current: {record['status']})")+ (verified platform identity) and the mapping supplies the permissions.++ Phase 2 — the order below is the control. Identity, then a tenant-narrowed+ authoritative read, then authorization, then an **atomic claim in durable+ storage**, and only then the vendor. This used to be a read, a status+ comparison in Python, the vendor call, and a save afterwards: two replicas+ both read ``awaiting_approval`` and both contained the host, and if the+ process died in between there was no record that either had tried.+ """+ bound = _resolve_identity(action_id, approver, chatops_approver, require=True)+ record = await _load_for_decision(action_id, _tenant_of(bound))- bound = _bind_approver(record, action_id, approver, chatops_approver, require=True)if bound is not None:+ _bind_approver(record, action_id, bound, None, require=True)record["approved_by_user_id"] = bound.user_id+ record["approved_at"] = datetime.now(UTC).isoformat()+ else:+ record["separation_of_duties"] = SOD_UNEVALUATED++ if record["status"] != ActionStatus.AWAITING_APPROVAL:+ raise HTTPException(status_code=400, detail=f"Action is not awaiting approval (current: {record['status']})")++ claim = await action_store.claim_for_execution(+ action_id,+ tenant_id=str(record["tenant_id"]),+ expected_status=ActionStatus.AWAITING_APPROVAL.value,+ claimed_by=_WORKER_ID,+ )+ if not claim.ok:+ raise _claim_refusal(action_id, claim)++ # The claimed record, read back from the store, carries the approval+ # stamps above only if they were written before the claim — they were not,+ # so re-apply them to the authoritative copy rather than to the one this+ # process happened to read.+ claimed = claim.record or record+ claimed["approved_by_user_id"] = record.get("approved_by_user_id")+ claimed["approved_at"] = record.get("approved_at")+ claimed["separation_of_duties"] = record.get("separation_of_duties", SOD_UNEVALUATED)# Reconstruct the request as submitted, parameters included. Dropping# them turned an approved action into a different action that still# reported success.request = ActionRequest(id=UUID(action_id),- incident_id=UUID(record["incident_id"]),- tenant_id=UUID(record["tenant_id"]),- action_type=ActionType(record["action_type"]),- target=record["target"],- rationale=record["rationale"],- parameters=dict(record.get("parameters") or {}),+ incident_id=UUID(claimed["incident_id"]),+ tenant_id=UUID(claimed["tenant_id"]),+ action_type=ActionType(claimed["action_type"]),+ target=claimed["target"],+ rationale=claimed.get("rationale") or "",+ parameters=dict(claimed.get("parameters") or {}),# The bound approver, not the original requester: the executor's# authorization has to evaluate the identity that authorised this run.principal=bound,)- executor = EXECUTOR_REGISTRY.get(request.action_type)- if executor:- result = await executor.execute(request)- record["status"] = result.status- record["output"] = result.output- else:- record["status"] = ActionStatus.FAILED- record["error"] = "No executor available"-- await action_store.save(record)- logger.info("Action approved and executed", action_id=action_id, status=record["status"])+ record = await _dispatch_claimed(claimed, request)+ logger.info(+ "Action approved and executed",+ action_id=action_id,+ status=record["status"],+ approved_by=bound.user_id if bound else None,+ )return record+def _claim_refusal(action_id: str, claim: action_store.Claim) -> HTTPException:+ """The refusal a losing or blocked claimant gets, by reason.++ Three different operator actions hide behind "the claim did not take", so+ they are three different answers: another worker has it (wait), the action+ is in a state no approval can move (reconcile or resubmit), or the store+ cannot be written (retry).+ """+ if claim.outcome is ClaimOutcome.STORE_UNAVAILABLE:+ logger.error("approval_refused_store_unavailable", action_id=action_id)+ return HTTPException(+ status_code=503,+ detail="the action store is unavailable, so execution was not authorized; nothing was sent to the vendor",+ )+ if claim.outcome is ClaimOutcome.NOT_FOUND:+ return HTTPException(status_code=404, detail="Action not found")++ current = claim.current_status or "unknown"+ if current == ActionStatus.RUNNING.value:+ detail = "this action is already being executed by another worker"+ elif current == ActionStatus.RECONCILIATION_REQUIRED.value:+ detail = (+ "this action's outcome is unknown and it will not be run again; "+ f"reconcile it with POST /api/v1/actions/{action_id}/reconcile before deciding anything further"+ )+ else:+ detail = f"Action is not awaiting approval (current: {current})"+ logger.info("approval_refused", action_id=action_id, current_status=current)+ return HTTPException(status_code=409, detail=detail)++@router.post("/actions/{action_id}/reject")async def reject_action(action_id: str,@@ -248,37 +520,166 @@ async def reject_action(to take no identity and record none, so "who declined to contain thishost, and were they entitled to" had no answer — which matters as much asthe approve side during an incident review.++ A rejection is a state transition like any other, so it is claimed rather+ than written over whatever is there: rejecting an action a worker is in+ the middle of executing would record a decision that did not happen."""- record = await action_store.get(action_id)- if not record:- raise HTTPException(status_code=404, detail="Action not found")+ bound = _resolve_identity(action_id, approver, chatops_approver, require=False)+ record = await _load_for_decision(action_id, _tenant_of(bound))- bound = _bind_approver(record, action_id, approver, chatops_approver, require=False)if bound is not None:+ _bind_approver(record, action_id, bound, None, require=False)record["rejected_by_user_id"] = bound.user_id- record["status"] = ActionStatus.REJECTED- await action_store.save(record)+ if record["status"] == ActionStatus.REJECTED:+ return record++ claim = await action_store.claim_for_execution(+ action_id,+ tenant_id=str(record["tenant_id"]),+ expected_status=ActionStatus.AWAITING_APPROVAL.value,+ claimed_by=_WORKER_ID,+ running_status=ActionStatus.REJECTED.value,+ )+ if not claim.ok:+ raise _claim_refusal(action_id, claim)++ rejected = claim.record or record+ rejected["status"] = ActionStatus.REJECTED+ rejected["rejected_by_user_id"] = record.get("rejected_by_user_id")+ rejected["rejected_at"] = datetime.now(UTC).isoformat()+ try:+ await action_store.save(rejected)+ except ActionIntentConflict: # pragma: no cover - the intent is unchanged here+ logger.error("action_store.rejection_write_refused", action_id=action_id)logger.info("Action rejected",action_id=action_id,rejected_by=bound.user_id if bound else None,)- return record+ return rejected+++#: What an unscoped read is allowed to see.+#:+#: A caller that does not declare a tenant still gets a usable status poll,+#: and does not get the host or account the action was aimed at, the+#: parameters it ran with, or the identities around it.+_UNSCOPED_READ_FIELDS: tuple[str, ...] = (+ "id",+ "status",+ "action_type",+ "blast_radius",+ "approval_tier",+ "tenant_id",+)@router.get("/actions/{action_id}")-async def get_action(action_id: str, _auth: None = Depends(require_service_auth)):+async def get_action(action_id: str, request: Request, _auth: None = Depends(require_service_auth)):"""Get action status and result.Behind the service guard: an action record names the host or account anaction was aimed at and the parameters it ran with, and ``GET`` by id was- reachable with no credential at all. ``aisoc_action_records`` carries no- RLS policy, so the id was the only thing standing in front of it.+ reachable with no credential at all.++ The service token is shared, and ``aisoc_action_records``' row-level+ policy is inert on this service's connections — it reads a GUC nothing+ here sets — so inside the trust boundary an id was still a capability over+ every tenant's response history. A caller that declares its tenant on+ ``X-Tenant-ID`` gets the record when it owns it and a 404 when it does+ not; never a 403, which would make the route an existence oracle over+ every tenant's action ids. A caller that declares none keeps working and+ is answered with status only.+ """+ tenant_id = (request.headers.get(_TENANT_HEADER) or "").strip() or None+ try:+ record = await action_store.get(action_id, tenant_id=tenant_id, authoritative=True)+ except ActionStoreUnavailable as exc:+ raise HTTPException(status_code=503, detail="the action store is unavailable") from exc+ if not record:+ raise HTTPException(status_code=404, detail="Action not found")+ if tenant_id is None:+ logger.info("action_read_unscoped", action_id=action_id)+ return {key: record.get(key) for key in _UNSCOPED_READ_FIELDS if key in record}+ return record+++class ReconcileRequest(BaseModel):+ """Credentials for the read-back, supplied per call.++ Not taken from the stored record alone on purpose: an operator reconciling+ days later should be able to present a current credential rather than+ depend on whatever was captured at submit time.+ """++ parameters: dict[str, Any] = Field(default_factory=dict)++[email protected]("/actions/{action_id}/reconcile")+async def reconcile_action(+ action_id: str,+ request: Request,+ body: ReconcileRequest | None = None,+ _auth: None = Depends(require_service_auth),+):+ """Resolve an action whose outcome was never established, by asking the vendor.++ The recovery path for the two states a dispatch can strand in: ``running``,+ where the worker that claimed it died before writing an outcome, and+ ``reconciliation_required``, where it finished and could not tell what had+ happened.++ It never re-executes. The reason those states exist is that the vendor may+ already have acted, and running the action again is the one response that+ turns an unknown into a real duplicate. What it does is the read-back+ probe: present means it completed, absent means it genuinely failed,+ unreadable means it stays unknown and a human has to look."""- record = await action_store.get(action_id)+ tenant_id = (request.headers.get(_TENANT_HEADER) or "").strip() or None+ try:+ record = await action_store.get(action_id, tenant_id=tenant_id, authoritative=True)+ except ActionStoreUnavailable as exc:+ raise HTTPException(status_code=503, detail="the action store is unavailable") from excif not record:raise HTTPException(status_code=404, detail="Action not found")++ current = str(getattr(record.get("status"), "value", record.get("status")))+ if current not in (ActionStatus.RUNNING.value, ActionStatus.RECONCILIATION_REQUIRED.value):+ raise HTTPException(+ status_code=409,+ detail=f"only a claimed or unresolved action can be reconciled (current: {current})",+ )++ parameters = {**(record.get("parameters") or {}), **((body.parameters if body else None) or {})}+ probe_request = ActionRequest(+ id=UUID(action_id),+ incident_id=UUID(record["incident_id"]),+ tenant_id=UUID(record["tenant_id"]),+ action_type=ActionType(record["action_type"]),+ target=record["target"],+ rationale=record.get("rationale") or "",+ parameters=parameters,+ )++ outcome, verification = await _verify_effect(probe_request)+ record["verification"] = verification+ record["reconciled_at"] = datetime.now(UTC).isoformat()+ if outcome is VerificationOutcome.VERIFIED:+ record["status"] = ActionStatus.COMPLETED+ record.pop("error", None)+ elif outcome is VerificationOutcome.FAILED:+ record["status"] = ActionStatus.FAILED+ record.setdefault("error", "the effect was not present when the vendor was read back")+ else:+ record["status"] = ActionStatus.RECONCILIATION_REQUIRED++ try:+ await action_store.save(record)+ except ActionIntentConflict: # pragma: no cover - reconciliation changes no intent+ logger.error("action_store.reconcile_write_refused", action_id=action_id)+ logger.info("action_reconciled", action_id=action_id, status=record["status"], outcome=outcome.value)return recorddiff --git a/services/actions/app/models/action.py b/services/actions/app/models/action.pyindex 3ce585bf8..4e1530981 100644--- a/services/actions/app/models/action.py+++ b/services/actions/app/models/action.py@@ -108,9 +108,24 @@ class ActionStatus(str, Enum):AWAITING_APPROVAL = "awaiting_approval"APPROVED = "approved"REJECTED = "rejected"+ #: Claimed for execution. Written to the durable store *before* the vendor+ #: is contacted and never only afterwards, so a replica that dies+ #: mid-dispatch leaves a record that cannot be mistaken for one nobody+ #: started. The member always existed; nothing wrote it.RUNNING = "running"COMPLETED = "completed"FAILED = "failed"+ #: The dispatch did not come back clean and the vendor could not be read+ #: back, so whether the estate changed is genuinely unknown.+ #:+ #: Its absence was the defect. A containment the EDR had accepted, whose+ #: response was then lost, was recorded as ``failed`` — the reading whose+ #: obvious remedy is to run it again. Nothing here may be re-approved;+ #: ``POST /actions/{id}/reconcile`` resolves it by asking the vendor.+ #:+ #: Not a terminal success: `services/api`'s usage meter counts `completed`+ #: as the one status meaning a vendor was touched, and that stays true.+ RECONCILIATION_REQUIRED = "reconciliation_required"ROLLED_BACK = "rolled_back"@@ -251,6 +266,23 @@ class ActionRequest(BaseModel):# W4.1 — structured invoking identity. Optional for back-compat with the# legacy `requested_by` string + system-initiated calls.principal: ActionPrincipal | None = None+ #: Who originally asked for this action, when that is not the caller.+ #:+ #: Three identities meet on an approved action and collapsing any two+ #: breaks a control. `principal` is the *invoking* identity whose+ #: permissions gate the submit; the approver arrives separately on the+ #: approve call; and this is the originator, read only for separation of+ #: duties and provenance, never for authorization.+ #:+ #: It exists because `services/api` had nowhere to put the originator and+ #: sent the deciding console user as `principal`, which became+ #: `requested_by_user_id`. Separation of duties then compared the approver+ #: against itself and refused: every approval raised in the responder app+ #: was answered 403 upstream and surfaced as a 502. The free-text+ #: `requested_by` beside this is a label and is deliberately not used for+ #: the comparison — it is caller-supplied and trusting it would make the+ #: control forgeable in the other direction.+ requester: ActionPrincipal | None = Nonerationale: str = ""auto_rollback: bool = Falserollback_after_seconds: int | None = Nonediff --git a/services/actions/app/security/authz.py b/services/actions/app/security/authz.pyindex a7dc7566b..e754b97ec 100644--- a/services/actions/app/security/authz.py+++ b/services/actions/app/security/authz.py@@ -81,24 +81,55 @@ def authorize_action(request: ActionRequest, *, require_principal: bool | None =raise ActionAuthzError(f"principal {principal.user_id} lacks '{required}' required for {request.action_type.value}")+#: Recorded on the action when separation of duties had nothing to compare.+#:+#: ``if separation_of_duties and requester and ...`` passed silently whenever+#: the submission carried no principal, which is possible while+#: ``AISOC_ACTIONS_REQUIRE_PRINCIPAL`` is false. The control reported itself+#: satisfied on exactly the requests it could not evaluate, and the record+#: afterwards was indistinguishable from one where it had been checked and+#: passed. Refusing instead would brick every legacy system-initiated action+#: during an incident, which is why this is a recorded fact rather than a+#: denial — but it has to *be* a fact, in the row, not an omission.+SOD_UNEVALUATED = "unevaluated"+SOD_ENFORCED = "enforced"++def authorize_approver(record: dict[str, Any],approver: ActionPrincipal,*,separation_of_duties: bool = True,-) -> None:+) -> str:"""Enforce that an approver may approve this action (W4.4).- The approver must hold the action's required permission and — under- separation of duties — must not be the principal who requested it."""+ Three conditions, and the tenant one was missing. An approver had to hold+ the action's permission tier and not be its requester, but nothing compared+ their tenant to the action's — so a principal holding+ ``actions:execute:high`` in *any* tenant could approve any other tenant's+ containment, against a store where an id was the only other control.+ :func:`authorize_action` had carried the same comparison since it was+ written; only the approve side went without.++ Returns which separation-of-duties state applied, for the caller to record.+ """action_type = ActionType(record["action_type"])required = _required_permission(action_type)if not has_action_permission(approver.permissions, required):raise ActionAuthzError(f"approver {approver.user_id} lacks '{required}' to approve {action_type.value}")+ record_tenant = str(record.get("tenant_id") or "").strip()+ if approver.tenant_id is not None and record_tenant and str(approver.tenant_id) != record_tenant:+ raise ActionAuthzError("approver tenant does not match the action's tenant")+requester = record.get("requested_by_user_id")- if separation_of_duties and requester and str(requester) == str(approver.user_id):+ if not separation_of_duties:+ return SOD_UNEVALUATED+ if not requester:+ return SOD_UNEVALUATED+ if str(requester) == str(approver.user_id):raise ActionAuthzError("approver cannot be the requester (separation of duties)")+ return SOD_ENFORCEDasync def require_service_auth(authorization: str | None = Header(default=None)) -> None:diff --git a/services/actions/app/services/action_store.py b/services/actions/app/services/action_store.pyindex b9dac1774..14ce8c5d7 100644--- a/services/actions/app/services/action_store.py+++ b/services/actions/app/services/action_store.py@@ -18,12 +18,49 @@ run without one would push every test onto a mock of itself.Writes are upserts on the action id, so a replay of the same submission isidempotent rather than a duplicate pending approval.++What a store has to guarantee before a vendor is contacted+----------------------------------------------------------+Durability was only half of it. The approve path read a record, checked its+status in Python, called the vendor and saved afterwards, which left three+gaps that an upsert alone cannot close:++* **Nobody claimed the execution.** Two replicas, or one replica and a client+ that retried, both read ``awaiting_approval`` and both contained the host.+ :func:`claim_for_execution` is one conditional ``UPDATE`` — Postgres takes a+ row lock, so a second claimant blocks, re-evaluates the predicate against the+ committed row, finds ``running`` and matches nothing. Same mechanism as+ ``scheduler_leases`` (migration 115), and for the same reason: a+ SELECT-then-UPDATE has no such re-evaluation.+* **The cache outranked the database.** ``get`` answered from ``_MEMORY``+ first, so the replica that handled the submission held an+ ``awaiting_approval`` copy for the life of the process and would execute+ again whatever the database had since recorded. Reads on a decision path ask+ for ``authoritative=True``, which skips the cache when a DSN is configured+ and *raises* rather than reporting a reachable store's outage as a missing+ action.+* **Intent was mutable under the approver.** ``ON CONFLICT (id) DO UPDATE SET+ record = EXCLUDED.record`` with no condition meant a second submission of the+ same id could retarget an action a human was in the middle of approving. The+ upsert now refuses to change the action type, target or parameters of a+ record that is awaiting approval or running, and says so.++Failures are not swallowed on those paths. ``save`` keeps its tolerant+behaviour — losing a *pending* action because the store blinked is the failure+this module was written to prevent, and an outcome write that raised would lose+the one record of a vendor that has already been touched — but a claim refuses+rather than proceeding, because that write is the authorization for an external+effect."""from __future__ import annotations+import asyncioimport jsonimport os+from dataclasses import dataclass+from datetime import UTC, datetime+from enum import Enumfrom typing import Anyimport structlog@@ -33,6 +70,80 @@ logger = structlog.get_logger()#: Process-local fallback, and the store the unit suite exercises._MEMORY: dict[str, dict[str, Any]] = {}+#: Serialises the in-memory claim. Only load-bearing when no DSN is configured;+#: with one, Postgres' row lock is the serialisation point and this lock is+#: held across no I/O.+_MEMORY_CLAIM_LOCK = asyncio.Lock()+++class ActionStoreUnavailable(RuntimeError):+ """A durable store is configured and could not be read or written.++ Distinct from "no such action" on purpose. Reporting a reachable store's+ outage as a 404 sends an operator to look for an action that exists, and+ reporting it as success would let an external effect proceed with no+ record of the authorization behind it.+ """+++class ActionIntentConflict(RuntimeError):+ """A submission tried to change what a pending action would do.++ Carries the stored intent so the caller can say what the approver is+ actually looking at rather than only that the write was refused.+ """++ def __init__(self, action_id: str, stored: dict[str, Any]) -> None:+ super().__init__(f"action {action_id} is already awaiting a decision on a different target or parameters")+ self.action_id = action_id+ self.stored = stored+++class ClaimOutcome(str, Enum):+ """Why a claim did or did not take.++ Every value except ``CLAIMED`` means *do not contact the vendor*, and they+ are kept apart because they need different answers: a losing replica is a+ 409, an action in an unknown state needs reconciliation rather than a+ retry, and an unwritable store is a 503 the caller may repeat.+ """++ CLAIMED = "claimed"+ NOT_FOUND = "not_found"+ WRONG_STATE = "wrong_state"+ STORE_UNAVAILABLE = "store_unavailable"+++@dataclass(frozen=True)+class Claim:+ outcome: ClaimOutcome+ #: The record as it stands *after* a successful claim, read back from the+ #: database rather than reconstructed, so what executes is what was+ #: claimed. ``None`` when the claim did not take and there is no row.+ record: dict[str, Any] | None = None+ #: The status that blocked the claim, for the caller's message.+ current_status: str | None = None++ @property+ def ok(self) -> bool:+ return self.outcome is ClaimOutcome.CLAIMED+++#: The fields that define *what an approver signed off*. A change to any of+#: them is a different action, whatever the id says.+#:+#: ``rationale`` is deliberately out: it is the explanation shown to a human+#: and an agent re-running its triage may word it differently without changing+#: what would happen to the estate. ``confidence`` is out for the same reason —+#: it is re-graded by the gate on every submission.+INTENT_FIELDS: tuple[str, ...] = ("action_type", "target", "parameters")+++def intent_of(record: dict[str, Any]) -> dict[str, Any]:+ """The immutable part of an action, for comparison and for an error."""+ return {field: record.get(field) for field in INTENT_FIELDS}++#: Set once a write has actually reached Postgres.#:#: This existed as a flag nothing read, which made the comment above it a@@ -81,30 +192,73 @@ def _column_text(value: Any) -> str:return str(getattr(value, "value", value))+#: Statuses whose intent an approver may already be looking at.+#:+#: Spelled as literals rather than imported from ``app.models.action`` because+#: this module is the storage layer and the column holds text; importing the+#: enum here would make the SQL depend on the executor contract, which is the+#: coupling 055 avoided by storing the record whole.+_INTENT_LOCKED_STATUSES = ("awaiting_approval", "running", "reconciliation_required")++#: The same tuple as a SQL list. Built rather than written twice so the two+#: cannot drift, and quoted here because these are fixed identifiers from a+#: closed set, never request data.+_INTENT_LOCKED_SQL_LIST = "(" + ", ".join(f"'{status}'" for status in _INTENT_LOCKED_STATUSES) + ")"++#: Refuse an intent change while a decision is outstanding, allow every other+#: update, and tell the caller which happened.+#:+#: ``RETURNING id`` is how: an ``ON CONFLICT DO UPDATE`` whose ``WHERE`` is not+#: satisfied returns no row, which is the only signal Postgres gives that the+#: upsert declined. ``IS NOT DISTINCT FROM`` rather than ``=`` because a NULL+#: target must compare equal to itself.+_UPSERT_SQL = f"""+ INSERT INTO aisoc_action_records (id, tenant_id, status, approval_tier, record, created_at, updated_at)+ VALUES ($1, $2, $3, $4, $5, now(), now())+ ON CONFLICT (id) DO UPDATE+ SET status = EXCLUDED.status,+ approval_tier = EXCLUDED.approval_tier,+ record = EXCLUDED.record,+ updated_at = now()+ WHERE aisoc_action_records.status NOT IN {_INTENT_LOCKED_SQL_LIST}+ OR (+ aisoc_action_records.record ->> 'action_type' IS NOT DISTINCT FROM EXCLUDED.record ->> 'action_type'+ AND aisoc_action_records.record ->> 'target' IS NOT DISTINCT FROM EXCLUDED.record ->> 'target'+ AND aisoc_action_records.record -> 'parameters' IS NOT DISTINCT FROM EXCLUDED.record -> 'parameters'+ )+ RETURNING id+"""++async def save(record: dict[str, Any]) -> None:- """Persist or update an action record."""+ """Persist or update an action record.++ Raises :class:`ActionIntentConflict` when the write would change what a+ pending action does. Everything else is still tolerated and logged: the+ whole reason this module exists is that losing a pending containment to a+ store blink is worse than running degraded, and an outcome write that+ raised would discard the only record of a vendor already touched.++ The durable write goes first. It used to be second, so a refused or failed+ write left the process-local copy holding a record the database had never+ accepted — and ``get`` preferred that copy.+ """action_id = str(record["id"])- _MEMORY[action_id] = recorddsn = _dsn()if dsn is None:+ _MEMORY[action_id] = recordreturn+try:conn = await _connect(dsn)except Exception as exc: # noqa: BLE001 — an unreachable store must not drop the action_report_degraded("connect_failed", action_id, exc)+ _MEMORY[action_id] = recordreturntry:- await conn.execute(- """- INSERT INTO aisoc_action_records (id, tenant_id, status, approval_tier, record, created_at, updated_at)- VALUES ($1, $2, $3, $4, $5, now(), now())- ON CONFLICT (id) DO UPDATE- SET status = EXCLUDED.status,- approval_tier = EXCLUDED.approval_tier,- record = EXCLUDED.record,- updated_at = now()- """,+ written = await conn.fetchval(+ _UPSERT_SQL,action_id,_column_text(record.get("tenant_id")),_column_text(record.get("status")),@@ -113,11 +267,26 @@ async def save(record: dict[str, Any]) -> None:)global _DB_CONFIRMED_DB_CONFIRMED = True+ if written is None:+ stored = await _fetch_record(conn, action_id, None)+ logger.warning(+ "action_store.intent_change_refused",+ action_id=action_id,+ stored_intent=intent_of(stored or {}),+ submitted_intent=intent_of(record),+ )+ raise ActionIntentConflict(action_id, intent_of(stored or {}))+ except ActionIntentConflict:+ raiseexcept Exception as exc: # noqa: BLE001 — see above_report_degraded("write_failed", action_id, exc)+ _MEMORY[action_id] = record+ returnfinally:await conn.close()+ _MEMORY[action_id] = record+def _report_degraded(event: str, action_id: str, exc: Exception) -> None:"""Log a store failure at the severity its consequence deserves.@@ -142,57 +311,179 @@ def _report_degraded(event: str, action_id: str, exc: Exception) -> None:)-async def get(action_id: str, tenant_id: str | None = None) -> dict[str, Any] | None:- """Fetch an action record, preferring the durable copy.-- The in-memory copy is checked first only as a cache: on the replica that- handled the submission it is the same object, and on any other replica it- is absent and the database answers.-- ``tenant_id`` narrows the lookup to one tenant and should always be- supplied from a request path. ``aisoc_action_records`` carries no RLS- policy, so without it an id is a capability: an action record names the- host or account an action was aimed at, and any caller who could guess or- observe a UUID would read another tenant's response history.+async def _fetch_record(conn: Any, action_id: str, tenant_id: str | None) -> dict[str, Any] | None:+ """One tenant-narrowed read, on a connection the caller owns."""+ if tenant_id is None:+ row = await conn.fetchrow("SELECT record FROM aisoc_action_records WHERE id = $1", action_id)+ else:+ row = await conn.fetchrow(+ "SELECT record FROM aisoc_action_records WHERE id = $1 AND tenant_id = $2",+ action_id,+ str(tenant_id),+ )+ return json.loads(row["record"]) if row is not None else None+++async def get(action_id: str, tenant_id: str | None = None, *, authoritative: bool = False) -> dict[str, Any] | None:+ """Fetch an action record.++ ``tenant_id`` narrows the lookup to one tenant and should be supplied from+ a request path wherever the caller has an authenticated identity to take+ it from. ``aisoc_action_records``' RLS policy is inert on this service's+ connections — it reads ``app.current_tenant_id``, which nothing here sets,+ and its unset arm is deliberately permissive so cross-tenant workers are+ not starved — so without a predicate an id is a capability: an action+ record names the host or account an action was aimed at.++ ``authoritative`` is for the decision paths. The in-memory copy is a cache,+ and on the replica that handled the submission it is a cache that never+ expires: it held ``awaiting_approval`` for the life of the process, so an+ approve on that replica would execute an action another replica had already+ run. An authoritative read skips it when a DSN is configured and raises+ :class:`ActionStoreUnavailable` rather than answering "no such action" for+ a store that is merely unreachable."""- cached = _MEMORY.get(action_id)- if cached is not None:- if tenant_id is not None and str(cached.get("tenant_id") or "") != str(tenant_id):+ dsn = _dsn()++ if dsn is None:+ cached = _MEMORY.get(action_id)+ if cached is not None and tenant_id is not None and str(cached.get("tenant_id") or "") != str(tenant_id):return Nonereturn cached- dsn = _dsn()- if dsn is None:- return None+ if not authoritative:+ cached = _MEMORY.get(action_id)+ if cached is not None:+ if tenant_id is not None and str(cached.get("tenant_id") or "") != str(tenant_id):+ return None+ return cached+try:conn = await _connect(dsn)except Exception as exc: # noqa: BLE001logger.warning("action_store.connect_failed", action_id=action_id, error=str(exc))+ if authoritative:+ raise ActionStoreUnavailable("the action store is configured and unreachable") from excreturn Nonetry:- if tenant_id is None:- row = await conn.fetchrow("SELECT record FROM aisoc_action_records WHERE id = $1", action_id)- else:- row = await conn.fetchrow(- "SELECT record FROM aisoc_action_records WHERE id = $1 AND tenant_id = $2",- action_id,- tenant_id,- )+ record = await _fetch_record(conn, action_id, tenant_id)except Exception as exc: # noqa: BLE001logger.warning("action_store.read_failed", action_id=action_id, error=str(exc))+ if authoritative:+ raise ActionStoreUnavailable("the action store is configured and could not be read") from excreturn Nonefinally:await conn.close()- if row is None:+ if record is None:+ # Drop any cached copy: the authoritative answer is that this tenant+ # has no such action, and leaving a stale one behind is how the cache+ # outranked the database in the first place.+ if authoritative:+ _MEMORY.pop(action_id, None)return None- record = json.loads(row["record"])- # Cache it so the approve path's subsequent mutation and save see one- # object rather than two diverging copies._MEMORY[action_id] = recordreturn record+async def claim_for_execution(+ action_id: str,+ *,+ tenant_id: str,+ expected_status: str,+ claimed_by: str,+ running_status: str = "running",+) -> Claim:+ """Take exclusive ownership of an action's execution, durably, or refuse.++ One conditional ``UPDATE``. Postgres locks the row, so a second claimant+ blocks until the first commits, then re-evaluates ``status =+ $expected_status`` against the committed row, finds ``running`` and matches+ nothing. That re-evaluation is the whole mechanism and it is why this is+ not a ``SELECT`` followed by an ``UPDATE``: that pair has no re-evaluation+ and elects two executors, which here means contacting a vendor twice.++ The tenant is a predicate rather than a check afterwards. A handler that+ fetches globally and compares in Python is the same defect one line later.++ An unreachable or unwritable store returns ``STORE_UNAVAILABLE`` and the+ caller must not proceed: this write *is* the authorization for an external+ effect, and an effect nobody can prove was authorized is worse than a+ refused approval an operator can repeat.+ """+ claimed_at = datetime.now(UTC).isoformat()+ dsn = _dsn()++ if dsn is None:+ async with _MEMORY_CLAIM_LOCK:+ record = _MEMORY.get(action_id)+ if record is None or str(record.get("tenant_id") or "") != str(tenant_id):+ return Claim(ClaimOutcome.NOT_FOUND)+ current = _column_text(record.get("status"))+ if current != expected_status:+ return Claim(ClaimOutcome.WRONG_STATE, current_status=current)+ record["status"] = running_status+ record["dispatch"] = {"claimed_by": claimed_by, "claimed_at": claimed_at, "state": "claimed"}+ return Claim(ClaimOutcome.CLAIMED, record=dict(record))++ try:+ conn = await _connect(dsn)+ except Exception as exc: # noqa: BLE001+ _report_degraded("claim_connect_failed", action_id, exc)+ return Claim(ClaimOutcome.STORE_UNAVAILABLE)++ try:+ row = await conn.fetchrow(+ """+ UPDATE aisoc_action_records+ SET status = $4,+ record = jsonb_set(+ jsonb_set(record, '{status}', to_jsonb($4::text), true),+ '{dispatch}', $5::jsonb, true+ ),+ updated_at = now()+ WHERE id = $1+ AND tenant_id = $2+ AND status = $3+ RETURNING record+ """,+ action_id,+ str(tenant_id),+ expected_status,+ running_status,+ json.dumps({"claimed_by": claimed_by, "claimed_at": claimed_at, "state": "claimed"}),+ )+ except Exception as exc: # noqa: BLE001+ _report_degraded("claim_failed", action_id, exc)+ await conn.close()+ return Claim(ClaimOutcome.STORE_UNAVAILABLE)++ try:+ if row is not None:+ global _DB_CONFIRMED+ _DB_CONFIRMED = True+ record = json.loads(row["record"])+ _MEMORY[action_id] = record+ return Claim(ClaimOutcome.CLAIMED, record=record)++ # Nothing matched. Say which of the three reasons it was, because an+ # operator's next step differs for each: another replica has it, the+ # action is in a state no approval can move, or this tenant has no+ # such action at all.+ existing = await _fetch_record(conn, action_id, str(tenant_id))+ if existing is None:+ _MEMORY.pop(action_id, None)+ return Claim(ClaimOutcome.NOT_FOUND)+ current = _column_text(existing.get("status"))+ _MEMORY[action_id] = existing+ return Claim(ClaimOutcome.WRONG_STATE, record=existing, current_status=current)+ except Exception as exc: # noqa: BLE001+ _report_degraded("claim_readback_failed", action_id, exc)+ return Claim(ClaimOutcome.STORE_UNAVAILABLE)+ finally:+ await conn.close()++def clear() -> None:"""Drop the in-memory copy. Used by tests."""_MEMORY.clear()diff --git a/services/actions/tests/test_action_store_columns.py b/services/actions/tests/test_action_store_columns.pyindex 7a26445e7..f2665f423 100644--- a/services/actions/tests/test_action_store_columns.py+++ b/services/actions/tests/test_action_store_columns.py@@ -37,15 +37,28 @@ _ID, _TENANT, _STATUS, _TIER, _RECORD = range(5)class _RecordingConnection:- """Captures the arguments `save()` binds, and nothing else."""+ """Captures the arguments `save()` binds, and nothing else.++ `fetchval` rather than `execute` since the upsert gained `RETURNING id`:+ an `ON CONFLICT DO UPDATE` whose `WHERE` is not satisfied returns no row,+ which is the only signal Postgres gives that it declined to change what a+ pending approval would do. The double answers "a row was written", which+ is the normal case these tests are about.+ """def __init__(self) -> None:self.args: tuple[Any, ...] = ()self.sql = ""+ #: What the upsert returns. `None` is a refused intent change.+ self.returns: Any = "written"- async def execute(self, sql: str, *args: Any) -> None:+ async def fetchval(self, sql: str, *args: Any) -> Any:self.sql = sqlself.args = args+ return self.returns++ async def fetchrow(self, _sql: str, *_args: Any) -> Any:+ return Noneasync def close(self) -> None:return None@@ -129,3 +142,31 @@ class TestTheApprovalTierIsPersisted:frozen on a row that is saved again after execution."""await action_store.save(_record())assert captured.sql.count("approval_tier") >= 2, captured.sql+++class TestTheUpsertRefusesToChangeAPendingAction:+ """`ON CONFLICT (id) DO UPDATE SET record = EXCLUDED.record` had no+ condition, so a second submission of the same id replaced the record an+ approver was looking at — a different target under the same approval."""++ @pytest.mark.asyncio+ async def test_a_declined_upsert_raises_rather_than_reporting_success(self, captured):+ captured.returns = None+ with pytest.raises(action_store.ActionIntentConflict):+ await action_store.save(_record(status=ActionStatus.AWAITING_APPROVAL))++ @pytest.mark.asyncio+ async def test_a_refused_write_does_not_land_in_the_process_cache(self, captured):+ """The durable store is authoritative; a refused write must not leave+ this replica holding a record the database never accepted."""+ captured.returns = None+ record = _record(status=ActionStatus.AWAITING_APPROVAL, target="dc-01")+ with pytest.raises(action_store.ActionIntentConflict):+ await action_store.save(record)+ assert action_store._MEMORY.get(str(record["id"])) is None++ @pytest.mark.asyncio+ async def test_the_statement_compares_every_field_that_defines_the_intent(self, captured):+ await action_store.save(_record())+ for field in action_store.INTENT_FIELDS:+ assert f"'{field}'" in captured.sql, f"{field} is declared as intent and the upsert never compares it"diff --git a/services/api/app/api/v1/endpoints/approvals.py b/services/api/app/api/v1/endpoints/approvals.pyindex 4ca15039c..3ab08a156 100644--- a/services/api/app/api/v1/endpoints/approvals.py+++ b/services/api/app/api/v1/endpoints/approvals.py@@ -105,6 +105,69 @@ class ApprovalDecisionRequest(BaseModel):comment: str | None = Field(default=None, max_length=2000)+#: Where the originating identity is kept on the approval row.+#:+#: Inside ``action`` rather than in a column of its own, following the two+#: server-written keys already there (``delivery``, ``dispatch``) and the+#: reasoning migration 055 gives for keeping the record whole: the shape is+#: owned by the thing that writes it, and a column per field makes every+#: contract change a migration.+#:+#: Always overwritten from the authenticated principal at create time and+#: never read out of the request body. ``requested_by`` beside it stays a+#: free-text label — it is caller-supplied, and separation of duties+#: evaluated against it would be forgeable from the body.+_REQUESTER_KEY = "requester_principal"++#: Dispatch states recorded on the approval row.+#:+#: ``dispatching`` is the one that did not exist. ``_dispatch_decision`` ran+#: before ``db.commit()``, so a crash in between left the approval ``pending``+#: against an action that had run — the row's own answer to "was this+#: executed" was silently wrong in the dangerous direction. The decision is+#: now committed first with the dispatch marked in flight, so an interrupted+#: decide is visible rather than invisible.+_DISPATCH_IN_FLIGHT = "dispatching"+#: The actions service was reached and answered.+_DISPATCH_EXECUTED = "executed"+_DISPATCH_DECLINED = "declined"+#: The actions service refused, with a reason.+_DISPATCH_FAILED = "failed"+#: We never heard back. Distinct from ``failed`` because the request may have+#: been processed, and "your action did not run" is a claim the evidence does+#: not support.+_DISPATCH_UNKNOWN = "unknown"+_DISPATCH_NOT_EXECUTABLE = "not_executable"++#: Dispatch states that mean the decision has not finished being carried out.+#: A repeat decide re-enters on these rather than reporting success: the+#: actions service claims an execution durably before contacting a vendor, so+#: a re-entry cannot double-execute — it is answered 409 — and the alternative+#: is an approval whose row says ``approved`` and whose action never ran.+_UNRESOLVED_DISPATCH = frozenset({_DISPATCH_IN_FLIGHT, _DISPATCH_UNKNOWN})+++def _requester_principal(user: AuthUser) -> dict[str, Any]:+ """The identity that raised this approval, from authenticated context.++ For an agent-raised approval this is the agents service's own service+ principal; for one a human raised it is that human, which is what keeps+ separation of duties biting on the half where it must.+ """+ return {+ "user_id": str(user.user_id),+ "tenant_id": str(user.tenant_id),+ "email": getattr(user, "email", None),+ "roles": [user.role] if user.role else [],+ "permissions": user.effective_permissions(),+ }+++def _dispatch_state(row: AgentApproval) -> str | None:+ dispatch = (row.action or {}).get("dispatch")+ return dispatch.get("state") if isinstance(dispatch, dict) else None++@router.get("", response_model=ApprovalListResponse)async def list_approvals(user: AuthUser,@@ -198,7 +261,7 @@ async def create_approval(title=body.title,summary=body.summary,risk_level=body.risk_level,- action=body.action,+ action={**(body.action or {}), _REQUESTER_KEY: _requester_principal(user)},expires_at=body.expires_at,status="pending",)@@ -254,10 +317,11 @@ async def decide_approval(raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Approval not found")new_status = "approved" if body.decision == "approve" else "denied"+ unresolved = _dispatch_state(row) in _UNRESOLVED_DISPATCH- if row.status == new_status:+ if row.status == new_status and not unresolved:return ApprovalResponse.model_validate(row)- if row.status not in {"pending", "expired"}:+ if row.status not in {"pending", "expired"} and row.status != new_status:raise HTTPException(status_code=status.HTTP_409_CONFLICT,detail=f"Approval already {row.status}; cannot change to {new_status}",@@ -268,12 +332,28 @@ async def decide_approval(row.decided_at = datetime.now(UTC)row.decision_comment = body.comment- # Dispatch before committing the decision, so an approval that the actions- # service refuses does not leave a row reading "approved" against an- # action that never ran. This endpoint used to flip the row and stop:- # every tap of Approve in the responder app recorded a decision and- # executed nothing, which is the most dangerous shape a security control- # can have — it reports that the host was contained.+ # Commit the decision with the dispatch marked in flight, *then* dispatch.+ #+ # The order used to be the other way round, with a comment explaining that+ # dispatching first stopped a refused action leaving a row reading+ # "approved". It did, and it opened a worse hole: the dispatch ran at one+ # line and `db.commit()` at the next, so a crash in between left the+ # approval `pending` against an action that had already contained a host.+ # Nothing in the row said a dispatch had been attempted, so the recovery+ # was to approve it again.+ #+ # Writing the attempt first costs the thing the old comment protected —+ # the window where the row says approved and the action has not run yet —+ # and that window is now *visible*, which is the property that matters:+ # `dispatch.state` says `dispatching`, a repeat decide re-enters, and the+ # actions service's durable claim is what stops a re-entry from executing+ # twice.+ row.action = {+ **(row.action or {}),+ "dispatch": {"state": _DISPATCH_IN_FLIGHT, "started_at": datetime.now(UTC).isoformat()},+ }+ await db.commit()+dispatch = await _dispatch_decision(row, user, approve=body.decision == "approve")row.action = {**(row.action or {}), "dispatch": dispatch}@@ -284,7 +364,7 @@ async def decide_approval(# resume — fire-and-forget pattern matches the rest of the surface.await _notify_realtime(row, event="approval_decided")- if dispatch.get("state") == "failed":+ if dispatch.get("state") == _DISPATCH_FAILED:# The decision is durable and the operator is told the truth: their# choice was recorded, the execution behind it was not accepted.raise HTTPException(@@ -294,6 +374,17 @@ async def decide_approval("The approval row reflects your decision; the action did not run."),)+ if dispatch.get("state") == _DISPATCH_UNKNOWN:+ # Deliberately not phrased as a failure. The action service may have+ # executed this; it claims an execution durably before touching a+ # vendor, so the answer exists and nobody should act on a guess.+ raise HTTPException(+ status_code=status.HTTP_504_GATEWAY_TIMEOUT,+ detail=(+ f"Decision recorded, but the action service did not answer: {dispatch.get('detail')}. "+ "Whether the action ran is not yet known — check the action's status before deciding again."+ ),+ )return ApprovalResponse.model_validate(row)@@ -316,15 +407,19 @@ async def _dispatch_decision(``declined`` the decision was to deny; the action was rejectedupstream so it cannot later be approved by replay.``executed`` the actions service accepted and ran it.- ``failed`` the actions service refused or was unreachable.+ ``failed`` the actions service refused, with a reason.+ ``unknown`` we never heard back. The request may have been+ processed; the actions service claims an execution+ durably before contacting a vendor, so the answer+ exists upstream and must be read rather than guessed."""action = row.action or {}action_type = action.get("action_type") or action.get("type")target = action.get("target")if not isinstance(action_type, str) or not action_type.strip():- return {"state": "not_executable", "reason": "approval carries no action_type"}+ return {"state": _DISPATCH_NOT_EXECUTABLE, "reason": "approval carries no action_type"}if not isinstance(target, str) or not target.strip():- return {"state": "not_executable", "reason": "approval carries no target"}+ return {"state": _DISPATCH_NOT_EXECUTABLE, "reason": "approval carries no target"}# The approval id is reused as the action id so the two systems share one# identifier. Without that, "which action did this approval authorise"@@ -353,6 +448,20 @@ async def _dispatch_decision("permissions": permissions,}+ # Who originally asked, stamped onto the row from authenticated context+ # when the approval was created. Sent as `requester`, a separate field+ # from `principal`, because the actions service records the requester for+ # separation of duties: sending the deciding user as `principal` made+ # them the requester too, so every approval was refused upstream as its+ # own request and surfaced here as a 502.+ #+ # Falls back to this caller only when the approval predates the stamp, in+ # which case separation of duties has nothing to compare and the actions+ # service records that rather than claiming it passed.+ requester = action.get(_REQUESTER_KEY)+ if not isinstance(requester, dict) or not requester.get("user_id"):+ requester = None+try:await submit_action(action_id=action_id,@@ -362,34 +471,38 @@ async def _dispatch_decision(incident_id=str(row.run_id or row.id),rationale=row.summary or row.title,parameters=action.get("parameters") or {},- # The requester is the agent, not the approver. Sending the- # approver here would make them both, and separation of duties- # would pass by accident.+ # A label, not an identity. The authorization basis is `requester`.requested_by=row.requested_by or "agent",# Identifies the caller to the gate. Omitting it was harmless only# while `AISOC_ACTIONS_REQUIRE_PRINCIPAL` defaults false; turning# that on would have broken the submit leg exactly as the decide# leg was broken.principal=principal,+ requester=requester,)except ActionsServiceError as exc:logger.warning("Approval dispatch: submit refused",extra={"approval_id": action_id, "status": exc.status_code, "detail": str(exc)},)- return {"state": "failed", "stage": "submit", "detail": str(exc)}+ # A submit that was never acknowledged has still not authorized+ # anything: the actions service gates before it executes, and the+ # decide leg below is what would have released it.+ state = _DISPATCH_UNKNOWN if exc.indeterminate else _DISPATCH_FAILED+ return {"state": state, "stage": "submit", "detail": str(exc)}try:result = await decide_action(action_id=action_id, approve=approve, approver=principal)except ActionsServiceError as exc:logger.warning(- "Approval dispatch: decision refused",- extra={"approval_id": action_id, "status": exc.status_code, "detail": str(exc)},+ "Approval dispatch: decision not confirmed",+ extra={"approval_id": action_id, "status": exc.status_code, "detail": str(exc), "indeterminate": exc.indeterminate},)- return {"state": "failed", "stage": "decide", "detail": str(exc)}+ state = _DISPATCH_UNKNOWN if exc.indeterminate else _DISPATCH_FAILED+ return {"state": state, "stage": "decide", "action_id": action_id, "detail": str(exc)}return {- "state": "executed" if approve else "declined",+ "state": _DISPATCH_EXECUTED if approve else _DISPATCH_DECLINED,"action_id": action_id,"action_status": result.get("status"),"blast_radius": result.get("blast_radius"),diff --git a/services/api/app/services/actions_client.py b/services/api/app/services/actions_client.pyindex 2b2503933..42c785a84 100644--- a/services/api/app/services/actions_client.py+++ b/services/api/app/services/actions_client.py@@ -44,6 +44,14 @@ class ActionsServiceError(RuntimeError):operator-facing string, which it deliberately builds as an error *type*rather than a raw exception. Rendering the wrong one into an HTML page ishow an unauthenticated email-approval link starts leaking topology.++ ``indeterminate`` is the difference between "it was refused" and "we never+ heard back". A refusal is a fact about the decision; a transport error on+ a POST that may already have been processed is not, and reporting the+ second as the first told an operator their action had not run when it may+ well have. The actions service answers the question properly — it claims+ the execution durably before contacting a vendor, so the outcome is+ readable afterwards — but this client must not assert what it cannot see."""def __init__(@@ -52,10 +60,12 @@ class ActionsServiceError(RuntimeError):*,status_code: int | None = None,upstream_detail: str | None = None,+ indeterminate: bool = False,) -> None:super().__init__(message)self.status_code = status_codeself.upstream_detail = upstream_detail or "The action service refused the decision."+ self.indeterminate = indeterminatedef base_url() -> str:@@ -81,8 +91,13 @@ async def _post(path: str, payload: dict[str, Any] | None) -> dict[str, Any]:try:async with httpx.AsyncClient(timeout=_TIMEOUT_SECONDS) as client:response = await client.post(url, headers=service_headers(), json=payload)- except httpx.HTTPError as exc:+ except httpx.ConnectError as exc:+ # Nothing was written to a socket, so nothing was processed.raise ActionsServiceError(f"actions service unreachable: {exc}") from exc+ except httpx.HTTPError as exc:+ # A timeout or a dropped connection on a POST: the request may have+ # been received and acted on. Say so rather than calling it a refusal.+ raise ActionsServiceError(f"actions service unreachable: {exc}", indeterminate=True) from excif response.status_code >= 400:detail = _detail(response)@@ -190,9 +205,19 @@ async def submit_action(rationale: str,parameters: dict[str, Any] | None = None,principal: dict[str, Any] | None = None,+ requester: dict[str, Any] | None = None,requested_by: str = "aisoc-api",) -> dict[str, Any]:- """Submit an action. The gate decides whether it executes or waits."""+ """Submit an action. The gate decides whether it executes or waits.++ ``principal`` is the *invoking* identity, whose permissions the actions+ service checks. ``requester`` is who originally asked, when that is+ somebody else — an agent that proposed a containment a human is about to+ approve. They are separate fields because collapsing them broke+ separation of duties: this client sent the approving console user as+ ``principal``, the actions service recorded it as the requester, and the+ approval that followed was refused for being its own request.+ """payload: dict[str, Any] = {"id": action_id,"action_type": action_type,@@ -205,6 +230,8 @@ async def submit_action(}if principal is not None:payload["principal"] = principal+ if requester is not None:+ payload["requester"] = requesterreturn await _post("/api/v1/actions", payload)diff --git a/tests/isolation/test_action_durability_live.py b/tests/isolation/test_action_durability_live.pynew file mode 100644index 000000000..1024d09c7--- /dev/null+++ b/tests/isolation/test_action_durability_live.py@@ -0,0 +1,1263 @@+"""An approved action reaches the vendor once, or visibly not at all.++Remediation phase 2. The approve path in ``services/actions`` was a plain+read-check-act with the durable write placed *after* the vendor call::++ record = await action_store.get(action_id) # no tenant, cache first+ if record["status"] != AWAITING_APPROVAL: ... # checked in Python+ result = await executor.execute(request) # the host is contained+ await action_store.save(record) # ... and only now recorded++Every guarantee an approval is supposed to carry sits in that gap.++* Two replicas, or one replica and a client that retried, both read+ ``awaiting_approval`` and both contain the host. On the submitting replica+ the race is invisible because ``action_store.get`` answers from a+ process-local dict and hands both callers the *same object*, so the local+ suite could never see it and a second container was required to.+* A vendor timeout raised out of the handler: ``save`` was never reached, the+ record stayed ``awaiting_approval`` while the vendor may well have acted,+ and the next approval ran it again. There was no status meaning "we do not+ know".+* The reads carried no tenant, so an approver holding the right permission tier+ in *any* tenant could approve another tenant's containment.+* ``save`` was an unconditional upsert, so re-submitting an id with a different+ target replaced the record underneath the approver looking at it.+* ``save`` swallowed connect and write failures, so an action could look+ persisted while existing only in one replica's memory.++And one defect that is not in any of those lists, found by running the path+rather than reading it: ``services/api``'s ``_dispatch_decision`` submitted+with ``principal=<the deciding console user>``, which the actions service+records as ``requested_by_user_id``. The same principal was then sent as the+approver, so separation of duties compared the approver against *itself* and+refused. **Every** approval raised through the responder/console path was+answered 403 upstream and surfaced as a 502 reading "Decision recorded, but+the action service refused it". The three identities — who asked, who relayed,+who approved — have to stay distinct for the control to work at all.++Why this file is here and not in a service's own suite+------------------------------------------------------+``services/api`` and ``services/actions`` both name their top-level package+``app``. Importing one into a process that has imported the other resolves to+whichever is on ``PYTHONPATH``, and a cross-service test written that way+degrades into skips — which report the same word as a pass.++So the test process is the **API** (it imports the real ``actions_client`` and+the real ``approvals._dispatch_decision``), the actions service runs as **two+separate processes** on loopback sockets sharing one Postgres, and the vendor+is a real HTTP server that records every request that reaches it. Nothing+about authorization, persistence or retry is replaced by a double: a mock that+answers whatever it is asked cannot notice an action that executed twice.++Two replicas rather than one is the whole point. A single process would pass+these tests against the pre-fix tree.+"""++from __future__ import annotations++import asyncio+import json+import os+import socket+import subprocess+import sys+import threading+import time+import uuid+from datetime import UTC, datetime+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer+from pathlib import Path+from types import SimpleNamespace+from typing import Any+from urllib.parse import parse_qs, urlparse++import httpx+import pytest++_REPO = Path(__file__).resolve().parents[2]+_ACTIONS = _REPO / "services" / "actions"++#: The durable store under test. Set by ``action-durability-live.yml``; absent+#: in the offline isolation job, which has no Postgres and must skip rather+#: than invent one.+_DSN = os.environ.get("ISOLATION_ACTIONS_DSN", "").strip()++SERVICE_TOKEN = "p2-actions-token-not-a-real-secret"++pytestmark = [+ pytest.mark.asyncio,+ pytest.mark.skipif(+ not _DSN,+ reason="ISOLATION_ACTIONS_DSN is not set; this suite needs a real Postgres and both services installed",+ ),+]+++# ── the vendor ─────────────────────────────────────────────────────────────+++class _VendorState:+ """What the stub does next, and what it has been asked so far.++ ``contain_calls`` is the number that decides every durability assertion in+ this file. It counts *requests that reached the vendor*, which is the only+ definition of "the host was contained" that does not depend on believing+ the thing under test.+ """++ def __init__(self) -> None:+ self.lock = threading.Lock()+ self.contain_calls: list[dict[str, Any]] = []+ self.token_calls = 0+ self.status_reads = 0+ #: Seconds to stall a containment request. Used to make a real client+ #: timeout, rather than raising a timeout the client never saw.+ self.contain_delay = 0.0+ #: Accept the containment and then hang up without answering. The+ #: vendor has acted and the caller will never learn the outcome, which+ #: is the condition reconciliation exists for — and it reproduces in+ #: milliseconds rather than by waiting out a 30-second client timeout.+ self.drop_response = False+ #: What the device reports when the verifier reads it back.+ self.device_status = "normal"+ #: Whether the read-back route answers at all. A vendor that accepted+ #: the containment and whose device API is then unavailable is the one+ #: case where the outcome genuinely cannot be established.+ self.probe_available = True+ #: What the containment route answers. A 500 is a vendor that received+ #: the request and refused it, which — unlike a dropped response — the+ #: read-back can and must settle as a genuine failure.+ self.contain_http_status = 200++ def reset(self) -> None:+ with self.lock:+ self.contain_calls.clear()+ self.token_calls = 0+ self.status_reads = 0+ self.contain_delay = 0.0+ self.drop_response = False+ self.device_status = "normal"+ self.probe_available = True+ self.contain_http_status = 200+++VENDOR = _VendorState()+++class _VendorHandler(BaseHTTPRequestHandler):+ """A real HTTP endpoint speaking enough of the EDR API to be used for real.++ The forward call and the read-back probe are different routes, so+ "reconcile without executing again" is a property this stub can actually+ distinguish rather than one the test has to take on trust.+ """++ protocol_version = "HTTP/1.1"++ def log_message(self, *_args: Any) -> None: # noqa: D102 - quiet+ return++ def _send(self, payload: dict[str, Any], status: int = 200) -> None:+ body = json.dumps(payload).encode()+ self.send_response(status)+ self.send_header("Content-Type", "application/json")+ self.send_header("Content-Length", str(len(body)))+ self.end_headers()+ self.wfile.write(body)++ def do_POST(self) -> None: # noqa: N802+ parsed = urlparse(self.path)+ length = int(self.headers.get("Content-Length") or 0)+ raw = self.rfile.read(length) if length else b""++ if parsed.path.endswith("/oauth2/token"):+ with VENDOR.lock:+ VENDOR.token_calls += 1+ self._send({"access_token": "vendor-token", "expires_in": 1799})+ return++ if parsed.path.endswith("/devices/entities/devices-actions/v2"):+ action = (parse_qs(parsed.query).get("action_name") or [""])[0]+ with VENDOR.lock:+ delay = VENDOR.contain_delay+ drop = VENDOR.drop_response+ http_status = VENDOR.contain_http_status+ VENDOR.contain_calls.append(+ {+ "action": action,+ "body": json.loads(raw or b"{}"),+ "at": datetime.now(UTC).isoformat(),+ }+ )+ if action == "contain" and http_status < 400:+ VENDOR.device_status = "contained"+ if delay:+ time.sleep(delay)+ if drop:+ self.close_connection = True+ return+ if http_status >= 400:+ self._send({"errors": [{"message": "containment refused"}]}, status=http_status)+ return+ self._send({"resources": [{"id": "dev-1"}], "errors": []})+ return++ self._send({"errors": [{"message": "unexpected route"}]}, status=404)++ def do_GET(self) -> None: # noqa: N802+ parsed = urlparse(self.path)+ if parsed.path.endswith("/devices/queries/devices/v1"):+ self._send({"resources": ["dev-1"]})+ return+ if parsed.path.endswith("/devices/entities/devices/v2"):+ with VENDOR.lock:+ VENDOR.status_reads += 1+ status = VENDOR.device_status+ available = VENDOR.probe_available+ if not available:+ self._send({"errors": [{"message": "device service unavailable"}]}, status=503)+ return+ self._send({"resources": [{"device_id": "dev-1", "status": status}]})+ return+ self._send({"errors": [{"message": "unexpected route"}]}, status=404)+++def _free_port() -> int:+ with socket.socket() as sock:+ sock.bind(("127.0.0.1", 0))+ return int(sock.getsockname()[1])++[email protected](scope="module")+def vendor_base_url():+ server = ThreadingHTTPServer(("127.0.0.1", _free_port()), _VendorHandler)+ thread = threading.Thread(target=server.serve_forever, daemon=True)+ thread.start()+ yield f"http://127.0.0.1:{server.server_address[1]}"+ server.shutdown()+ server.server_close()++[email protected](autouse=True)+def _clean_vendor():+ VENDOR.reset()+ yield+ VENDOR.reset()+++# ── the actions service, twice ─────────────────────────────────────────────+++class Replica:+ """One actions container on a loopback socket."""++ def __init__(self, name: str, port: int, proc: subprocess.Popen) -> None:+ self.name = name+ self.port = port+ self.proc = proc+ self.base_url = f"http://127.0.0.1:{port}"++ def kill(self) -> None:+ self.proc.kill()+ self.proc.wait(timeout=10)+++def _spawn_replica(name: str, dsn: str) -> Replica:+ if not (_ACTIONS / "app" / "main.py").is_file():+ pytest.fail(f"the actions service is not at {_ACTIONS}; this test must not be skipped into passing")++ port = _free_port()+ env = {+ **os.environ,+ "AISOC_ACTIONS_SERVICE_TOKEN": SERVICE_TOKEN,+ # Every approval in this file carries an identity. The default is on;+ # it is set explicitly so a change to the default cannot silently turn+ # the separation-of-duties assertions into no-ops.+ "AISOC_ACTIONS_REQUIRE_APPROVER": "true",+ "DATABASE_DSN": dsn,+ "DATABASE_URL": dsn,+ "OTEL_SDK_DISABLED": "true",+ "ENVIRONMENT": "test",+ "PYTHONPATH": str(_ACTIONS),+ }+ env.pop("AISOC_DEV_MODE", None)++ proc = subprocess.Popen(+ [+ sys.executable,+ "-c",+ "import uvicorn\nfrom app.main import app as actions_app\n"+ f"uvicorn.run(actions_app, host='127.0.0.1', port={port}, log_level='error')\n",+ ],+ cwd=str(_ACTIONS),+ env=env,+ stdout=subprocess.PIPE,+ stderr=subprocess.STDOUT,+ )++ deadline = time.time() + 45+ while time.time() < deadline:+ if proc.poll() is not None:+ output = ((proc.stdout.read() if proc.stdout else b"") or b"").decode()[-2000:]+ pytest.fail(f"actions replica {name} exited before serving:\n{output}")+ try:+ httpx.get(f"http://127.0.0.1:{port}/api/v1/health", timeout=1.0)+ return Replica(name, port, proc)+ except httpx.HTTPError:+ time.sleep(0.2)+ proc.terminate()+ pytest.fail(f"actions replica {name} did not start within 45s")+ raise AssertionError("unreachable")++[email protected](scope="module")+def replicas():+ """Two containers behind one database, which is the deployed shape.++ One process cannot demonstrate the defect: ``action_store`` answers from a+ module-global dict, so a single replica's two concurrent approvals mutate+ one object and the race hides. The second replica has an empty cache and+ has to ask the database, which is exactly the question under test.+ """+ a = _spawn_replica("a", _DSN)+ b = _spawn_replica("b", _DSN)+ yield a, b+ for replica in (a, b):+ replica.proc.terminate()+ try:+ replica.proc.wait(timeout=10)+ except subprocess.TimeoutExpired: # pragma: no cover - teardown guard+ replica.proc.kill()+++# ── identities, as the console actually builds them ────────────────────────+++def _principal(tenant: str, *, user_id: str | None = None, permissions: list[str] | None = None) -> dict[str, Any]:+ return {+ "user_id": user_id or str(uuid.uuid4()),+ "tenant_id": tenant,+ "email": f"{(user_id or 'user').split('@')[0]}@example.com",+ "roles": ["analyst"],+ "permissions": permissions if permissions is not None else ["actions:execute:high"],+ }+++def _vendor_params(vendor_base_url: str) -> dict[str, Any]:+ """Credentials that make the executor take its real CrowdStrike arm.++ Without these the executor falls through to its simulation branch, which+ touches no socket — so every "did the vendor get called" assertion in this+ file would pass for a reason that has nothing to do with the fix.+ """+ return {+ "cs_client_id": "p2-client",+ "cs_client_secret": "p2-secret",+ "cs_base_url": vendor_base_url,+ }+++async def _submit(+ replica: Replica,+ *,+ tenant: str,+ action_id: str,+ requester: dict[str, Any] | None,+ principal: dict[str, Any],+ target: str,+ params: dict[str, Any],+) -> httpx.Response:+ body: dict[str, Any] = {+ "id": action_id,+ "action_type": "isolate_host",+ "target": target,+ "tenant_id": tenant,+ "incident_id": str(uuid.uuid4()),+ "rationale": "phase 2 regression",+ "requested_by": "agent",+ "parameters": params,+ "principal": principal,+ }+ if requester is not None:+ body["requester"] = requester+ async with httpx.AsyncClient(timeout=30.0) as client:+ return await client.post(+ f"{replica.base_url}/api/v1/actions",+ json=body,+ headers={"Authorization": f"Bearer {SERVICE_TOKEN}"},+ )+++async def _approve(replica: Replica, action_id: str, approver: dict[str, Any], *, timeout: float = 30.0) -> httpx.Response:+ async with httpx.AsyncClient(timeout=timeout) as client:+ return await client.post(+ f"{replica.base_url}/api/v1/actions/{action_id}/approve",+ json={"approver": approver},+ headers={"Authorization": f"Bearer {SERVICE_TOKEN}"},+ )+++async def _read_record(dsn: str, action_id: str) -> dict[str, Any] | None:+ """The database's answer, not any replica's.++ Every status assertion reads through here on purpose: a replica's own+ ``GET`` can be served from the process-local cache that caused the defect,+ so asking it whether the durable record is right is asking the suspect.+ """+ import asyncpg++ conn = await asyncpg.connect(dsn.replace("postgresql+asyncpg://", "postgresql://"), timeout=10.0)+ try:+ row = await conn.fetchrow(+ "SELECT status, record FROM aisoc_action_records WHERE id = $1",+ action_id,+ )+ finally:+ await conn.close()+ if row is None:+ return None+ record = json.loads(row["record"])+ record["_column_status"] = row["status"]+ return record+++def _contain_calls() -> int:+ with VENDOR.lock:+ return len([c for c in VENDOR.contain_calls if c["action"] == "contain"])+++# ── A. the three identities ────────────────────────────────────────────────+++class TestTheThreeIdentitiesStayDistinct:+ """Who asked, who relayed and who approved are three facts, not one.++ Against the pre-fix tree the first test here fails with 403+ ``approver cannot be the requester``, which is what every approval tapped+ in the responder app received.+ """++ async def test_an_agent_originated_request_is_approved_by_a_human(self, replicas, vendor_base_url) -> None:+ a, _b = replicas+ tenant = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ analyst = _principal(tenant, user_id="dana")+ action_id = str(uuid.uuid4())++ submitted = await _submit(+ a,+ tenant=tenant,+ action_id=action_id,+ requester=agent,+ principal=analyst,+ target="ws-4471",+ params=_vendor_params(vendor_base_url),+ )+ assert submitted.status_code == 200, submitted.text+ assert submitted.json()["status"] == "awaiting_approval"++ approved = await _approve(a, action_id, analyst)+ assert approved.status_code == 200, (+ "a human approving an agent-originated action was refused: "+ f"{approved.status_code} {approved.text}. The requester identity did not survive the submit."+ )+ assert approved.json()["status"] == "completed"+ assert _contain_calls() == 1++ stored = await _read_record(_DSN, action_id)+ assert stored is not None+ assert stored["requested_by_user_id"] == "agents-service"+ assert stored["approved_by_user_id"] == analyst["user_id"]++ async def test_the_real_console_dispatch_path_approves(self, replicas, vendor_base_url, monkeypatch) -> None:+ """``approvals._dispatch_decision`` itself, not a hand-built stand-in.++ This is the function the responder app reaches, and it is where the+ identity collapse happened. A test that posted its own body would have+ passed on the defective tree.+ """+ a, _b = replicas+ approvals = pytest.importorskip(+ "app.api.v1.endpoints.approvals",+ reason="services/api is not installed in this job",+ )+ actions_client = pytest.importorskip("app.services.actions_client")+ monkeypatch.setattr(actions_client.settings, "AISOC_ACTIONS_BASE_URL", a.base_url, raising=False)+ monkeypatch.setattr(actions_client.settings, "AISOC_ACTIONS_SERVICE_TOKEN", SERVICE_TOKEN, raising=False)++ tenant = uuid.uuid4()+ approval_id = uuid.uuid4()+ row = SimpleNamespace(+ id=approval_id,+ tenant_id=tenant,+ run_id=None,+ summary="isolate the beachhead",+ title="isolate ws-4471",+ requested_by="triage-agent",+ action={+ "action_type": "isolate_host",+ "target": "ws-4471",+ "parameters": _vendor_params(vendor_base_url),+ # Written by `create_approval` from the authenticated principal+ # that raised the approval, never read from the request body.+ "requester_principal": {+ "user_id": "agents-service",+ "tenant_id": str(tenant),+ "permissions": ["actions:execute:high"],+ },+ },+ )+ analyst_id = uuid.uuid4()+ user = SimpleNamespace(+ user_id=analyst_id,+ tenant_id=tenant,+ email="[email protected]",+ role="analyst",+ effective_permissions=lambda: ["actions:execute:high"],+ )++ dispatch = await approvals._dispatch_decision(row, user, approve=True)++ assert dispatch["state"] == "executed", (+ f"the console approval path did not execute: {dispatch}. Every tap of Approve in the responder app surfaced this as a 502."+ )+ assert _contain_calls() == 1++ async def test_a_human_cannot_approve_the_action_they_raised(self, replicas, vendor_base_url) -> None:+ """Separation of duties, on the half where it must still bite."""+ a, _b = replicas+ tenant = str(uuid.uuid4())+ analyst = _principal(tenant, user_id="dana")+ action_id = str(uuid.uuid4())++ await _submit(+ a,+ tenant=tenant,+ action_id=action_id,+ requester=analyst,+ principal=analyst,+ target="ws-4471",+ params=_vendor_params(vendor_base_url),+ )+ refused = await _approve(a, action_id, analyst)++ assert refused.status_code == 403, refused.text+ assert "separation of duties" in refused.text+ assert _contain_calls() == 0++ async def test_separation_of_duties_is_not_waived_by_an_absent_requester(self, replicas, vendor_base_url) -> None:+ """An action nobody is recorded as requesting is not a free approval.++ ``if separation_of_duties and requester and ...`` passed silently when+ the submit carried no principal at all, so the control reported itself+ satisfied on exactly the requests it could not evaluate.+ """+ a, _b = replicas+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ analyst = _principal(tenant, user_id="dana")++ async with httpx.AsyncClient(timeout=30.0) as client:+ submitted = await client.post(+ f"{a.base_url}/api/v1/actions",+ json={+ "id": action_id,+ "action_type": "isolate_host",+ "target": "ws-4471",+ "tenant_id": tenant,+ "incident_id": str(uuid.uuid4()),+ "rationale": "phase 2 regression",+ "parameters": _vendor_params(vendor_base_url),+ },+ headers={"Authorization": f"Bearer {SERVICE_TOKEN}"},+ )+ assert submitted.status_code == 200, submitted.text++ approved = await _approve(a, action_id, analyst)+ assert approved.status_code == 200, approved.text++ stored = await _read_record(_DSN, action_id)+ assert stored is not None+ assert stored.get("separation_of_duties") == "unevaluated", (+ "an approval whose requester is unknown recorded nothing about it, so the audit "+ f"trail cannot distinguish an evaluated control from a skipped one: {stored.get('separation_of_duties')!r}"+ )++ async def test_an_approver_from_another_tenant_is_refused(self, replicas, vendor_base_url) -> None:+ """``aisoc_action_records`` is reachable by id, and the id was the only control."""+ a, b = replicas+ victim = str(uuid.uuid4())+ attacker_tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())++ await _submit(+ a,+ tenant=victim,+ action_id=action_id,+ requester=_principal(victim, user_id="agents-service"),+ principal=_principal(victim, user_id="agents-service"),+ target="ws-victim",+ params=_vendor_params(vendor_base_url),+ )++ # From the *other* replica, so the read has to come from the database+ # rather than the submitting process's cache.+ refused = await _approve(b, action_id, _principal(attacker_tenant, user_id="mallory"))++ assert refused.status_code in (403, 404), (+ f"an approver in tenant {attacker_tenant} was served tenant {victim}'s action: {refused.status_code} {refused.text}"+ )+ assert _contain_calls() == 0, "another tenant's host was contained"++ async def test_an_approver_below_the_tier_is_refused(self, replicas, vendor_base_url) -> None:+ """The control that already worked, kept so the fix cannot quietly drop it."""+ a, _b = replicas+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())++ await _submit(+ a,+ tenant=tenant,+ action_id=action_id,+ requester=_principal(tenant, user_id="agents-service"),+ principal=_principal(tenant, user_id="agents-service"),+ target="ws-4471",+ params=_vendor_params(vendor_base_url),+ )+ refused = await _approve(a, action_id, _principal(tenant, user_id="sam", permissions=["actions:execute:low"]))++ assert refused.status_code == 403+ assert _contain_calls() == 0+++# ── B. the intent an approver signed off ───────────────────────────────────+++class TestApprovalIsBoundToTheIntent:+ async def test_a_resubmission_cannot_retarget_a_pending_approval(self, replicas, vendor_base_url) -> None:+ """The approval card says ws-4471; the record must not become dc-01.++ ``save`` was ``ON CONFLICT (id) DO UPDATE SET record = EXCLUDED.record``+ with no condition, so anything that could submit could rewrite what a+ human was in the middle of approving.+ """+ a, b = replicas+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ analyst = _principal(tenant, user_id="dana")+ params = _vendor_params(vendor_base_url)++ first = await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)+ assert first.status_code == 200+ assert first.json()["status"] == "awaiting_approval"++ swapped = await _submit(b, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="dc-01", params=params)+ assert swapped.status_code == 409, (+ f"the pending action was retargeted to dc-01 while awaiting approval: {swapped.status_code} {swapped.text}"+ )++ stored = await _read_record(_DSN, action_id)+ assert stored is not None+ assert stored["target"] == "ws-4471"++ approved = await _approve(a, action_id, analyst)+ assert approved.status_code == 200, approved.text+ with VENDOR.lock:+ assert VENDOR.contain_calls, "nothing reached the vendor"++ async def test_a_resubmission_cannot_change_the_parameters_either(self, replicas, vendor_base_url) -> None:+ """The subtler half, and the one a target-only guard would miss.++ An action gated *because of what it targets* can be rewritten through+ its parameters alone — same verb, same host, a different thing done to+ it — and the approval card would not change at all.+ """+ a, b = replicas+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ params = _vendor_params(vendor_base_url)++ first = await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)+ assert first.status_code == 200+ assert "scope" not in first.json()["parameters"]++ altered = await _submit(+ b,+ tenant=tenant,+ action_id=action_id,+ requester=agent,+ principal=agent,+ target="ws-4471",+ params={**params, "scope": "entire-subnet"},+ )+ assert altered.status_code == 409, (+ f"a pending action's parameters were rewritten while it awaited approval: {altered.status_code} {altered.text}"+ )++ stored = await _read_record(_DSN, action_id)+ assert stored is not None+ assert "scope" not in (stored.get("parameters") or {}), stored.get("parameters")++ async def test_an_identical_resubmission_is_still_idempotent(self, replicas, vendor_base_url) -> None:+ """A retried submit of the same intent must not become an error.++ The store's contract has always been "a replay of the same submission+ is idempotent rather than a duplicate pending approval", and refusing+ a changed intent must not cost that.+ """+ a, b = replicas+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ params = _vendor_params(vendor_base_url)++ first = await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)+ again = await _submit(b, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)+ assert first.status_code == 200+ assert again.status_code == 200, again.text+ assert again.json()["status"] == "awaiting_approval"+++# ── C. one vendor call, or none ────────────────────────────────────────────+++class TestExecutionIsClaimedBeforeTheVendorIsContacted:+ async def test_two_replicas_cannot_both_execute_one_approval(self, replicas, vendor_base_url) -> None:+ """The defect, in the shape that produced it.++ Both replicas read ``awaiting_approval`` — one from its cache, one from+ the database — and both called the EDR. The host was contained twice,+ two audit trails were written, and the only durable evidence of either+ was whichever ``save`` landed last.+ """+ a, b = replicas+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ analyst = _principal(tenant, user_id="dana")+ params = _vendor_params(vendor_base_url)++ await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)++ # Stall the vendor so both approvals are genuinely in flight together.+ # Without this the first can finish before the second reads, and the+ # race closes by timing rather than by the fix.+ with VENDOR.lock:+ VENDOR.contain_delay = 1.5++ outcomes: list[Any] = list(+ await asyncio.gather(+ _approve(a, action_id, analyst),+ _approve(b, action_id, analyst),+ return_exceptions=True,+ )+ )++ assert _contain_calls() == 1, (+ f"the host was contained {_contain_calls()} times for one approval. "+ f"replicas answered {[getattr(o, 'status_code', o) for o in outcomes]}"+ )+ codes = sorted(getattr(r, "status_code", 500) for r in outcomes)+ assert codes[0] == 200, f"neither replica executed: {codes}"+ assert codes[1] in (400, 409), f"the losing replica did not refuse cleanly: {codes}"++ async def test_a_repeated_approval_does_not_execute_twice(self, replicas, vendor_base_url) -> None:+ """A client that retries — a double tap, a proxy retry, a resumed run."""+ a, _b = replicas+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ analyst = _principal(tenant, user_id="dana")+ params = _vendor_params(vendor_base_url)++ await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)++ first = await _approve(a, action_id, analyst)+ second = await _approve(a, action_id, analyst)++ assert first.status_code == 200, first.text+ assert second.status_code in (400, 409), second.text+ assert _contain_calls() == 1++ async def test_the_vendor_is_not_contacted_when_the_claim_cannot_be_written(self, replicas, vendor_base_url) -> None:+ """A store that cannot record the authorization must not authorize one.++ ``save`` logged and returned on a write failure, so an approval could+ contain a host and leave no durable trace that it had. The claim is the+ pre-dispatch write, and it fails closed.+ """+ import asyncpg++ a, _b = replicas+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ analyst = _principal(tenant, user_id="dana")+ params = _vendor_params(vendor_base_url)++ await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)++ plain = _DSN.replace("postgresql+asyncpg://", "postgresql://")+ admin = await asyncpg.connect(plain.replace("aisoc_app", "aisoc"), timeout=10.0)+ try:+ await admin.execute("REVOKE UPDATE, INSERT ON aisoc_action_records FROM aisoc_app")+ refused = await _approve(a, action_id, analyst)+ finally:+ await admin.execute("GRANT INSERT, UPDATE ON aisoc_action_records TO aisoc_app")+ await admin.close()++ assert _contain_calls() == 0, "a host was contained while the durable record could not be written"+ assert refused.status_code >= 500, f"an unwritable store was reported as a client error: {refused.status_code} {refused.text}"++ async def test_a_stale_cache_cannot_resurrect_an_executed_action(self, replicas, vendor_base_url) -> None:+ """The submitting replica holds the record in memory forever.++ ``action_store.get`` checked that dict first, so a replica whose cache+ said ``awaiting_approval`` would execute again regardless of what the+ database had recorded. Replica A submits and caches; replica B executes;+ A must then refuse.+ """+ a, b = replicas+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ analyst = _principal(tenant, user_id="dana")+ params = _vendor_params(vendor_base_url)++ await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)++ executed = await _approve(b, action_id, analyst)+ assert executed.status_code == 200, executed.text++ stale = await _approve(a, action_id, analyst)+ assert stale.status_code in (400, 409), (+ f"replica A's cached copy let an already-executed action run again: {stale.status_code} {stale.text}"+ )+ assert _contain_calls() == 1+++# ── D. when the outcome is genuinely unknown ───────────────────────────────+++class TestAnUnknownOutcomeIsRecordedAsUnknown:+ """A dispatch that did not come back clean is unknown until the vendor says.++ The executors catch their own vendor errors and return+ ``ActionResult(status=FAILED, error=str(exc))``, so the record read+ ``failed`` for a containment the EDR had accepted — a stronger claim than+ the evidence supports, and the dangerous direction, because the obvious+ response to a failure is to run it again.++ The resolution is not to guess from the exception type. It is to ask the+ vendor, with the read-back probe this service already ships: present means+ it completed, absent means it genuinely failed, and unreadable is the only+ honest ``reconciliation_required``.+ """++ async def test_a_lost_response_is_resolved_by_reading_the_vendor_back(self, replicas, vendor_base_url) -> None:+ """Vendor receipt, response lost, device reachable. The host *is* contained."""+ a, _b = replicas+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ analyst = _principal(tenant, user_id="dana")+ params = _vendor_params(vendor_base_url)++ await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)+ with VENDOR.lock:+ VENDOR.drop_response = True++ response = await _approve(a, action_id, analyst, timeout=60.0)+ assert response.status_code in (200, 202), response.text+ assert _contain_calls() == 1, "the vendor never received the containment; this test proves nothing"++ stored = await _read_record(_DSN, action_id)+ assert stored is not None+ assert stored["_column_status"] == "completed", (+ f"a containment the EDR confirms is in force was recorded as {stored['_column_status']!r}"+ )+ assert stored["verification"]["outcome"] == "verified", stored.get("verification")++ async def test_an_effect_that_cannot_be_read_back_is_recorded_as_unknown(self, replicas, vendor_base_url) -> None:+ """Vendor receipt, response lost, device API down. Nobody knows."""+ a, _b = replicas+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ analyst = _principal(tenant, user_id="dana")+ params = _vendor_params(vendor_base_url)++ await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)+ with VENDOR.lock:+ VENDOR.drop_response = True+ VENDOR.probe_available = False++ response = await _approve(a, action_id, analyst, timeout=60.0)+ assert response.status_code in (200, 202), response.text+ assert _contain_calls() == 1++ stored = await _read_record(_DSN, action_id)+ assert stored is not None+ assert stored["_column_status"] == "reconciliation_required", (+ "a vendor call that may have landed, against a device API that cannot be read, was recorded as "+ f"{stored['_column_status']!r}; the host's real state is unknown and the record must say so"+ )++ async def test_a_refused_dispatch_is_still_a_failure(self, replicas, vendor_base_url) -> None:+ """The control. Not everything becomes "unknown" — that would be its own lie.++ The vendor answered the containment with a 500 and the device is+ readable and reports ``normal``, so the containment demonstrably did+ not take and ``failed`` is the honest record. A change that resolved+ every imperfect dispatch to ``reconciliation_required`` would pass+ every other test in this class and fail this one.+ """+ a, _b = replicas+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ analyst = _principal(tenant, user_id="dana")+ params = _vendor_params(vendor_base_url)++ await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)+ with VENDOR.lock:+ VENDOR.contain_http_status = 500++ response = await _approve(a, action_id, analyst, timeout=60.0)+ assert response.status_code in (200, 202), response.text++ stored = await _read_record(_DSN, action_id)+ assert stored is not None+ assert stored["_column_status"] == "failed", stored["_column_status"]+ assert stored["verification"]["outcome"] == "failed", stored.get("verification")++ async def test_a_vendor_timeout_is_not_recorded_as_a_failure(self, replicas, vendor_base_url) -> None:+ """ "The request timed out" and "the action did not happen" are different facts.++ The slow sibling of the lost-response case, kept because a real+ ``httpx`` read timeout and a dropped connection are different code+ paths inside the client and only one of them is cheap to reproduce.+ """+ a, _b = replicas+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ analyst = _principal(tenant, user_id="dana")+ params = _vendor_params(vendor_base_url)++ await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)++ # Longer than the client's 30s containment timeout, so the client+ # really times out on a request the vendor really received. The device+ # API is down too, so the outcome is genuinely unestablished.+ with VENDOR.lock:+ VENDOR.contain_delay = 34.0+ VENDOR.probe_available = False++ response = await _approve(a, action_id, analyst, timeout=120.0)+ assert response.status_code in (200, 202), response.text++ stored = await _read_record(_DSN, action_id)+ assert stored is not None+ assert stored["_column_status"] == "reconciliation_required", (+ "a vendor call that may have landed was recorded as "+ f"{stored['_column_status']!r}; the host's real state is unknown and the record must say so"+ )+ assert _contain_calls() == 1++ async def test_an_unknown_action_is_not_re_executed_on_approval(self, replicas, vendor_base_url) -> None:+ """Retrying an action that may already have run is the thing to prevent."""+ a, _b = replicas+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ analyst = _principal(tenant, user_id="dana")+ params = _vendor_params(vendor_base_url)++ await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)+ with VENDOR.lock:+ VENDOR.drop_response = True+ VENDOR.probe_available = False+ await _approve(a, action_id, analyst, timeout=60.0)+ with VENDOR.lock:+ VENDOR.drop_response = False+ VENDOR.probe_available = True++ retry = await _approve(a, action_id, analyst)+ assert retry.status_code in (400, 409), f"an action in an unknown state was re-approved and run: {retry.text}"+ assert _contain_calls() == 1++ async def test_a_replica_killed_mid_dispatch_leaves_a_claimed_record(self, replicas, vendor_base_url) -> None:+ """Restart recovery. A process that dies after contacting the vendor+ must not leave an action that looks un-started."""+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ analyst = _principal(tenant, user_id="dana")+ params = _vendor_params(vendor_base_url)++ doomed = _spawn_replica("doomed", _DSN)+ try:+ await _submit(doomed, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)+ with VENDOR.lock:+ VENDOR.contain_delay = 8.0++ task = asyncio.create_task(_approve(doomed, action_id, analyst, timeout=60.0))+ # Wait until the vendor has actually been reached, then kill the+ # replica. Sleeping a fixed interval would race the dispatch.+ deadline = time.time() + 20+ while _contain_calls() == 0 and time.time() < deadline:+ await asyncio.sleep(0.1)+ assert _contain_calls() == 1, "the vendor was never reached; this test proves nothing"+ doomed.kill()+ task.cancel()+ try:+ await task+ except (asyncio.CancelledError, httpx.HTTPError):+ # Expected and carries no information: the replica was killed+ # mid-request on purpose, so the in-flight approve either+ # cancels or loses its connection. The assertion this test+ # makes is about the durable record, read from Postgres below.+ pass+ finally:+ if doomed.proc.poll() is None: # pragma: no cover - teardown guard+ doomed.kill()+ with VENDOR.lock:+ VENDOR.contain_delay = 0.0++ stored = await _read_record(_DSN, action_id)+ assert stored is not None+ assert stored["_column_status"] in ("running", "reconciliation_required"), (+ f"a replica died after contacting the vendor and the record reads {stored['_column_status']!r}, "+ "which is indistinguishable from an action nobody started"+ )+ assert stored.get("dispatch", {}).get("claimed_by"), "the record does not say which replica claimed it"++ async def test_a_claimed_record_survives_the_restart_and_still_refuses(self, replicas, vendor_base_url) -> None:+ """A fresh replica with an empty cache reads the claim and holds."""+ a, _b = replicas+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ analyst = _principal(tenant, user_id="dana")+ params = _vendor_params(vendor_base_url)++ doomed = _spawn_replica("restart-a", _DSN)+ try:+ await _submit(doomed, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)+ with VENDOR.lock:+ VENDOR.contain_delay = 8.0+ task = asyncio.create_task(_approve(doomed, action_id, analyst, timeout=60.0))+ deadline = time.time() + 20+ while _contain_calls() == 0 and time.time() < deadline:+ await asyncio.sleep(0.1)+ doomed.kill()+ task.cancel()+ try:+ await task+ except (asyncio.CancelledError, httpx.HTTPError):+ # Expected and carries no information: the replica was killed+ # mid-request on purpose, so the in-flight approve either+ # cancels or loses its connection. The assertion this test+ # makes is about the durable record, read from Postgres below.+ pass+ finally:+ if doomed.proc.poll() is None: # pragma: no cover - teardown guard+ doomed.kill()+ with VENDOR.lock:+ VENDOR.contain_delay = 0.0++ replacement = await _approve(a, action_id, analyst)+ assert replacement.status_code in (400, 409), (+ f"after a restart the action was run a second time: {replacement.status_code} {replacement.text}"+ )+ assert _contain_calls() == 1+++# ── E. reconciling, without guessing ───────────────────────────────────────+++class TestReconciliationReadsTheVendorRatherThanGuessing:+ async def _strand(self, replica: Replica, vendor_base_url: str) -> tuple[str, dict[str, Any], dict[str, Any]]:+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ analyst = _principal(tenant, user_id="dana")+ params = _vendor_params(vendor_base_url)+ await _submit(replica, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)+ with VENDOR.lock:+ VENDOR.drop_response = True+ VENDOR.probe_available = False+ await _approve(replica, action_id, analyst, timeout=60.0)+ with VENDOR.lock:+ VENDOR.drop_response = False+ VENDOR.probe_available = True+ return action_id, analyst, params++ async def test_a_confirmed_effect_closes_the_record_without_acting_again(self, replicas, vendor_base_url) -> None:+ """The EDR says the host is contained, so the action completed.++ ``PostActionVerifier`` already existed and already reads the device's+ real containment state. Reconciliation is that probe, not a retry.+ """+ a, _b = replicas+ action_id, _analyst, params = await self._strand(a, vendor_base_url)+ before = _contain_calls()++ async with httpx.AsyncClient(timeout=60.0) as client:+ reconciled = await client.post(+ f"{a.base_url}/api/v1/actions/{action_id}/reconcile",+ json={"parameters": params},+ headers={"Authorization": f"Bearer {SERVICE_TOKEN}"},+ )++ assert reconciled.status_code == 200, reconciled.text+ body = reconciled.json()+ assert body["status"] == "completed", body+ assert body["verification"]["outcome"] == "verified", body+ assert _contain_calls() == before, "reconciliation executed the action again instead of reading it back"++ stored = await _read_record(_DSN, action_id)+ assert stored is not None and stored["_column_status"] == "completed"++ async def test_an_absent_effect_is_recorded_as_failed_not_guessed_complete(self, replicas, vendor_base_url) -> None:+ a, _b = replicas+ action_id, _analyst, params = await self._strand(a, vendor_base_url)+ before = _contain_calls()+ # The vendor accepted the request and the host is nonetheless not+ # contained — a real outcome, and the one a retry would be wrong about.+ with VENDOR.lock:+ VENDOR.device_status = "normal"++ async with httpx.AsyncClient(timeout=60.0) as client:+ reconciled = await client.post(+ f"{a.base_url}/api/v1/actions/{action_id}/reconcile",+ json={"parameters": params},+ headers={"Authorization": f"Bearer {SERVICE_TOKEN}"},+ )++ assert reconciled.status_code == 200, reconciled.text+ assert reconciled.json()["status"] == "failed", reconciled.json()+ assert _contain_calls() == before++ async def test_an_unreadable_vendor_leaves_the_record_unknown(self, replicas, vendor_base_url) -> None:+ """A reconciliation that cannot read the vendor resolves nothing.++ The tempting failure is to treat an unanswerable probe as a clean+ result in whichever direction is convenient. It stays unknown, and the+ operator is told so rather than told it is handled.+ """+ a, _b = replicas+ action_id, _analyst, params = await self._strand(a, vendor_base_url)+ before = _contain_calls()+ with VENDOR.lock:+ VENDOR.probe_available = False++ async with httpx.AsyncClient(timeout=60.0) as client:+ reconciled = await client.post(+ f"{a.base_url}/api/v1/actions/{action_id}/reconcile",+ json={"parameters": params},+ headers={"Authorization": f"Bearer {SERVICE_TOKEN}"},+ )++ assert reconciled.status_code == 200, reconciled.text+ body = reconciled.json()+ assert body["status"] == "reconciliation_required", body+ assert body["verification"]["outcome"] == "unverified", body+ assert _contain_calls() == before++ async def test_reconcile_will_not_touch_another_tenants_action(self, replicas, vendor_base_url) -> None:+ a, b = replicas+ action_id, _analyst, params = await self._strand(a, vendor_base_url)++ async with httpx.AsyncClient(timeout=60.0) as client:+ refused = await client.post(+ f"{b.base_url}/api/v1/actions/{action_id}/reconcile",+ json={"parameters": params},+ headers={+ "Authorization": f"Bearer {SERVICE_TOKEN}",+ "X-Tenant-ID": str(uuid.uuid4()),+ },+ )+ assert refused.status_code == 404, refused.text+++# ── F. the read surface ────────────────────────────────────────────────────+++class TestReadingAnActionRecordIsTenantScoped:+ async def test_a_matching_tenant_sees_the_whole_record(self, replicas, vendor_base_url) -> None:+ a, _b = replicas+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ await _submit(+ a,+ tenant=tenant,+ action_id=action_id,+ requester=agent,+ principal=agent,+ target="ws-4471",+ params=_vendor_params(vendor_base_url),+ )++ async with httpx.AsyncClient(timeout=20.0) as client:+ seen = await client.get(+ f"{a.base_url}/api/v1/actions/{action_id}",+ headers={"Authorization": f"Bearer {SERVICE_TOKEN}", "X-Tenant-ID": tenant},+ )+ assert seen.status_code == 200+ assert seen.json()["target"] == "ws-4471"++ async def test_another_tenant_is_told_nothing_about_it(self, replicas, vendor_base_url) -> None:+ a, b = replicas+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ await _submit(+ a,+ tenant=tenant,+ action_id=action_id,+ requester=agent,+ principal=agent,+ target="ws-victim",+ params=_vendor_params(vendor_base_url),+ )++ async with httpx.AsyncClient(timeout=20.0) as client:+ seen = await client.get(+ f"{b.base_url}/api/v1/actions/{action_id}",+ headers={"Authorization": f"Bearer {SERVICE_TOKEN}", "X-Tenant-ID": str(uuid.uuid4())},+ )+ assert seen.status_code == 404, seen.text++ async def test_an_unscoped_read_does_not_disclose_the_target(self, replicas, vendor_base_url) -> None:+ """No caller in the tree sends this route a tenant, so it stays usable —+ but an id is not a licence to read what a host or account was aimed at."""+ a, _b = replicas+ tenant = str(uuid.uuid4())+ action_id = str(uuid.uuid4())+ agent = _principal(tenant, user_id="agents-service")+ await _submit(+ a,+ tenant=tenant,+ action_id=action_id,+ requester=agent,+ principal=agent,+ target="ws-secret-name",+ params=_vendor_params(vendor_base_url),+ )++ async with httpx.AsyncClient(timeout=20.0) as client:+ seen = await client.get(+ f"{a.base_url}/api/v1/actions/{action_id}",+ headers={"Authorization": f"Bearer {SERVICE_TOKEN}"},+ )+ assert seen.status_code == 200, seen.text+ body = seen.json()+ assert body["status"] == "awaiting_approval"+ assert "ws-secret-name" not in json.dumps(body), f"an unscoped read disclosed the target: {body}"+ assert "parameters" not in body