Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

fix(actions): an approved action reaches the vendor once, or visibly not at all (#1528)

Beenu Arora54deb0338003
diff --git a/.github/workflows/action-durability-live.yml b/.github/workflows/action-durability-live.yml
new file mode 100644
index 000000000..298f9c3c5
--- /dev/null
+++ b/.github/workflows/action-durability-live.yml
@@ -0,0 +1,237 @@
+# An approved action reaches the vendor once, or visibly not at all.
+#
+# Remediation phase 2. The approve path in services/actions read a record,
+# compared its status in Python, called the vendor, and saved afterwards.
+# Everything an approval is supposed to guarantee lived in that gap: two
+# replicas both executed, a vendor that accepted a containment and dropped the
+# response was recorded as "failed", an approver in any tenant could approve
+# any tenant's containment, and a crash left a record indistinguishable from
+# an action nobody started.
+#
+# Why this job and not a unit test
+# --------------------------------
+# `action_store` answers from a process-local dict, so one process hands two
+# concurrent approvals the *same object* and the race cannot be observed at
+# all. The decisive case needs two containers behind one database — which is
+# the deployed shape — and a real Postgres, because the whole mechanism is a
+# conditional UPDATE and a row lock.
+#
+# The vendor is a stub, and only the vendor. Authorization, persistence and
+# retry are the things under test and are not replaced by anything: a mock
+# that answers whatever it is asked cannot notice an action that ran twice.
+#
+# No `paths:` filter. This grades a governed response capability, and a path
+# filter is how a required check reports success having examined nothing.
+name: Action durability (live)
+
+on:
+ push:
+ branches: [main]
+ pull_request:
+ workflow_dispatch:
+
+concurrency:
+ group: action-durability-live-${{ github.ref }}-${{ github.event_name }}-${{ github.event.pull_request.number || github.sha }}
+ cancel-in-progress: ${{ github.event_name == 'pull_request' }}
+
+permissions:
+ contents: read
+
+jobs:
+ action-durability-live:
+ name: One approval, one vendor call, or an honest unknown
+ runs-on: ubuntu-latest
+ timeout-minutes: 25
+
+ env:
+ # The runtime role, not the owner. A superuser ignores every RLS policy
+ # in the schema, and a live suite that runs as one is measuring a
+ # posture nothing ships with.
+ ACTIONS_DSN: postgresql://aisoc_app:aisoc_app_ci_only@localhost:5432/aisoc
+ PGPASSWORD: aisoc_ci_only
+ ENVIRONMENT: test
+ OTEL_SDK_DISABLED: "true"
+
+ steps:
+ - uses: actions/checkout@3d3c42e5aac5ba805825da76410c181273ba90b1 # v7
+
+ - name: Start PostgreSQL with registry fallback
+ uses: ./.github/actions/start-postgres
+ with:
+ password: aisoc_ci_only
+
+ - uses: actions/setup-python@e9d6f990972a57673cdb72ec29e19d42ba28880f # v6
+ with:
+ python-version: "3.11"
+
+ # Both services' own lockfiles, not a hand-curated pip list. The test
+ # process imports services/api's actions client and approval dispatch;
+ # the subprocesses it starts are services/actions. One interpreter runs
+ # both, selected per step by `working-directory`, because each names its
+ # top-level package `app` and importing one shadows the other.
+ - name: Install api from its lockfile
+ working-directory: services/api
+ run: |
+ set -euo pipefail
+ python -m pip install --quiet --upgrade pip "poetry==2.4.1"
+ poetry config virtualenvs.create false
+ poetry install --no-root --no-interaction
+
+ - name: Install actions from its lockfile
+ working-directory: services/actions
+ run: |
+ set -euo pipefail
+ poetry config virtualenvs.create false
+ poetry install --no-root --no-interaction
+ python -m pip install --quiet pytest pytest-asyncio greenlet
+
+ - name: Apply every migration, in order
+ run: |
+ set -euo pipefail
+ for f in services/api/migrations/*.sql; do
+ psql -h localhost -U aisoc -d aisoc -v ON_ERROR_STOP=1 -q -f "$f"
+ done
+
+ - name: Give the runtime role its password
+ run: |
+ set -euo pipefail
+ psql -h localhost -U aisoc -d aisoc -v ON_ERROR_STOP=1 -q -c \
+ "ALTER ROLE aisoc_app WITH PASSWORD 'aisoc_app_ci_only';"
+ psql -h localhost -U aisoc -d aisoc -v ON_ERROR_STOP=1 -q -c \
+ "GRANT USAGE ON SCHEMA public TO aisoc_app;"
+
+ - name: Identity, intent, the claim, and reconciliation
+ working-directory: services/api
+ env:
+ ISOLATION_ACTIONS_DSN: ${{ env.ACTIONS_DSN }}
+ PYTHONPATH: .
+ run: python -m pytest ../../tests/isolation/test_action_durability_live.py -v --tb=short
+
+ - name: The suite actually ran
+ working-directory: services/api
+ env:
+ ISOLATION_ACTIONS_DSN: ${{ env.ACTIONS_DSN }}
+ PYTHONPATH: .
+ run: |
+ set -euo pipefail
+ python -m pytest ../../tests/isolation/test_action_durability_live.py -q 2>&1 | tee /tmp/durability-run.txt
+ if grep -qE '[0-9]+ skipped' /tmp/durability-run.txt; then
+ echo "FAIL: a case skipped in the job whose whole purpose is to run it live."
+ exit 1
+ fi
+ if ! grep -qE '[0-9]+ passed' /tmp/durability-run.txt; then
+ echo "FAIL: no case reported as passed."
+ exit 1
+ fi
+
+ # ── negative controls ────────────────────────────────────────────────
+ #
+ # Each removes exactly one line of the fix and requires the suite to go
+ # red. A gate nobody has watched fail is a gate nobody has tested.
+
+ - name: The gate fails when the claim stops checking the status
+ working-directory: services/api
+ env:
+ ISOLATION_ACTIONS_DSN: ${{ env.ACTIONS_DSN }}
+ PYTHONPATH: .
+ run: |
+ set -euo pipefail
+ store=../actions/app/services/action_store.py
+ cp "$store" /tmp/action_store.bak
+ python - <<'PY'
+ import pathlib, sys
+ p = pathlib.Path("../actions/app/services/action_store.py")
+ s = p.read_text()
+ needle = " AND status = $3\n"
+ if needle not in s:
+ print("negative control could not find the claim's status predicate; the claim has "
+ "been reshaped and this control no longer proves anything")
+ sys.exit(1)
+ p.write_text(s.replace(needle, "", 1))
+ PY
+ set +e
+ python -m pytest \
+ "../../tests/isolation/test_action_durability_live.py::TestExecutionIsClaimedBeforeTheVendorIsContacted" \
+ -q >/dev/null 2>&1
+ rc=$?
+ set -e
+ cp /tmp/action_store.bak "$store"
+ if [ "$rc" -eq 0 ]; then
+ echo "FAIL: without the status predicate both replicas claimed and the suite stayed green."
+ exit 1
+ fi
+ echo "OK: without the predicate both replicas execute and the suite goes red, as it must."
+
+ - name: The gate fails when the upsert stops protecting a pending intent
+ working-directory: services/api
+ env:
+ ISOLATION_ACTIONS_DSN: ${{ env.ACTIONS_DSN }}
+ PYTHONPATH: .
+ run: |
+ set -euo pipefail
+ store=../actions/app/services/action_store.py
+ cp "$store" /tmp/action_store.bak
+ python - <<'PY'
+ import pathlib, sys
+ p = pathlib.Path("../actions/app/services/action_store.py")
+ s = p.read_text()
+ needle = " WHERE aisoc_action_records.status NOT IN {_INTENT_LOCKED_SQL_LIST}\n"
+ if needle not in s:
+ print("negative control could not find the upsert's intent guard")
+ sys.exit(1)
+ p.write_text(s.replace(needle, " WHERE true\n", 1))
+ PY
+ set +e
+ python -m pytest \
+ "../../tests/isolation/test_action_durability_live.py::TestApprovalIsBoundToTheIntent" \
+ -q >/dev/null 2>&1
+ rc=$?
+ set -e
+ cp /tmp/action_store.bak "$store"
+ if [ "$rc" -eq 0 ]; then
+ echo "FAIL: a pending approval was retargeted and the suite stayed green."
+ exit 1
+ fi
+ echo "OK: without the guard a pending approval is retargeted, and the suite goes red."
+
+ # Tenant isolation on the approve path is two layers, so a one-line
+ # removal proves nothing: take out the narrowing and the authorization
+ # check refuses, take out the check and the narrowed read 404s. The
+ # control removes both, which is also the honest statement of what is
+ # being claimed — that the pair is load-bearing, not either alone.
+ - name: The gate fails when nothing scopes an approval to a tenant
+ working-directory: services/api
+ env:
+ ISOLATION_ACTIONS_DSN: ${{ env.ACTIONS_DSN }}
+ PYTHONPATH: .
+ run: |
+ set -euo pipefail
+ cp ../actions/app/security/authz.py /tmp/authz.bak
+ cp ../actions/app/api/router.py /tmp/router.bak
+ python - <<'PY'
+ import pathlib, sys
+ authz = pathlib.Path("../actions/app/security/authz.py")
+ router = pathlib.Path("../actions/app/api/router.py")
+ a, r = authz.read_text(), router.read_text()
+ check = " if approver.tenant_id is not None and record_tenant and str(approver.tenant_id) != record_tenant:\n"
+ narrow = " record = await _load_for_decision(action_id, _tenant_of(bound))\n"
+ if check not in a or narrow not in r:
+ print("negative control could not find both tenant layers; they have been reshaped "
+ "and this control no longer proves anything")
+ sys.exit(1)
+ authz.write_text(a.replace(check, " if False:\n", 1))
+ router.write_text(r.replace(narrow, " record = await _load_for_decision(action_id, None)\n"))
+ PY
+ set +e
+ python -m pytest \
+ "../../tests/isolation/test_action_durability_live.py::TestTheThreeIdentitiesStayDistinct::test_an_approver_from_another_tenant_is_refused" \
+ -q >/dev/null 2>&1
+ rc=$?
+ set -e
+ cp /tmp/authz.bak ../actions/app/security/authz.py
+ cp /tmp/router.bak ../actions/app/api/router.py
+ if [ "$rc" -eq 0 ]; then
+ echo "FAIL: with neither layer present another tenant's containment was still refused."
+ exit 1
+ fi
+ echo "OK: with neither layer present another tenant's host is contained, and the suite goes red."
diff --git a/docs/remediation/phase-2-action-durability.md b/docs/remediation/phase-2-action-durability.md
new file mode 100644
index 000000000..974f87abf
--- /dev/null
+++ b/docs/remediation/phase-2-action-durability.md
@@ -0,0 +1,483 @@
+# Phase 2 — approval identity and durable action execution
+
+`services/actions` is the only part of the platform that changes somebody
+else's estate. Its approve path was a read, a status comparison in Python, the
+vendor call, and a durable write afterwards:
+
+```text
+record = await action_store.get(action_id) # no tenant, cache first
+if record["status"] != ActionStatus.AWAITING_APPROVAL: ... # checked in Python
+result = await executor.execute(request) # the host is contained
+await action_store.save(record) # ... and only now recorded
+```
+
+Every guarantee an approval is supposed to carry lived in the gap between the
+third line and the fourth. This document records what was reproduced, what was
+changed, how it was verified, and what is still open.
+
+All seven findings below are **reproduced** — each was observed failing on the
+tree as it stood, not inferred from reading it.
+
+| # | Finding | Status | Regression test |
+| --- | --- | --- | --- |
+| 8 | No atomic claim; durable write after the vendor call | reproduced | `TestExecutionIsClaimedBeforeTheVendorIsContacted` |
+| 9 | No unknown state; no `try/except` on the approve dispatch | reproduced | `TestAnUnknownOutcomeIsRecordedAsUnknown` |
+| 12 | Decision and read paths carry no tenant | reproduced | `test_an_approver_from_another_tenant_is_refused`, `TestReadingAnActionRecordIsTenantScoped` |
+| 13 | Unconditional upsert replaces a pending action's intent | reproduced | `TestApprovalIsBoundToTheIntent` |
+| 15 | `requested_by` is caller-supplied free text | source-confirmed | — (see below) |
+| 16 | Separation of duties passes silently with no requester | reproduced | `test_separation_of_duties_is_not_waived_by_an_absent_requester` |
+| **17** | **The console approval path refuses every approval** | **reproduced (new)** | `test_an_agent_originated_request_is_approved_by_a_human`, `test_the_real_console_dispatch_path_approves` |
+
+Everything lives in one live suite, `tests/isolation/test_action_durability_live.py`,
+graded by `.github/workflows/action-durability-live.yml`.
+
+---
+
+## Finding 17 — every approval raised in the console was refused upstream
+
+Not on the brief's list. Found by running the path rather than reading it, and
+it is the most consequential item here.
+
+**Affected production path.** `POST /api/v1/approvals/{id}/decide` →
+`approvals._dispatch_decision` → `actions_client.submit_action` →
+`POST /api/v1/actions` → `actions_client.decide_action` →
+`POST /api/v1/actions/{id}/approve`.
+
+**What happened.** `_dispatch_decision` built one principal — the deciding
+console user — and sent it as the submit's `principal`. `router.submit_action`
+records `principal.user_id` as `requested_by_user_id`. The same principal was
+then sent as the approver, so `authorize_approver` compared the approver
+against itself and raised *"approver cannot be the requester (separation of
+duties)"*. The 403 became an `ActionsServiceError`, which `_dispatch_decision`
+recorded as `{"state": "failed", "stage": "decide"}`, which surfaced to the
+operator as a 502 reading *"Decision recorded, but the action service refused
+it"*.
+
+The code was aware of the distinction and solved it in the wrong field: it
+passed `requested_by="agent"` with a comment saying *"Sending the approver here
+would make them both, and separation of duties would pass by accident."*
+`requested_by` is a free-text label that nothing compares; `principal` is what
+the comparison reads.
+
+Reproduced directly against the shipped app before anything was changed:
+
+```
+submit: 200 awaiting_approval requested_by_user_id= e84b8dfe-980c-4210-bda6-0741cea66aa8
+approve: 403 {"detail":"approver cannot be the requester (separation of duties)"}
+```
+
+**Fix.** `ActionRequest` gains `requester`, the originating identity, separate
+from `principal`, the invoking identity whose permissions gate the submit. The
+three identities are now distinct by construction:
+
+| Identity | Field | Used for |
+| --- | --- | --- |
+| who originally asked | `requester` | separation of duties, provenance |
+| who relayed the call | `principal` | least-privilege check on the submit |
+| who approved | `approver` on the approve call | permission tier, tenant, separation of duties |
+
+`approvals.create_approval` stamps the authenticated principal that raised the
+approval into `action["requester_principal"]` — server-written, always
+overwritten, never read from the request body — and `_dispatch_decision` sends
+it as `requester`. For an agent-raised approval that is the agents service's
+own service principal, so the agent identity survives a human approving it;
+for one a human raised it is that human, so self-approval is still refused.
+
+**Compatibility.** `requester` is optional. When absent, `requested_by_user_id`
+falls back to `principal.user_id`, which is exactly what shipped — so a caller
+that submits on its own behalf is unchanged. Approvals created before this
+change carry no stamp; their dispatch sends no `requester`, separation of
+duties has nothing to compare, and the action records
+`separation_of_duties: "unevaluated"` rather than claiming it passed.
+
+---
+
+## Finding 8 — two replicas could both contain the host
+
+**Affected production path.** `POST /api/v1/actions/{id}/approve`, and the
+auto-execute branch of `POST /api/v1/actions`.
+
+**Reproduced.** Two approvals in flight against one action, from two replicas
+behind one Postgres, produced **two** containment requests at the recording
+vendor. On a single replica the race is invisible: `action_store.get` answered
+from a module-global dict and handed both callers *the same object*, so the
+status check and the mutation operated on one record. That is why a second
+container is required to see it at all, and why this suite starts two.
+
+**Fix.** `action_store.claim_for_execution` — one conditional `UPDATE` moving
+`awaiting_approval → running`, scoped by id, tenant and expected status, which
+writes the claim and the claiming worker into the record and returns the row:
+
+```sql
+UPDATE aisoc_action_records
+ SET status = $4, record = jsonb_set(...), updated_at = now()
+ WHERE id = $1 AND tenant_id = $2 AND status = $3
+RETURNING record
+```
+
+Postgres takes a row lock, so a second claimant blocks until the first commits
+and then re-evaluates the predicate against the committed row, finds `running`
+and matches nothing. That re-evaluation is the whole mechanism; it is the same
+one `scheduler_leases` (migration 115) uses, and it is why this is not a
+`SELECT` followed by an `UPDATE` — that pair has no re-evaluation and elects
+two executors.
+
+The claim is also the fail-closed point. `save` still tolerates a store
+failure, deliberately: losing a *pending* containment because the store blinked
+is the failure the module was written to prevent, and an outcome write that
+raised would discard the only record of a vendor already touched. A claim is
+different — it *is* the authorization for an external effect — so it returns
+`STORE_UNAVAILABLE` and the route answers 503 having contacted nobody.
+
+Finally, the durable write now goes first and decision reads pass
+`authoritative=True`, which skips the process-local cache when a DSN is
+configured. That cache never expired: the submitting replica held an
+`awaiting_approval` copy for the life of the process and would re-execute an
+action another replica had already run.
+
+**Compatibility.** `RUNNING` already existed in `ActionStatus` and nothing
+wrote it; it is now written before every dispatch. With no DSN configured —
+the single-process and unit-test configuration — the claim is an in-memory
+compare-and-set under an `asyncio.Lock`, so the 1,237-case offline suite runs
+unchanged.
+
+---
+
+## Finding 9 — a vendor that acted was recorded as having failed
+
+**Affected production path.** The dispatch in both `approve_action` and the
+auto-execute branch of `submit_action`.
+
+**Reproduced, and the shape is not quite what the brief describes.** There was
+indeed no `try/except` around `approve_action`'s `executor.execute`, and no
+status meaning "unknown". But the executors catch their *own* vendor errors and
+return `ActionResult(status=FAILED, error=str(exc))`, so the dominant symptom
+was not a 500 — it was a record reading `failed` for a containment the EDR had
+accepted. That is the dangerous direction: the obvious response to a failure is
+to run it again.
+
+**Fix, and why not exception classification.** Classifying the exception
+answers "did the client raise", and the question is "did the estate change" —
+for the case that matters, a vendor that accepted a containment and then
+dropped the response, those two answers disagree. So a dispatch that does not
+come back clean is resolved by *reading the vendor back*, using
+`PostActionVerifier`, which already re-queries with the same clients the
+forward action uses and already refuses to invent a `VERIFIED` it did not earn:
+
+| Read-back | Recorded as |
+| --- | --- |
+| effect present | `completed` (the response was lost, not the action) |
+| effect absent | `failed` |
+| cannot be read | `reconciliation_required` |
+
+`ActionStatus.RECONCILIATION_REQUIRED` is the new member. An action in it
+cannot be approved again — the claim's `expected_status` does not match, and
+the 409 names the reconcile route.
+
+**One deliberate limit.** Below MEDIUM blast radius the executor's own verdict
+stands and no probe runs. A reconciliation queue filled with retryable Slack
+messages and tickets is one an operator learns to ignore, and a control nobody
+reads is not a control. `_UNKNOWN_OUTCOME_FLOOR` in `router.py` is where that
+line is drawn, and `test_a_refused_dispatch_is_still_a_failure` is the control
+proving the rule does not collapse into "everything is unknown".
+
+**The crash window in `approvals.py`** was real: `_dispatch_decision` ran at one
+line and `db.commit()` at the next, so a crash in between left the approval
+`pending` against an action that had run, with nothing in the row saying a
+dispatch had been attempted. The decision is now committed first with
+`dispatch.state = "dispatching"`, then dispatched, then committed again with
+the outcome. That costs the property the old ordering protected — a window
+where the row says approved and the action has not run — and makes that window
+*visible*, which is the one that matters: a repeat decide re-enters on an
+unresolved dispatch, and the actions service's claim is what stops a re-entry
+from executing twice.
+
+`actions_client` now distinguishes a refusal from silence: a `ConnectError`
+wrote nothing to a socket, while a timeout or a dropped connection on a POST
+may well have been processed. The second raises with `indeterminate=True`, and
+`_dispatch_decision` records `{"state": "unknown"}` and answers **504** with
+"whether the action ran is not yet known" rather than 502 "the action did not
+run".
+
+**Compatibility.** `reconciliation_required` is a new status string. Checked
+against every consumer:
+
+* `services/api`'s usage meter keys `EXECUTED_STATUS` on `"completed"`, so an
+ unresolved action is correctly *not* counted as a vendor touch.
+* `services/slack-bot`'s `commands.py` tests for
+ `{"awaiting_approval", "pending_approval", "pending"}` to decide whether to
+ post an approval card; anything else renders as a status line, which is
+ right.
+* `packages/types`' `AlertAction.status` union is a console-side concept that
+ already does not mirror `ActionStatus` (it has no `awaiting_approval`,
+ `approved` or `rejected`, and its `type` union contains verbs `ActionType`
+ has never had), so it is not a consumer of this enum.
+* The status column is written through `_column_text`, so the new member lands
+ as `reconciliation_required` and not `ActionStatus.RECONCILIATION_REQUIRED` —
+ the defect migration 095 had to repair.
+
+---
+
+## Finding 12 — an id was a capability across tenants
+
+**Affected production path.** `approve_action`, `reject_action` and
+`get_action`.
+
+**Reproduced.** An approver holding `actions:execute:high` in a freshly-made
+tenant approved another tenant's `isolate_host` from the second replica, and
+the host was contained. `GET /api/v1/actions/{id}` returned the full record —
+target, parameters and all — to any holder of the shared service token.
+
+`action_store.get`'s docstring said a tenant "should always be supplied from a
+request path"; no request path supplied one. Its claim that the table "carries
+no RLS policy" is **imprecise**: migration 060 does create
+`aisoc_action_records_tenant`. The consequence stated is still right, for a
+different reason — the policy reads `app.current_tenant_id`, which nothing in
+this service sets, and its unset arm is deliberately permissive so cross-tenant
+workers are not starved. The policy is inert here, so the query predicate is
+the only control.
+
+**Fix.** Two layers on the decision paths:
+
+1. The deciding identity is resolved **before** the record is read
+ (`_resolve_identity`), and the read is narrowed by its tenant. Fetching
+ globally and comparing in Python is the same defect one line later.
+2. `authorize_approver` compares the approver's tenant to the action's, which
+ `authorize_action` has done since it was written and only the approve side
+ went without.
+
+The claim carries `AND tenant_id = $2` as well, so the write cannot land
+cross-tenant even if both reads were wrong.
+
+On `get_action`: a caller declaring `X-Tenant-ID` gets the record when it owns
+it and **404** when it does not — never 403, which would make the route an
+existence oracle over every tenant's action ids. A caller declaring none keeps
+working and is answered with status only (`_UNSCOPED_READ_FIELDS`), so the
+route stays usable for polling without handing out the host an action was
+aimed at. The header is read off `request.headers` rather than declared as a
+`Header(...)` parameter, because one such parameter adds a parameter and a 422
+response to the published OpenAPI document for every operation it guards.
+
+**Compatibility.** No caller of `GET /api/v1/actions/{id}` exists anywhere in
+the tree — `services/slack-bot` and `services/api` call only `/approve` and
+`/reject` — so the redaction breaks nothing today and the full record remains
+one header away. `docs/openapi.yaml` is byte-identical after the change
+(`scripts/export_openapi.py --check`: *"docs/openapi.yaml is up to date"*).
+
+---
+
+## Finding 13 — a pending approval could be retargeted
+
+**Affected production path.** `POST /api/v1/actions` with an id that already
+exists.
+
+**Reproduced.** Submitting `ws-4471`, then the same id with `target: "dc-01"`,
+left the record naming `dc-01` while the approval card said `ws-4471`.
+
+**Fix.** The upsert's `DO UPDATE` is now conditional, comparing the three
+fields that define what an approver signed off:
+
+```sql
+ WHERE aisoc_action_records.status NOT IN ('awaiting_approval','running','reconciliation_required')
+ OR ( record->>'action_type' IS NOT DISTINCT FROM EXCLUDED.record->>'action_type'
+ AND record->>'target' IS NOT DISTINCT FROM EXCLUDED.record->>'target'
+ AND record->'parameters' IS NOT DISTINCT FROM EXCLUDED.record->'parameters' )
+RETURNING id
+```
+
+`RETURNING id` is how the refusal is detected: an `ON CONFLICT DO UPDATE` whose
+`WHERE` is unsatisfied returns no row, which is the only signal Postgres gives.
+`save` raises `ActionIntentConflict` and the route answers 409.
+
+`rationale` and `confidence` are deliberately **not** part of the intent: the
+first is prose shown to a human, the second is re-graded by the gate on every
+submission, and neither changes what happens to the estate.
+
+**Compatibility.** The store's standing contract — *"a replay of the same
+submission is idempotent rather than a duplicate pending approval"* — is
+preserved and pinned by `test_an_identical_resubmission_is_still_idempotent`.
+Only a *changed* intent is refused, and only while a decision is outstanding.
+
+---
+
+## Finding 16 — separation of duties passed on the requests it could not evaluate
+
+`if separation_of_duties and requester and str(requester) == str(approver.user_id)`
+— the `and requester` arm meant a submission carrying no principal produced an
+approval the control reported as satisfied, and the row afterwards was
+indistinguishable from one where it had been checked and passed.
+
+**Fix.** `authorize_approver` returns which state applied and the route records
+it on the action as `separation_of_duties: "enforced" | "unevaluated"`.
+
+Refusing instead was considered and rejected: `AISOC_ACTIONS_REQUIRE_PRINCIPAL`
+defaults false, and a hard denial would brick every legacy system-initiated
+action during an incident. The honest fix is that it must *be a fact in the
+row*, not an omission.
+
+## Finding 15 — `requested_by` is caller-supplied free text
+
+**Source-confirmed, deliberately not "fixed" by trusting it.** It remains a
+label. Separation of duties reads `requester` / `principal`, which arrive from
+authenticated context, and the fix for finding 17 is what makes the originating
+identity trustworthy rather than cosmetic. Using `requested_by` to *refuse* an
+approval was considered and rejected: a caller controlling that string could
+name an analyst and block them from approving during an incident, which trades
+a non-bypass for a denial of service on the containment path.
+
+---
+
+## No migration
+
+`aisoc_action_records` needs no schema change, and that is a decision rather
+than an omission:
+
+* The claim's predicates are the existing `id`, `tenant_id` and `status`
+ columns, and `ix_action_records_tenant_status` already serves the operator's
+ `status = 'running'` scan.
+* Dispatch provenance (`dispatch.claimed_by`, `dispatch.claimed_at`,
+ `verification`, `approved_at`) lives in the JSONB record, which is the
+ argument migration 055 makes for storing it whole: *"mirroring it as columns
+ here would make every contract change a migration"*.
+* The intent comparison is a JSONB expression, so there is no second definition
+ of "the intent" to drift from the Python one and no backfill to get wrong.
+* Migration 061 already grants the runtime role DML on every table, and 060
+ already carries the RLS policy.
+
+## Recovery and reconciliation — what an operator runs
+
+### 1. Find actions that were claimed and never resolved
+
+```sql
+SELECT id, tenant_id, status, updated_at,
+ record->'dispatch'->>'claimed_by' AS claimed_by,
+ record->'dispatch'->>'claimed_at' AS claimed_at,
+ record->>'action_type' AS action_type,
+ record->>'target' AS target
+ FROM aisoc_action_records
+ WHERE status IN ('running', 'reconciliation_required')
+ AND updated_at < now() - interval '10 minutes'
+ ORDER BY updated_at;
+```
+
+`running` means a worker took the action and has not written an outcome — it
+either died mid-dispatch or is still inside a vendor call.
+`reconciliation_required` means it finished and could not establish what
+happened. `claimed_by` is `hostname/pid`, which is how you find the container.
+
+**A stale claim is never stolen automatically.** A timeout that reassigns a
+claim is exactly the duplicate execution this work exists to prevent: the
+previous holder may have reached the vendor. Resolution is always the probe
+below.
+
+### 2. Reconcile by asking the vendor
+
+```bash
+curl -sS -X POST "$ACTIONS_URL/api/v1/actions/$ACTION_ID/reconcile" \
+ -H "Authorization: Bearer $AISOC_ACTIONS_SERVICE_TOKEN" \
+ -H "X-Tenant-ID: $TENANT_ID" \
+ -H 'Content-Type: application/json' \
+ -d '{"parameters": {"cs_client_id": "...", "cs_client_secret": "..."}}'
+```
+
+`parameters` is merged over the stored ones and is how you present a current
+credential rather than depending on whatever was captured at submit time. The
+route **never re-executes** — it runs the read-back probe only.
+
+| Response `status` | Meaning | Next step |
+| --- | --- | --- |
+| `completed` | the effect is in force; the response was lost, not the action | nothing |
+| `failed` | the vendor was read and the effect is absent | submit a **new action id** if it is still wanted |
+| `reconciliation_required` | the vendor could not be read | fix the credential or vendor reachability and run it again; check by hand if it stays |
+
+`verification.reason` on the returned record says which probe ran and what it
+saw. A verb with no registered probe reports
+`no read-back verifier for <verb>`, which stays `reconciliation_required` — the
+honest answer, and never a guess in either direction.
+
+### 3. Never do this
+
+* Do not re-approve. The claim refuses it (409) and the refusal is the point.
+* Do not hand-edit `status` to `completed` without a probe that said so — that
+ is the fabricated-verdict shape the project disavows everywhere else.
+* Do not resubmit the **same** action id with different parameters: the upsert
+ refuses it (409) while a decision is outstanding, by design.
+
+## Verification performed
+
+Against PostgreSQL 16 with all 135 migrations applied, replicas connecting as
+the runtime role `aisoc_app` (`NOBYPASSRLS`, not the owner), two actions
+containers on loopback sockets, and a recording HTTP vendor.
+
+| Check | Result |
+| --- | --- |
+| `tests/isolation/test_action_durability_live.py` **before** the fix | 20 failed, 7 passed |
+| the same suite **after** | **27 passed** |
+| `services/actions` offline suite | 1,237 passed (baseline 1,234 + 3 new) |
+| `services/api` offline suite | 4,964 passed, 164 skipped |
+| `ruff check services/ scripts/ tests/ tools/` | All checks passed |
+| `ruff format --check .` (0.16.10, the pin) | 2,596 files already formatted |
+| `scripts/check_mypy_baseline.py` | OK — no tree got worse |
+| `scripts/export_openapi.py --check` | docs/openapi.yaml is up to date |
+| `check_tenant_query_predicates`, `check_raw_sql_columns`, `check_route_authz`, `check_action_contract`, `check_attribution`, `check_competitor_names`, `check_machine_paths`, `check_gate_coverage`, `check_text_bind_casts`, `check_comment_paths`, `check_logger_kwargs` | all pass |
+
+### Negative controls
+
+Each removes one piece of the fix and requires the suite to go red. All three
+were executed, not asserted, and are wired into the workflow.
+
+| Control | Result |
+| --- | --- |
+| drop `AND status = $3` from the claim | 3 of 4 cases fail, including both replicas containing the host |
+| make the upsert unconditional (`WHERE true`) | the pending approval is retargeted to `dc-01` — 409 becomes 200 |
+| remove both tenant layers | tenant A's approver contains tenant B's host and is returned the full record |
+
+The third removes two lines rather than one deliberately: tenant isolation here
+is the narrowed read *plus* the authorization check, and removing either alone
+leaves the other refusing, so a one-line control would prove nothing. Saying
+which pair is load-bearing is the accurate claim.
+
+### Two test expectations were corrected after seeing the implementation
+
+Written against a weaker design, both now assert a stronger property, and both
+were re-run against the pre-fix tree to confirm they still fail there (they are
+in the 20).
+
+* A lost vendor response was expected to produce `reconciliation_required`. The
+ implementation reads the device back, finds it contained, and records
+ `completed` — which is better, and the test now asserts that plus a separate
+ case where the device API is down.
+* "No probe available" was expressed by sending empty `parameters`, which the
+ reconcile route merges over the stored ones, so the probe ran. It now makes
+ the vendor's read-back route unavailable instead.
+
+## Remaining limitations
+
+* **No hosted vendor has been exercised.** The forward call and the read-back
+ both go to a stub on a loopback socket. That proves the client constructs,
+ authenticates, and that the two are distinct routes; it does not prove any
+ vendor's real API behaves as modelled.
+* **Vendor-side idempotency keys are not used**, because none of the clients in
+ `services/actions/app/clients/` accepts one. The duplicate-execution
+ guarantee is therefore entirely ours — the claim — rather than ours plus the
+ vendor's. Adding `Idempotency-Key` per client is tracked work, not done here.
+* **The unknown-outcome rule has a floor.** Below MEDIUM blast radius an
+ unverifiable failure is still recorded as `failed`. Stated above with its
+ reasoning; it is a judgement, not an oversight.
+* **Reconciliation is operator-driven.** There is no worker sweeping stale
+ claims, by choice: an automatic sweep that resolved an unreadable probe would
+ be guessing, and one that re-dispatched would duplicate. The SQL above is the
+ queue.
+* **`chatops_callback` still mutates the record without saving it.** Reached
+ only by the ChatOps verification link, pre-existing, and outside this phase's
+ scope — but it means a user's reply updates the in-process copy and not the
+ durable row on any other replica.
+* **The API-side crash window is narrowed, not closed.** A crash between the
+ first commit and the dispatch leaves `dispatch.state = "dispatching"`, which
+ a repeat decide re-enters safely. Nothing automatically re-enters it; a
+ stalled approval needs a human or a client retry.
+* **`services/api`'s own route layer is exercised by its offline suite, not
+ here.** The live suite drives `_dispatch_decision` with a stand-in ORM row
+ and principal, because the question it answers is what crosses the wire to
+ the actions service. The session, auth dependency and ORM behind
+ `decide_approval` are covered by `services/api/tests/`.
diff --git a/services/actions/app/api/router.py b/services/actions/app/api/router.py
index b87dca05c..0c512d455 100644
--- a/services/actions/app/api/router.py
+++ b/services/actions/app/api/router.py
@@ -4,23 +4,30 @@ Action Execution Service REST API.
from __future__ import annotations
+import os
+import socket
from datetime import UTC, datetime
from typing import Any
from uuid import UUID
import structlog
-from fastapi import APIRouter, Depends, HTTPException, Query
+from fastapi import APIRouter, Depends, HTTPException, Query, Request
from fastapi.responses import HTMLResponse
+from pydantic import BaseModel, Field
from app.core.config import get_settings
from app.models.action import (
+ ACTION_BLAST_RADIUS,
ActionPrincipal,
ActionRequest,
+ ActionResult,
ActionStatus,
ActionType,
+ BlastRadius,
ChatOpsApprover,
)
from app.security.authz import (
+ SOD_UNEVALUATED,
ActionAuthzError,
authorize_action,
authorize_approver,
@@ -29,15 +36,39 @@ from app.security.authz import (
from app.security.chatops_identity import resolve_approver
from app.security.chatops_token import ChatOpsTokenError, verify_token
from app.services import action_store
+from app.services.action_store import ActionIntentConflict, ActionStoreUnavailable, ClaimOutcome
from app.services.approval_gate import apply_matrix
from app.services.blast_radius import BlastRadiusGate
from app.services.executor_registry import EXECUTOR_REGISTRY
from app.services.timeline_client import TimelineClientError, post_timeline_event
+from app.services.verification import PostActionVerifier, VerificationOutcome
logger = structlog.get_logger()
router = APIRouter()
gate = BlastRadiusGate()
+#: Which replica claimed an execution. A hostname and a pid: enough for an
+#: operator holding a stuck `running` record to find the container that took
+#: it, and nothing a caller supplies.
+_WORKER_ID = f"{socket.gethostname()}/{os.getpid()}"
+
+#: The header a service-to-service caller declares its tenant on.
+#:
+#: Read off the raw request rather than declared as a `Header(...)` parameter:
+#: one such parameter on a route rewrites the published OpenAPI document with
+#: a parameter and a 422 response, and a tenant that services exchange between
+#: themselves does not belong in the public contract.
+_TENANT_HEADER = "X-Tenant-ID"
+
+#: Below this, an unverifiable failure is recorded as a failure.
+#:
+#: The rule above it is "unknown until the vendor says so", which is right for
+#: anything that changes an estate. Applied to a Slack message or a ticket it
+#: would fill the reconciliation queue with items whose duplicate costs
+#: nothing — and a queue an operator learns to ignore is not a control. So
+#: MINIMAL and LOW keep the executor's own verdict, and MEDIUM upward do not.
+_UNKNOWN_OUTCOME_FLOOR: frozenset[BlastRadius] = frozenset({BlastRadius.MEDIUM, BlastRadius.HIGH, BlastRadius.CRITICAL})
+
# The module-global `_actions` dict that used to live here is gone; storage is
# `app.services.action_store`, which persists to Postgres when a DSN is
# configured. A restart used to lose every action awaiting approval, so an
@@ -50,6 +81,127 @@ gate = BlastRadiusGate()
_chatops_replied: set[str] = set()
+def _requester_id(request: ActionRequest) -> str | None:
+ """Who asked for this action, for separation of duties.
+
+ The explicit originator when the caller supplied one, otherwise the
+ invoking principal — which is what this has always recorded, and is still
+ right for a caller that submits on its own behalf.
+
+ Never ``request.requested_by``: that is a free-text label a caller picks,
+ and a separation-of-duties comparison against it would be forgeable from
+ the request body.
+ """
+ if request.requester is not None:
+ return request.requester.user_id
+ if request.principal is not None:
+ return request.principal.user_id
+ return None
+
+
+async def _verify_effect(request: ActionRequest) -> tuple[VerificationOutcome, dict[str, Any]]:
+ """Ask the vendor whether the action's effect is actually present.
+
+ ``PostActionVerifier`` already re-queries the vendor with the same clients
+ the forward action uses, and already refuses to invent a ``VERIFIED`` it
+ did not earn. It is used here rather than classifying the exception,
+ because an exception type answers "did the client raise" and the question
+ is "did the estate change" — and for the case that matters, a vendor that
+ accepted a containment and then dropped the response, those two answers
+ disagree.
+ """
+ verdict = await PostActionVerifier().verify(request.action_type, request.target, dict(request.parameters or {}))
+ return verdict.outcome, {
+ "outcome": verdict.outcome.value,
+ "reason": verdict.reason,
+ "vendor": verdict.vendor,
+ "checked_at": datetime.now(UTC).isoformat(),
+ }
+
+
+async def _apply_outcome(
+ record: dict[str, Any],
+ request: ActionRequest,
+ result: ActionResult | None,
+ error: str | None,
+) -> None:
+ """Turn a dispatch into a status the record can be trusted on.
+
+ A clean success is a success. Anything else is unknown until the vendor is
+ read back: the executors catch their own vendor errors and return
+ ``ActionResult(status=FAILED, error=str(exc))``, so a containment the EDR
+ had accepted was recorded as ``failed`` — the reading whose obvious remedy
+ is to run it again.
+ """
+ if result is not None and result.status == ActionStatus.COMPLETED:
+ record["status"] = ActionStatus.COMPLETED
+ record["output"] = result.output
+ record["rollback_data"] = result.rollback_data
+ record.pop("error", None)
+ return
+
+ reported = error or (result.error if result is not None else None) or "the executor reported no outcome"
+ if result is not None and result.output:
+ record["output"] = result.output
+
+ blast = ACTION_BLAST_RADIUS.get(request.action_type, BlastRadius.HIGH)
+ if blast not in _UNKNOWN_OUTCOME_FLOOR:
+ record["status"] = ActionStatus.FAILED
+ record["error"] = reported
+ return
+
+ outcome, verification = await _verify_effect(request)
+ record["verification"] = verification
+ if outcome is VerificationOutcome.VERIFIED:
+ # The dispatch reported a problem and the effect is in force anyway:
+ # the response was lost, not the action.
+ record["status"] = ActionStatus.COMPLETED
+ record["dispatch_error"] = reported
+ record.pop("error", None)
+ return
+ if outcome is VerificationOutcome.FAILED:
+ record["status"] = ActionStatus.FAILED
+ record["error"] = reported
+ return
+ record["status"] = ActionStatus.RECONCILIATION_REQUIRED
+ record["error"] = reported
+
+
+async def _dispatch_claimed(record: dict[str, Any], request: ActionRequest) -> dict[str, Any]:
+ """Run an action whose execution this process has already claimed.
+
+ The claim is the durable authorization for the vendor call, so by the time
+ control reaches here the record says ``running`` in Postgres and no other
+ replica, retry or approval can start the same action. What is left is to
+ run it and write down what happened — including when "what happened" is
+ that nobody knows.
+ """
+ executor = EXECUTOR_REGISTRY.get(request.action_type)
+ if executor is None:
+ record["status"] = ActionStatus.FAILED
+ record["error"] = f"No executor found for action type: {request.action_type}"
+ else:
+ try:
+ result = await executor.execute(request)
+ await _apply_outcome(record, request, result, None)
+ except Exception as exc: # noqa: BLE001 — never leave a claimed action without an outcome
+ # The message is logged in full and never echoed: it can carry
+ # internal hostnames and stack detail.
+ logger.error("Action execution failed", action_id=str(request.id), error=str(exc), exc_info=True)
+ await _apply_outcome(record, request, None, f"execution failed ({type(exc).__name__})")
+
+ dispatch = dict(record.get("dispatch") or {})
+ dispatch["state"] = "resolved"
+ dispatch["resolved_at"] = datetime.now(UTC).isoformat()
+ record["dispatch"] = dispatch
+
+ try:
+ await action_store.save(record)
+ except ActionIntentConflict: # pragma: no cover - the intent cannot change under a claim
+ logger.error("action_store.outcome_write_refused", action_id=str(request.id))
+ return record
+
+
@router.post("/actions", response_model=dict)
async def submit_action(request: ActionRequest, _auth: None = Depends(require_service_auth)):
"""Submit an action for execution (may require approval)."""
@@ -84,8 +236,10 @@ async def submit_action(request: ActionRequest, _auth: None = Depends(require_se
"tenant_id": str(request.tenant_id),
"rationale": request.rationale,
# W4.4 — remember who requested it so an approver can't approve their
- # own action (separation of duties).
- "requested_by_user_id": request.principal.user_id if request.principal else None,
+ # own action (separation of duties). The originator when the caller
+ # distinguishes one, because a relaying service sending its own
+ # principal here makes the requester and the approver the same person.
+ "requested_by_user_id": _requester_id(request),
# Kept so approve can rebuild the request faithfully. Without this the
# approve path reconstructed an ActionRequest with an empty
# `parameters`, so an action that was gated *because* of what it
@@ -93,32 +247,40 @@ async def submit_action(request: ActionRequest, _auth: None = Depends(require_se
# approval that silently changes what it approved is not an approval.
"parameters": dict(request.parameters or {}),
}
- await action_store.save(record)
+ try:
+ await action_store.save(record)
+ except ActionIntentConflict as exc:
+ # A second submission of an id that is already awaiting a decision, with
+ # a different target or parameters. Refused rather than applied: an
+ # approval card showing one host while the record names another is an
+ # approval of something nobody read.
+ raise HTTPException(
+ status_code=409,
+ detail=("this action id is already awaiting a decision on a different target or parameters; submit a new action id"),
+ ) from exc
# Auto-execute if approved
if status == ActionStatus.APPROVED:
- executor = EXECUTOR_REGISTRY.get(request.action_type)
- if executor:
- try:
- result = await executor.execute(request)
- record["status"] = result.status
- record["output"] = result.output
- record["rollback_data"] = result.rollback_data
- if result.error:
- record["error"] = result.error
- except Exception as exc:
- # Log the full detail server-side, but never echo the raw
- # exception message (which can carry internal paths / stack
- # detail) back to the API caller — return only the error type.
- logger.error("Action execution failed", error=str(exc), exc_info=True)
- record["status"] = ActionStatus.FAILED
- record["error"] = f"execution failed ({type(exc).__name__})"
+ claim = await action_store.claim_for_execution(
+ str(request.id),
+ tenant_id=str(request.tenant_id),
+ expected_status=ActionStatus.APPROVED.value,
+ claimed_by=_WORKER_ID,
+ )
+ if claim.ok and claim.record is not None:
+ record = await _dispatch_claimed(claim.record, request)
+ elif claim.outcome is ClaimOutcome.STORE_UNAVAILABLE:
+ # The gate said this may run unattended and the store cannot record
+ # that it did. Refuse: an external effect nobody can prove was
+ # authorized is worse than an action an operator can resubmit.
+ raise HTTPException(
+ status_code=503,
+ detail="the action store is unavailable, so execution was not authorized; retry when it is reachable",
+ )
else:
- record["status"] = ActionStatus.FAILED
- record["error"] = f"No executor found for action type: {request.action_type}"
-
- # Persist again so the executed status is durable, not just the pending one.
- await action_store.save(record)
+ # Another replica took it, or it is no longer in a state that can
+ # run. Report what the store holds rather than executing.
+ record = claim.record or record
logger.info(
"Action submitted",
@@ -155,6 +317,33 @@ def _bind_approver(
human at all when it comes from the approval-timeout scheduler — requiring
one would leave expired requests stuck in ``awaiting_approval`` forever.
"""
+ approver = _resolve_identity(action_id, approver, assertion, require=require)
+ if approver is None:
+ return None
+
+ try:
+ record["separation_of_duties"] = authorize_approver(record, approver)
+ except ActionAuthzError as exc:
+ logger.warning("Approval denied", action_id=action_id, reason=str(exc))
+ raise HTTPException(status_code=403, detail=str(exc)) from exc
+ return approver
+
+
+def _resolve_identity(
+ action_id: str,
+ approver: ActionPrincipal | None,
+ assertion: ChatOpsApprover | None,
+ *,
+ require: bool,
+) -> ActionPrincipal | None:
+ """The identity half of :func:`_bind_approver`, with no record in hand.
+
+ Split out so the route can learn the deciding principal's tenant *before*
+ it reads the action, and narrow the read by it. Fetching globally and
+ comparing afterwards leaves the lookup itself unscoped, which on a table
+ whose row-level policy is inert on this service's connections means the id
+ is the only control.
+ """
if approver is None and assertion is not None:
approver = resolve_approver(assertion.platform, assertion.platform_user_id)
if approver is None:
@@ -171,13 +360,40 @@ def _bind_approver(
if require and (settings.AISOC_ACTIONS_REQUIRE_APPROVER or settings.AISOC_ACTIONS_REQUIRE_PRINCIPAL):
raise HTTPException(status_code=403, detail="an approver identity is required")
return None
+ return approver
+
+def _tenant_of(principal: ActionPrincipal | None) -> str | None:
+ """The tenant to narrow a read by, or None when the identity carries none.
+
+ A ChatOps approver comes from operator configuration and may legitimately
+ have no tenant; the narrowing is then skipped and
+ :func:`authorize_approver` is the control. The tenant is never taken from
+ the request body.
+ """
+ if principal is None or principal.tenant_id is None:
+ return None
+ return str(principal.tenant_id)
+
+
+async def _load_for_decision(action_id: str, tenant_id: str | None) -> dict[str, Any]:
+ """Read the record a decision will act on, from the database.
+
+ ``authoritative`` because the process-local cache is what let an approve
+ on the submitting replica resurrect an action another replica had already
+ executed, and because a store that is configured and unreachable must not
+ be reported as "no such action".
+ """
try:
- authorize_approver(record, approver)
- except ActionAuthzError as exc:
- logger.warning("Approval denied", action_id=action_id, reason=str(exc))
- raise HTTPException(status_code=403, detail=str(exc)) from exc
- return approver
+ record = await action_store.get(action_id, tenant_id=tenant_id, authoritative=True)
+ except ActionStoreUnavailable as exc:
+ raise HTTPException(
+ status_code=503,
+ detail="the action store is unavailable; the decision was not recorded and nothing was executed",
+ ) from exc
+ if not record:
+ raise HTTPException(status_code=404, detail="Action not found")
+ return record
@router.post("/actions/{action_id}/approve")
@@ -194,47 +410,103 @@ async def approve_action(
no resolvable identity is refused by default
(``AISOC_ACTIONS_REQUIRE_APPROVER``), because separation of duties cannot
be evaluated against nobody. ChatOps callers send ``chatops_approver``
- (verified platform identity) and the mapping supplies the permissions."""
- record = await action_store.get(action_id)
- if not record:
- raise HTTPException(status_code=404, detail="Action not found")
- if record["status"] != ActionStatus.AWAITING_APPROVAL:
- raise HTTPException(status_code=400, detail=f"Action is not awaiting approval (current: {record['status']})")
+ (verified platform identity) and the mapping supplies the permissions.
+
+ Phase 2 — the order below is the control. Identity, then a tenant-narrowed
+ authoritative read, then authorization, then an **atomic claim in durable
+ storage**, and only then the vendor. This used to be a read, a status
+ comparison in Python, the vendor call, and a save afterwards: two replicas
+ both read ``awaiting_approval`` and both contained the host, and if the
+ process died in between there was no record that either had tried.
+ """
+ bound = _resolve_identity(action_id, approver, chatops_approver, require=True)
+ record = await _load_for_decision(action_id, _tenant_of(bound))
- bound = _bind_approver(record, action_id, approver, chatops_approver, require=True)
if bound is not None:
+ _bind_approver(record, action_id, bound, None, require=True)
record["approved_by_user_id"] = bound.user_id
+ record["approved_at"] = datetime.now(UTC).isoformat()
+ else:
+ record["separation_of_duties"] = SOD_UNEVALUATED
+
+ if record["status"] != ActionStatus.AWAITING_APPROVAL:
+ raise HTTPException(status_code=400, detail=f"Action is not awaiting approval (current: {record['status']})")
+
+ claim = await action_store.claim_for_execution(
+ action_id,
+ tenant_id=str(record["tenant_id"]),
+ expected_status=ActionStatus.AWAITING_APPROVAL.value,
+ claimed_by=_WORKER_ID,
+ )
+ if not claim.ok:
+ raise _claim_refusal(action_id, claim)
+
+ # The claimed record, read back from the store, carries the approval
+ # stamps above only if they were written before the claim — they were not,
+ # so re-apply them to the authoritative copy rather than to the one this
+ # process happened to read.
+ claimed = claim.record or record
+ claimed["approved_by_user_id"] = record.get("approved_by_user_id")
+ claimed["approved_at"] = record.get("approved_at")
+ claimed["separation_of_duties"] = record.get("separation_of_duties", SOD_UNEVALUATED)
# Reconstruct the request as submitted, parameters included. Dropping
# them turned an approved action into a different action that still
# reported success.
request = ActionRequest(
id=UUID(action_id),
- incident_id=UUID(record["incident_id"]),
- tenant_id=UUID(record["tenant_id"]),
- action_type=ActionType(record["action_type"]),
- target=record["target"],
- rationale=record["rationale"],
- parameters=dict(record.get("parameters") or {}),
+ incident_id=UUID(claimed["incident_id"]),
+ tenant_id=UUID(claimed["tenant_id"]),
+ action_type=ActionType(claimed["action_type"]),
+ target=claimed["target"],
+ rationale=claimed.get("rationale") or "",
+ parameters=dict(claimed.get("parameters") or {}),
# The bound approver, not the original requester: the executor's
# authorization has to evaluate the identity that authorised this run.
principal=bound,
)
- executor = EXECUTOR_REGISTRY.get(request.action_type)
- if executor:
- result = await executor.execute(request)
- record["status"] = result.status
- record["output"] = result.output
- else:
- record["status"] = ActionStatus.FAILED
- record["error"] = "No executor available"
-
- await action_store.save(record)
- logger.info("Action approved and executed", action_id=action_id, status=record["status"])
+ record = await _dispatch_claimed(claimed, request)
+ logger.info(
+ "Action approved and executed",
+ action_id=action_id,
+ status=record["status"],
+ approved_by=bound.user_id if bound else None,
+ )
return record
+def _claim_refusal(action_id: str, claim: action_store.Claim) -> HTTPException:
+ """The refusal a losing or blocked claimant gets, by reason.
+
+ Three different operator actions hide behind "the claim did not take", so
+ they are three different answers: another worker has it (wait), the action
+ is in a state no approval can move (reconcile or resubmit), or the store
+ cannot be written (retry).
+ """
+ if claim.outcome is ClaimOutcome.STORE_UNAVAILABLE:
+ logger.error("approval_refused_store_unavailable", action_id=action_id)
+ return HTTPException(
+ status_code=503,
+ detail="the action store is unavailable, so execution was not authorized; nothing was sent to the vendor",
+ )
+ if claim.outcome is ClaimOutcome.NOT_FOUND:
+ return HTTPException(status_code=404, detail="Action not found")
+
+ current = claim.current_status or "unknown"
+ if current == ActionStatus.RUNNING.value:
+ detail = "this action is already being executed by another worker"
+ elif current == ActionStatus.RECONCILIATION_REQUIRED.value:
+ detail = (
+ "this action's outcome is unknown and it will not be run again; "
+ f"reconcile it with POST /api/v1/actions/{action_id}/reconcile before deciding anything further"
+ )
+ else:
+ detail = f"Action is not awaiting approval (current: {current})"
+ logger.info("approval_refused", action_id=action_id, current_status=current)
+ return HTTPException(status_code=409, detail=detail)
+
+
@router.post("/actions/{action_id}/reject")
async def reject_action(
action_id: str,
@@ -248,37 +520,166 @@ async def reject_action(
to take no identity and record none, so "who declined to contain this
host, and were they entitled to" had no answer — which matters as much as
the approve side during an incident review.
+
+ A rejection is a state transition like any other, so it is claimed rather
+ than written over whatever is there: rejecting an action a worker is in
+ the middle of executing would record a decision that did not happen.
"""
- record = await action_store.get(action_id)
- if not record:
- raise HTTPException(status_code=404, detail="Action not found")
+ bound = _resolve_identity(action_id, approver, chatops_approver, require=False)
+ record = await _load_for_decision(action_id, _tenant_of(bound))
- bound = _bind_approver(record, action_id, approver, chatops_approver, require=False)
if bound is not None:
+ _bind_approver(record, action_id, bound, None, require=False)
record["rejected_by_user_id"] = bound.user_id
- record["status"] = ActionStatus.REJECTED
- await action_store.save(record)
+ if record["status"] == ActionStatus.REJECTED:
+ return record
+
+ claim = await action_store.claim_for_execution(
+ action_id,
+ tenant_id=str(record["tenant_id"]),
+ expected_status=ActionStatus.AWAITING_APPROVAL.value,
+ claimed_by=_WORKER_ID,
+ running_status=ActionStatus.REJECTED.value,
+ )
+ if not claim.ok:
+ raise _claim_refusal(action_id, claim)
+
+ rejected = claim.record or record
+ rejected["status"] = ActionStatus.REJECTED
+ rejected["rejected_by_user_id"] = record.get("rejected_by_user_id")
+ rejected["rejected_at"] = datetime.now(UTC).isoformat()
+ try:
+ await action_store.save(rejected)
+ except ActionIntentConflict: # pragma: no cover - the intent is unchanged here
+ logger.error("action_store.rejection_write_refused", action_id=action_id)
logger.info(
"Action rejected",
action_id=action_id,
rejected_by=bound.user_id if bound else None,
)
- return record
+ return rejected
+
+
+#: What an unscoped read is allowed to see.
+#:
+#: A caller that does not declare a tenant still gets a usable status poll,
+#: and does not get the host or account the action was aimed at, the
+#: parameters it ran with, or the identities around it.
+_UNSCOPED_READ_FIELDS: tuple[str, ...] = (
+ "id",
+ "status",
+ "action_type",
+ "blast_radius",
+ "approval_tier",
+ "tenant_id",
+)
@router.get("/actions/{action_id}")
-async def get_action(action_id: str, _auth: None = Depends(require_service_auth)):
+async def get_action(action_id: str, request: Request, _auth: None = Depends(require_service_auth)):
"""Get action status and result.
Behind the service guard: an action record names the host or account an
action was aimed at and the parameters it ran with, and ``GET`` by id was
- reachable with no credential at all. ``aisoc_action_records`` carries no
- RLS policy, so the id was the only thing standing in front of it.
+ reachable with no credential at all.
+
+ The service token is shared, and ``aisoc_action_records``' row-level
+ policy is inert on this service's connections — it reads a GUC nothing
+ here sets — so inside the trust boundary an id was still a capability over
+ every tenant's response history. A caller that declares its tenant on
+ ``X-Tenant-ID`` gets the record when it owns it and a 404 when it does
+ not; never a 403, which would make the route an existence oracle over
+ every tenant's action ids. A caller that declares none keeps working and
+ is answered with status only.
+ """
+ tenant_id = (request.headers.get(_TENANT_HEADER) or "").strip() or None
+ try:
+ record = await action_store.get(action_id, tenant_id=tenant_id, authoritative=True)
+ except ActionStoreUnavailable as exc:
+ raise HTTPException(status_code=503, detail="the action store is unavailable") from exc
+ if not record:
+ raise HTTPException(status_code=404, detail="Action not found")
+ if tenant_id is None:
+ logger.info("action_read_unscoped", action_id=action_id)
+ return {key: record.get(key) for key in _UNSCOPED_READ_FIELDS if key in record}
+ return record
+
+
+class ReconcileRequest(BaseModel):
+ """Credentials for the read-back, supplied per call.
+
+ Not taken from the stored record alone on purpose: an operator reconciling
+ days later should be able to present a current credential rather than
+ depend on whatever was captured at submit time.
+ """
+
+ parameters: dict[str, Any] = Field(default_factory=dict)
+
+
[email protected]("/actions/{action_id}/reconcile")
+async def reconcile_action(
+ action_id: str,
+ request: Request,
+ body: ReconcileRequest | None = None,
+ _auth: None = Depends(require_service_auth),
+):
+ """Resolve an action whose outcome was never established, by asking the vendor.
+
+ The recovery path for the two states a dispatch can strand in: ``running``,
+ where the worker that claimed it died before writing an outcome, and
+ ``reconciliation_required``, where it finished and could not tell what had
+ happened.
+
+ It never re-executes. The reason those states exist is that the vendor may
+ already have acted, and running the action again is the one response that
+ turns an unknown into a real duplicate. What it does is the read-back
+ probe: present means it completed, absent means it genuinely failed,
+ unreadable means it stays unknown and a human has to look.
"""
- record = await action_store.get(action_id)
+ tenant_id = (request.headers.get(_TENANT_HEADER) or "").strip() or None
+ try:
+ record = await action_store.get(action_id, tenant_id=tenant_id, authoritative=True)
+ except ActionStoreUnavailable as exc:
+ raise HTTPException(status_code=503, detail="the action store is unavailable") from exc
if not record:
raise HTTPException(status_code=404, detail="Action not found")
+
+ current = str(getattr(record.get("status"), "value", record.get("status")))
+ if current not in (ActionStatus.RUNNING.value, ActionStatus.RECONCILIATION_REQUIRED.value):
+ raise HTTPException(
+ status_code=409,
+ detail=f"only a claimed or unresolved action can be reconciled (current: {current})",
+ )
+
+ parameters = {**(record.get("parameters") or {}), **((body.parameters if body else None) or {})}
+ probe_request = ActionRequest(
+ id=UUID(action_id),
+ incident_id=UUID(record["incident_id"]),
+ tenant_id=UUID(record["tenant_id"]),
+ action_type=ActionType(record["action_type"]),
+ target=record["target"],
+ rationale=record.get("rationale") or "",
+ parameters=parameters,
+ )
+
+ outcome, verification = await _verify_effect(probe_request)
+ record["verification"] = verification
+ record["reconciled_at"] = datetime.now(UTC).isoformat()
+ if outcome is VerificationOutcome.VERIFIED:
+ record["status"] = ActionStatus.COMPLETED
+ record.pop("error", None)
+ elif outcome is VerificationOutcome.FAILED:
+ record["status"] = ActionStatus.FAILED
+ record.setdefault("error", "the effect was not present when the vendor was read back")
+ else:
+ record["status"] = ActionStatus.RECONCILIATION_REQUIRED
+
+ try:
+ await action_store.save(record)
+ except ActionIntentConflict: # pragma: no cover - reconciliation changes no intent
+ logger.error("action_store.reconcile_write_refused", action_id=action_id)
+ logger.info("action_reconciled", action_id=action_id, status=record["status"], outcome=outcome.value)
return record
diff --git a/services/actions/app/models/action.py b/services/actions/app/models/action.py
index 3ce585bf8..4e1530981 100644
--- a/services/actions/app/models/action.py
+++ b/services/actions/app/models/action.py
@@ -108,9 +108,24 @@ class ActionStatus(str, Enum):
AWAITING_APPROVAL = "awaiting_approval"
APPROVED = "approved"
REJECTED = "rejected"
+ #: Claimed for execution. Written to the durable store *before* the vendor
+ #: is contacted and never only afterwards, so a replica that dies
+ #: mid-dispatch leaves a record that cannot be mistaken for one nobody
+ #: started. The member always existed; nothing wrote it.
RUNNING = "running"
COMPLETED = "completed"
FAILED = "failed"
+ #: The dispatch did not come back clean and the vendor could not be read
+ #: back, so whether the estate changed is genuinely unknown.
+ #:
+ #: Its absence was the defect. A containment the EDR had accepted, whose
+ #: response was then lost, was recorded as ``failed`` — the reading whose
+ #: obvious remedy is to run it again. Nothing here may be re-approved;
+ #: ``POST /actions/{id}/reconcile`` resolves it by asking the vendor.
+ #:
+ #: Not a terminal success: `services/api`'s usage meter counts `completed`
+ #: as the one status meaning a vendor was touched, and that stays true.
+ RECONCILIATION_REQUIRED = "reconciliation_required"
ROLLED_BACK = "rolled_back"
@@ -251,6 +266,23 @@ class ActionRequest(BaseModel):
# W4.1 — structured invoking identity. Optional for back-compat with the
# legacy `requested_by` string + system-initiated calls.
principal: ActionPrincipal | None = None
+ #: Who originally asked for this action, when that is not the caller.
+ #:
+ #: Three identities meet on an approved action and collapsing any two
+ #: breaks a control. `principal` is the *invoking* identity whose
+ #: permissions gate the submit; the approver arrives separately on the
+ #: approve call; and this is the originator, read only for separation of
+ #: duties and provenance, never for authorization.
+ #:
+ #: It exists because `services/api` had nowhere to put the originator and
+ #: sent the deciding console user as `principal`, which became
+ #: `requested_by_user_id`. Separation of duties then compared the approver
+ #: against itself and refused: every approval raised in the responder app
+ #: was answered 403 upstream and surfaced as a 502. The free-text
+ #: `requested_by` beside this is a label and is deliberately not used for
+ #: the comparison — it is caller-supplied and trusting it would make the
+ #: control forgeable in the other direction.
+ requester: ActionPrincipal | None = None
rationale: str = ""
auto_rollback: bool = False
rollback_after_seconds: int | None = None
diff --git a/services/actions/app/security/authz.py b/services/actions/app/security/authz.py
index a7dc7566b..e754b97ec 100644
--- a/services/actions/app/security/authz.py
+++ b/services/actions/app/security/authz.py
@@ -81,24 +81,55 @@ def authorize_action(request: ActionRequest, *, require_principal: bool | None =
raise ActionAuthzError(f"principal {principal.user_id} lacks '{required}' required for {request.action_type.value}")
+#: Recorded on the action when separation of duties had nothing to compare.
+#:
+#: ``if separation_of_duties and requester and ...`` passed silently whenever
+#: the submission carried no principal, which is possible while
+#: ``AISOC_ACTIONS_REQUIRE_PRINCIPAL`` is false. The control reported itself
+#: satisfied on exactly the requests it could not evaluate, and the record
+#: afterwards was indistinguishable from one where it had been checked and
+#: passed. Refusing instead would brick every legacy system-initiated action
+#: during an incident, which is why this is a recorded fact rather than a
+#: denial — but it has to *be* a fact, in the row, not an omission.
+SOD_UNEVALUATED = "unevaluated"
+SOD_ENFORCED = "enforced"
+
+
def authorize_approver(
record: dict[str, Any],
approver: ActionPrincipal,
*,
separation_of_duties: bool = True,
-) -> None:
+) -> str:
"""Enforce that an approver may approve this action (W4.4).
- The approver must hold the action's required permission and — under
- separation of duties — must not be the principal who requested it."""
+ Three conditions, and the tenant one was missing. An approver had to hold
+ the action's permission tier and not be its requester, but nothing compared
+ their tenant to the action's — so a principal holding
+ ``actions:execute:high`` in *any* tenant could approve any other tenant's
+ containment, against a store where an id was the only other control.
+ :func:`authorize_action` had carried the same comparison since it was
+ written; only the approve side went without.
+
+ Returns which separation-of-duties state applied, for the caller to record.
+ """
action_type = ActionType(record["action_type"])
required = _required_permission(action_type)
if not has_action_permission(approver.permissions, required):
raise ActionAuthzError(f"approver {approver.user_id} lacks '{required}' to approve {action_type.value}")
+ record_tenant = str(record.get("tenant_id") or "").strip()
+ if approver.tenant_id is not None and record_tenant and str(approver.tenant_id) != record_tenant:
+ raise ActionAuthzError("approver tenant does not match the action's tenant")
+
requester = record.get("requested_by_user_id")
- if separation_of_duties and requester and str(requester) == str(approver.user_id):
+ if not separation_of_duties:
+ return SOD_UNEVALUATED
+ if not requester:
+ return SOD_UNEVALUATED
+ if str(requester) == str(approver.user_id):
raise ActionAuthzError("approver cannot be the requester (separation of duties)")
+ return SOD_ENFORCED
async def require_service_auth(authorization: str | None = Header(default=None)) -> None:
diff --git a/services/actions/app/services/action_store.py b/services/actions/app/services/action_store.py
index b9dac1774..14ce8c5d7 100644
--- a/services/actions/app/services/action_store.py
+++ b/services/actions/app/services/action_store.py
@@ -18,12 +18,49 @@ run without one would push every test onto a mock of itself.
Writes are upserts on the action id, so a replay of the same submission is
idempotent rather than a duplicate pending approval.
+
+What a store has to guarantee before a vendor is contacted
+----------------------------------------------------------
+Durability was only half of it. The approve path read a record, checked its
+status in Python, called the vendor and saved afterwards, which left three
+gaps that an upsert alone cannot close:
+
+* **Nobody claimed the execution.** Two replicas, or one replica and a client
+ that retried, both read ``awaiting_approval`` and both contained the host.
+ :func:`claim_for_execution` is one conditional ``UPDATE`` — Postgres takes a
+ row lock, so a second claimant blocks, re-evaluates the predicate against the
+ committed row, finds ``running`` and matches nothing. Same mechanism as
+ ``scheduler_leases`` (migration 115), and for the same reason: a
+ SELECT-then-UPDATE has no such re-evaluation.
+* **The cache outranked the database.** ``get`` answered from ``_MEMORY``
+ first, so the replica that handled the submission held an
+ ``awaiting_approval`` copy for the life of the process and would execute
+ again whatever the database had since recorded. Reads on a decision path ask
+ for ``authoritative=True``, which skips the cache when a DSN is configured
+ and *raises* rather than reporting a reachable store's outage as a missing
+ action.
+* **Intent was mutable under the approver.** ``ON CONFLICT (id) DO UPDATE SET
+ record = EXCLUDED.record`` with no condition meant a second submission of the
+ same id could retarget an action a human was in the middle of approving. The
+ upsert now refuses to change the action type, target or parameters of a
+ record that is awaiting approval or running, and says so.
+
+Failures are not swallowed on those paths. ``save`` keeps its tolerant
+behaviour — losing a *pending* action because the store blinked is the failure
+this module was written to prevent, and an outcome write that raised would lose
+the one record of a vendor that has already been touched — but a claim refuses
+rather than proceeding, because that write is the authorization for an external
+effect.
"""
from __future__ import annotations
+import asyncio
import json
import os
+from dataclasses import dataclass
+from datetime import UTC, datetime
+from enum import Enum
from typing import Any
import structlog
@@ -33,6 +70,80 @@ logger = structlog.get_logger()
#: Process-local fallback, and the store the unit suite exercises.
_MEMORY: dict[str, dict[str, Any]] = {}
+#: Serialises the in-memory claim. Only load-bearing when no DSN is configured;
+#: with one, Postgres' row lock is the serialisation point and this lock is
+#: held across no I/O.
+_MEMORY_CLAIM_LOCK = asyncio.Lock()
+
+
+class ActionStoreUnavailable(RuntimeError):
+ """A durable store is configured and could not be read or written.
+
+ Distinct from "no such action" on purpose. Reporting a reachable store's
+ outage as a 404 sends an operator to look for an action that exists, and
+ reporting it as success would let an external effect proceed with no
+ record of the authorization behind it.
+ """
+
+
+class ActionIntentConflict(RuntimeError):
+ """A submission tried to change what a pending action would do.
+
+ Carries the stored intent so the caller can say what the approver is
+ actually looking at rather than only that the write was refused.
+ """
+
+ def __init__(self, action_id: str, stored: dict[str, Any]) -> None:
+ super().__init__(f"action {action_id} is already awaiting a decision on a different target or parameters")
+ self.action_id = action_id
+ self.stored = stored
+
+
+class ClaimOutcome(str, Enum):
+ """Why a claim did or did not take.
+
+ Every value except ``CLAIMED`` means *do not contact the vendor*, and they
+ are kept apart because they need different answers: a losing replica is a
+ 409, an action in an unknown state needs reconciliation rather than a
+ retry, and an unwritable store is a 503 the caller may repeat.
+ """
+
+ CLAIMED = "claimed"
+ NOT_FOUND = "not_found"
+ WRONG_STATE = "wrong_state"
+ STORE_UNAVAILABLE = "store_unavailable"
+
+
+@dataclass(frozen=True)
+class Claim:
+ outcome: ClaimOutcome
+ #: The record as it stands *after* a successful claim, read back from the
+ #: database rather than reconstructed, so what executes is what was
+ #: claimed. ``None`` when the claim did not take and there is no row.
+ record: dict[str, Any] | None = None
+ #: The status that blocked the claim, for the caller's message.
+ current_status: str | None = None
+
+ @property
+ def ok(self) -> bool:
+ return self.outcome is ClaimOutcome.CLAIMED
+
+
+#: The fields that define *what an approver signed off*. A change to any of
+#: them is a different action, whatever the id says.
+#:
+#: ``rationale`` is deliberately out: it is the explanation shown to a human
+#: and an agent re-running its triage may word it differently without changing
+#: what would happen to the estate. ``confidence`` is out for the same reason —
+#: it is re-graded by the gate on every submission.
+INTENT_FIELDS: tuple[str, ...] = ("action_type", "target", "parameters")
+
+
+def intent_of(record: dict[str, Any]) -> dict[str, Any]:
+ """The immutable part of an action, for comparison and for an error."""
+ return {field: record.get(field) for field in INTENT_FIELDS}
+
+
#: Set once a write has actually reached Postgres.
#:
#: This existed as a flag nothing read, which made the comment above it a
@@ -81,30 +192,73 @@ def _column_text(value: Any) -> str:
return str(getattr(value, "value", value))
+#: Statuses whose intent an approver may already be looking at.
+#:
+#: Spelled as literals rather than imported from ``app.models.action`` because
+#: this module is the storage layer and the column holds text; importing the
+#: enum here would make the SQL depend on the executor contract, which is the
+#: coupling 055 avoided by storing the record whole.
+_INTENT_LOCKED_STATUSES = ("awaiting_approval", "running", "reconciliation_required")
+
+#: The same tuple as a SQL list. Built rather than written twice so the two
+#: cannot drift, and quoted here because these are fixed identifiers from a
+#: closed set, never request data.
+_INTENT_LOCKED_SQL_LIST = "(" + ", ".join(f"'{status}'" for status in _INTENT_LOCKED_STATUSES) + ")"
+
+#: Refuse an intent change while a decision is outstanding, allow every other
+#: update, and tell the caller which happened.
+#:
+#: ``RETURNING id`` is how: an ``ON CONFLICT DO UPDATE`` whose ``WHERE`` is not
+#: satisfied returns no row, which is the only signal Postgres gives that the
+#: upsert declined. ``IS NOT DISTINCT FROM`` rather than ``=`` because a NULL
+#: target must compare equal to itself.
+_UPSERT_SQL = f"""
+ INSERT INTO aisoc_action_records (id, tenant_id, status, approval_tier, record, created_at, updated_at)
+ VALUES ($1, $2, $3, $4, $5, now(), now())
+ ON CONFLICT (id) DO UPDATE
+ SET status = EXCLUDED.status,
+ approval_tier = EXCLUDED.approval_tier,
+ record = EXCLUDED.record,
+ updated_at = now()
+ WHERE aisoc_action_records.status NOT IN {_INTENT_LOCKED_SQL_LIST}
+ OR (
+ aisoc_action_records.record ->> 'action_type' IS NOT DISTINCT FROM EXCLUDED.record ->> 'action_type'
+ AND aisoc_action_records.record ->> 'target' IS NOT DISTINCT FROM EXCLUDED.record ->> 'target'
+ AND aisoc_action_records.record -> 'parameters' IS NOT DISTINCT FROM EXCLUDED.record -> 'parameters'
+ )
+ RETURNING id
+"""
+
+
async def save(record: dict[str, Any]) -> None:
- """Persist or update an action record."""
+ """Persist or update an action record.
+
+ Raises :class:`ActionIntentConflict` when the write would change what a
+ pending action does. Everything else is still tolerated and logged: the
+ whole reason this module exists is that losing a pending containment to a
+ store blink is worse than running degraded, and an outcome write that
+ raised would discard the only record of a vendor already touched.
+
+ The durable write goes first. It used to be second, so a refused or failed
+ write left the process-local copy holding a record the database had never
+ accepted — and ``get`` preferred that copy.
+ """
action_id = str(record["id"])
- _MEMORY[action_id] = record
dsn = _dsn()
if dsn is None:
+ _MEMORY[action_id] = record
return
+
try:
conn = await _connect(dsn)
except Exception as exc: # noqa: BLE001 — an unreachable store must not drop the action
_report_degraded("connect_failed", action_id, exc)
+ _MEMORY[action_id] = record
return
try:
- await conn.execute(
- """
- INSERT INTO aisoc_action_records (id, tenant_id, status, approval_tier, record, created_at, updated_at)
- VALUES ($1, $2, $3, $4, $5, now(), now())
- ON CONFLICT (id) DO UPDATE
- SET status = EXCLUDED.status,
- approval_tier = EXCLUDED.approval_tier,
- record = EXCLUDED.record,
- updated_at = now()
- """,
+ written = await conn.fetchval(
+ _UPSERT_SQL,
action_id,
_column_text(record.get("tenant_id")),
_column_text(record.get("status")),
@@ -113,11 +267,26 @@ async def save(record: dict[str, Any]) -> None:
)
global _DB_CONFIRMED
_DB_CONFIRMED = True
+ if written is None:
+ stored = await _fetch_record(conn, action_id, None)
+ logger.warning(
+ "action_store.intent_change_refused",
+ action_id=action_id,
+ stored_intent=intent_of(stored or {}),
+ submitted_intent=intent_of(record),
+ )
+ raise ActionIntentConflict(action_id, intent_of(stored or {}))
+ except ActionIntentConflict:
+ raise
except Exception as exc: # noqa: BLE001 — see above
_report_degraded("write_failed", action_id, exc)
+ _MEMORY[action_id] = record
+ return
finally:
await conn.close()
+ _MEMORY[action_id] = record
+
def _report_degraded(event: str, action_id: str, exc: Exception) -> None:
"""Log a store failure at the severity its consequence deserves.
@@ -142,57 +311,179 @@ def _report_degraded(event: str, action_id: str, exc: Exception) -> None:
)
-async def get(action_id: str, tenant_id: str | None = None) -> dict[str, Any] | None:
- """Fetch an action record, preferring the durable copy.
-
- The in-memory copy is checked first only as a cache: on the replica that
- handled the submission it is the same object, and on any other replica it
- is absent and the database answers.
-
- ``tenant_id`` narrows the lookup to one tenant and should always be
- supplied from a request path. ``aisoc_action_records`` carries no RLS
- policy, so without it an id is a capability: an action record names the
- host or account an action was aimed at, and any caller who could guess or
- observe a UUID would read another tenant's response history.
+async def _fetch_record(conn: Any, action_id: str, tenant_id: str | None) -> dict[str, Any] | None:
+ """One tenant-narrowed read, on a connection the caller owns."""
+ if tenant_id is None:
+ row = await conn.fetchrow("SELECT record FROM aisoc_action_records WHERE id = $1", action_id)
+ else:
+ row = await conn.fetchrow(
+ "SELECT record FROM aisoc_action_records WHERE id = $1 AND tenant_id = $2",
+ action_id,
+ str(tenant_id),
+ )
+ return json.loads(row["record"]) if row is not None else None
+
+
+async def get(action_id: str, tenant_id: str | None = None, *, authoritative: bool = False) -> dict[str, Any] | None:
+ """Fetch an action record.
+
+ ``tenant_id`` narrows the lookup to one tenant and should be supplied from
+ a request path wherever the caller has an authenticated identity to take
+ it from. ``aisoc_action_records``' RLS policy is inert on this service's
+ connections — it reads ``app.current_tenant_id``, which nothing here sets,
+ and its unset arm is deliberately permissive so cross-tenant workers are
+ not starved — so without a predicate an id is a capability: an action
+ record names the host or account an action was aimed at.
+
+ ``authoritative`` is for the decision paths. The in-memory copy is a cache,
+ and on the replica that handled the submission it is a cache that never
+ expires: it held ``awaiting_approval`` for the life of the process, so an
+ approve on that replica would execute an action another replica had already
+ run. An authoritative read skips it when a DSN is configured and raises
+ :class:`ActionStoreUnavailable` rather than answering "no such action" for
+ a store that is merely unreachable.
"""
- cached = _MEMORY.get(action_id)
- if cached is not None:
- if tenant_id is not None and str(cached.get("tenant_id") or "") != str(tenant_id):
+ dsn = _dsn()
+
+ if dsn is None:
+ cached = _MEMORY.get(action_id)
+ if cached is not None and tenant_id is not None and str(cached.get("tenant_id") or "") != str(tenant_id):
return None
return cached
- dsn = _dsn()
- if dsn is None:
- return None
+ if not authoritative:
+ cached = _MEMORY.get(action_id)
+ if cached is not None:
+ if tenant_id is not None and str(cached.get("tenant_id") or "") != str(tenant_id):
+ return None
+ return cached
+
try:
conn = await _connect(dsn)
except Exception as exc: # noqa: BLE001
logger.warning("action_store.connect_failed", action_id=action_id, error=str(exc))
+ if authoritative:
+ raise ActionStoreUnavailable("the action store is configured and unreachable") from exc
return None
try:
- if tenant_id is None:
- row = await conn.fetchrow("SELECT record FROM aisoc_action_records WHERE id = $1", action_id)
- else:
- row = await conn.fetchrow(
- "SELECT record FROM aisoc_action_records WHERE id = $1 AND tenant_id = $2",
- action_id,
- tenant_id,
- )
+ record = await _fetch_record(conn, action_id, tenant_id)
except Exception as exc: # noqa: BLE001
logger.warning("action_store.read_failed", action_id=action_id, error=str(exc))
+ if authoritative:
+ raise ActionStoreUnavailable("the action store is configured and could not be read") from exc
return None
finally:
await conn.close()
- if row is None:
+ if record is None:
+ # Drop any cached copy: the authoritative answer is that this tenant
+ # has no such action, and leaving a stale one behind is how the cache
+ # outranked the database in the first place.
+ if authoritative:
+ _MEMORY.pop(action_id, None)
return None
- record = json.loads(row["record"])
- # Cache it so the approve path's subsequent mutation and save see one
- # object rather than two diverging copies.
_MEMORY[action_id] = record
return record
+async def claim_for_execution(
+ action_id: str,
+ *,
+ tenant_id: str,
+ expected_status: str,
+ claimed_by: str,
+ running_status: str = "running",
+) -> Claim:
+ """Take exclusive ownership of an action's execution, durably, or refuse.
+
+ One conditional ``UPDATE``. Postgres locks the row, so a second claimant
+ blocks until the first commits, then re-evaluates ``status =
+ $expected_status`` against the committed row, finds ``running`` and matches
+ nothing. That re-evaluation is the whole mechanism and it is why this is
+ not a ``SELECT`` followed by an ``UPDATE``: that pair has no re-evaluation
+ and elects two executors, which here means contacting a vendor twice.
+
+ The tenant is a predicate rather than a check afterwards. A handler that
+ fetches globally and compares in Python is the same defect one line later.
+
+ An unreachable or unwritable store returns ``STORE_UNAVAILABLE`` and the
+ caller must not proceed: this write *is* the authorization for an external
+ effect, and an effect nobody can prove was authorized is worse than a
+ refused approval an operator can repeat.
+ """
+ claimed_at = datetime.now(UTC).isoformat()
+ dsn = _dsn()
+
+ if dsn is None:
+ async with _MEMORY_CLAIM_LOCK:
+ record = _MEMORY.get(action_id)
+ if record is None or str(record.get("tenant_id") or "") != str(tenant_id):
+ return Claim(ClaimOutcome.NOT_FOUND)
+ current = _column_text(record.get("status"))
+ if current != expected_status:
+ return Claim(ClaimOutcome.WRONG_STATE, current_status=current)
+ record["status"] = running_status
+ record["dispatch"] = {"claimed_by": claimed_by, "claimed_at": claimed_at, "state": "claimed"}
+ return Claim(ClaimOutcome.CLAIMED, record=dict(record))
+
+ try:
+ conn = await _connect(dsn)
+ except Exception as exc: # noqa: BLE001
+ _report_degraded("claim_connect_failed", action_id, exc)
+ return Claim(ClaimOutcome.STORE_UNAVAILABLE)
+
+ try:
+ row = await conn.fetchrow(
+ """
+ UPDATE aisoc_action_records
+ SET status = $4,
+ record = jsonb_set(
+ jsonb_set(record, '{status}', to_jsonb($4::text), true),
+ '{dispatch}', $5::jsonb, true
+ ),
+ updated_at = now()
+ WHERE id = $1
+ AND tenant_id = $2
+ AND status = $3
+ RETURNING record
+ """,
+ action_id,
+ str(tenant_id),
+ expected_status,
+ running_status,
+ json.dumps({"claimed_by": claimed_by, "claimed_at": claimed_at, "state": "claimed"}),
+ )
+ except Exception as exc: # noqa: BLE001
+ _report_degraded("claim_failed", action_id, exc)
+ await conn.close()
+ return Claim(ClaimOutcome.STORE_UNAVAILABLE)
+
+ try:
+ if row is not None:
+ global _DB_CONFIRMED
+ _DB_CONFIRMED = True
+ record = json.loads(row["record"])
+ _MEMORY[action_id] = record
+ return Claim(ClaimOutcome.CLAIMED, record=record)
+
+ # Nothing matched. Say which of the three reasons it was, because an
+ # operator's next step differs for each: another replica has it, the
+ # action is in a state no approval can move, or this tenant has no
+ # such action at all.
+ existing = await _fetch_record(conn, action_id, str(tenant_id))
+ if existing is None:
+ _MEMORY.pop(action_id, None)
+ return Claim(ClaimOutcome.NOT_FOUND)
+ current = _column_text(existing.get("status"))
+ _MEMORY[action_id] = existing
+ return Claim(ClaimOutcome.WRONG_STATE, record=existing, current_status=current)
+ except Exception as exc: # noqa: BLE001
+ _report_degraded("claim_readback_failed", action_id, exc)
+ return Claim(ClaimOutcome.STORE_UNAVAILABLE)
+ finally:
+ await conn.close()
+
+
def clear() -> None:
"""Drop the in-memory copy. Used by tests."""
_MEMORY.clear()
diff --git a/services/actions/tests/test_action_store_columns.py b/services/actions/tests/test_action_store_columns.py
index 7a26445e7..f2665f423 100644
--- a/services/actions/tests/test_action_store_columns.py
+++ b/services/actions/tests/test_action_store_columns.py
@@ -37,15 +37,28 @@ _ID, _TENANT, _STATUS, _TIER, _RECORD = range(5)
class _RecordingConnection:
- """Captures the arguments `save()` binds, and nothing else."""
+ """Captures the arguments `save()` binds, and nothing else.
+
+ `fetchval` rather than `execute` since the upsert gained `RETURNING id`:
+ an `ON CONFLICT DO UPDATE` whose `WHERE` is not satisfied returns no row,
+ which is the only signal Postgres gives that it declined to change what a
+ pending approval would do. The double answers "a row was written", which
+ is the normal case these tests are about.
+ """
def __init__(self) -> None:
self.args: tuple[Any, ...] = ()
self.sql = ""
+ #: What the upsert returns. `None` is a refused intent change.
+ self.returns: Any = "written"
- async def execute(self, sql: str, *args: Any) -> None:
+ async def fetchval(self, sql: str, *args: Any) -> Any:
self.sql = sql
self.args = args
+ return self.returns
+
+ async def fetchrow(self, _sql: str, *_args: Any) -> Any:
+ return None
async def close(self) -> None:
return None
@@ -129,3 +142,31 @@ class TestTheApprovalTierIsPersisted:
frozen on a row that is saved again after execution."""
await action_store.save(_record())
assert captured.sql.count("approval_tier") >= 2, captured.sql
+
+
+class TestTheUpsertRefusesToChangeAPendingAction:
+ """`ON CONFLICT (id) DO UPDATE SET record = EXCLUDED.record` had no
+ condition, so a second submission of the same id replaced the record an
+ approver was looking at — a different target under the same approval."""
+
+ @pytest.mark.asyncio
+ async def test_a_declined_upsert_raises_rather_than_reporting_success(self, captured):
+ captured.returns = None
+ with pytest.raises(action_store.ActionIntentConflict):
+ await action_store.save(_record(status=ActionStatus.AWAITING_APPROVAL))
+
+ @pytest.mark.asyncio
+ async def test_a_refused_write_does_not_land_in_the_process_cache(self, captured):
+ """The durable store is authoritative; a refused write must not leave
+ this replica holding a record the database never accepted."""
+ captured.returns = None
+ record = _record(status=ActionStatus.AWAITING_APPROVAL, target="dc-01")
+ with pytest.raises(action_store.ActionIntentConflict):
+ await action_store.save(record)
+ assert action_store._MEMORY.get(str(record["id"])) is None
+
+ @pytest.mark.asyncio
+ async def test_the_statement_compares_every_field_that_defines_the_intent(self, captured):
+ await action_store.save(_record())
+ for field in action_store.INTENT_FIELDS:
+ assert f"'{field}'" in captured.sql, f"{field} is declared as intent and the upsert never compares it"
diff --git a/services/api/app/api/v1/endpoints/approvals.py b/services/api/app/api/v1/endpoints/approvals.py
index 4ca15039c..3ab08a156 100644
--- a/services/api/app/api/v1/endpoints/approvals.py
+++ b/services/api/app/api/v1/endpoints/approvals.py
@@ -105,6 +105,69 @@ class ApprovalDecisionRequest(BaseModel):
comment: str | None = Field(default=None, max_length=2000)
+#: Where the originating identity is kept on the approval row.
+#:
+#: Inside ``action`` rather than in a column of its own, following the two
+#: server-written keys already there (``delivery``, ``dispatch``) and the
+#: reasoning migration 055 gives for keeping the record whole: the shape is
+#: owned by the thing that writes it, and a column per field makes every
+#: contract change a migration.
+#:
+#: Always overwritten from the authenticated principal at create time and
+#: never read out of the request body. ``requested_by`` beside it stays a
+#: free-text label — it is caller-supplied, and separation of duties
+#: evaluated against it would be forgeable from the body.
+_REQUESTER_KEY = "requester_principal"
+
+#: Dispatch states recorded on the approval row.
+#:
+#: ``dispatching`` is the one that did not exist. ``_dispatch_decision`` ran
+#: before ``db.commit()``, so a crash in between left the approval ``pending``
+#: against an action that had run — the row's own answer to "was this
+#: executed" was silently wrong in the dangerous direction. The decision is
+#: now committed first with the dispatch marked in flight, so an interrupted
+#: decide is visible rather than invisible.
+_DISPATCH_IN_FLIGHT = "dispatching"
+#: The actions service was reached and answered.
+_DISPATCH_EXECUTED = "executed"
+_DISPATCH_DECLINED = "declined"
+#: The actions service refused, with a reason.
+_DISPATCH_FAILED = "failed"
+#: We never heard back. Distinct from ``failed`` because the request may have
+#: been processed, and "your action did not run" is a claim the evidence does
+#: not support.
+_DISPATCH_UNKNOWN = "unknown"
+_DISPATCH_NOT_EXECUTABLE = "not_executable"
+
+#: Dispatch states that mean the decision has not finished being carried out.
+#: A repeat decide re-enters on these rather than reporting success: the
+#: actions service claims an execution durably before contacting a vendor, so
+#: a re-entry cannot double-execute — it is answered 409 — and the alternative
+#: is an approval whose row says ``approved`` and whose action never ran.
+_UNRESOLVED_DISPATCH = frozenset({_DISPATCH_IN_FLIGHT, _DISPATCH_UNKNOWN})
+
+
+def _requester_principal(user: AuthUser) -> dict[str, Any]:
+ """The identity that raised this approval, from authenticated context.
+
+ For an agent-raised approval this is the agents service's own service
+ principal; for one a human raised it is that human, which is what keeps
+ separation of duties biting on the half where it must.
+ """
+ return {
+ "user_id": str(user.user_id),
+ "tenant_id": str(user.tenant_id),
+ "email": getattr(user, "email", None),
+ "roles": [user.role] if user.role else [],
+ "permissions": user.effective_permissions(),
+ }
+
+
+def _dispatch_state(row: AgentApproval) -> str | None:
+ dispatch = (row.action or {}).get("dispatch")
+ return dispatch.get("state") if isinstance(dispatch, dict) else None
+
+
@router.get("", response_model=ApprovalListResponse)
async def list_approvals(
user: AuthUser,
@@ -198,7 +261,7 @@ async def create_approval(
title=body.title,
summary=body.summary,
risk_level=body.risk_level,
- action=body.action,
+ action={**(body.action or {}), _REQUESTER_KEY: _requester_principal(user)},
expires_at=body.expires_at,
status="pending",
)
@@ -254,10 +317,11 @@ async def decide_approval(
raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="Approval not found")
new_status = "approved" if body.decision == "approve" else "denied"
+ unresolved = _dispatch_state(row) in _UNRESOLVED_DISPATCH
- if row.status == new_status:
+ if row.status == new_status and not unresolved:
return ApprovalResponse.model_validate(row)
- if row.status not in {"pending", "expired"}:
+ if row.status not in {"pending", "expired"} and row.status != new_status:
raise HTTPException(
status_code=status.HTTP_409_CONFLICT,
detail=f"Approval already {row.status}; cannot change to {new_status}",
@@ -268,12 +332,28 @@ async def decide_approval(
row.decided_at = datetime.now(UTC)
row.decision_comment = body.comment
- # Dispatch before committing the decision, so an approval that the actions
- # service refuses does not leave a row reading "approved" against an
- # action that never ran. This endpoint used to flip the row and stop:
- # every tap of Approve in the responder app recorded a decision and
- # executed nothing, which is the most dangerous shape a security control
- # can have — it reports that the host was contained.
+ # Commit the decision with the dispatch marked in flight, *then* dispatch.
+ #
+ # The order used to be the other way round, with a comment explaining that
+ # dispatching first stopped a refused action leaving a row reading
+ # "approved". It did, and it opened a worse hole: the dispatch ran at one
+ # line and `db.commit()` at the next, so a crash in between left the
+ # approval `pending` against an action that had already contained a host.
+ # Nothing in the row said a dispatch had been attempted, so the recovery
+ # was to approve it again.
+ #
+ # Writing the attempt first costs the thing the old comment protected —
+ # the window where the row says approved and the action has not run yet —
+ # and that window is now *visible*, which is the property that matters:
+ # `dispatch.state` says `dispatching`, a repeat decide re-enters, and the
+ # actions service's durable claim is what stops a re-entry from executing
+ # twice.
+ row.action = {
+ **(row.action or {}),
+ "dispatch": {"state": _DISPATCH_IN_FLIGHT, "started_at": datetime.now(UTC).isoformat()},
+ }
+ await db.commit()
+
dispatch = await _dispatch_decision(row, user, approve=body.decision == "approve")
row.action = {**(row.action or {}), "dispatch": dispatch}
@@ -284,7 +364,7 @@ async def decide_approval(
# resume — fire-and-forget pattern matches the rest of the surface.
await _notify_realtime(row, event="approval_decided")
- if dispatch.get("state") == "failed":
+ if dispatch.get("state") == _DISPATCH_FAILED:
# The decision is durable and the operator is told the truth: their
# choice was recorded, the execution behind it was not accepted.
raise HTTPException(
@@ -294,6 +374,17 @@ async def decide_approval(
"The approval row reflects your decision; the action did not run."
),
)
+ if dispatch.get("state") == _DISPATCH_UNKNOWN:
+ # Deliberately not phrased as a failure. The action service may have
+ # executed this; it claims an execution durably before touching a
+ # vendor, so the answer exists and nobody should act on a guess.
+ raise HTTPException(
+ status_code=status.HTTP_504_GATEWAY_TIMEOUT,
+ detail=(
+ f"Decision recorded, but the action service did not answer: {dispatch.get('detail')}. "
+ "Whether the action ran is not yet known — check the action's status before deciding again."
+ ),
+ )
return ApprovalResponse.model_validate(row)
@@ -316,15 +407,19 @@ async def _dispatch_decision(
``declined`` the decision was to deny; the action was rejected
upstream so it cannot later be approved by replay.
``executed`` the actions service accepted and ran it.
- ``failed`` the actions service refused or was unreachable.
+ ``failed`` the actions service refused, with a reason.
+ ``unknown`` we never heard back. The request may have been
+ processed; the actions service claims an execution
+ durably before contacting a vendor, so the answer
+ exists upstream and must be read rather than guessed.
"""
action = row.action or {}
action_type = action.get("action_type") or action.get("type")
target = action.get("target")
if not isinstance(action_type, str) or not action_type.strip():
- return {"state": "not_executable", "reason": "approval carries no action_type"}
+ return {"state": _DISPATCH_NOT_EXECUTABLE, "reason": "approval carries no action_type"}
if not isinstance(target, str) or not target.strip():
- return {"state": "not_executable", "reason": "approval carries no target"}
+ return {"state": _DISPATCH_NOT_EXECUTABLE, "reason": "approval carries no target"}
# The approval id is reused as the action id so the two systems share one
# identifier. Without that, "which action did this approval authorise"
@@ -353,6 +448,20 @@ async def _dispatch_decision(
"permissions": permissions,
}
+ # Who originally asked, stamped onto the row from authenticated context
+ # when the approval was created. Sent as `requester`, a separate field
+ # from `principal`, because the actions service records the requester for
+ # separation of duties: sending the deciding user as `principal` made
+ # them the requester too, so every approval was refused upstream as its
+ # own request and surfaced here as a 502.
+ #
+ # Falls back to this caller only when the approval predates the stamp, in
+ # which case separation of duties has nothing to compare and the actions
+ # service records that rather than claiming it passed.
+ requester = action.get(_REQUESTER_KEY)
+ if not isinstance(requester, dict) or not requester.get("user_id"):
+ requester = None
+
try:
await submit_action(
action_id=action_id,
@@ -362,34 +471,38 @@ async def _dispatch_decision(
incident_id=str(row.run_id or row.id),
rationale=row.summary or row.title,
parameters=action.get("parameters") or {},
- # The requester is the agent, not the approver. Sending the
- # approver here would make them both, and separation of duties
- # would pass by accident.
+ # A label, not an identity. The authorization basis is `requester`.
requested_by=row.requested_by or "agent",
# Identifies the caller to the gate. Omitting it was harmless only
# while `AISOC_ACTIONS_REQUIRE_PRINCIPAL` defaults false; turning
# that on would have broken the submit leg exactly as the decide
# leg was broken.
principal=principal,
+ requester=requester,
)
except ActionsServiceError as exc:
logger.warning(
"Approval dispatch: submit refused",
extra={"approval_id": action_id, "status": exc.status_code, "detail": str(exc)},
)
- return {"state": "failed", "stage": "submit", "detail": str(exc)}
+ # A submit that was never acknowledged has still not authorized
+ # anything: the actions service gates before it executes, and the
+ # decide leg below is what would have released it.
+ state = _DISPATCH_UNKNOWN if exc.indeterminate else _DISPATCH_FAILED
+ return {"state": state, "stage": "submit", "detail": str(exc)}
try:
result = await decide_action(action_id=action_id, approve=approve, approver=principal)
except ActionsServiceError as exc:
logger.warning(
- "Approval dispatch: decision refused",
- extra={"approval_id": action_id, "status": exc.status_code, "detail": str(exc)},
+ "Approval dispatch: decision not confirmed",
+ extra={"approval_id": action_id, "status": exc.status_code, "detail": str(exc), "indeterminate": exc.indeterminate},
)
- return {"state": "failed", "stage": "decide", "detail": str(exc)}
+ state = _DISPATCH_UNKNOWN if exc.indeterminate else _DISPATCH_FAILED
+ return {"state": state, "stage": "decide", "action_id": action_id, "detail": str(exc)}
return {
- "state": "executed" if approve else "declined",
+ "state": _DISPATCH_EXECUTED if approve else _DISPATCH_DECLINED,
"action_id": action_id,
"action_status": result.get("status"),
"blast_radius": result.get("blast_radius"),
diff --git a/services/api/app/services/actions_client.py b/services/api/app/services/actions_client.py
index 2b2503933..42c785a84 100644
--- a/services/api/app/services/actions_client.py
+++ b/services/api/app/services/actions_client.py
@@ -44,6 +44,14 @@ class ActionsServiceError(RuntimeError):
operator-facing string, which it deliberately builds as an error *type*
rather than a raw exception. Rendering the wrong one into an HTML page is
how an unauthenticated email-approval link starts leaking topology.
+
+ ``indeterminate`` is the difference between "it was refused" and "we never
+ heard back". A refusal is a fact about the decision; a transport error on
+ a POST that may already have been processed is not, and reporting the
+ second as the first told an operator their action had not run when it may
+ well have. The actions service answers the question properly — it claims
+ the execution durably before contacting a vendor, so the outcome is
+ readable afterwards — but this client must not assert what it cannot see.
"""
def __init__(
@@ -52,10 +60,12 @@ class ActionsServiceError(RuntimeError):
*,
status_code: int | None = None,
upstream_detail: str | None = None,
+ indeterminate: bool = False,
) -> None:
super().__init__(message)
self.status_code = status_code
self.upstream_detail = upstream_detail or "The action service refused the decision."
+ self.indeterminate = indeterminate
def base_url() -> str:
@@ -81,8 +91,13 @@ async def _post(path: str, payload: dict[str, Any] | None) -> dict[str, Any]:
try:
async with httpx.AsyncClient(timeout=_TIMEOUT_SECONDS) as client:
response = await client.post(url, headers=service_headers(), json=payload)
- except httpx.HTTPError as exc:
+ except httpx.ConnectError as exc:
+ # Nothing was written to a socket, so nothing was processed.
raise ActionsServiceError(f"actions service unreachable: {exc}") from exc
+ except httpx.HTTPError as exc:
+ # A timeout or a dropped connection on a POST: the request may have
+ # been received and acted on. Say so rather than calling it a refusal.
+ raise ActionsServiceError(f"actions service unreachable: {exc}", indeterminate=True) from exc
if response.status_code >= 400:
detail = _detail(response)
@@ -190,9 +205,19 @@ async def submit_action(
rationale: str,
parameters: dict[str, Any] | None = None,
principal: dict[str, Any] | None = None,
+ requester: dict[str, Any] | None = None,
requested_by: str = "aisoc-api",
) -> dict[str, Any]:
- """Submit an action. The gate decides whether it executes or waits."""
+ """Submit an action. The gate decides whether it executes or waits.
+
+ ``principal`` is the *invoking* identity, whose permissions the actions
+ service checks. ``requester`` is who originally asked, when that is
+ somebody else — an agent that proposed a containment a human is about to
+ approve. They are separate fields because collapsing them broke
+ separation of duties: this client sent the approving console user as
+ ``principal``, the actions service recorded it as the requester, and the
+ approval that followed was refused for being its own request.
+ """
payload: dict[str, Any] = {
"id": action_id,
"action_type": action_type,
@@ -205,6 +230,8 @@ async def submit_action(
}
if principal is not None:
payload["principal"] = principal
+ if requester is not None:
+ payload["requester"] = requester
return await _post("/api/v1/actions", payload)
diff --git a/tests/isolation/test_action_durability_live.py b/tests/isolation/test_action_durability_live.py
new file mode 100644
index 000000000..1024d09c7
--- /dev/null
+++ b/tests/isolation/test_action_durability_live.py
@@ -0,0 +1,1263 @@
+"""An approved action reaches the vendor once, or visibly not at all.
+
+Remediation phase 2. The approve path in ``services/actions`` was a plain
+read-check-act with the durable write placed *after* the vendor call::
+
+ record = await action_store.get(action_id) # no tenant, cache first
+ if record["status"] != AWAITING_APPROVAL: ... # checked in Python
+ result = await executor.execute(request) # the host is contained
+ await action_store.save(record) # ... and only now recorded
+
+Every guarantee an approval is supposed to carry sits in that gap.
+
+* Two replicas, or one replica and a client that retried, both read
+ ``awaiting_approval`` and both contain the host. On the submitting replica
+ the race is invisible because ``action_store.get`` answers from a
+ process-local dict and hands both callers the *same object*, so the local
+ suite could never see it and a second container was required to.
+* A vendor timeout raised out of the handler: ``save`` was never reached, the
+ record stayed ``awaiting_approval`` while the vendor may well have acted,
+ and the next approval ran it again. There was no status meaning "we do not
+ know".
+* The reads carried no tenant, so an approver holding the right permission tier
+ in *any* tenant could approve another tenant's containment.
+* ``save`` was an unconditional upsert, so re-submitting an id with a different
+ target replaced the record underneath the approver looking at it.
+* ``save`` swallowed connect and write failures, so an action could look
+ persisted while existing only in one replica's memory.
+
+And one defect that is not in any of those lists, found by running the path
+rather than reading it: ``services/api``'s ``_dispatch_decision`` submitted
+with ``principal=<the deciding console user>``, which the actions service
+records as ``requested_by_user_id``. The same principal was then sent as the
+approver, so separation of duties compared the approver against *itself* and
+refused. **Every** approval raised through the responder/console path was
+answered 403 upstream and surfaced as a 502 reading "Decision recorded, but
+the action service refused it". The three identities — who asked, who relayed,
+who approved — have to stay distinct for the control to work at all.
+
+Why this file is here and not in a service's own suite
+------------------------------------------------------
+``services/api`` and ``services/actions`` both name their top-level package
+``app``. Importing one into a process that has imported the other resolves to
+whichever is on ``PYTHONPATH``, and a cross-service test written that way
+degrades into skips — which report the same word as a pass.
+
+So the test process is the **API** (it imports the real ``actions_client`` and
+the real ``approvals._dispatch_decision``), the actions service runs as **two
+separate processes** on loopback sockets sharing one Postgres, and the vendor
+is a real HTTP server that records every request that reaches it. Nothing
+about authorization, persistence or retry is replaced by a double: a mock that
+answers whatever it is asked cannot notice an action that executed twice.
+
+Two replicas rather than one is the whole point. A single process would pass
+these tests against the pre-fix tree.
+"""
+
+from __future__ import annotations
+
+import asyncio
+import json
+import os
+import socket
+import subprocess
+import sys
+import threading
+import time
+import uuid
+from datetime import UTC, datetime
+from http.server import BaseHTTPRequestHandler, ThreadingHTTPServer
+from pathlib import Path
+from types import SimpleNamespace
+from typing import Any
+from urllib.parse import parse_qs, urlparse
+
+import httpx
+import pytest
+
+_REPO = Path(__file__).resolve().parents[2]
+_ACTIONS = _REPO / "services" / "actions"
+
+#: The durable store under test. Set by ``action-durability-live.yml``; absent
+#: in the offline isolation job, which has no Postgres and must skip rather
+#: than invent one.
+_DSN = os.environ.get("ISOLATION_ACTIONS_DSN", "").strip()
+
+SERVICE_TOKEN = "p2-actions-token-not-a-real-secret"
+
+pytestmark = [
+ pytest.mark.asyncio,
+ pytest.mark.skipif(
+ not _DSN,
+ reason="ISOLATION_ACTIONS_DSN is not set; this suite needs a real Postgres and both services installed",
+ ),
+]
+
+
+# ── the vendor ─────────────────────────────────────────────────────────────
+
+
+class _VendorState:
+ """What the stub does next, and what it has been asked so far.
+
+ ``contain_calls`` is the number that decides every durability assertion in
+ this file. It counts *requests that reached the vendor*, which is the only
+ definition of "the host was contained" that does not depend on believing
+ the thing under test.
+ """
+
+ def __init__(self) -> None:
+ self.lock = threading.Lock()
+ self.contain_calls: list[dict[str, Any]] = []
+ self.token_calls = 0
+ self.status_reads = 0
+ #: Seconds to stall a containment request. Used to make a real client
+ #: timeout, rather than raising a timeout the client never saw.
+ self.contain_delay = 0.0
+ #: Accept the containment and then hang up without answering. The
+ #: vendor has acted and the caller will never learn the outcome, which
+ #: is the condition reconciliation exists for — and it reproduces in
+ #: milliseconds rather than by waiting out a 30-second client timeout.
+ self.drop_response = False
+ #: What the device reports when the verifier reads it back.
+ self.device_status = "normal"
+ #: Whether the read-back route answers at all. A vendor that accepted
+ #: the containment and whose device API is then unavailable is the one
+ #: case where the outcome genuinely cannot be established.
+ self.probe_available = True
+ #: What the containment route answers. A 500 is a vendor that received
+ #: the request and refused it, which — unlike a dropped response — the
+ #: read-back can and must settle as a genuine failure.
+ self.contain_http_status = 200
+
+ def reset(self) -> None:
+ with self.lock:
+ self.contain_calls.clear()
+ self.token_calls = 0
+ self.status_reads = 0
+ self.contain_delay = 0.0
+ self.drop_response = False
+ self.device_status = "normal"
+ self.probe_available = True
+ self.contain_http_status = 200
+
+
+VENDOR = _VendorState()
+
+
+class _VendorHandler(BaseHTTPRequestHandler):
+ """A real HTTP endpoint speaking enough of the EDR API to be used for real.
+
+ The forward call and the read-back probe are different routes, so
+ "reconcile without executing again" is a property this stub can actually
+ distinguish rather than one the test has to take on trust.
+ """
+
+ protocol_version = "HTTP/1.1"
+
+ def log_message(self, *_args: Any) -> None: # noqa: D102 - quiet
+ return
+
+ def _send(self, payload: dict[str, Any], status: int = 200) -> None:
+ body = json.dumps(payload).encode()
+ self.send_response(status)
+ self.send_header("Content-Type", "application/json")
+ self.send_header("Content-Length", str(len(body)))
+ self.end_headers()
+ self.wfile.write(body)
+
+ def do_POST(self) -> None: # noqa: N802
+ parsed = urlparse(self.path)
+ length = int(self.headers.get("Content-Length") or 0)
+ raw = self.rfile.read(length) if length else b""
+
+ if parsed.path.endswith("/oauth2/token"):
+ with VENDOR.lock:
+ VENDOR.token_calls += 1
+ self._send({"access_token": "vendor-token", "expires_in": 1799})
+ return
+
+ if parsed.path.endswith("/devices/entities/devices-actions/v2"):
+ action = (parse_qs(parsed.query).get("action_name") or [""])[0]
+ with VENDOR.lock:
+ delay = VENDOR.contain_delay
+ drop = VENDOR.drop_response
+ http_status = VENDOR.contain_http_status
+ VENDOR.contain_calls.append(
+ {
+ "action": action,
+ "body": json.loads(raw or b"{}"),
+ "at": datetime.now(UTC).isoformat(),
+ }
+ )
+ if action == "contain" and http_status < 400:
+ VENDOR.device_status = "contained"
+ if delay:
+ time.sleep(delay)
+ if drop:
+ self.close_connection = True
+ return
+ if http_status >= 400:
+ self._send({"errors": [{"message": "containment refused"}]}, status=http_status)
+ return
+ self._send({"resources": [{"id": "dev-1"}], "errors": []})
+ return
+
+ self._send({"errors": [{"message": "unexpected route"}]}, status=404)
+
+ def do_GET(self) -> None: # noqa: N802
+ parsed = urlparse(self.path)
+ if parsed.path.endswith("/devices/queries/devices/v1"):
+ self._send({"resources": ["dev-1"]})
+ return
+ if parsed.path.endswith("/devices/entities/devices/v2"):
+ with VENDOR.lock:
+ VENDOR.status_reads += 1
+ status = VENDOR.device_status
+ available = VENDOR.probe_available
+ if not available:
+ self._send({"errors": [{"message": "device service unavailable"}]}, status=503)
+ return
+ self._send({"resources": [{"device_id": "dev-1", "status": status}]})
+ return
+ self._send({"errors": [{"message": "unexpected route"}]}, status=404)
+
+
+def _free_port() -> int:
+ with socket.socket() as sock:
+ sock.bind(("127.0.0.1", 0))
+ return int(sock.getsockname()[1])
+
+
[email protected](scope="module")
+def vendor_base_url():
+ server = ThreadingHTTPServer(("127.0.0.1", _free_port()), _VendorHandler)
+ thread = threading.Thread(target=server.serve_forever, daemon=True)
+ thread.start()
+ yield f"http://127.0.0.1:{server.server_address[1]}"
+ server.shutdown()
+ server.server_close()
+
+
[email protected](autouse=True)
+def _clean_vendor():
+ VENDOR.reset()
+ yield
+ VENDOR.reset()
+
+
+# ── the actions service, twice ─────────────────────────────────────────────
+
+
+class Replica:
+ """One actions container on a loopback socket."""
+
+ def __init__(self, name: str, port: int, proc: subprocess.Popen) -> None:
+ self.name = name
+ self.port = port
+ self.proc = proc
+ self.base_url = f"http://127.0.0.1:{port}"
+
+ def kill(self) -> None:
+ self.proc.kill()
+ self.proc.wait(timeout=10)
+
+
+def _spawn_replica(name: str, dsn: str) -> Replica:
+ if not (_ACTIONS / "app" / "main.py").is_file():
+ pytest.fail(f"the actions service is not at {_ACTIONS}; this test must not be skipped into passing")
+
+ port = _free_port()
+ env = {
+ **os.environ,
+ "AISOC_ACTIONS_SERVICE_TOKEN": SERVICE_TOKEN,
+ # Every approval in this file carries an identity. The default is on;
+ # it is set explicitly so a change to the default cannot silently turn
+ # the separation-of-duties assertions into no-ops.
+ "AISOC_ACTIONS_REQUIRE_APPROVER": "true",
+ "DATABASE_DSN": dsn,
+ "DATABASE_URL": dsn,
+ "OTEL_SDK_DISABLED": "true",
+ "ENVIRONMENT": "test",
+ "PYTHONPATH": str(_ACTIONS),
+ }
+ env.pop("AISOC_DEV_MODE", None)
+
+ proc = subprocess.Popen(
+ [
+ sys.executable,
+ "-c",
+ "import uvicorn\nfrom app.main import app as actions_app\n"
+ f"uvicorn.run(actions_app, host='127.0.0.1', port={port}, log_level='error')\n",
+ ],
+ cwd=str(_ACTIONS),
+ env=env,
+ stdout=subprocess.PIPE,
+ stderr=subprocess.STDOUT,
+ )
+
+ deadline = time.time() + 45
+ while time.time() < deadline:
+ if proc.poll() is not None:
+ output = ((proc.stdout.read() if proc.stdout else b"") or b"").decode()[-2000:]
+ pytest.fail(f"actions replica {name} exited before serving:\n{output}")
+ try:
+ httpx.get(f"http://127.0.0.1:{port}/api/v1/health", timeout=1.0)
+ return Replica(name, port, proc)
+ except httpx.HTTPError:
+ time.sleep(0.2)
+ proc.terminate()
+ pytest.fail(f"actions replica {name} did not start within 45s")
+ raise AssertionError("unreachable")
+
+
[email protected](scope="module")
+def replicas():
+ """Two containers behind one database, which is the deployed shape.
+
+ One process cannot demonstrate the defect: ``action_store`` answers from a
+ module-global dict, so a single replica's two concurrent approvals mutate
+ one object and the race hides. The second replica has an empty cache and
+ has to ask the database, which is exactly the question under test.
+ """
+ a = _spawn_replica("a", _DSN)
+ b = _spawn_replica("b", _DSN)
+ yield a, b
+ for replica in (a, b):
+ replica.proc.terminate()
+ try:
+ replica.proc.wait(timeout=10)
+ except subprocess.TimeoutExpired: # pragma: no cover - teardown guard
+ replica.proc.kill()
+
+
+# ── identities, as the console actually builds them ────────────────────────
+
+
+def _principal(tenant: str, *, user_id: str | None = None, permissions: list[str] | None = None) -> dict[str, Any]:
+ return {
+ "user_id": user_id or str(uuid.uuid4()),
+ "tenant_id": tenant,
+ "email": f"{(user_id or 'user').split('@')[0]}@example.com",
+ "roles": ["analyst"],
+ "permissions": permissions if permissions is not None else ["actions:execute:high"],
+ }
+
+
+def _vendor_params(vendor_base_url: str) -> dict[str, Any]:
+ """Credentials that make the executor take its real CrowdStrike arm.
+
+ Without these the executor falls through to its simulation branch, which
+ touches no socket — so every "did the vendor get called" assertion in this
+ file would pass for a reason that has nothing to do with the fix.
+ """
+ return {
+ "cs_client_id": "p2-client",
+ "cs_client_secret": "p2-secret",
+ "cs_base_url": vendor_base_url,
+ }
+
+
+async def _submit(
+ replica: Replica,
+ *,
+ tenant: str,
+ action_id: str,
+ requester: dict[str, Any] | None,
+ principal: dict[str, Any],
+ target: str,
+ params: dict[str, Any],
+) -> httpx.Response:
+ body: dict[str, Any] = {
+ "id": action_id,
+ "action_type": "isolate_host",
+ "target": target,
+ "tenant_id": tenant,
+ "incident_id": str(uuid.uuid4()),
+ "rationale": "phase 2 regression",
+ "requested_by": "agent",
+ "parameters": params,
+ "principal": principal,
+ }
+ if requester is not None:
+ body["requester"] = requester
+ async with httpx.AsyncClient(timeout=30.0) as client:
+ return await client.post(
+ f"{replica.base_url}/api/v1/actions",
+ json=body,
+ headers={"Authorization": f"Bearer {SERVICE_TOKEN}"},
+ )
+
+
+async def _approve(replica: Replica, action_id: str, approver: dict[str, Any], *, timeout: float = 30.0) -> httpx.Response:
+ async with httpx.AsyncClient(timeout=timeout) as client:
+ return await client.post(
+ f"{replica.base_url}/api/v1/actions/{action_id}/approve",
+ json={"approver": approver},
+ headers={"Authorization": f"Bearer {SERVICE_TOKEN}"},
+ )
+
+
+async def _read_record(dsn: str, action_id: str) -> dict[str, Any] | None:
+ """The database's answer, not any replica's.
+
+ Every status assertion reads through here on purpose: a replica's own
+ ``GET`` can be served from the process-local cache that caused the defect,
+ so asking it whether the durable record is right is asking the suspect.
+ """
+ import asyncpg
+
+ conn = await asyncpg.connect(dsn.replace("postgresql+asyncpg://", "postgresql://"), timeout=10.0)
+ try:
+ row = await conn.fetchrow(
+ "SELECT status, record FROM aisoc_action_records WHERE id = $1",
+ action_id,
+ )
+ finally:
+ await conn.close()
+ if row is None:
+ return None
+ record = json.loads(row["record"])
+ record["_column_status"] = row["status"]
+ return record
+
+
+def _contain_calls() -> int:
+ with VENDOR.lock:
+ return len([c for c in VENDOR.contain_calls if c["action"] == "contain"])
+
+
+# ── A. the three identities ────────────────────────────────────────────────
+
+
+class TestTheThreeIdentitiesStayDistinct:
+ """Who asked, who relayed and who approved are three facts, not one.
+
+ Against the pre-fix tree the first test here fails with 403
+ ``approver cannot be the requester``, which is what every approval tapped
+ in the responder app received.
+ """
+
+ async def test_an_agent_originated_request_is_approved_by_a_human(self, replicas, vendor_base_url) -> None:
+ a, _b = replicas
+ tenant = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ analyst = _principal(tenant, user_id="dana")
+ action_id = str(uuid.uuid4())
+
+ submitted = await _submit(
+ a,
+ tenant=tenant,
+ action_id=action_id,
+ requester=agent,
+ principal=analyst,
+ target="ws-4471",
+ params=_vendor_params(vendor_base_url),
+ )
+ assert submitted.status_code == 200, submitted.text
+ assert submitted.json()["status"] == "awaiting_approval"
+
+ approved = await _approve(a, action_id, analyst)
+ assert approved.status_code == 200, (
+ "a human approving an agent-originated action was refused: "
+ f"{approved.status_code} {approved.text}. The requester identity did not survive the submit."
+ )
+ assert approved.json()["status"] == "completed"
+ assert _contain_calls() == 1
+
+ stored = await _read_record(_DSN, action_id)
+ assert stored is not None
+ assert stored["requested_by_user_id"] == "agents-service"
+ assert stored["approved_by_user_id"] == analyst["user_id"]
+
+ async def test_the_real_console_dispatch_path_approves(self, replicas, vendor_base_url, monkeypatch) -> None:
+ """``approvals._dispatch_decision`` itself, not a hand-built stand-in.
+
+ This is the function the responder app reaches, and it is where the
+ identity collapse happened. A test that posted its own body would have
+ passed on the defective tree.
+ """
+ a, _b = replicas
+ approvals = pytest.importorskip(
+ "app.api.v1.endpoints.approvals",
+ reason="services/api is not installed in this job",
+ )
+ actions_client = pytest.importorskip("app.services.actions_client")
+ monkeypatch.setattr(actions_client.settings, "AISOC_ACTIONS_BASE_URL", a.base_url, raising=False)
+ monkeypatch.setattr(actions_client.settings, "AISOC_ACTIONS_SERVICE_TOKEN", SERVICE_TOKEN, raising=False)
+
+ tenant = uuid.uuid4()
+ approval_id = uuid.uuid4()
+ row = SimpleNamespace(
+ id=approval_id,
+ tenant_id=tenant,
+ run_id=None,
+ summary="isolate the beachhead",
+ title="isolate ws-4471",
+ requested_by="triage-agent",
+ action={
+ "action_type": "isolate_host",
+ "target": "ws-4471",
+ "parameters": _vendor_params(vendor_base_url),
+ # Written by `create_approval` from the authenticated principal
+ # that raised the approval, never read from the request body.
+ "requester_principal": {
+ "user_id": "agents-service",
+ "tenant_id": str(tenant),
+ "permissions": ["actions:execute:high"],
+ },
+ },
+ )
+ analyst_id = uuid.uuid4()
+ user = SimpleNamespace(
+ user_id=analyst_id,
+ tenant_id=tenant,
+ role="analyst",
+ effective_permissions=lambda: ["actions:execute:high"],
+ )
+
+ dispatch = await approvals._dispatch_decision(row, user, approve=True)
+
+ assert dispatch["state"] == "executed", (
+ f"the console approval path did not execute: {dispatch}. Every tap of Approve in the responder app surfaced this as a 502."
+ )
+ assert _contain_calls() == 1
+
+ async def test_a_human_cannot_approve_the_action_they_raised(self, replicas, vendor_base_url) -> None:
+ """Separation of duties, on the half where it must still bite."""
+ a, _b = replicas
+ tenant = str(uuid.uuid4())
+ analyst = _principal(tenant, user_id="dana")
+ action_id = str(uuid.uuid4())
+
+ await _submit(
+ a,
+ tenant=tenant,
+ action_id=action_id,
+ requester=analyst,
+ principal=analyst,
+ target="ws-4471",
+ params=_vendor_params(vendor_base_url),
+ )
+ refused = await _approve(a, action_id, analyst)
+
+ assert refused.status_code == 403, refused.text
+ assert "separation of duties" in refused.text
+ assert _contain_calls() == 0
+
+ async def test_separation_of_duties_is_not_waived_by_an_absent_requester(self, replicas, vendor_base_url) -> None:
+ """An action nobody is recorded as requesting is not a free approval.
+
+ ``if separation_of_duties and requester and ...`` passed silently when
+ the submit carried no principal at all, so the control reported itself
+ satisfied on exactly the requests it could not evaluate.
+ """
+ a, _b = replicas
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ analyst = _principal(tenant, user_id="dana")
+
+ async with httpx.AsyncClient(timeout=30.0) as client:
+ submitted = await client.post(
+ f"{a.base_url}/api/v1/actions",
+ json={
+ "id": action_id,
+ "action_type": "isolate_host",
+ "target": "ws-4471",
+ "tenant_id": tenant,
+ "incident_id": str(uuid.uuid4()),
+ "rationale": "phase 2 regression",
+ "parameters": _vendor_params(vendor_base_url),
+ },
+ headers={"Authorization": f"Bearer {SERVICE_TOKEN}"},
+ )
+ assert submitted.status_code == 200, submitted.text
+
+ approved = await _approve(a, action_id, analyst)
+ assert approved.status_code == 200, approved.text
+
+ stored = await _read_record(_DSN, action_id)
+ assert stored is not None
+ assert stored.get("separation_of_duties") == "unevaluated", (
+ "an approval whose requester is unknown recorded nothing about it, so the audit "
+ f"trail cannot distinguish an evaluated control from a skipped one: {stored.get('separation_of_duties')!r}"
+ )
+
+ async def test_an_approver_from_another_tenant_is_refused(self, replicas, vendor_base_url) -> None:
+ """``aisoc_action_records`` is reachable by id, and the id was the only control."""
+ a, b = replicas
+ victim = str(uuid.uuid4())
+ attacker_tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+
+ await _submit(
+ a,
+ tenant=victim,
+ action_id=action_id,
+ requester=_principal(victim, user_id="agents-service"),
+ principal=_principal(victim, user_id="agents-service"),
+ target="ws-victim",
+ params=_vendor_params(vendor_base_url),
+ )
+
+ # From the *other* replica, so the read has to come from the database
+ # rather than the submitting process's cache.
+ refused = await _approve(b, action_id, _principal(attacker_tenant, user_id="mallory"))
+
+ assert refused.status_code in (403, 404), (
+ f"an approver in tenant {attacker_tenant} was served tenant {victim}'s action: {refused.status_code} {refused.text}"
+ )
+ assert _contain_calls() == 0, "another tenant's host was contained"
+
+ async def test_an_approver_below_the_tier_is_refused(self, replicas, vendor_base_url) -> None:
+ """The control that already worked, kept so the fix cannot quietly drop it."""
+ a, _b = replicas
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+
+ await _submit(
+ a,
+ tenant=tenant,
+ action_id=action_id,
+ requester=_principal(tenant, user_id="agents-service"),
+ principal=_principal(tenant, user_id="agents-service"),
+ target="ws-4471",
+ params=_vendor_params(vendor_base_url),
+ )
+ refused = await _approve(a, action_id, _principal(tenant, user_id="sam", permissions=["actions:execute:low"]))
+
+ assert refused.status_code == 403
+ assert _contain_calls() == 0
+
+
+# ── B. the intent an approver signed off ───────────────────────────────────
+
+
+class TestApprovalIsBoundToTheIntent:
+ async def test_a_resubmission_cannot_retarget_a_pending_approval(self, replicas, vendor_base_url) -> None:
+ """The approval card says ws-4471; the record must not become dc-01.
+
+ ``save`` was ``ON CONFLICT (id) DO UPDATE SET record = EXCLUDED.record``
+ with no condition, so anything that could submit could rewrite what a
+ human was in the middle of approving.
+ """
+ a, b = replicas
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ analyst = _principal(tenant, user_id="dana")
+ params = _vendor_params(vendor_base_url)
+
+ first = await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)
+ assert first.status_code == 200
+ assert first.json()["status"] == "awaiting_approval"
+
+ swapped = await _submit(b, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="dc-01", params=params)
+ assert swapped.status_code == 409, (
+ f"the pending action was retargeted to dc-01 while awaiting approval: {swapped.status_code} {swapped.text}"
+ )
+
+ stored = await _read_record(_DSN, action_id)
+ assert stored is not None
+ assert stored["target"] == "ws-4471"
+
+ approved = await _approve(a, action_id, analyst)
+ assert approved.status_code == 200, approved.text
+ with VENDOR.lock:
+ assert VENDOR.contain_calls, "nothing reached the vendor"
+
+ async def test_a_resubmission_cannot_change_the_parameters_either(self, replicas, vendor_base_url) -> None:
+ """The subtler half, and the one a target-only guard would miss.
+
+ An action gated *because of what it targets* can be rewritten through
+ its parameters alone — same verb, same host, a different thing done to
+ it — and the approval card would not change at all.
+ """
+ a, b = replicas
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ params = _vendor_params(vendor_base_url)
+
+ first = await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)
+ assert first.status_code == 200
+ assert "scope" not in first.json()["parameters"]
+
+ altered = await _submit(
+ b,
+ tenant=tenant,
+ action_id=action_id,
+ requester=agent,
+ principal=agent,
+ target="ws-4471",
+ params={**params, "scope": "entire-subnet"},
+ )
+ assert altered.status_code == 409, (
+ f"a pending action's parameters were rewritten while it awaited approval: {altered.status_code} {altered.text}"
+ )
+
+ stored = await _read_record(_DSN, action_id)
+ assert stored is not None
+ assert "scope" not in (stored.get("parameters") or {}), stored.get("parameters")
+
+ async def test_an_identical_resubmission_is_still_idempotent(self, replicas, vendor_base_url) -> None:
+ """A retried submit of the same intent must not become an error.
+
+ The store's contract has always been "a replay of the same submission
+ is idempotent rather than a duplicate pending approval", and refusing
+ a changed intent must not cost that.
+ """
+ a, b = replicas
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ params = _vendor_params(vendor_base_url)
+
+ first = await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)
+ again = await _submit(b, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)
+ assert first.status_code == 200
+ assert again.status_code == 200, again.text
+ assert again.json()["status"] == "awaiting_approval"
+
+
+# ── C. one vendor call, or none ────────────────────────────────────────────
+
+
+class TestExecutionIsClaimedBeforeTheVendorIsContacted:
+ async def test_two_replicas_cannot_both_execute_one_approval(self, replicas, vendor_base_url) -> None:
+ """The defect, in the shape that produced it.
+
+ Both replicas read ``awaiting_approval`` — one from its cache, one from
+ the database — and both called the EDR. The host was contained twice,
+ two audit trails were written, and the only durable evidence of either
+ was whichever ``save`` landed last.
+ """
+ a, b = replicas
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ analyst = _principal(tenant, user_id="dana")
+ params = _vendor_params(vendor_base_url)
+
+ await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)
+
+ # Stall the vendor so both approvals are genuinely in flight together.
+ # Without this the first can finish before the second reads, and the
+ # race closes by timing rather than by the fix.
+ with VENDOR.lock:
+ VENDOR.contain_delay = 1.5
+
+ outcomes: list[Any] = list(
+ await asyncio.gather(
+ _approve(a, action_id, analyst),
+ _approve(b, action_id, analyst),
+ return_exceptions=True,
+ )
+ )
+
+ assert _contain_calls() == 1, (
+ f"the host was contained {_contain_calls()} times for one approval. "
+ f"replicas answered {[getattr(o, 'status_code', o) for o in outcomes]}"
+ )
+ codes = sorted(getattr(r, "status_code", 500) for r in outcomes)
+ assert codes[0] == 200, f"neither replica executed: {codes}"
+ assert codes[1] in (400, 409), f"the losing replica did not refuse cleanly: {codes}"
+
+ async def test_a_repeated_approval_does_not_execute_twice(self, replicas, vendor_base_url) -> None:
+ """A client that retries — a double tap, a proxy retry, a resumed run."""
+ a, _b = replicas
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ analyst = _principal(tenant, user_id="dana")
+ params = _vendor_params(vendor_base_url)
+
+ await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)
+
+ first = await _approve(a, action_id, analyst)
+ second = await _approve(a, action_id, analyst)
+
+ assert first.status_code == 200, first.text
+ assert second.status_code in (400, 409), second.text
+ assert _contain_calls() == 1
+
+ async def test_the_vendor_is_not_contacted_when_the_claim_cannot_be_written(self, replicas, vendor_base_url) -> None:
+ """A store that cannot record the authorization must not authorize one.
+
+ ``save`` logged and returned on a write failure, so an approval could
+ contain a host and leave no durable trace that it had. The claim is the
+ pre-dispatch write, and it fails closed.
+ """
+ import asyncpg
+
+ a, _b = replicas
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ analyst = _principal(tenant, user_id="dana")
+ params = _vendor_params(vendor_base_url)
+
+ await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)
+
+ plain = _DSN.replace("postgresql+asyncpg://", "postgresql://")
+ admin = await asyncpg.connect(plain.replace("aisoc_app", "aisoc"), timeout=10.0)
+ try:
+ await admin.execute("REVOKE UPDATE, INSERT ON aisoc_action_records FROM aisoc_app")
+ refused = await _approve(a, action_id, analyst)
+ finally:
+ await admin.execute("GRANT INSERT, UPDATE ON aisoc_action_records TO aisoc_app")
+ await admin.close()
+
+ assert _contain_calls() == 0, "a host was contained while the durable record could not be written"
+ assert refused.status_code >= 500, f"an unwritable store was reported as a client error: {refused.status_code} {refused.text}"
+
+ async def test_a_stale_cache_cannot_resurrect_an_executed_action(self, replicas, vendor_base_url) -> None:
+ """The submitting replica holds the record in memory forever.
+
+ ``action_store.get`` checked that dict first, so a replica whose cache
+ said ``awaiting_approval`` would execute again regardless of what the
+ database had recorded. Replica A submits and caches; replica B executes;
+ A must then refuse.
+ """
+ a, b = replicas
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ analyst = _principal(tenant, user_id="dana")
+ params = _vendor_params(vendor_base_url)
+
+ await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)
+
+ executed = await _approve(b, action_id, analyst)
+ assert executed.status_code == 200, executed.text
+
+ stale = await _approve(a, action_id, analyst)
+ assert stale.status_code in (400, 409), (
+ f"replica A's cached copy let an already-executed action run again: {stale.status_code} {stale.text}"
+ )
+ assert _contain_calls() == 1
+
+
+# ── D. when the outcome is genuinely unknown ───────────────────────────────
+
+
+class TestAnUnknownOutcomeIsRecordedAsUnknown:
+ """A dispatch that did not come back clean is unknown until the vendor says.
+
+ The executors catch their own vendor errors and return
+ ``ActionResult(status=FAILED, error=str(exc))``, so the record read
+ ``failed`` for a containment the EDR had accepted — a stronger claim than
+ the evidence supports, and the dangerous direction, because the obvious
+ response to a failure is to run it again.
+
+ The resolution is not to guess from the exception type. It is to ask the
+ vendor, with the read-back probe this service already ships: present means
+ it completed, absent means it genuinely failed, and unreadable is the only
+ honest ``reconciliation_required``.
+ """
+
+ async def test_a_lost_response_is_resolved_by_reading_the_vendor_back(self, replicas, vendor_base_url) -> None:
+ """Vendor receipt, response lost, device reachable. The host *is* contained."""
+ a, _b = replicas
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ analyst = _principal(tenant, user_id="dana")
+ params = _vendor_params(vendor_base_url)
+
+ await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)
+ with VENDOR.lock:
+ VENDOR.drop_response = True
+
+ response = await _approve(a, action_id, analyst, timeout=60.0)
+ assert response.status_code in (200, 202), response.text
+ assert _contain_calls() == 1, "the vendor never received the containment; this test proves nothing"
+
+ stored = await _read_record(_DSN, action_id)
+ assert stored is not None
+ assert stored["_column_status"] == "completed", (
+ f"a containment the EDR confirms is in force was recorded as {stored['_column_status']!r}"
+ )
+ assert stored["verification"]["outcome"] == "verified", stored.get("verification")
+
+ async def test_an_effect_that_cannot_be_read_back_is_recorded_as_unknown(self, replicas, vendor_base_url) -> None:
+ """Vendor receipt, response lost, device API down. Nobody knows."""
+ a, _b = replicas
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ analyst = _principal(tenant, user_id="dana")
+ params = _vendor_params(vendor_base_url)
+
+ await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)
+ with VENDOR.lock:
+ VENDOR.drop_response = True
+ VENDOR.probe_available = False
+
+ response = await _approve(a, action_id, analyst, timeout=60.0)
+ assert response.status_code in (200, 202), response.text
+ assert _contain_calls() == 1
+
+ stored = await _read_record(_DSN, action_id)
+ assert stored is not None
+ assert stored["_column_status"] == "reconciliation_required", (
+ "a vendor call that may have landed, against a device API that cannot be read, was recorded as "
+ f"{stored['_column_status']!r}; the host's real state is unknown and the record must say so"
+ )
+
+ async def test_a_refused_dispatch_is_still_a_failure(self, replicas, vendor_base_url) -> None:
+ """The control. Not everything becomes "unknown" — that would be its own lie.
+
+ The vendor answered the containment with a 500 and the device is
+ readable and reports ``normal``, so the containment demonstrably did
+ not take and ``failed`` is the honest record. A change that resolved
+ every imperfect dispatch to ``reconciliation_required`` would pass
+ every other test in this class and fail this one.
+ """
+ a, _b = replicas
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ analyst = _principal(tenant, user_id="dana")
+ params = _vendor_params(vendor_base_url)
+
+ await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)
+ with VENDOR.lock:
+ VENDOR.contain_http_status = 500
+
+ response = await _approve(a, action_id, analyst, timeout=60.0)
+ assert response.status_code in (200, 202), response.text
+
+ stored = await _read_record(_DSN, action_id)
+ assert stored is not None
+ assert stored["_column_status"] == "failed", stored["_column_status"]
+ assert stored["verification"]["outcome"] == "failed", stored.get("verification")
+
+ async def test_a_vendor_timeout_is_not_recorded_as_a_failure(self, replicas, vendor_base_url) -> None:
+ """ "The request timed out" and "the action did not happen" are different facts.
+
+ The slow sibling of the lost-response case, kept because a real
+ ``httpx`` read timeout and a dropped connection are different code
+ paths inside the client and only one of them is cheap to reproduce.
+ """
+ a, _b = replicas
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ analyst = _principal(tenant, user_id="dana")
+ params = _vendor_params(vendor_base_url)
+
+ await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)
+
+ # Longer than the client's 30s containment timeout, so the client
+ # really times out on a request the vendor really received. The device
+ # API is down too, so the outcome is genuinely unestablished.
+ with VENDOR.lock:
+ VENDOR.contain_delay = 34.0
+ VENDOR.probe_available = False
+
+ response = await _approve(a, action_id, analyst, timeout=120.0)
+ assert response.status_code in (200, 202), response.text
+
+ stored = await _read_record(_DSN, action_id)
+ assert stored is not None
+ assert stored["_column_status"] == "reconciliation_required", (
+ "a vendor call that may have landed was recorded as "
+ f"{stored['_column_status']!r}; the host's real state is unknown and the record must say so"
+ )
+ assert _contain_calls() == 1
+
+ async def test_an_unknown_action_is_not_re_executed_on_approval(self, replicas, vendor_base_url) -> None:
+ """Retrying an action that may already have run is the thing to prevent."""
+ a, _b = replicas
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ analyst = _principal(tenant, user_id="dana")
+ params = _vendor_params(vendor_base_url)
+
+ await _submit(a, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)
+ with VENDOR.lock:
+ VENDOR.drop_response = True
+ VENDOR.probe_available = False
+ await _approve(a, action_id, analyst, timeout=60.0)
+ with VENDOR.lock:
+ VENDOR.drop_response = False
+ VENDOR.probe_available = True
+
+ retry = await _approve(a, action_id, analyst)
+ assert retry.status_code in (400, 409), f"an action in an unknown state was re-approved and run: {retry.text}"
+ assert _contain_calls() == 1
+
+ async def test_a_replica_killed_mid_dispatch_leaves_a_claimed_record(self, replicas, vendor_base_url) -> None:
+ """Restart recovery. A process that dies after contacting the vendor
+ must not leave an action that looks un-started."""
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ analyst = _principal(tenant, user_id="dana")
+ params = _vendor_params(vendor_base_url)
+
+ doomed = _spawn_replica("doomed", _DSN)
+ try:
+ await _submit(doomed, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)
+ with VENDOR.lock:
+ VENDOR.contain_delay = 8.0
+
+ task = asyncio.create_task(_approve(doomed, action_id, analyst, timeout=60.0))
+ # Wait until the vendor has actually been reached, then kill the
+ # replica. Sleeping a fixed interval would race the dispatch.
+ deadline = time.time() + 20
+ while _contain_calls() == 0 and time.time() < deadline:
+ await asyncio.sleep(0.1)
+ assert _contain_calls() == 1, "the vendor was never reached; this test proves nothing"
+ doomed.kill()
+ task.cancel()
+ try:
+ await task
+ except (asyncio.CancelledError, httpx.HTTPError):
+ # Expected and carries no information: the replica was killed
+ # mid-request on purpose, so the in-flight approve either
+ # cancels or loses its connection. The assertion this test
+ # makes is about the durable record, read from Postgres below.
+ pass
+ finally:
+ if doomed.proc.poll() is None: # pragma: no cover - teardown guard
+ doomed.kill()
+ with VENDOR.lock:
+ VENDOR.contain_delay = 0.0
+
+ stored = await _read_record(_DSN, action_id)
+ assert stored is not None
+ assert stored["_column_status"] in ("running", "reconciliation_required"), (
+ f"a replica died after contacting the vendor and the record reads {stored['_column_status']!r}, "
+ "which is indistinguishable from an action nobody started"
+ )
+ assert stored.get("dispatch", {}).get("claimed_by"), "the record does not say which replica claimed it"
+
+ async def test_a_claimed_record_survives_the_restart_and_still_refuses(self, replicas, vendor_base_url) -> None:
+ """A fresh replica with an empty cache reads the claim and holds."""
+ a, _b = replicas
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ analyst = _principal(tenant, user_id="dana")
+ params = _vendor_params(vendor_base_url)
+
+ doomed = _spawn_replica("restart-a", _DSN)
+ try:
+ await _submit(doomed, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)
+ with VENDOR.lock:
+ VENDOR.contain_delay = 8.0
+ task = asyncio.create_task(_approve(doomed, action_id, analyst, timeout=60.0))
+ deadline = time.time() + 20
+ while _contain_calls() == 0 and time.time() < deadline:
+ await asyncio.sleep(0.1)
+ doomed.kill()
+ task.cancel()
+ try:
+ await task
+ except (asyncio.CancelledError, httpx.HTTPError):
+ # Expected and carries no information: the replica was killed
+ # mid-request on purpose, so the in-flight approve either
+ # cancels or loses its connection. The assertion this test
+ # makes is about the durable record, read from Postgres below.
+ pass
+ finally:
+ if doomed.proc.poll() is None: # pragma: no cover - teardown guard
+ doomed.kill()
+ with VENDOR.lock:
+ VENDOR.contain_delay = 0.0
+
+ replacement = await _approve(a, action_id, analyst)
+ assert replacement.status_code in (400, 409), (
+ f"after a restart the action was run a second time: {replacement.status_code} {replacement.text}"
+ )
+ assert _contain_calls() == 1
+
+
+# ── E. reconciling, without guessing ───────────────────────────────────────
+
+
+class TestReconciliationReadsTheVendorRatherThanGuessing:
+ async def _strand(self, replica: Replica, vendor_base_url: str) -> tuple[str, dict[str, Any], dict[str, Any]]:
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ analyst = _principal(tenant, user_id="dana")
+ params = _vendor_params(vendor_base_url)
+ await _submit(replica, tenant=tenant, action_id=action_id, requester=agent, principal=agent, target="ws-4471", params=params)
+ with VENDOR.lock:
+ VENDOR.drop_response = True
+ VENDOR.probe_available = False
+ await _approve(replica, action_id, analyst, timeout=60.0)
+ with VENDOR.lock:
+ VENDOR.drop_response = False
+ VENDOR.probe_available = True
+ return action_id, analyst, params
+
+ async def test_a_confirmed_effect_closes_the_record_without_acting_again(self, replicas, vendor_base_url) -> None:
+ """The EDR says the host is contained, so the action completed.
+
+ ``PostActionVerifier`` already existed and already reads the device's
+ real containment state. Reconciliation is that probe, not a retry.
+ """
+ a, _b = replicas
+ action_id, _analyst, params = await self._strand(a, vendor_base_url)
+ before = _contain_calls()
+
+ async with httpx.AsyncClient(timeout=60.0) as client:
+ reconciled = await client.post(
+ f"{a.base_url}/api/v1/actions/{action_id}/reconcile",
+ json={"parameters": params},
+ headers={"Authorization": f"Bearer {SERVICE_TOKEN}"},
+ )
+
+ assert reconciled.status_code == 200, reconciled.text
+ body = reconciled.json()
+ assert body["status"] == "completed", body
+ assert body["verification"]["outcome"] == "verified", body
+ assert _contain_calls() == before, "reconciliation executed the action again instead of reading it back"
+
+ stored = await _read_record(_DSN, action_id)
+ assert stored is not None and stored["_column_status"] == "completed"
+
+ async def test_an_absent_effect_is_recorded_as_failed_not_guessed_complete(self, replicas, vendor_base_url) -> None:
+ a, _b = replicas
+ action_id, _analyst, params = await self._strand(a, vendor_base_url)
+ before = _contain_calls()
+ # The vendor accepted the request and the host is nonetheless not
+ # contained — a real outcome, and the one a retry would be wrong about.
+ with VENDOR.lock:
+ VENDOR.device_status = "normal"
+
+ async with httpx.AsyncClient(timeout=60.0) as client:
+ reconciled = await client.post(
+ f"{a.base_url}/api/v1/actions/{action_id}/reconcile",
+ json={"parameters": params},
+ headers={"Authorization": f"Bearer {SERVICE_TOKEN}"},
+ )
+
+ assert reconciled.status_code == 200, reconciled.text
+ assert reconciled.json()["status"] == "failed", reconciled.json()
+ assert _contain_calls() == before
+
+ async def test_an_unreadable_vendor_leaves_the_record_unknown(self, replicas, vendor_base_url) -> None:
+ """A reconciliation that cannot read the vendor resolves nothing.
+
+ The tempting failure is to treat an unanswerable probe as a clean
+ result in whichever direction is convenient. It stays unknown, and the
+ operator is told so rather than told it is handled.
+ """
+ a, _b = replicas
+ action_id, _analyst, params = await self._strand(a, vendor_base_url)
+ before = _contain_calls()
+ with VENDOR.lock:
+ VENDOR.probe_available = False
+
+ async with httpx.AsyncClient(timeout=60.0) as client:
+ reconciled = await client.post(
+ f"{a.base_url}/api/v1/actions/{action_id}/reconcile",
+ json={"parameters": params},
+ headers={"Authorization": f"Bearer {SERVICE_TOKEN}"},
+ )
+
+ assert reconciled.status_code == 200, reconciled.text
+ body = reconciled.json()
+ assert body["status"] == "reconciliation_required", body
+ assert body["verification"]["outcome"] == "unverified", body
+ assert _contain_calls() == before
+
+ async def test_reconcile_will_not_touch_another_tenants_action(self, replicas, vendor_base_url) -> None:
+ a, b = replicas
+ action_id, _analyst, params = await self._strand(a, vendor_base_url)
+
+ async with httpx.AsyncClient(timeout=60.0) as client:
+ refused = await client.post(
+ f"{b.base_url}/api/v1/actions/{action_id}/reconcile",
+ json={"parameters": params},
+ headers={
+ "Authorization": f"Bearer {SERVICE_TOKEN}",
+ "X-Tenant-ID": str(uuid.uuid4()),
+ },
+ )
+ assert refused.status_code == 404, refused.text
+
+
+# ── F. the read surface ────────────────────────────────────────────────────
+
+
+class TestReadingAnActionRecordIsTenantScoped:
+ async def test_a_matching_tenant_sees_the_whole_record(self, replicas, vendor_base_url) -> None:
+ a, _b = replicas
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ await _submit(
+ a,
+ tenant=tenant,
+ action_id=action_id,
+ requester=agent,
+ principal=agent,
+ target="ws-4471",
+ params=_vendor_params(vendor_base_url),
+ )
+
+ async with httpx.AsyncClient(timeout=20.0) as client:
+ seen = await client.get(
+ f"{a.base_url}/api/v1/actions/{action_id}",
+ headers={"Authorization": f"Bearer {SERVICE_TOKEN}", "X-Tenant-ID": tenant},
+ )
+ assert seen.status_code == 200
+ assert seen.json()["target"] == "ws-4471"
+
+ async def test_another_tenant_is_told_nothing_about_it(self, replicas, vendor_base_url) -> None:
+ a, b = replicas
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ await _submit(
+ a,
+ tenant=tenant,
+ action_id=action_id,
+ requester=agent,
+ principal=agent,
+ target="ws-victim",
+ params=_vendor_params(vendor_base_url),
+ )
+
+ async with httpx.AsyncClient(timeout=20.0) as client:
+ seen = await client.get(
+ f"{b.base_url}/api/v1/actions/{action_id}",
+ headers={"Authorization": f"Bearer {SERVICE_TOKEN}", "X-Tenant-ID": str(uuid.uuid4())},
+ )
+ assert seen.status_code == 404, seen.text
+
+ async def test_an_unscoped_read_does_not_disclose_the_target(self, replicas, vendor_base_url) -> None:
+ """No caller in the tree sends this route a tenant, so it stays usable —
+ but an id is not a licence to read what a host or account was aimed at."""
+ a, _b = replicas
+ tenant = str(uuid.uuid4())
+ action_id = str(uuid.uuid4())
+ agent = _principal(tenant, user_id="agents-service")
+ await _submit(
+ a,
+ tenant=tenant,
+ action_id=action_id,
+ requester=agent,
+ principal=agent,
+ target="ws-secret-name",
+ params=_vendor_params(vendor_base_url),
+ )
+
+ async with httpx.AsyncClient(timeout=20.0) as client:
+ seen = await client.get(
+ f"{a.base_url}/api/v1/actions/{action_id}",
+ headers={"Authorization": f"Bearer {SERVICE_TOKEN}"},
+ )
+ assert seen.status_code == 200, seen.text
+ body = seen.json()
+ assert body["status"] == "awaiting_approval"
+ assert "ws-secret-name" not in json.dumps(body), f"an unscoped read disclosed the target: {body}"
+ assert "parameters" not in body