Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

Make the published playbook schema describe the engine, and contract the last ActionType (#807)

Beenu Arora20fd2b3ebd63
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index c5adf2ff6..66e796b2a 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -987,8 +987,19 @@ jobs:
git push origin eval-results || echo "push skipped"
# ──────────────────────────────────────────────
- # Pillar-2: Playbook JSON Schema lint
- # Validates all playbook JSON files against schemas/playbook.schema.json
+ # Pillar-2: Playbook JSON Schema lint + schema/engine parity
+ #
+ # The lint validates every *.playbook.json in the tree against
+ # schemas/playbook.schema.json. It used to scan only the two files under
+ # services/agents/data/playbooks and report "2/2 passed" while the 62
+ # playbooks in playbooks/packs/v1 went unchecked — 32 of which did not match.
+ #
+ # The parity check asks the other question, which nothing asked before: does
+ # the published schema describe the engine that runs these files. It compares
+ # the schema, the StepType enum, the engine's handler table and the pack
+ # validator's trigger list in both directions, and carries a --self-test that
+ # injects drift each way so a passing run cannot mean "the comparison is
+ # broken".
# ──────────────────────────────────────────────
p2-playbook-schema-lint:
name: P2 — Playbook Schema Lint
@@ -998,10 +1009,17 @@ jobs:
- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0
with:
python-version: '3.12'
- - name: Install jsonschema
- run: pip install jsonschema
+ - name: Install deps
+ # pydantic + httpx let the parity check import the real engine rather
+ # than parse it; a parser can disagree with the handler table that is
+ # actually built at import time.
+ run: pip install jsonschema pydantic httpx pytest
- name: Lint playbook files
run: python3 scripts/lint_playbooks.py
+ - name: Schema/engine parity (both directions, with self-test)
+ run: python3 scripts/check_playbook_schema_parity.py --self-test
+ - name: Parity gate tests
+ run: python3 -m pytest tests/test_playbook_schema_parity_gate.py -q
# ──────────────────────────────────────────────
# Docker Compose smoke test
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 4bacee901..dd7e70d06 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -98,6 +98,77 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
or parses empty is a hard error rather than a quiet pass — a gate that
reports OK about a tree it never opened is worse than no gate.
+- **The published playbook schema now describes the engine that runs
+ playbooks, and a gate keeps it that way in both directions.**
+ `schemas/playbook.schema.json` is the contract authors are told to trust,
+ and it had drifted from `services/agents/app/playbook/` in every available
+ direction at once. Two schema files existed with different step
+ vocabularies — 15 types at the repo root, 9 under `schemas/` — against a
+ `StepType` enum of 22, and the NL drafter silently fell back from one to
+ the other if the primary was missing. Eleven step types were declared by a
+ schema and implemented nowhere (`trigger`, `action`, `loop`, `parallel`,
+ `human_approval`, `wait`, `isolate`, `block`, `create_case`,
+ `run_playbook`, `script`); thirteen were accepted by the engine and
+ declared by neither schema. Six step fields (`blast_radius`, `depends_on`,
+ `output_key`, `retry.max_attempts`, `retry.backoff_seconds`,
+ `retry.backoff_multiplier`) were declared and never read — `blast_radius`
+ carried the description "engine enforces analyst approval for destructive
+ steps", and the engine could not see the field at all.
+
+ Resolved by making `schemas/playbook.schema.json` the only schema, widened
+ to the full `StepType` range with the condition-as-string form the engine
+ already accepted, bounds matched to `bounds.py` (3600s / 25 retries, not
+ 600s / 5), and the two authored-but-inert playbook keys (`inputs`,
+ `dry_run_support`) declared as the documentation they are. The root
+ duplicate is deleted. The schema also carries `x-aisoc-execution`, a
+ machine-checked map recording whether each step type is `executed`,
+ `simulated`, or vocabulary with no handler — so an author can tell what
+ will happen before writing the playbook rather than after running it.
+
+ `run_playbook` is deliberately **not** implemented as a step, on its own
+ merits rather than by inheriting the argument that removed it as an action.
+ A nested playbook's steps are not visible where the parent declares its
+ step-level policy, so the parent cannot bound them; the engine reads no
+ step-level approval or blast-radius field today, so nesting would let one
+ ungated parent pull in an arbitrary tree; and twelve of the engine's own
+ step types have no handler, so a verb whose purpose is to execute more
+ steps would multiply that. It can return when playbook steps are graded
+ individually — recursion depth and an ancestor set are the easy part.
+
+ `scripts/check_playbook_schema_parity.py` compares the schema enum, the
+ `StepType` enum, the engine's handler table, the execution map, the bounds
+ module and the pack validator's trigger list — every pair in both
+ directions, because the characteristic failure here is a check that asks
+ only whether the schema declares something the engine lacks and never the
+ reverse, which is the direction things actually drift. It carries a
+ `--self-test` that injects drift each way and fails if any goes undetected,
+ refuses to run at all on a tree missing its marker files rather than
+ printing OK about files it never opened, and names the root, schema, step
+ counts and playbook count it inspected.
+
+- **Every `ActionType` now resolves a capability contract.** `notify_slack`
+ was the last one without, and it was a naming gap rather than a missing
+ capability: the verb is `notify`, it has had a contract throughout, and the
+ `SlackNotify` adapter already bridged the two names. Nothing connected a
+ lookup *by `ActionType` value* to that bridge, so `approval_gate` found no
+ contract and skipped the confidence matrix — a 10%-confidence
+ `notify_slack` was approved for auto-execution, and the verb most likely to
+ auto-execute was the one graded without reference to confidence. It now
+ requires an analyst under the default L1 tier.
+
+ Closed with a one-entry alias (`ACTION_TYPE_CAPABILITY_ALIASES`) rather
+ than a rename, because `action_type` is persisted operator intent:
+ `remediation_whitelist` (migration 015) stores per-tenant pre-approvals
+ keyed `UNIQUE (tenant_id, action_type)`, so renaming the member silently
+ orphans every row an operator created for `notify_slack`. It is also a
+ documented request field and a member of the `ActionType` union in
+ `packages/types`. The retirement condition is recorded rather than left
+ open-ended: the map goes when `ActionType` does. `check_action_contract.py`
+ gains two directions — every `ActionType` must resolve a contract, and
+ every alias must name a real `ActionType`, point at a real contract, not
+ shadow a capability of the same name, and describe a bridge some adapter
+ actually implements.
+
- **Competitor product names removed from the docs portal, the benchmark page
and the archived plan subtree, and a CI gate added to keep them out.** AiSOC
names no competitor product, but two published comparison tables and the
@@ -519,6 +590,59 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
cover. The one MITRE figure the project does publish — 97.0% in
`BenchmarkBand` — is labelled "substrate" and is unchanged.
+- **A playbook step the engine could not run reported success.** Twelve of
+ the twenty-two `StepType` members had no entry in the engine's handler
+ table. The run loop answered those with `{"skipped": true}` and left the
+ step's status at its `SUCCESS` default, so a playbook containing them ran
+ to `COMPLETED` having done nothing it said it did. `approval` is the one
+ that mattered: it is a human decision point, it appears in 14 steps across
+ the shipped packs, and it passed on its own — the run continued straight
+ into the action an analyst was meant to authorise. An unimplemented step
+ type now fails closed with `unimplemented: true` and an error naming the
+ verb, is not retried (a missing handler will still be missing next
+ attempt), and halts the run under the default `on_failure: abort` while
+ still honouring an explicit `continue`. A dry run reports `would_fail` for
+ such a step instead of a bare `dry_run: true`.
+
+ `apps/docs/docs/concepts/playbooks.md` had documented the safe behaviour
+ all along — "recorded as `SKIPPED` … so unknown actions never silently
+ succeed" — and has been corrected to describe what the code now does. The
+ same page described a manual approval gate backed by
+ `POST /v1/playbook-runs/{id}/approve`, where "the engine pauses on the
+ condition until the field flips, then resumes". No such endpoint exists and
+ the engine has no pause or resume; that section now says so and points at
+ the actions service, which does hold an action for an analyst.
+
+- **The playbook lint job validated two files and reported "2/2 passed".**
+ `scripts/lint_playbooks.py` claimed in its own docstring to check "any
+ `*.playbook.json` files anywhere in the repo" and only ever scanned the two
+ under `services/agents/data/playbooks/`. Pointed at the whole tree, 32 of
+ the 62 playbooks in `playbooks/packs/v1/` did not match the published
+ schema. It now scans recursively, treats finding no files as a broken scan
+ rather than a clean bill of health, and no longer crashes in its own error
+ path when a file passed on argv sits outside the repo.
+
+- **20 shipped playbooks carried a duplicated tag.**
+ `scripts/generate_playbooks.py` emitted `[category, *tags]` where several
+ categories already lead their tag list with the category name, producing
+ e.g. `["supply-chain", "supply-chain", "npm", …]`. Fixed in the generator,
+ which is what the reproducibility gate diffs, rather than in the 20
+ generated files.
+
+- **The NL drafter rewrote steps to make its own output pass lint.** Because
+ the schema declared 9 of 22 step types, `_collapse_step_types_for_schema`
+ mapped the other 13 onto a "nearest neighbour" — `run_av_scan` and
+ `revoke_session` both became `investigate`, `approval` became `condition` —
+ keeping the original in `params.original_type`. The playbook that shipped
+ said it would investigate when the author had asked to disable an account,
+ and an approval gate came out as an ungated branch. The projection and its
+ second validation pass are removed; the schema covers the full range, so a
+ validation failure is now a real failure. The drafter's system prompt also
+ restated the vocabulary by hand and had drifted from it — offering
+ `webhook` as a trigger, which no validator in the repo accepts, and capping
+ `retry_max` at 5 against a model allowing 25 — and is now generated from
+ `StepType` and `bounds.py`.
+
- **The agent recommended evidence acquisition the platform could not
perform.** `capture_forensics` was an `ActionType` with no executor
anywhere, and `services/agents/app/agents/investigation_agent.py` proposes
diff --git a/ROADMAP.md b/ROADMAP.md
index e2d877375..df60431cf 100644
--- a/ROADMAP.md
+++ b/ROADMAP.md
@@ -56,9 +56,18 @@ The claim-to-gate matrix stands at **108 rows — 99 GATED / 9 PARTIAL / 0 NO GA
- [x] Eval harness: 20 synthetic incidents, ≥80% MITRE-tactic accuracy CI gate
### Visual SOAR studio
-- [x] React Flow playbook editor with full node palette (Trigger, Condition, Action, Loop, Parallel, Human Approval, Wait, Notify)
-- [x] DAG playbook engine with retries, idempotency, blast-radius checks
-- [x] `playbook.schema.json` (JSON Schema 2020-12) for portability and CI linting
+- [x] React Flow playbook editor over the step types the engine implements
+ (enrich, investigate, notify, block_ip, isolate_host, create_ticket,
+ close_case, http, condition). Loop, Parallel, Wait and Human Approval
+ were listed here and exist in neither the palette nor the engine.
+- [x] Sequential playbook engine with conditions, branching, retries and
+ cycle detection. Not a DAG: there is no `depends_on` and no parallel
+ execution. No idempotency keys and no step-level blast-radius check —
+ `blast_radius` was declared by a schema the engine could not read, and
+ that schema has been removed rather than left as a promise.
+- [x] `schemas/playbook.schema.json` (JSON Schema draft-07) for portability
+ and CI linting, held to the engine in both directions by
+ `scripts/check_playbook_schema_parity.py`
- [x] Detection-as-Code: `detections/` directory with Sigma + AiSOC YAML, GitHub Action deploy-on-merge
- [x] 12 starter playbook templates
- [x] Community playbook marketplace (static index v4.0; publishing flow v4.1)
diff --git a/apps/docs/docs/compliance/evidence-pack.md b/apps/docs/docs/compliance/evidence-pack.md
index 148a9416b..4f7d9c042 100644
--- a/apps/docs/docs/compliance/evidence-pack.md
+++ b/apps/docs/docs/compliance/evidence-pack.md
@@ -33,7 +33,7 @@ and says so.
| Prompt-injection resistance | 32-payload adversarial corpus with a recall floor and a false-positive ceiling | `services/agents/tests/adversarial/` |
| Claim-to-gate traceability | Every product claim mapped to the CI job that fails when it stops being true | `docs/audit/CLAIM_TO_GATE_MATRIX.md` |
-The last row is the one worth reading first. It is the honest index: 109 rows
+The last row is the one worth reading first. It is the honest index: 112 rows
`GATED`, 9 `PARTIAL` with the specific gap named, and a ratchet that refuses
to let a row sit at `NO GATE`.
diff --git a/apps/docs/docs/concepts/live-actions.md b/apps/docs/docs/concepts/live-actions.md
index 3e3345a4a..5f92182ee 100644
--- a/apps/docs/docs/concepts/live-actions.md
+++ b/apps/docs/docs/concepts/live-actions.md
@@ -235,15 +235,15 @@ dispatched under governance:
| `KNOWN_CAPABILITIES` | The vocabulary everything validates against |
| `ProposedAction(action_type=...)` | What the agent puts in front of an analyst |
-A verb can be complete in three of the four and unreachable, and nothing in
+A verb can be complete in four of the five and unreachable, and nothing in
the failure message says which one is missing. `ack_alert` and
`suppress_alert` were exactly that for several releases: real executors with
Splunk, Elastic and Defender arms, wired into `EXECUTOR_REGISTRY`, and absent
-from the other three. Governed dispatch answered `executor_not_found` for code
+from the other four. Governed dispatch answered `executor_not_found` for code
that worked, which reads as a misconfigured integration rather than a
capability nobody connected.
-`scripts/check_action_contract.py` now compares all four, **in both
+`scripts/check_action_contract.py` now compares all five, **in both
directions**, on every pull request. Both directions is the part that matters.
The characteristic failure of a drift check is that it compares A against B
and never B against A, so drift in the direction things actually change passes
@@ -261,7 +261,13 @@ The gate reports:
- an adapter pointing at an `ActionType` with no executor, which would fail at
execution rather than at registration;
- an `ActionType` with no executor at all;
-- a verb the agent proposes by name that no executor implements.
+- a verb the agent proposes by name that no executor implements;
+- an `ActionType` that resolves no capability contract, so the approval
+ matrix is skipped for it and blast radius decides alone;
+- an alias in `ACTION_TYPE_CAPABILITY_ALIASES` that points at a capability
+ with no contract, keys an `ActionType` that does not exist, shadows a real
+ capability of the same name, or claims two names are one verb when no
+ adapter pairs them.
That last one is the check that spans two services, and it is the one that
mattered most. `investigation_agent` proposes `capture_forensics` whenever an
@@ -290,6 +296,34 @@ the `ActionType` surface rather than sit on it dead. The API accepts any
member of the enum, so a member with nothing behind it is a request that is
accepted and then fails with a message that reads like a broken deployment.
+### The one alias, and when it goes
+
+`notify_slack` was the last `ActionType` with no capability contract, and it
+was never a missing capability: the verb is `notify`, it has had a contract
+throughout, and the `SlackNotify` adapter already bridges the two names. What
+was missing was anything connecting a lookup *by `ActionType` value* to that
+bridge — so `approval_gate` found no contract, skipped the confidence matrix,
+and the verb most likely to auto-execute was the one verb graded without
+reference to confidence. A 10%-confidence `notify_slack` was approved for
+auto-execution; it now needs an analyst under the default tier.
+
+It is closed with an alias rather than a rename. `action_type` is persisted
+operator intent: `remediation_whitelist` (migration 015) stores per-tenant
+pre-approvals keyed `UNIQUE (tenant_id, action_type)`, so renaming the member
+silently orphans every row an operator created for `notify_slack` — their
+pre-approval stops matching and nobody is told. It is also a documented
+request field and a member of the `ActionType` union in `packages/types`.
+
+A permanent alias nothing ever retires is debt, so the retirement condition
+is stated: this map goes when `ActionType` does. The enum is already the
+legacy half of the registry — every adapter carries a `_legacy_action_type`
+pointing back at it, and governed dispatch works in capability strings. When
+`POST /actions` takes a capability and the `remediation_whitelist` rows have
+been migrated, the map and the enum go together. Until then a new entry needs
+the same justification: a *naming* difference for a verb that already has a
+contract, never a stand-in for a capability nobody wrote. The gate enforces
+each of those conditions rather than trusting the note.
+
### Actions that are not finished when they return
Most executors are synchronous: the vendor call returns and the action is
diff --git a/apps/docs/docs/concepts/playbooks.md b/apps/docs/docs/concepts/playbooks.md
index 681638ac6..c3eb6ae93 100644
--- a/apps/docs/docs/concepts/playbooks.md
+++ b/apps/docs/docs/concepts/playbooks.md
@@ -83,20 +83,51 @@ collapsed into this set in each connector's `normalize()`.
## Steps
-Each step has a `type` that maps to a handler in
-[`engine.py`](https://github.com/beenuar/AiSOC/tree/main/services/agents/app/playbook/engine.py).
+Each step has a `type`. The authoring contract is
+[`schemas/playbook.schema.json`](https://github.com/beenuar/AiSOC/tree/main/schemas/playbook.schema.json),
+and its `x-aisoc-execution` map records what
+[`engine.py`](https://github.com/beenuar/AiSOC/tree/main/services/agents/app/playbook/engine.py)
+actually does with each one. `scripts/check_playbook_schema_parity.py` holds
+the two together in both directions, so the table below cannot quietly stop
+being true.
+
+**Executed** — a handler runs and has a real effect:
| `type` | What it does |
|--------|--------------|
| `enrich` | Calls the enrichment service for IOC reputation, geo, ASN, GreyNoise, VT, OTX. |
| `investigate` | Triggers the AI investigator agent with focus areas (`forensics`, `lateral_movement`, etc.). |
-| `notify` | Sends a webhook, Slack, PagerDuty, or email notification. |
-| `block_ip` | Calls a firewall/EDR connector to block an IP (currently simulated for safety in OSS builds). |
-| `isolate_host` | Calls EDR to isolate a host (simulated by default). |
-| `create_ticket` | Opens a ticket in Jira / ServiceNow / Linear via connector. |
+| `notify` | Sends a webhook notification. SSRF-guarded. |
+| `http` | Generic outbound HTTP request — `method`, `url`, `body`, `headers`. SSRF-guarded. |
| `close_case` | Marks the AiSOC case as closed via the API service. |
-| `http` | Generic outbound HTTP request — `method`, `url`, `body`, `headers`. |
| `condition` | Pure branching node. Evaluates `condition` and routes to `next_true` / `next_false`. |
+| `osquery_live_query` | Distributed osquery via osctrl / FleetDM / aisoc-direct, against an allow-listed template. |
+
+**Simulated** — a handler runs and returns `{"simulated": true, ...}` without
+touching a vendor. Wire them to real connectors by editing the handler, or
+replace the step with an `http` step that calls your connector's enforcement
+endpoint:
+
+| `type` | What it does |
+|--------|--------------|
+| `block_ip` | Reports the IP it would block. |
+| `isolate_host` | Reports the host it would isolate. |
+| `create_ticket` | Reports the ticket it would open. |
+
+**Vocabulary without an engine handler** — these parse, validate and are used
+by the shipped packs, but no handler is registered, so the engine **fails the
+step closed** rather than reporting a success it did not achieve. With the
+default `on_failure: abort` that halts the run:
+
+`approval`, `block_ioc`, `disable_user`, `reset_password`, `revoke_session`,
+`force_mfa`, `kill_process`, `quarantine_file`, `run_av_scan`, `run_script`,
+`search_siem`, `create_notable_event`.
+
+Most of these verbs *do* have working executors in the actions service, which
+grades each one against its own capability contract before it runs. What does
+not exist yet is a bridge from a playbook step to that governed dispatch, so
+today they are authoring vocabulary rather than execution. A run that reaches
+one stops and says so.
Common step fields:
@@ -143,29 +174,47 @@ handling.
backoff before falling back to whatever you set as the next-failure mode.
- Cycle detection — if the engine revisits the same `step.id`, it aborts with
`error: "cycle at step <id>"` rather than looping forever.
-- Unknown step type — the step is recorded as `SKIPPED` with
- `reason: "no handler for <type>"` so unknown actions never silently succeed.
+- Step type with no handler — the step is recorded as `FAILED` with
+ `unimplemented: true` and an error naming the verb, so a step the engine
+ cannot run never reports success. It is not retried: a handler that is
+ missing now will still be missing on the next attempt.
+
+ This page previously said such a step was recorded as `SKIPPED` "so unknown
+ actions never silently succeed". The engine did put `skipped: true` in the
+ result, but left the step's *status* at `SUCCESS` and the run completed —
+ so the documented safety property was the opposite of the behaviour. It now
+ does what this section always claimed.
Each step result includes `_elapsed_ms` so you can see per-step latency in the
UI and in run history.
## Approvals and dry-runs
-Two patterns are supported today for human-in-the-loop steps:
-
-1. **`dry_run` flag** — pass `dry_run: true` when calling
- `PlaybookEngine.run(...)` (or set `dry_run` in step params for individual
- destructive actions). Handlers short-circuit with
- `{"dry_run": true, "step": <name>}` and emit the same realtime events, so
- you can preview an entire run without touching production.
-2. **Manual approval gate** — model the gate as a `condition` step backed by a
- field that an operator sets via the API (`POST /v1/playbook-runs/{id}/approve`).
- The engine pauses on the condition until the field flips, then resumes.
-
-`block_ip`, `isolate_host`, and `create_ticket` are simulated by default in OSS
-builds (they return `{"simulated": true, ...}`). Wire them to real connectors
-by editing the corresponding handler in `engine.py` or by replacing the step
-with an `http` step that calls your connector's enforcement endpoint.
+**`dry_run` flag** — pass `dry_run: true` when calling
+`PlaybookEngine.run(...)`. Handlers short-circuit with
+`{"dry_run": true, "step": <name>}` and emit the same realtime events, so you
+can preview an entire run without touching production. A step whose type has
+no handler is additionally reported as `unimplemented: true, would_fail: true`,
+so a preview tells you which steps a real run would stop on.
+
+**Human-in-the-loop is not implemented.** The engine is a single pass with no
+pause or resume, and an `approval` step fails closed (see
+[Steps](#steps) above), which halts the run under the default
+`on_failure: abort`.
+
+This section previously described a second supported pattern: model the gate
+as a `condition` step backed by a field an operator sets via
+`POST /v1/playbook-runs/{id}/approve`, and "the engine pauses on the condition
+until the field flips, then resumes". No such endpoint exists, and the engine
+has no pause or resume — the loop evaluates each condition once against the
+run context and moves on. Worse, until recently an `approval` step reported
+`SUCCESS` without doing anything, so a playbook that modelled a gate ran
+straight through it into whatever it was gating.
+
+If you need an approval gate today, split the playbook: run the
+investigative half, and submit the response verb to the actions service
+(`POST /actions`), which grades it against its capability contract and holds
+it at `awaiting_approval` for an analyst. That path is real and audited.
## Realtime events
diff --git a/apps/docs/docs/operations/notifications.md b/apps/docs/docs/operations/notifications.md
index cd84b6624..c214091ca 100644
--- a/apps/docs/docs/operations/notifications.md
+++ b/apps/docs/docs/operations/notifications.md
@@ -122,18 +122,29 @@ Slack tokens **must** come from your secret store (Doppler / Vault / k8s Secret)
The `notify_slack` action executor is the simplest possible Slack surface: a one-shot incoming-webhook POST. It does not require the Slack bot to be running.
+The playbook step type is `notify` — `notify_slack` is the `action_type` on
+the actions API, and the two names are bridged by an alias (see
+[Live Actions](../concepts/live-actions)). A step keys its arguments under
+`params`, not `parameters`:
+
```yaml
-# playbooks/auto-isolate-on-p0.yaml
steps:
- id: notify
- type: notify_slack
- parameters:
- webhook_url: "https://hooks.slack.com/services/..."
- channel: "#security-alerts"
+ name: Post to the SOC channel
+ type: notify
+ params:
+ channel: webhook
+ url: "https://hooks.slack.com/services/..."
message: |
- AiSOC isolated host {{ context.host }} after P0 alert {{ context.alert_id }}.
+ AiSOC isolated host {{ host }} after alert {{ alert_id }}.
```
+`schemas/playbook.schema.json` is the authoring contract, and
+`scripts/check_playbook_schema_parity.py` keeps it matched to the engine in
+both directions. Its `x-aisoc-execution` map records what the engine does
+with each step type, so you can tell before authoring whether a verb
+executes, simulates, or is vocabulary the engine will fail closed on.
+
Implementation: [`services/actions/app/executors/notification.py`](https://github.com/beenuar/AiSOC/blob/main/services/actions/app/executors/notification.py).
Failure mode is non-fatal but visible: if the webhook returns non-2xx the executor returns `ActionStatus.FAILED` with the underlying error string, the playbook step is marked failed, and the rest of the playbook runs as authored (typically with the failure path branching to `escalate_to_human`).
diff --git a/apps/web/public/marketplace/index.json b/apps/web/public/marketplace/index.json
index e1bd31249..2a3cccfe1 100644
--- a/apps/web/public/marketplace/index.json
+++ b/apps/web/public/marketplace/index.json
@@ -1,7 +1,7 @@
{
"$schema": "https://example.com/schemas/marketplace/v1.json",
"version": "1.0.0",
- "generated": "2026-09-23T14:14:28Z",
+ "generated": "2026-09-24T07:22:47Z",
"categories": [
{
"id": "playbooks",
@@ -222980,7 +222980,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "bec",
"bec",
"identity",
"conditional-access"
@@ -223004,7 +223003,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "bec",
"bec",
"email"
],
@@ -223029,7 +223027,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "bec",
"bec",
"email"
],
@@ -223054,7 +223051,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "bec",
"bec",
"identity",
"token-theft"
@@ -223080,7 +223076,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "bec",
"bec",
"finance"
],
@@ -223445,7 +223440,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ddos",
"ddos",
"dns"
],
@@ -223470,7 +223464,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ddos",
"ddos",
"waf"
],
@@ -223495,7 +223488,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ddos",
"ddos",
"ato"
],
@@ -223520,7 +223512,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ddos",
"ddos",
"network"
],
@@ -223545,7 +223536,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ddos",
"ddos",
"network"
],
@@ -224108,7 +224098,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ransomware",
"ransomware",
"c2"
],
@@ -224133,7 +224122,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ransomware",
"ransomware",
"fileserver"
],
@@ -224158,7 +224146,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ransomware",
"ransomware",
"endpoint"
],
@@ -224183,7 +224170,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ransomware",
"ransomware",
"exposure"
],
@@ -224208,7 +224194,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ransomware",
"ransomware",
"endpoint"
],
@@ -224233,7 +224218,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "supply-chain",
"supply-chain",
"github"
],
@@ -224258,7 +224242,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "supply-chain",
"supply-chain",
"iac",
"drift"
@@ -224282,7 +224265,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "supply-chain",
"supply-chain",
"npm"
],
@@ -224307,7 +224289,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "supply-chain",
"supply-chain",
"pypi"
],
@@ -224332,7 +224313,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "supply-chain",
"supply-chain",
"vendor"
],
diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.md
index b6173058b..bcff3ff22 100644
--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md
+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md
@@ -126,10 +126,13 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`
| No OSS default resolves to a specific deployment's hostname, so a self-hosted install never presents another deployment's host as its own | self-host contract; `apps/docs/docs/deployment/env-vars.md` (CORS default) | `hosted-hostname.yml` (`scripts/check_hosted_hostname.py` — an exact occurrence count per path, checked in **both** directions: a new or grown occurrence and a stale exemption whose occurrence is gone both fail, so an allow-list entry cannot outlive what justified it; the root is resolved from the working directory rather than `__file__` and verified against sentinels, a zero-match scan is reported as broken rather than clean, and `--self-test` runs first proving the detector separates known-bad from known-good and that the comparison rejects a vacuous pass; it also pins the six vendored `cors.py` copies byte-identical) + `tests/test_hosted_hostname_gate.py` (39 cases, including the ten runtime defaults that leaked and a check that the CORS-tuple extractor itself would notice the origin coming back) | GATED | - |
| A dry run never reaches a customer's vendor | live-action dry-run contract | `ci.yml` actions job (`test_dry_run_credential_strip.py` — each client factory's read set is re-derived from its own source and compared with the adapter's strip list, stripping that set is asserted to make the factory return `None`, and an end-to-end dry run with full credentials trips a wire if any factory returns a client) | GATED | the wire covers the four SIEM factories the writeback path uses; other vendors' adapters are covered by the strip-list equality check only |
| Documentation and marketing name no competitor product, while vendors AiSOC integrates with stay named | AGENTS.md (naming rule); docs portal + benchmark comparison tables | `competitor-names.yml` (`scripts/check_competitor_names.py` — an explicit name list plus a path allow-list for integration surfaces, both checked in both directions: a dead pattern, a stale allow entry that no longer excuses anything, and a drifted comparison-table marker each fail rather than reporting a clean tree; the scan prints the absolute root and file count so it cannot report on a tree it never read) + `tests/test_competitor_names_gate.py` (42 cases, including `Torq` asserted as a competitor in prose *and* left untouched across its connector module, manifest, docs page and tests) | GATED | the name list is AI-SOC/SOAR rivals; vendors that are purely integration targets are deliberately absent from it and are instead held out of the pinned comparison-table regions, which are checked against a wider vendor list |
+| The published playbook schema describes the engine that runs playbooks | concepts/playbooks.md (step table + `x-aisoc-execution`); `schemas/playbook.schema.json` | `ci.yml` P2 job (`scripts/check_playbook_schema_parity.py --self-test` — compares the schema enum, the `StepType` enum, the engine's `_HANDLERS` table, the `x-aisoc-execution` map, `bounds.py` and the pack validator's trigger list, **every pair in both directions**; refuses to run on a tree missing its marker files instead of printing OK about files it never opened, and names the root, schema, step counts and playbook count it inspected) + `scripts/lint_playbooks.py` (every `*.playbook.json` in the tree, not the two it used to scan while reporting "2/2 passed") + `tests/test_playbook_schema_parity_gate.py` (18 cases; drift injected in each direction and the gate asserted to name it) | GATED | twelve step types are declared vocabulary with no engine handler; the schema says so per type and the engine fails them closed, so the gap is published rather than hidden. Closes when playbook steps dispatch through the actions service's per-capability contract |
+| A step a playbook engine cannot run never reports that it ran | concepts/playbooks.md (error handling) | `ci.yml` agents job (`test_playbook_engine_correctness.py::TestUnimplementedStepTypesFailClosed` — an `approval` gate the engine cannot honour halts the run instead of passing it, every handler-less `StepType` is asserted to fail rather than succeed, a missing handler is not retried, an explicit `on_failure: continue` is still honoured, and a dry run reports `would_fail` rather than a bare `dry_run: true`) | GATED | - |
+| Every `ActionType` resolves a capability contract, so the confidence matrix is never skipped | concepts/live-actions.md (the one alias, and when it goes) | `ci.yml` actions job (`scripts/check_action_contract.py` directions 7 and 8 — every `ActionType` must resolve a contract by name or alias, and every alias must key a real `ActionType`, point at a real contract, not shadow a capability of the same name, and describe a bridge some adapter actually implements) + `test_approval_gate_wiring.py` (`notify_slack` at 10% confidence was approved for auto-execution and now requires an analyst; the alias is asserted unable to *lower* a requirement) | GATED | one alias entry (`notify_slack` -> `notify`), with its retirement condition recorded: it goes when `ActionType` does |
## Summary
-- GATED: 108
+- GATED: 112
- PARTIAL: 9
- NO GATE: 0 (**every claim is backed by a failing test.** The last NO GATE — the weekly benchmark scoreboard running live against `main` — closed in Phase E1: `scripts/check_scoreboard.py` ties the published scoreboard to a deterministic per-PR live-agent MITRE-accuracy run, while the funded weekly `wet-eval.yml` appends the LLM-tier rows. The remaining PARTIAL rows are honest, named deferrals — each states the specific gap and what closes it — not unproven claims.)
diff --git a/docs/upgrade/MIGRATION.md b/docs/upgrade/MIGRATION.md
index 113d79703..245501c8d 100644
--- a/docs/upgrade/MIGRATION.md
+++ b/docs/upgrade/MIGRATION.md
@@ -28,7 +28,7 @@ Read it before running `docker compose pull` or deploying to production.
| Area | v3 | v4 |
|------|----|----|
| Core engine | Rule-based SOAR engine | LangGraph multi-agent investigator |
-| Playbook format | Custom JSON (no schema) | JSON Schema 2020-12 (`playbook.schema.json`) |
+| Playbook format | Custom JSON (no schema) | JSON Schema draft-07 (`schemas/playbook.schema.json`) |
| Plugin manifest | `aisoc-plugin.json` only | `plugin.yaml` (preferred) + `aisoc-plugin.json` (legacy) |
| Plugin types | `enricher`, `action`, `connector` | + `responder`, `detection`, `widget` |
| Plugin distribution | Local directory only | Local directory + OCI images (via `oras`) |
@@ -171,7 +171,7 @@ config_schema:
### Step 5 — Migrate playbooks
-Playbooks are now validated against `playbook.schema.json` (JSON Schema 2020-12).
+Playbooks are now validated against `schemas/playbook.schema.json` (JSON Schema draft-07).
**Check your existing playbooks:**
diff --git a/marketplace/index.json b/marketplace/index.json
index e1bd31249..2a3cccfe1 100644
--- a/marketplace/index.json
+++ b/marketplace/index.json
@@ -1,7 +1,7 @@
{
"$schema": "https://example.com/schemas/marketplace/v1.json",
"version": "1.0.0",
- "generated": "2026-09-23T14:14:28Z",
+ "generated": "2026-09-24T07:22:47Z",
"categories": [
{
"id": "playbooks",
@@ -222980,7 +222980,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "bec",
"bec",
"identity",
"conditional-access"
@@ -223004,7 +223003,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "bec",
"bec",
"email"
],
@@ -223029,7 +223027,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "bec",
"bec",
"email"
],
@@ -223054,7 +223051,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "bec",
"bec",
"identity",
"token-theft"
@@ -223080,7 +223076,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "bec",
"bec",
"finance"
],
@@ -223445,7 +223440,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ddos",
"ddos",
"dns"
],
@@ -223470,7 +223464,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ddos",
"ddos",
"waf"
],
@@ -223495,7 +223488,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ddos",
"ddos",
"ato"
],
@@ -223520,7 +223512,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ddos",
"ddos",
"network"
],
@@ -223545,7 +223536,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ddos",
"ddos",
"network"
],
@@ -224108,7 +224098,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ransomware",
"ransomware",
"c2"
],
@@ -224133,7 +224122,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ransomware",
"ransomware",
"fileserver"
],
@@ -224158,7 +224146,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ransomware",
"ransomware",
"endpoint"
],
@@ -224183,7 +224170,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ransomware",
"ransomware",
"exposure"
],
@@ -224208,7 +224194,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "ransomware",
"ransomware",
"endpoint"
],
@@ -224233,7 +224218,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "supply-chain",
"supply-chain",
"github"
],
@@ -224258,7 +224242,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "supply-chain",
"supply-chain",
"iac",
"drift"
@@ -224282,7 +224265,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "supply-chain",
"supply-chain",
"npm"
],
@@ -224307,7 +224289,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "supply-chain",
"supply-chain",
"pypi"
],
@@ -224332,7 +224313,6 @@
"version": "1.0.0",
"author": "AiSOC",
"tags": [
- "supply-chain",
"supply-chain",
"vendor"
],
diff --git a/playbook.schema.json b/playbook.schema.json
deleted file mode 100644
index c396d19a5..000000000
--- a/playbook.schema.json
+++ /dev/null
@@ -1,249 +0,0 @@
-{
- "$schema": "https://json-schema.org/draft/2020-12/schema",
- "$id": "https://github.com/beenuar/aisoc/blob/main/playbook.schema.json",
- "title": "AiSOC Playbook",
- "description": "JSON Schema 2020-12 for AiSOC playbook definitions (Pillar 2, v4.0). Portable across file format (YAML / JSON) and CI-lintable.",
- "type": "object",
- "required": ["id", "name", "version", "trigger", "steps"],
- "additionalProperties": false,
- "properties": {
- "id": {
- "type": "string",
- "pattern": "^[a-z0-9][a-z0-9-]{2,62}$",
- "description": "Stable, globally-unique kebab-case identifier."
- },
- "name": { "type": "string", "minLength": 3, "maxLength": 120 },
- "description": { "type": "string", "maxLength": 1000 },
- "version": {
- "type": "string",
- "pattern": "^\\d+\\.\\d+\\.\\d+$",
- "description": "Semantic version string."
- },
- "author": { "type": "string" },
- "tags": {
- "type": "array",
- "items": { "type": "string", "pattern": "^[a-z0-9-]+$" },
- "uniqueItems": true
- },
- "enabled": { "type": "boolean", "default": true },
- "trigger": {
- "type": "object",
- "required": ["on"],
- "additionalProperties": false,
- "properties": {
- "on": {
- "type": "string",
- "enum": ["alert", "case", "schedule", "manual", "webhook"]
- },
- "severity": {
- "type": "array",
- "items": { "type": "string", "enum": ["info", "low", "medium", "high", "critical"] }
- },
- "alert_types": { "type": "array", "items": { "type": "string" } },
- "cron": {
- "type": "string",
- "description": "Cron expression; only valid when on=schedule."
- },
- "filters": {
- "type": "object",
- "description": "Key/value filters applied to the trigger event.",
- "additionalProperties": { "type": "string" }
- }
- }
- },
- "steps": {
- "type": "array",
- "minItems": 1,
- "items": { "$ref": "#/$defs/step" }
- },
- "on_failure": { "$ref": "#/$defs/failure_policy" },
- "timeout_seconds": {
- "type": "integer",
- "minimum": 1,
- "maximum": 86400,
- "default": 3600
- },
- "metadata": {
- "type": "object",
- "description": "Arbitrary key/value metadata; not processed by the engine.",
- "additionalProperties": true
- }
- },
-
- "$defs": {
- "step": {
- "type": "object",
- "required": ["id", "name", "type"],
- "additionalProperties": false,
- "properties": {
- "id": {
- "type": "string",
- "pattern": "^[a-z0-9][a-z0-9-]{1,62}$"
- },
- "name": { "type": "string", "minLength": 1, "maxLength": 120 },
- "type": {
- "type": "string",
- "enum": [
- "trigger",
- "condition",
- "action",
- "loop",
- "parallel",
- "human_approval",
- "wait",
- "notify",
- "enrich",
- "isolate",
- "block",
- "close_case",
- "create_case",
- "run_playbook",
- "script"
- ]
- },
- "description": { "type": "string" },
- "params": {
- "type": "object",
- "description": "Step-type-specific parameters; validated at runtime.",
- "additionalProperties": true
- },
- "condition": { "$ref": "#/$defs/condition_expr" },
- "on_failure": { "$ref": "#/$defs/failure_policy" },
- "retry": { "$ref": "#/$defs/retry_policy" },
- "timeout_seconds": { "type": "integer", "minimum": 1, "maximum": 3600 },
- "depends_on": {
- "type": "array",
- "items": { "type": "string" },
- "description": "Step IDs that must complete before this step starts (DAG edges)."
- },
- "blast_radius": {
- "type": "string",
- "enum": ["dry_run", "reversible", "destructive"],
- "default": "dry_run",
- "description": "Declared impact level; engine enforces analyst approval for destructive steps."
- },
- "output_key": {
- "type": "string",
- "description": "Variable name under which this step's output is stored in the run context."
- }
- }
- },
-
- "condition_expr": {
- "oneOf": [
- {
- "type": "object",
- "required": ["field", "op", "value"],
- "additionalProperties": false,
- "properties": {
- "field": { "type": "string" },
- "op": {
- "type": "string",
- "enum": ["eq", "ne", "lt", "lte", "gt", "gte", "contains", "startswith", "endswith", "in", "not_in", "exists", "not_exists"]
- },
- "value": {}
- }
- },
- {
- "type": "object",
- "required": ["and"],
- "additionalProperties": false,
- "properties": {
- "and": { "type": "array", "items": { "$ref": "#/$defs/condition_expr" }, "minItems": 2 }
- }
- },
- {
- "type": "object",
- "required": ["or"],
- "additionalProperties": false,
- "properties": {
- "or": { "type": "array", "items": { "$ref": "#/$defs/condition_expr" }, "minItems": 2 }
- }
- },
- {
- "type": "object",
- "required": ["not"],
- "additionalProperties": false,
- "properties": {
- "not": { "$ref": "#/$defs/condition_expr" }
- }
- }
- ]
- },
-
- "retry_policy": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "max_attempts": { "type": "integer", "minimum": 1, "maximum": 10, "default": 3 },
- "backoff_seconds": { "type": "number", "minimum": 0, "default": 2 },
- "backoff_multiplier": { "type": "number", "minimum": 1, "default": 2 },
- "retry_on": {
- "type": "array",
- "items": { "type": "string" },
- "description": "Exception class names that trigger a retry; empty = retry on any error."
- }
- }
- },
-
- "failure_policy": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "action": {
- "type": "string",
- "enum": ["abort", "continue", "fallback"],
- "default": "abort"
- },
- "fallback_step": {
- "type": "string",
- "description": "Step ID to jump to when action=fallback."
- },
- "notify": {
- "type": "object",
- "additionalProperties": false,
- "properties": {
- "channel": { "type": "string" },
- "message": { "type": "string" }
- }
- }
- }
- }
- },
-
- "examples": [
- {
- "id": "phishing-triage-v1",
- "name": "Phishing Email Triage",
- "description": "Enrich sender, notify SOC, auto-close low-risk.",
- "version": "1.0.0",
- "author": "AiSOC Core Team",
- "tags": ["phishing", "email", "triage"],
- "trigger": { "on": "alert", "severity": ["low", "medium"] },
- "steps": [
- {
- "id": "enrich-ip",
- "name": "Enrich sender IP",
- "type": "enrich",
- "params": { "ioc_type": "ip", "providers": ["virustotal", "greynoise"] },
- "blast_radius": "dry_run"
- },
- {
- "id": "notify-soc",
- "name": "Notify SOC channel",
- "type": "notify",
- "params": { "channel": "webhook", "template": "alert_summary" },
- "depends_on": ["enrich-ip"]
- },
- {
- "id": "close-if-clean",
- "name": "Auto-close clean alerts",
- "type": "close_case",
- "condition": { "field": "enrich-ip.malicious_score", "op": "lte", "value": 20 },
- "depends_on": ["notify-soc"],
- "blast_radius": "reversible"
- }
- ]
- }
- ]
-}
diff --git a/playbooks/packs/v1/bec/bec-conditional-access-bypass.playbook.json b/playbooks/packs/v1/bec/bec-conditional-access-bypass.playbook.json
index 6f337075f..f07c10572 100644
--- a/playbooks/packs/v1/bec/bec-conditional-access-bypass.playbook.json
+++ b/playbooks/packs/v1/bec/bec-conditional-access-bypass.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered when a BEC-related sign-in succeeds from a region that should be blocked by Conditional Access. Restores policy.",
"version": "1.0.0",
"tags": [
- "bec",
"bec",
"identity",
"conditional-access"
diff --git a/playbooks/packs/v1/bec/bec-impersonation-domain.playbook.json b/playbooks/packs/v1/bec/bec-impersonation-domain.playbook.json
index d4865246f..0a18b4c2f 100644
--- a/playbooks/packs/v1/bec/bec-impersonation-domain.playbook.json
+++ b/playbooks/packs/v1/bec/bec-impersonation-domain.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered when an inbound email originates from a domain that look-alikes an executive or vendor. Quarantines and blocks.",
"version": "1.0.0",
"tags": [
- "bec",
"bec",
"email",
"mitre.t1656"
diff --git a/playbooks/packs/v1/bec/bec-inbox-rule-malicious.playbook.json b/playbooks/packs/v1/bec/bec-inbox-rule-malicious.playbook.json
index 3e3774a0b..3559b8668 100644
--- a/playbooks/packs/v1/bec/bec-inbox-rule-malicious.playbook.json
+++ b/playbooks/packs/v1/bec/bec-inbox-rule-malicious.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered when a forwarding/filter rule is created that hides vendor or finance keywords. Removes the rule and resets creds.",
"version": "1.0.0",
"tags": [
- "bec",
"bec",
"email",
"mitre.t1564.008"
diff --git a/playbooks/packs/v1/bec/bec-token-theft.playbook.json b/playbooks/packs/v1/bec/bec-token-theft.playbook.json
index c774ea207..5f1d4c767 100644
--- a/playbooks/packs/v1/bec/bec-token-theft.playbook.json
+++ b/playbooks/packs/v1/bec/bec-token-theft.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered when a refresh token is observed from a new ASN with no interactive sign-in. Revokes refresh tokens and forces MFA.",
"version": "1.0.0",
"tags": [
- "bec",
"bec",
"identity",
"token-theft",
diff --git a/playbooks/packs/v1/bec/bec-vendor-payment-redirect.playbook.json b/playbooks/packs/v1/bec/bec-vendor-payment-redirect.playbook.json
index 912cbb424..a2fdbc26d 100644
--- a/playbooks/packs/v1/bec/bec-vendor-payment-redirect.playbook.json
+++ b/playbooks/packs/v1/bec/bec-vendor-payment-redirect.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered when a vendor banking detail change happens via email. Freezes pending payments and pages CFO.",
"version": "1.0.0",
"tags": [
- "bec",
"bec",
"finance",
"mitre.t1565.001"
diff --git a/playbooks/packs/v1/ddos/ddos-amplification-dns.playbook.json b/playbooks/packs/v1/ddos/ddos-amplification-dns.playbook.json
index 8112a786e..aedc32ca7 100644
--- a/playbooks/packs/v1/ddos/ddos-amplification-dns.playbook.json
+++ b/playbooks/packs/v1/ddos/ddos-amplification-dns.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered when an inbound spike of crafted DNS queries hits an authoritative server. Sinkholes spoofed sources.",
"version": "1.0.0",
"tags": [
- "ddos",
"ddos",
"dns",
"mitre.t1498.002"
diff --git a/playbooks/packs/v1/ddos/ddos-app-layer-l7.playbook.json b/playbooks/packs/v1/ddos/ddos-app-layer-l7.playbook.json
index a7612e1ad..9a58a6a2e 100644
--- a/playbooks/packs/v1/ddos/ddos-app-layer-l7.playbook.json
+++ b/playbooks/packs/v1/ddos/ddos-app-layer-l7.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered by anomalous L7 request rate or pattern. Engages WAF rate-limit rule and challenge-page.",
"version": "1.0.0",
"tags": [
- "ddos",
"ddos",
"waf",
"mitre.t1498.001"
diff --git a/playbooks/packs/v1/ddos/ddos-credential-stuffing-fraud.playbook.json b/playbooks/packs/v1/ddos/ddos-credential-stuffing-fraud.playbook.json
index 287c81131..3b417e578 100644
--- a/playbooks/packs/v1/ddos/ddos-credential-stuffing-fraud.playbook.json
+++ b/playbooks/packs/v1/ddos/ddos-credential-stuffing-fraud.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered when /login takes a sustained spike of failed attempts. Combines DDoS + credential-stuffing response: WAF challenge, rate-limit, source-IP block.",
"version": "1.0.0",
"tags": [
- "ddos",
"ddos",
"ato",
"mitre.t1110.004"
diff --git a/playbooks/packs/v1/ddos/ddos-syn-flood.playbook.json b/playbooks/packs/v1/ddos/ddos-syn-flood.playbook.json
index 5e1d517ca..fc0ff3843 100644
--- a/playbooks/packs/v1/ddos/ddos-syn-flood.playbook.json
+++ b/playbooks/packs/v1/ddos/ddos-syn-flood.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered by SYN flood pattern at edge. Enables SYN cookies and increases backend capacity.",
"version": "1.0.0",
"tags": [
- "ddos",
"ddos",
"network",
"mitre.t1498"
diff --git a/playbooks/packs/v1/ddos/ddos-volumetric-l3.playbook.json b/playbooks/packs/v1/ddos/ddos-volumetric-l3.playbook.json
index 2995bef38..050caa724 100644
--- a/playbooks/packs/v1/ddos/ddos-volumetric-l3.playbook.json
+++ b/playbooks/packs/v1/ddos/ddos-volumetric-l3.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered by volumetric flood detection at edge. Engages scrubbing provider and pages on-call.",
"version": "1.0.0",
"tags": [
- "ddos",
"ddos",
"network",
"mitre.t1498"
diff --git a/playbooks/packs/v1/ransomware/ransomware-c2-block.playbook.json b/playbooks/packs/v1/ransomware/ransomware-c2-block.playbook.json
index 9a3881391..d508802e9 100644
--- a/playbooks/packs/v1/ransomware/ransomware-c2-block.playbook.json
+++ b/playbooks/packs/v1/ransomware/ransomware-c2-block.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered by C2 beaconing detections that match ransomware-family TTPs. Blocks C2 destinations at edge, isolates source host.",
"version": "1.0.0",
"tags": [
- "ransomware",
"ransomware",
"c2",
"mitre.t1071"
diff --git a/playbooks/packs/v1/ransomware/ransomware-fileserver-encrypt.playbook.json b/playbooks/packs/v1/ransomware/ransomware-fileserver-encrypt.playbook.json
index 7f7b4657c..ecf5079a5 100644
--- a/playbooks/packs/v1/ransomware/ransomware-fileserver-encrypt.playbook.json
+++ b/playbooks/packs/v1/ransomware/ransomware-fileserver-encrypt.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered by mass-rename / mass-extension-change on a file server. Network-segments the host to halt SMB lateral encryption.",
"version": "1.0.0",
"tags": [
- "ransomware",
"ransomware",
"fileserver",
"mitre.t1486"
diff --git a/playbooks/packs/v1/ransomware/ransomware-host-isolate.playbook.json b/playbooks/packs/v1/ransomware/ransomware-host-isolate.playbook.json
index eb9414a7c..4e0b01704 100644
--- a/playbooks/packs/v1/ransomware/ransomware-host-isolate.playbook.json
+++ b/playbooks/packs/v1/ransomware/ransomware-host-isolate.playbook.json
@@ -4,7 +4,6 @@
"description": "Immediate host isolation, disk snapshot for forensics, page on-call. Standard first-response for endpoint ransomware.",
"version": "1.0.0",
"tags": [
- "ransomware",
"ransomware",
"endpoint",
"mitre.t1486"
diff --git a/playbooks/packs/v1/ransomware/ransomware-public-exposure.playbook.json b/playbooks/packs/v1/ransomware/ransomware-public-exposure.playbook.json
index 763c55414..6bb6c9b56 100644
--- a/playbooks/packs/v1/ransomware/ransomware-public-exposure.playbook.json
+++ b/playbooks/packs/v1/ransomware/ransomware-public-exposure.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered when a host with public exposure shows ransomware indicators. Pulls the asset off the internet immediately.",
"version": "1.0.0",
"tags": [
- "ransomware",
"ransomware",
"exposure",
"mitre.t1486"
diff --git a/playbooks/packs/v1/ransomware/ransomware-shadow-copy-deletion.playbook.json b/playbooks/packs/v1/ransomware/ransomware-shadow-copy-deletion.playbook.json
index 83c3aba3b..a1a18d8b9 100644
--- a/playbooks/packs/v1/ransomware/ransomware-shadow-copy-deletion.playbook.json
+++ b/playbooks/packs/v1/ransomware/ransomware-shadow-copy-deletion.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered on `vssadmin delete shadows` or wmic shadowcopy delete. Pre-encryption signal \u2014 isolate fast and verify backups.",
"version": "1.0.0",
"tags": [
- "ransomware",
"ransomware",
"endpoint",
"mitre.t1490"
diff --git a/playbooks/packs/v1/supply-chain/supply-github-action-compromise.playbook.json b/playbooks/packs/v1/supply-chain/supply-github-action-compromise.playbook.json
index 3f22beb22..a6e36c86c 100644
--- a/playbooks/packs/v1/supply-chain/supply-github-action-compromise.playbook.json
+++ b/playbooks/packs/v1/supply-chain/supply-github-action-compromise.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered when a referenced GitHub Action SHA changes unexpectedly or matches a compromise advisory. Pin to last-known-good and audit.",
"version": "1.0.0",
"tags": [
- "supply-chain",
"supply-chain",
"github",
"mitre.t1195.002"
diff --git a/playbooks/packs/v1/supply-chain/supply-iac-drift.playbook.json b/playbooks/packs/v1/supply-chain/supply-iac-drift.playbook.json
index 8c878a129..2f82707d8 100644
--- a/playbooks/packs/v1/supply-chain/supply-iac-drift.playbook.json
+++ b/playbooks/packs/v1/supply-chain/supply-iac-drift.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered when production infra deviates from IaC source of truth in a security-relevant way. Reverts to declared state.",
"version": "1.0.0",
"tags": [
- "supply-chain",
"supply-chain",
"iac",
"drift"
diff --git a/playbooks/packs/v1/supply-chain/supply-malicious-npm.playbook.json b/playbooks/packs/v1/supply-chain/supply-malicious-npm.playbook.json
index ed7487f77..e00d494e2 100644
--- a/playbooks/packs/v1/supply-chain/supply-malicious-npm.playbook.json
+++ b/playbooks/packs/v1/supply-chain/supply-malicious-npm.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered when a published npm package matches a known-bad list. Blocks installs at registry-mirror, audits org installs.",
"version": "1.0.0",
"tags": [
- "supply-chain",
"supply-chain",
"npm",
"mitre.t1195"
diff --git a/playbooks/packs/v1/supply-chain/supply-pypi-typosquat.playbook.json b/playbooks/packs/v1/supply-chain/supply-pypi-typosquat.playbook.json
index 6eb0bc12e..745c07b30 100644
--- a/playbooks/packs/v1/supply-chain/supply-pypi-typosquat.playbook.json
+++ b/playbooks/packs/v1/supply-chain/supply-pypi-typosquat.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered when a typosquat package is observed in CI installs. Pulls install lock to canonical name and audits.",
"version": "1.0.0",
"tags": [
- "supply-chain",
"supply-chain",
"pypi",
"mitre.t1195"
diff --git a/playbooks/packs/v1/supply-chain/supply-vendor-breach.playbook.json b/playbooks/packs/v1/supply-chain/supply-vendor-breach.playbook.json
index 5f6793c06..ce4c34760 100644
--- a/playbooks/packs/v1/supply-chain/supply-vendor-breach.playbook.json
+++ b/playbooks/packs/v1/supply-chain/supply-vendor-breach.playbook.json
@@ -4,7 +4,6 @@
"description": "Triggered when a vendor publishes a breach advisory matching internal vendor inventory. Blocks vendor IPs, opens procurement review.",
"version": "1.0.0",
"tags": [
- "supply-chain",
"supply-chain",
"vendor",
"mitre.t1199"
diff --git a/schemas/playbook.schema.json b/schemas/playbook.schema.json
index 8ccfed06e..18e19c3fe 100644
--- a/schemas/playbook.schema.json
+++ b/schemas/playbook.schema.json
@@ -2,10 +2,37 @@
"$schema": "http://json-schema.org/draft-07/schema#",
"$id": "https://github.com/beenuar/AiSOC/schemas/playbook.schema.json",
"title": "AiSOC Playbook",
- "description": "Schema for AiSOC v4 Pillar-2 playbook definitions.",
+ "description": "The authoring contract for AiSOC playbooks. Every step type listed in `properties.steps.items.properties.type.enum` is accepted by the runtime model (`services/agents/app/playbook/models.py::StepType`), and `x-aisoc-execution` records what the engine actually does with it. Kept in lockstep with the engine by `scripts/check_playbook_schema_parity.py`.",
"type": "object",
"required": ["name", "steps"],
"additionalProperties": false,
+
+ "x-aisoc-execution": {
+ "description": "What `services/agents/app/playbook/engine.py` does with each step type. `executed` = a handler runs and has a real effect (an outbound call to a service, or engine-internal control flow). `simulated` = a handler runs and returns `simulated: true` without touching a vendor. `unimplemented` = no handler is registered; the engine fails the step closed rather than reporting a success it did not achieve. This map is machine-checked against the engine's handler table, so it cannot drift into an aspirational claim.",
+ "enrich": "executed",
+ "investigate": "executed",
+ "notify": "executed",
+ "http": "executed",
+ "close_case": "executed",
+ "condition": "executed",
+ "osquery_live_query": "executed",
+ "block_ip": "simulated",
+ "isolate_host": "simulated",
+ "create_ticket": "simulated",
+ "approval": "unimplemented",
+ "block_ioc": "unimplemented",
+ "disable_user": "unimplemented",
+ "reset_password": "unimplemented",
+ "revoke_session": "unimplemented",
+ "force_mfa": "unimplemented",
+ "kill_process": "unimplemented",
+ "quarantine_file": "unimplemented",
+ "run_av_scan": "unimplemented",
+ "run_script": "unimplemented",
+ "search_siem": "unimplemented",
+ "create_notable_event": "unimplemented"
+ },
+
"properties": {
"id": {
"type": "string",
@@ -79,12 +106,33 @@
"updated_at": {
"type": "string",
"format": "date-time"
+ },
+ "inputs": {
+ "type": "object",
+ "description": "Authored documentation of the trigger-context fields this playbook expects. The engine does NOT bind, inject or validate these — a step reads its own values from `params` and from the run context. Present so a pack can describe its interface to a reader; do not treat it as a parameter-binding mechanism.",
+ "additionalProperties": {
+ "type": "object",
+ "properties": {
+ "description": { "type": "string" },
+ "field": {
+ "type": "string",
+ "description": "Dot-path on the trigger event where this input is expected to be found."
+ },
+ "required": { "type": "boolean" }
+ },
+ "additionalProperties": true
+ }
+ },
+ "dry_run_support": {
+ "type": "boolean",
+ "description": "Authored assertion that this playbook is safe to run with the engine's `dry_run` flag. The engine's `dry_run` applies uniformly to every step regardless of this value, so it is catalogue metadata rather than a runtime switch."
}
},
+
"definitions": {
"StepCondition": {
"type": "object",
- "required": ["field", "operator"],
+ "description": "Structured condition form. Every field has a runtime default, so none is required; an empty object is a condition that always passes.",
"additionalProperties": false,
"properties": {
"field": {
@@ -97,6 +145,10 @@
},
"value": {
"description": "Expected value (not required for 'exists' operator)."
+ },
+ "expression": {
+ "type": "string",
+ "description": "Expression form, e.g. \"inputs.source_ip != null\". Mutually exclusive with field/operator/value; evaluated at runtime by the engine."
}
}
},
@@ -115,16 +167,30 @@
},
"type": {
"type": "string",
+ "description": "The step verb. See `x-aisoc-execution` at the root of this schema for what the engine does with each one — a type listed there as `unimplemented` is a valid authoring vocabulary word that the engine will fail closed on rather than execute.",
"enum": [
"enrich",
"investigate",
"notify",
+ "http",
+ "close_case",
+ "condition",
+ "osquery_live_query",
"block_ip",
"isolate_host",
"create_ticket",
- "close_case",
- "http",
- "condition"
+ "approval",
+ "block_ioc",
+ "disable_user",
+ "reset_password",
+ "revoke_session",
+ "force_mfa",
+ "kill_process",
+ "quarantine_file",
+ "run_av_scan",
+ "run_script",
+ "search_siem",
+ "create_notable_event"
]
},
"params": {
@@ -133,7 +199,14 @@
"additionalProperties": true
},
"condition": {
- "$ref": "#/definitions/StepCondition"
+ "description": "Guard that must pass before the step runs. Accepts either an expression string or the structured form; the engine coerces a bare string into `{\"expression\": ...}`.",
+ "oneOf": [
+ {
+ "type": "string",
+ "description": "Expression form, e.g. \"steps.approval_isolate.outcome == 'approved'\"."
+ },
+ { "$ref": "#/definitions/StepCondition" }
+ ]
},
"on_failure": {
"type": "string",
@@ -143,14 +216,16 @@
"retry_max": {
"type": "integer",
"minimum": 0,
- "maximum": 5,
- "default": 0
+ "maximum": 25,
+ "default": 0,
+ "description": "Retry attempts. Ceiling mirrors `bounds.ABSOLUTE_MAX_RETRIES`."
},
"timeout_seconds": {
"type": "integer",
"minimum": 1,
- "maximum": 600,
- "default": 30
+ "maximum": 3600,
+ "default": 30,
+ "description": "Per-step timeout. Ceiling mirrors `bounds.ABSOLUTE_MAX_TIMEOUT_SECONDS`; long values exist for `approval` steps that wait on a human. Note that handlers reading `params.timeout_seconds` are separately clamped to `bounds.ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS`."
},
"next_true": {
"type": "string",
diff --git a/scripts/check_action_contract.py b/scripts/check_action_contract.py
index 92715e30a..11e31a243 100644
--- a/scripts/check_action_contract.py
+++ b/scripts/check_action_contract.py
@@ -466,6 +466,70 @@ def check_capability_reachability() -> list[str]:
f"deployment rather than a verb that was never built."
)
+ # ── Direction 7: ActionType → capability contract ──────────────────────
+ # An ActionType with no reachable contract is not a cosmetic gap. The
+ # submit path looks the contract up *by ActionType value* to run the
+ # confidence matrix; a miss means the matrix is skipped and blast radius
+ # decides alone, so a 40%-confidence guess is graded like a corroborated
+ # finding. That is what happened to `notify_slack` for as long as the
+ # capability was called `notify` and nothing connected the two.
+ from app.live_actions.capability_contracts import (
+ ACTION_TYPE_CAPABILITY_ALIASES,
+ CAPABILITY_CONTRACTS,
+ contract_for_action_type,
+ )
+
+ for action_type in sorted(ActionType, key=lambda a: a.value):
+ if contract_for_action_type(action_type.value) is None:
+ errors.append(
+ f"{action_type.value}: no capability contract resolves for this "
+ f"ActionType. The approval matrix has no impact to reason about, "
+ f"so it is skipped and blast radius decides alone. Give the verb a "
+ f"contract, or — if this is only a naming difference for a verb "
+ f"that already has one — add it to ACTION_TYPE_CAPABILITY_ALIASES."
+ )
+
+ # ── Direction 8: alias map → the rest of the world, both ways ──────────
+ # An alias is a claim that two names mean one verb. Three ways it can rot,
+ # each checked here, because an alias nothing validates is just a lookup
+ # that happens to succeed.
+ known_action_type_values = {a.value for a in ActionType}
+ adapter_bridges = {
+ (getattr(cls, "capability", ""), getattr(cls, "_legacy_action_type", None))
+ for cls in adapters
+ if getattr(cls, "_legacy_action_type", None) is not None
+ }
+ for legacy_value, capability in sorted(ACTION_TYPE_CAPABILITY_ALIASES.items()):
+ if legacy_value not in known_action_type_values:
+ errors.append(
+ f"ACTION_TYPE_CAPABILITY_ALIASES maps {legacy_value!r}, which is not an "
+ f"ActionType. The alias can never fire; delete it or fix the key."
+ )
+ if capability not in CAPABILITY_CONTRACTS:
+ errors.append(
+ f"ACTION_TYPE_CAPABILITY_ALIASES points {legacy_value!r} at capability "
+ f"{capability!r}, which has no contract — the alias resolves to nothing."
+ )
+ # The alias must describe a bridge that actually exists. If no adapter
+ # pairs this capability with this ActionType, the two names are not
+ # two names for one verb and the alias is asserting something false.
+ if not any(cap == capability and lat is not None and lat.value == legacy_value for cap, lat in adapter_bridges):
+ errors.append(
+ f"ACTION_TYPE_CAPABILITY_ALIASES claims {legacy_value!r} and {capability!r} "
+ f"are the same verb, but no adapter pairs them (no executor with "
+ f"capability={capability!r} and _legacy_action_type={legacy_value!r}). "
+ f"An alias asserting a bridge nobody built is worse than no alias."
+ )
+
+ # And the other way: an alias must not shadow a real capability of the
+ # same name, which would silently redirect a verb that has its own contract.
+ for legacy_value in sorted(ACTION_TYPE_CAPABILITY_ALIASES):
+ if legacy_value in CAPABILITY_CONTRACTS:
+ errors.append(
+ f"{legacy_value!r} has its own capability contract AND an alias entry. "
+ f"The alias is dead code at best and a silent redirect at worst; remove it."
+ )
+
errors.extend(_ratchet("KNOWN_UNGOVERNED_EXECUTORS", KNOWN_UNGOVERNED_EXECUTORS, {a.value for a in adapter_action_types}))
errors.extend(
_ratchet(
diff --git a/scripts/check_playbook_schema_parity.py b/scripts/check_playbook_schema_parity.py
new file mode 100644
index 000000000..b08dbdd42
--- /dev/null
+++ b/scripts/check_playbook_schema_parity.py
@@ -0,0 +1,515 @@
+#!/usr/bin/env python3
+"""Gate: the published playbook schema and the engine that runs playbooks
+must describe the same product.
+
+``schemas/playbook.schema.json`` is the contract authors are told to trust.
+When it drifts from ``services/agents/app/playbook/``, a playbook validates
+and then fails at runtime — the worst shape for a schema to have, because the
+failure lands on the person who did exactly what the documentation said.
+
+Before this gate the two had drifted in every available direction at once:
+
+ * two schema files existed with different step vocabularies (15 types and
+ 9 types) against an engine with 22, and the drafter silently fell back
+ from one to the other;
+ * 11 step types were declared by a schema and implemented nowhere;
+ * 13 step types were accepted by the engine and declared by no schema;
+ * 12 of those had no handler and the engine reported them ``SUCCESS``,
+ including ``approval`` — a human gate that auto-passed;
+ * 32 of 64 shipped playbooks failed the published schema, while the lint
+ job printed ``2/2 passed`` because it only ever looked at two files.
+
+Every comparison here runs in **both** directions. A check that only asks
+"does the schema declare something the engine lacks" passes forever while
+drift accumulates in the direction things actually change — the engine grows
+a verb and nobody tells the schema. That is how a sibling gate in this repo
+reported "OK" on a YAML file declaring 17 labels against Go code with 28.
+
+Run:
+ python3 scripts/check_playbook_schema_parity.py
+ python3 scripts/check_playbook_schema_parity.py --self-test
+"""
+
+from __future__ import annotations
+
+import argparse
+import inspect
+import json
+import sys
+from dataclasses import dataclass, field, replace
+from pathlib import Path
+
+#: Playbook-level keys the schema declares that the runtime model does not
+#: bind. Both are authored documentation rather than engine inputs, and the
+#: schema says so in their ``description``. This list is a named exemption,
+#: not a silent pass: anything else appearing in one and not the other is an
+#: error. Keep it empty-by-default in spirit — an entry here is a promise
+#: that the key is inert, and the schema text must say the same.
+INERT_AUTHORED_KEYS: frozenset[str] = frozenset({"inputs", "dry_run_support"})
+
+#: Execution classes usable in the schema's ``x-aisoc-execution`` map.
+_RUNS = frozenset({"executed", "simulated"})
+_EXECUTION_CLASSES = _RUNS | {"unimplemented"}
+
+
+class GateError(RuntimeError):
+ """The gate could not inspect what it is supposed to inspect.
+
+ Raised instead of returning "no problems found", because a gate that
+ cannot read its inputs and prints OK is worse than no gate at all.
+ """
+
+
+@dataclass(frozen=True)
+class Registries:
+ """Every declaration of the playbook vocabulary, read from its source."""
+
+ schema_step_types: frozenset[str]
+ model_step_types: frozenset[str]
+ handler_step_types: frozenset[str]
+ inline_step_types: frozenset[str]
+ execution: dict[str, str]
+ schema_timeout_max: int
+ schema_timeout_min: int
+ schema_retry_max: int
+ engine_timeout_max: int
+ engine_timeout_min: int
+ engine_retry_max: int
+ schema_triggers: frozenset[str]
+ validator_triggers: frozenset[str]
+ schema_playbook_keys: frozenset[str]
+ model_playbook_keys: frozenset[str]
+ schema_files: tuple[str, ...] = ()
+ scanned_playbooks: tuple[str, ...] = field(default=())
+
+ @property
+ def runnable(self) -> frozenset[str]:
+ """Step types the engine can actually act on."""
+ return self.handler_step_types | self.inline_step_types
+
+
+# ---------------------------------------------------------------------------
+# Collection — read each registry from the file that owns it
+# ---------------------------------------------------------------------------
+
+
+def _find_repo_root(explicit: str | None) -> Path:
+ """Resolve the tree to inspect, and prove it is the right one.
+
+ Resolving from ``__file__`` alone is how a gate ends up confidently
+ reporting OK about a tree it never opened: copy the script somewhere, or
+ run it from an installed package, and it grades whatever happens to sit
+ two directories up. Every marker below must exist or we refuse to run.
+ """
+ root = Path(explicit).resolve() if explicit else Path(__file__).resolve().parent.parent
+ markers = (
+ Path("schemas/playbook.schema.json"),
+ Path("services/agents/app/playbook/engine.py"),
+ Path("services/agents/app/playbook/models.py"),
+ Path("scripts/validate_playbooks.py"),
+ )
+ missing = [str(m) for m in markers if not (root / m).exists()]
+ if missing:
+ raise GateError(
+ f"{root} does not look like the AiSOC repository — missing: "
+ f"{', '.join(missing)}. Pass --repo-root to point at the tree to check."
+ )
+ return root
+
+
+def _load_schema(root: Path) -> dict:
+ return json.loads((root / "schemas" / "playbook.schema.json").read_text())
+
+
+def _schema_step_types(schema: dict) -> frozenset[str]:
+ try:
+ enum = schema["definitions"]["PlaybookStep"]["properties"]["type"]["enum"]
+ except (KeyError, TypeError) as exc:
+ raise GateError(f"schema step-type enum not found at the expected path: {exc}") from exc
+ if not enum:
+ raise GateError("schema step-type enum is empty; refusing to compare against nothing")
+ return frozenset(enum)
+
+
+def _schema_execution(schema: dict) -> dict[str, str]:
+ raw = schema.get("x-aisoc-execution")
+ if not isinstance(raw, dict):
+ raise GateError("schema has no `x-aisoc-execution` map; the engine's behaviour would be undeclared")
+ mapping = {k: v for k, v in raw.items() if k != "description"}
+ if not mapping:
+ raise GateError("`x-aisoc-execution` is empty; refusing to compare against nothing")
+ return mapping
+
+
+def _schema_step_bounds(schema: dict) -> tuple[int, int, int]:
+ try:
+ props = schema["definitions"]["PlaybookStep"]["properties"]
+ return (
+ int(props["timeout_seconds"]["maximum"]),
+ int(props["timeout_seconds"]["minimum"]),
+ int(props["retry_max"]["maximum"]),
+ )
+ except (KeyError, TypeError, ValueError) as exc:
+ raise GateError(f"schema step bounds not found at the expected path: {exc}") from exc
+
+
+def _import_engine(root: Path):
+ """Import the real engine module rather than parsing it.
+
+ Parsing would let a handler table that is built at import time disagree
+ with what the parser sees. The import is cheap: the engine documents
+ "zero external dependencies beyond httpx + stdlib".
+ """
+ agents = str(root / "services" / "agents")
+ if agents not in sys.path:
+ sys.path.insert(0, agents)
+ try:
+ from app.playbook import bounds, engine, models # type: ignore[import-not-found]
+ except Exception as exc: # noqa: BLE001 — surfaced, never swallowed
+ raise GateError(
+ f"could not import the playbook engine from {agents}: {type(exc).__name__}: {exc}. "
+ "Install httpx and pydantic, or the gate is grading nothing."
+ ) from exc
+ return engine, models, bounds
+
+
+def _inline_step_types(engine_mod, models_mod) -> frozenset[str]:
+ """Step types handled inside the run loop rather than via ``_HANDLERS``.
+
+ ``condition`` never reaches the handler table — it branches in the loop
+ body. Reading the source keeps that visible to the gate; if someone
+ deletes the branch, the type stops counting as runnable here too.
+ """
+ try:
+ source = inspect.getsource(engine_mod.PlaybookEngine.run)
+ except (OSError, TypeError, AttributeError) as exc:
+ raise GateError(f"could not read PlaybookEngine.run source to find inline step handling: {exc}") from exc
+ found = {st.value for st in models_mod.StepType if f"StepType.{st.name}" in source}
+ return frozenset(found)
+
+
+def _model_playbook_keys(models_mod) -> frozenset[str]:
+ return frozenset(models_mod.Playbook.model_fields.keys())
+
+
+def _validator_triggers(root: Path) -> frozenset[str]:
+ """``scripts/validate_playbooks.py`` keeps its own trigger allow-list."""
+ scripts = str(root / "scripts")
+ if scripts not in sys.path:
+ sys.path.insert(0, scripts)
+ try:
+ import validate_playbooks # type: ignore[import-not-found]
+ except Exception as exc: # noqa: BLE001
+ raise GateError(f"could not import scripts/validate_playbooks.py: {type(exc).__name__}: {exc}") from exc
+ triggers = getattr(validate_playbooks, "SUPPORTED_TRIGGERS", None)
+ if not triggers:
+ raise GateError("validate_playbooks.SUPPORTED_TRIGGERS is missing or empty")
+ return frozenset(triggers)
+
+
+def _schema_files(root: Path) -> tuple[str, ...]:
+ hits = [
+ p
+ for p in root.rglob("playbook.schema.json")
+ if "node_modules" not in p.parts and ".git" not in p.parts and ".venv-gate" not in p.parts
+ ]
+ return tuple(sorted(str(p.relative_to(root)) for p in hits))
+
+
+def _playbook_files(root: Path) -> list[Path]:
+ return sorted(p for p in root.rglob("*.playbook.json") if "node_modules" not in p.parts and ".git" not in p.parts)
+
+
+def collect(root: Path) -> Registries:
+ schema = _load_schema(root)
+ engine_mod, models_mod, bounds_mod = _import_engine(root)
+ s_tmax, s_tmin, s_rmax = _schema_step_bounds(schema)
+ return Registries(
+ schema_step_types=_schema_step_types(schema),
+ model_step_types=frozenset(st.value for st in models_mod.StepType),
+ handler_step_types=frozenset(st.value for st in engine_mod._HANDLERS),
+ inline_step_types=_inline_step_types(engine_mod, models_mod),
+ execution=_schema_execution(schema),
+ schema_timeout_max=s_tmax,
+ schema_timeout_min=s_tmin,
+ schema_retry_max=s_rmax,
+ engine_timeout_max=bounds_mod.ABSOLUTE_MAX_TIMEOUT_SECONDS,
+ engine_timeout_min=bounds_mod.MIN_TIMEOUT_SECONDS,
+ engine_retry_max=bounds_mod.ABSOLUTE_MAX_RETRIES,
+ schema_triggers=frozenset(schema["properties"]["trigger"]["properties"]["on"]["enum"]),
+ validator_triggers=_validator_triggers(root),
+ schema_playbook_keys=frozenset(schema["properties"].keys()),
+ model_playbook_keys=_model_playbook_keys(models_mod),
+ schema_files=_schema_files(root),
+ )
+
+
+# ---------------------------------------------------------------------------
+# Comparison — pure, so the self-test can perturb it
+# ---------------------------------------------------------------------------
+
+
+def compare(reg: Registries) -> list[str]:
+ """Every check, in both directions. Returns human-readable failures."""
+ errors: list[str] = []
+
+ if not reg.schema_step_types or not reg.model_step_types:
+ # Defence against a future refactor making the whole gate vacuous.
+ return ["one of the step-type registries is empty; the comparison would pass on nothing"]
+
+ # 1. schema enum <-> StepType, both ways.
+ for missing in sorted(reg.schema_step_types - reg.model_step_types):
+ errors.append(
+ f"schema declares step type {missing!r} that StepType does not implement — "
+ f"a playbook using it validates and then fails to parse at runtime. "
+ f"Build it or remove it from the schema."
+ )
+ for missing in sorted(reg.model_step_types - reg.schema_step_types):
+ errors.append(
+ f"StepType implements {missing!r} and the schema does not declare it — "
+ f"an author following the published contract cannot use it, and the "
+ f"drafter has to collapse it onto some other verb. Declare it or delete it."
+ )
+
+ # 2. execution map <-> schema enum, both ways.
+ exec_keys = frozenset(reg.execution)
+ for missing in sorted(reg.schema_step_types - exec_keys):
+ errors.append(f"step type {missing!r} is declared but `x-aisoc-execution` does not say what the engine does with it")
+ for extra in sorted(exec_keys - reg.schema_step_types):
+ errors.append(f"`x-aisoc-execution` describes {extra!r}, which the schema does not declare as a step type")
+
+ bad = {k: v for k, v in reg.execution.items() if v not in _EXECUTION_CLASSES}
+ for k, v in sorted(bad.items()):
+ errors.append(f"`x-aisoc-execution[{k}]` is {v!r}; expected one of {sorted(_EXECUTION_CLASSES)}")
+
+ # 3. execution claims <-> the engine's real handler table, both ways.
+ claims_to_run = frozenset(k for k, v in reg.execution.items() if v in _RUNS)
+ for k in sorted(claims_to_run - reg.runnable):
+ errors.append(
+ f"`x-aisoc-execution` claims {k!r} runs, but the engine has no handler for it — "
+ f"this is the claim the schema exists to make true."
+ )
+ for k in sorted(reg.runnable - claims_to_run):
+ errors.append(
+ f"the engine has a handler for {k!r} but `x-aisoc-execution` marks it "
+ f"{reg.execution.get(k, '(absent)')!r} — the schema is understating what the product does."
+ )
+
+ # 4. bounds, both ways (a schema ceiling below the engine's rejects valid
+ # playbooks; above it promises headroom Pydantic will refuse).
+ if reg.schema_timeout_max != reg.engine_timeout_max:
+ errors.append(
+ f"step timeout ceiling disagrees: schema {reg.schema_timeout_max} vs "
+ f"bounds.ABSOLUTE_MAX_TIMEOUT_SECONDS {reg.engine_timeout_max}"
+ )
+ if reg.schema_timeout_min != reg.engine_timeout_min:
+ errors.append(
+ f"step timeout floor disagrees: schema {reg.schema_timeout_min} vs bounds.MIN_TIMEOUT_SECONDS {reg.engine_timeout_min}"
+ )
+ if reg.schema_retry_max != reg.engine_retry_max:
+ errors.append(f"retry ceiling disagrees: schema {reg.schema_retry_max} vs bounds.ABSOLUTE_MAX_RETRIES {reg.engine_retry_max}")
+
+ # 5. trigger vocabulary <-> the pack validator's allow-list, both ways.
+ for missing in sorted(reg.schema_triggers - reg.validator_triggers):
+ errors.append(f"schema allows trigger.on={missing!r} that validate_playbooks.py rejects")
+ for missing in sorted(reg.validator_triggers - reg.schema_triggers):
+ errors.append(f"validate_playbooks.py allows trigger.on={missing!r} that the schema rejects")
+
+ # 6. playbook-level keys <-> Playbook model fields, both ways.
+ for missing in sorted(reg.schema_playbook_keys - reg.model_playbook_keys - INERT_AUTHORED_KEYS):
+ errors.append(
+ f"schema declares playbook key {missing!r} that the Playbook model drops on load — "
+ f"add the field, remove the key, or list it in INERT_AUTHORED_KEYS and say so in its description"
+ )
+ for missing in sorted(reg.model_playbook_keys - reg.schema_playbook_keys):
+ errors.append(f"Playbook model has field {missing!r} that the schema does not declare, so `additionalProperties: false` rejects it")
+
+ # 7. exactly one schema file. Two is how the vocabularies diverged.
+ if len(reg.schema_files) > 1:
+ errors.append(
+ f"more than one playbook.schema.json in the tree ({', '.join(reg.schema_files)}) — "
+ f"a second copy is a second source of truth and the drafter silently fell back to it once already"
+ )
+
+ return errors
+
+
+# ---------------------------------------------------------------------------
+# Content check — the contract must be true of what ships
+# ---------------------------------------------------------------------------
+
+
+def validate_shipped_playbooks(root: Path) -> tuple[list[str], list[Path]]:
+ try:
+ import jsonschema
+ except ImportError as exc:
+ raise GateError("jsonschema is not installed; the content half of this gate cannot run") from exc
+
+ schema = _load_schema(root)
+ validator = jsonschema.Draft7Validator(schema)
+ files = _playbook_files(root)
+ if not files:
+ raise GateError("no *.playbook.json found; refusing to report success having validated nothing")
+
+ errors: list[str] = []
+ for path in files:
+ rel = path.relative_to(root)
+ try:
+ data = json.loads(path.read_text())
+ except json.JSONDecodeError as exc:
+ errors.append(f"{rel}: JSON parse error: {exc}")
+ continue
+ for err in sorted(validator.iter_errors(data), key=lambda e: list(e.path)):
+ loc = ".".join(str(p) for p in err.path) or "(root)"
+ errors.append(f"{rel}: {loc}: {err.message}")
+ return errors, files
+
+
+# ---------------------------------------------------------------------------
+# Self-test — prove the gate fires, in each direction
+# ---------------------------------------------------------------------------
+
+
+def self_test(reg: Registries) -> list[str]:
+ """Inject drift in each direction and assert the gate notices.
+
+ A gate nobody has seen fail is indistinguishable from a gate that cannot.
+ """
+ failures: list[str] = []
+
+ def expect(label: str, perturbed: Registries, needle: str) -> None:
+ errs = compare(perturbed)
+ if not any(needle in e for e in errs):
+ failures.append(f"self-test: {label}: injected drift was NOT detected (needle {needle!r}); got {errs or 'no errors'}")
+
+ baseline = compare(reg)
+ if baseline:
+ failures.append(f"self-test: the unperturbed tree already fails, so a pass proves nothing: {baseline}")
+
+ victim = sorted(reg.schema_step_types)[0]
+ absent = "step_type_that_exists_nowhere"
+
+ expect(
+ "schema declares a type the engine lacks",
+ replace(
+ reg,
+ schema_step_types=reg.schema_step_types | {absent},
+ execution={**reg.execution, absent: "unimplemented"},
+ ),
+ "does not implement",
+ )
+ expect(
+ "engine implements a type the schema lacks",
+ replace(reg, model_step_types=reg.model_step_types | {absent}),
+ "the schema does not declare it",
+ )
+ expect(
+ "declared type with no execution annotation",
+ replace(reg, execution={k: v for k, v in reg.execution.items() if k != victim}),
+ "does not say what the engine does with it",
+ )
+ expect(
+ "execution annotation for an undeclared type",
+ replace(reg, execution={**reg.execution, absent: "executed"}),
+ "which the schema does not declare",
+ )
+ runs_now = sorted(k for k, v in reg.execution.items() if v in _RUNS)
+ if not runs_now:
+ failures.append("self-test: no step type is annotated as running; cannot test the handler directions")
+ else:
+ running = runs_now[0]
+ expect(
+ "annotation claims execution the engine cannot deliver",
+ replace(reg, handler_step_types=reg.handler_step_types - {running}, inline_step_types=reg.inline_step_types - {running}),
+ "the engine has no handler for it",
+ )
+ expect(
+ "engine gained a handler the annotation calls unimplemented",
+ replace(reg, execution={**reg.execution, running: "unimplemented"}),
+ "understating what the product does",
+ )
+ expect("timeout ceiling drift", replace(reg, schema_timeout_max=reg.engine_timeout_max + 1), "timeout ceiling disagrees")
+ expect("retry ceiling drift", replace(reg, schema_retry_max=reg.engine_retry_max + 1), "retry ceiling disagrees")
+ expect(
+ "schema allows a trigger the validator rejects",
+ replace(reg, schema_triggers=reg.schema_triggers | {"webhook"}),
+ "that validate_playbooks.py rejects",
+ )
+ expect(
+ "validator allows a trigger the schema rejects",
+ replace(reg, validator_triggers=reg.validator_triggers | {"webhook"}),
+ "that the schema rejects",
+ )
+ expect("schema key the model drops", replace(reg, schema_playbook_keys=reg.schema_playbook_keys | {"ghost_key"}), "drops on load")
+ expect(
+ "model field the schema rejects", replace(reg, model_playbook_keys=reg.model_playbook_keys | {"ghost_field"}), "does not declare"
+ )
+ expect(
+ "a second schema file reappears", replace(reg, schema_files=(*reg.schema_files, "playbook.schema.json")), "second source of truth"
+ )
+
+ return failures
+
+
+# ---------------------------------------------------------------------------
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)
+ parser.add_argument("--repo-root", default=None, help="Tree to inspect (default: the repo this script lives in)")
+ parser.add_argument("--self-test", action="store_true", help="Prove the gate detects injected drift in each direction")
+ args = parser.parse_args()
+
+ try:
+ root = _find_repo_root(args.repo_root)
+ reg = collect(root)
+ content_errors, scanned = validate_shipped_playbooks(root)
+ except GateError as exc:
+ print(f"FATAL: {exc}", file=sys.stderr)
+ return 2
+
+ # Name what was inspected. A gate that does not say what it read cannot
+ # be distinguished from one that read nothing.
+ print(f"repo root {root}")
+ print(f"schema {', '.join(reg.schema_files) or '(none)'}")
+ print("engine services/agents/app/playbook/engine.py")
+ print(f"step types schema {len(reg.schema_step_types)} | StepType {len(reg.model_step_types)} | runnable {len(reg.runnable)}")
+ counts = {c: sum(1 for v in reg.execution.values() if v == c) for c in sorted(_EXECUTION_CLASSES)}
+ print("execution " + " | ".join(f"{k} {v}" for k, v in counts.items()))
+ print(f"bounds timeout {reg.schema_timeout_min}..{reg.schema_timeout_max}s | retries <={reg.schema_retry_max}")
+ print(f"playbooks scanned {len(scanned)}")
+ print()
+
+ errors = compare(reg)
+
+ if args.self_test:
+ st = self_test(reg)
+ if st:
+ print("SELF-TEST FAILED")
+ for f in st:
+ print(f" {f}")
+ return 1
+ print("self-test OK — every direction detected its injected drift, and the clean tree passed")
+ print()
+
+ if content_errors:
+ print(f"{len(content_errors)} shipped playbook(s) do not match the published schema:")
+ for e in content_errors[:40]:
+ print(f" {e}")
+ if len(content_errors) > 40:
+ print(f" ... and {len(content_errors) - 40} more")
+
+ if errors:
+ print(f"{len(errors)} schema/engine disagreement(s):")
+ for e in errors:
+ print(f" {e}")
+
+ if errors or content_errors:
+ return 1
+
+ print("OK — schema and engine agree in both directions, and every shipped playbook matches the schema.")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/scripts/generate_playbooks.py b/scripts/generate_playbooks.py
index 38f0f2ee5..504fce9b8 100755
--- a/scripts/generate_playbooks.py
+++ b/scripts/generate_playbooks.py
@@ -204,7 +204,12 @@ def make_playbook(
"name": name,
"description": description,
"version": version,
- "tags": [category, *tags],
+ # dict.fromkeys de-duplicates while preserving order. Several
+ # categories already lead their tag list with the category name, so
+ # the plain `[category, *tags]` emitted e.g.
+ # ["supply-chain", "supply-chain", "npm", ...] in 20 of the 62 packs
+ # — which the schema rejects under `uniqueItems`.
+ "tags": list(dict.fromkeys([category, *tags])),
"trigger": {
"on": trigger_on,
"severity": severity,
diff --git a/scripts/lint_playbooks.py b/scripts/lint_playbooks.py
index 856a4c9c8..d2df40876 100755
--- a/scripts/lint_playbooks.py
+++ b/scripts/lint_playbooks.py
@@ -13,6 +13,7 @@ Usage:
python3 scripts/lint_playbooks.py
python3 scripts/lint_playbooks.py path/to/my-playbook.json
"""
+
from __future__ import annotations
import json
@@ -28,12 +29,20 @@ except ImportError:
REPO_ROOT = Path(__file__).resolve().parent.parent
SCHEMA_PATH = REPO_ROOT / "schemas" / "playbook.schema.json"
-# Directories / glob patterns to scan automatically
+# Directories scanned for loose ``*.json`` playbooks.
SCAN_DIRS = [
REPO_ROOT / "services" / "agents" / "data" / "playbooks",
]
SCAN_GLOB = "*.json"
+# ...plus every ``*.playbook.json`` in the tree, which is what the docstring
+# above always claimed. It did not: only the two files in SCAN_DIRS were ever
+# checked, so the job printed "2/2 playbook files passed" while the 62
+# playbooks under ``playbooks/packs/v1/`` were never looked at — 32 of them
+# did not match the schema at all.
+RECURSIVE_GLOB = "*.playbook.json"
+SKIP_DIR_PARTS = {"node_modules", ".git", ".venv", "venv", "dist", "build"}
+
def load_schema() -> dict:
if not SCHEMA_PATH.exists():
@@ -47,9 +56,23 @@ def collect_files(extra: list[str]) -> list[Path]:
for d in SCAN_DIRS:
if d.is_dir():
files.extend(sorted(d.glob(SCAN_GLOB)))
+ for p in REPO_ROOT.rglob(RECURSIVE_GLOB):
+ if SKIP_DIR_PARTS.isdisjoint(p.parts):
+ files.append(p)
for p in extra:
- files.append(Path(p))
- return files
+ files.append(Path(p).resolve())
+ # A file can match both SCAN_DIRS and the recursive glob.
+ return sorted(set(files))
+
+
+def _display(path: Path) -> str:
+ """Repo-relative where possible. ``Path.relative_to`` raises for a path
+ outside the repo, which used to crash the reporter on the first failure
+ of a file passed on argv."""
+ try:
+ return str(path.relative_to(REPO_ROOT))
+ except ValueError:
+ return str(path)
def validate_file(path: Path, schema: dict, validator_cls) -> list[str]:
@@ -76,19 +99,21 @@ def main() -> None:
files = collect_files(extra_args)
if not files:
- print("No playbook files found — skipping lint (exit 0).")
- sys.exit(0)
+ # Previously exit 0. A scan that finds nothing in a repo that ships
+ # 64 playbooks is a broken scan, not a clean bill of health.
+ print("ERROR: no playbook files found — the scan is broken, not the tree.", file=sys.stderr)
+ sys.exit(1)
fail_count = 0
for path in files:
errs = validate_file(path, schema, validator_cls)
if errs:
- print(f"FAIL {path.relative_to(REPO_ROOT)}")
+ print(f"FAIL {_display(path)}")
for e in errs:
print(e)
fail_count += 1
else:
- print(f"OK {path.relative_to(REPO_ROOT)}")
+ print(f"OK {_display(path)}")
total = len(files)
print(f"\n{total - fail_count}/{total} playbook files passed schema validation.")
diff --git a/services/actions/app/live_actions/capability_contracts.py b/services/actions/app/live_actions/capability_contracts.py
index 843bbda06..16757fad8 100644
--- a/services/actions/app/live_actions/capability_contracts.py
+++ b/services/actions/app/live_actions/capability_contracts.py
@@ -526,6 +526,56 @@ CAPABILITY_CONTRACTS: dict[str, CapabilityContract] = {
}
+#: Legacy ``ActionType`` value -> capability name, for the cases where the
+#: two vocabularies chose different words for the same verb.
+#:
+#: Only one entry, and it is a naming gap rather than a missing capability:
+#: the public API verb is ``notify_slack`` (it predates the capability
+#: registry and names the transport), the capability is ``notify`` (it names
+#: the verb, which is the whole point of a per-capability contract), and
+#: ``SlackNotify`` in ``builtins.py`` already bridges them. What was missing
+#: was anything that told a *lookup by ActionType value* about the bridge, so
+#: ``approval_gate`` found no contract, skipped the confidence matrix
+#: entirely and let blast radius decide alone.
+#:
+#: Why an alias and not a rename. ``action_type`` is persisted as free text:
+#: ``remediation_whitelist`` (migration 015) stores operator pre-approvals
+#: keyed ``UNIQUE (tenant_id, action_type)``, so renaming the enum member
+#: silently orphans every row an operator created for ``notify_slack`` —
+#: their pre-approval stops matching and nobody is told. It is also a
+#: documented request field (``{"action_type": "notify_slack"}`` in
+#: ``apps/docs/docs/operations/notifications.md``) and a member of the
+#: ``ActionType`` union in ``packages/types``.
+#:
+#: When to delete this. It retires with ``ActionType`` itself: the enum is
+#: already the legacy half of the registry (every adapter carries a
+#: ``_legacy_action_type`` pointing back at it) and the live-actions path
+#: dispatches on capability strings. When the ``POST /actions`` body takes a
+#: capability and the ``remediation_whitelist`` rows have been migrated,
+#: this map and the enum go together. Until then a new entry here needs the
+#: same justification: a *naming* difference for a verb that already has a
+#: contract, never a stand-in for a capability nobody wrote.
+ACTION_TYPE_CAPABILITY_ALIASES: dict[str, str] = {
+ "notify_slack": "notify",
+}
+
+
+def contract_for_action_type(action_type_value: str) -> CapabilityContract | None:
+ """Resolve a legacy ``ActionType`` value to its capability contract.
+
+ Tries the value as a capability name first, then the alias map. Returns
+ ``None`` when the verb genuinely has no contract, so callers can say so
+ rather than invent an impact for it.
+ """
+ contract = CAPABILITY_CONTRACTS.get(action_type_value)
+ if contract is not None:
+ return contract
+ aliased = ACTION_TYPE_CAPABILITY_ALIASES.get(action_type_value)
+ if aliased is None:
+ return None
+ return CAPABILITY_CONTRACTS.get(aliased)
+
+
def apply_contract(cls: type) -> type:
"""Class decorator: stamp the capability's contract onto an executor.
diff --git a/services/actions/app/services/approval_gate.py b/services/actions/app/services/approval_gate.py
index 2277093d5..cc530ea4d 100644
--- a/services/actions/app/services/approval_gate.py
+++ b/services/actions/app/services/approval_gate.py
@@ -16,18 +16,22 @@ the matrix already states for its own three inputs: each can raise a
requirement, none can lower it. So switching it on cannot make anything
auto-execute that did not before.
-Two honest limits, both logged rather than papered over:
-
-One of the 24 ``ActionType`` members has no capability contract, down from
-five: ``capture_forensics`` and ``chatops_verify`` gained one, and
-``add_ioc_to_blocklist`` and ``run_playbook`` left the enum because neither
-had an implementation path. The one left is ``notify_slack``, and it is a
-naming gap rather than an undeclared verb — the lookup below is by
-``ActionType`` value and the matching capability is called ``notify``, which
-does have a contract. For an unmapped verb the matrix has no impact to reason
-about and blast radius decides alone. That is recorded at debug rather than
-guessed at, because inventing an impact for an unmapped verb is how a gate
-starts certifying things it never examined.
+Every ``ActionType`` member now resolves to a capability contract, and
+``scripts/check_action_contract.py`` fails if one stops doing so.
+
+The last hold-out was ``notify_slack``. It was never a missing capability:
+the verb is ``notify``, it has a contract, and ``SlackNotify`` already
+bridged the two. What was missing was anything connecting a lookup *by
+ActionType value* to that bridge, so this gate found no contract, skipped
+the confidence matrix and let blast radius decide alone — a 40%-confidence
+guess and a corroborated finding got the same answer for the one verb most
+likely to be auto-executed. ``contract_for_action_type`` closes it by alias
+rather than by rename, because ``action_type`` is persisted operator intent
+(see the note on ``ACTION_TYPE_CAPABILITY_ALIASES``).
+
+An unmapped verb is still possible in principle, and is still recorded at
+debug rather than guessed at: inventing an impact for a verb nobody declared
+is how a gate starts certifying things it never examined.
A request with no ``confidence`` is treated as the lowest band. For anything
above READ_ONLY impact that means analyst approval, which is a real tightening
@@ -38,7 +42,7 @@ from __future__ import annotations
import structlog
-from app.live_actions.capability_contracts import CAPABILITY_CONTRACTS
+from app.live_actions.capability_contracts import contract_for_action_type
from app.live_actions.contract import ApprovalRequirement
from app.models.action import ActionRequest, ActionStatus, BlastRadius
from app.services.approval_matrix import evaluate as evaluate_matrix
@@ -90,7 +94,7 @@ async def apply_matrix(
Never lowers: a blast-radius gate that already demands approval keeps
demanding it whatever the confidence says.
"""
- contract = CAPABILITY_CONTRACTS.get(request.action_type.value)
+ contract = contract_for_action_type(request.action_type.value)
if contract is None:
logger.debug(
"approval_gate.no_capability_contract",
diff --git a/services/actions/tests/test_approval_gate_wiring.py b/services/actions/tests/test_approval_gate_wiring.py
index 869279773..70933298d 100644
--- a/services/actions/tests/test_approval_gate_wiring.py
+++ b/services/actions/tests/test_approval_gate_wiring.py
@@ -17,7 +17,8 @@ from __future__ import annotations
from uuid import uuid4
import pytest
-from app.live_actions.capability_contracts import CAPABILITY_CONTRACTS
+from app.live_actions import capability_contracts
+from app.live_actions.capability_contracts import CAPABILITY_CONTRACTS, contract_for_action_type
from app.models.action import ActionRequest, ActionStatus, ActionType, BlastRadius
from app.services.approval_gate import apply_matrix
from app.services.blast_radius import BlastRadiusGate
@@ -143,13 +144,68 @@ class TestTheTierIsHonoured:
assert "L1" in reason
+class TestTheAliasedVerbIsGatedToo:
+ """``notify_slack`` was the last ActionType the matrix never saw.
+
+ It was never a missing capability — the verb is ``notify``, it has had a
+ contract throughout, and ``SlackNotify`` bridges the two names. What was
+ missing was anything telling a lookup *by ActionType value* about that
+ bridge, so ``apply_matrix`` found nothing, returned the blast-radius
+ verdict unchanged, and the one verb most likely to auto-execute was the
+ one verb graded without reference to confidence.
+ """
+
+ def test_the_two_names_are_not_the_same_string(self):
+ """The premise. If these ever converge the alias becomes dead code,
+ and the contract gate says so rather than leaving it to rot."""
+ assert ActionType.NOTIFY_SLACK.value == "notify_slack"
+ assert ActionType.NOTIFY_SLACK.value not in CAPABILITY_CONTRACTS
+ assert "notify" in CAPABILITY_CONTRACTS
+
+ def test_the_alias_resolves_to_the_notify_contract(self):
+ assert contract_for_action_type(ActionType.NOTIFY_SLACK.value) is CAPABILITY_CONTRACTS["notify"]
+
+ @pytest.mark.asyncio
+ async def test_a_low_confidence_notify_no_longer_auto_executes(self):
+ """Fails against the pre-change tree, which approved this outright.
+
+ ``notify_slack`` is MINIMAL blast radius, so ``BlastRadiusGate``
+ approves it on its own. With the contract now reachable the matrix
+ gets a say, and at 10% confidence under the default L1 tier it
+ demands an analyst.
+ """
+ request = _request(ActionType.NOTIFY_SLACK, confidence=0.1)
+ before, br, before_reason = BlastRadiusGate().evaluate(request)
+ after, reason = await apply_matrix(request, before, br, before_reason)
+
+ assert before == ActionStatus.APPROVED
+ assert after == ActionStatus.AWAITING_APPROVAL
+ assert reason != before_reason
+
+ @pytest.mark.asyncio
+ async def test_the_alias_cannot_lower_a_requirement(self):
+ """An alias route into the matrix must obey the same composition rule
+ as a direct one: each input can raise a requirement, none can lower
+ it. A LOW-impact contract must not talk a blocked action into running.
+ """
+ request = _request(ActionType.NOTIFY_SLACK, confidence=1.0)
+ after, _ = await apply_matrix(request, ActionStatus.AWAITING_APPROVAL, BlastRadius.MINIMAL, "held by policy")
+
+ assert after == ActionStatus.AWAITING_APPROVAL
+
+
class TestVerbsWithNoContract:
@pytest.mark.asyncio
- async def test_an_unmapped_verb_is_left_to_blast_radius(self):
- """Seven ActionType members have no capability contract. Inventing an
- impact for them is how a gate starts certifying things it never
- examined, so blast radius decides alone and the gap is logged."""
- assert ActionType.NOTIFY_SLACK.value not in CAPABILITY_CONTRACTS
+ async def test_an_unmapped_verb_is_still_left_to_blast_radius(self, monkeypatch: pytest.MonkeyPatch):
+ """Every ActionType resolves a contract today and a gate keeps it that
+ way, so this path is no longer reachable through the enum. It stays
+ tested because the fallback is what stops the gate inventing an impact
+ for a verb nobody declared — a gate that certifies what it never
+ examined is the failure this whole module exists to avoid.
+
+ Pulling the alias out is the smallest way to reach the branch.
+ """
+ monkeypatch.setattr(capability_contracts, "ACTION_TYPE_CAPABILITY_ALIASES", {})
request = _request(ActionType.NOTIFY_SLACK, confidence=0.1)
before, br, before_reason = BlastRadiusGate().evaluate(request)
@@ -158,6 +214,12 @@ class TestVerbsWithNoContract:
assert after == before
assert reason == before_reason
+ def test_no_action_type_is_left_unmapped(self):
+ """The property the contract gate enforces, asserted here too so it
+ fails in the service's own suite and not only in a CI script."""
+ unmapped = [a.value for a in ActionType if contract_for_action_type(a.value) is None]
+ assert unmapped == [], f"ActionType members with no reachable capability contract: {unmapped}"
+
class TestTierLabelMapping:
@pytest.mark.parametrize(
diff --git a/services/agents/app/playbook/engine.py b/services/agents/app/playbook/engine.py
index 45faf835b..134613440 100644
--- a/services/agents/app/playbook/engine.py
+++ b/services/agents/app/playbook/engine.py
@@ -627,28 +627,56 @@ class PlaybookEngine:
attempt = 0
handler = _HANDLERS.get(step.type)
- while True:
- attempt += 1
- t0 = time.perf_counter()
- try:
- if dry_run:
- result = {"dry_run": True, "step": step.name}
- elif handler:
- result = await handler(step, pr.context, http)
- else:
- result = {"skipped": True, "reason": f"no handler for {step.type}"}
- elapsed = time.perf_counter() - t0
- result["_elapsed_ms"] = round(elapsed * 1000)
- break # success
- except Exception as exc: # noqa: BLE001
- elapsed = time.perf_counter() - t0
- logger.error("Step %s attempt %d failed: %s", step.name, attempt, exc)
- if attempt <= step.retry_max:
- await asyncio.sleep(min(2**attempt, 30))
- else:
- step_status = StepStatus.FAILED
- result = {"error": str(exc), "_elapsed_ms": round(elapsed * 1000)}
- break
+ if handler is None and not dry_run:
+ # Fail closed, and skip the retry loop — a missing handler
+ # will still be missing on the next attempt.
+ #
+ # This branch used to return ``{"skipped": True}`` while
+ # leaving step_status at SUCCESS, so twelve of the
+ # twenty-two declared step types reported that they had run
+ # when nothing had. The worst of them was ``approval``: a
+ # human decision point that passed on its own and let the
+ # run continue into the very action an analyst was supposed
+ # to authorise. Falling through to the shared tail below
+ # means the default ``on_failure: abort`` halts the run.
+ step_status = StepStatus.FAILED
+ result = {
+ "error": f"step type {step.type.value!r} has no handler in this engine",
+ "unimplemented": True,
+ "_elapsed_ms": 0,
+ }
+ logger.error(
+ "Step %s (%s) has no handler; failing closed rather than reporting success",
+ step.name,
+ step.type.value,
+ )
+ else:
+ while True:
+ attempt += 1
+ t0 = time.perf_counter()
+ try:
+ if dry_run:
+ result = {"dry_run": True, "step": step.name}
+ if handler is None:
+ # A dry run exists to tell the author what
+ # would happen. "dry_run: true" alone would
+ # imply this step is fine.
+ result["unimplemented"] = True
+ result["would_fail"] = True
+ else:
+ result = await handler(step, pr.context, http)
+ elapsed = time.perf_counter() - t0
+ result["_elapsed_ms"] = round(elapsed * 1000)
+ break # success
+ except Exception as exc: # noqa: BLE001
+ elapsed = time.perf_counter() - t0
+ logger.error("Step %s attempt %d failed: %s", step.name, attempt, exc)
+ if attempt <= step.retry_max:
+ await asyncio.sleep(min(2**attempt, 30))
+ else:
+ step_status = StepStatus.FAILED
+ result = {"error": str(exc), "_elapsed_ms": round(elapsed * 1000)}
+ break
pr.step_results.append({"step_id": step.id, "name": step.name, "status": step_status, "result": result})
# Merge result into context for downstream steps. The namespaced
diff --git a/services/agents/app/playbook/nl_drafter.py b/services/agents/app/playbook/nl_drafter.py
index 53053e098..8cee751f9 100644
--- a/services/agents/app/playbook/nl_drafter.py
+++ b/services/agents/app/playbook/nl_drafter.py
@@ -23,9 +23,9 @@ Two execution paths:
:func:`services.agents.app.llm.safe_ainvoke` and asks for a
JSON playbook back. The output is validated against the canonical
:class:`Playbook` Pydantic model AND against the project's
- `playbook.schema.json` (JSON Schema 2020-12) before being
- returned. Any validation failure falls back to the substrate
- draft so the UX never hangs.
+ `schemas/playbook.schema.json` (JSON Schema draft-07) before
+ being returned. Any validation failure falls back to the
+ substrate draft so the UX never hangs.
The drafter is the single Pydantic-aware Python entry point for
NL → playbook drafting; the API route at
@@ -44,6 +44,7 @@ from datetime import UTC, datetime
from pathlib import Path
from typing import Any
+from .bounds import ABSOLUTE_MAX_RETRIES, ABSOLUTE_MAX_TIMEOUT_SECONDS
from .models import Playbook, PlaybookStep, StepType
logger = logging.getLogger("aisoc.playbook.nl_drafter")
@@ -392,63 +393,39 @@ def _summarise_prompt_for_name(prompt: str) -> str:
def _schema_path() -> Path:
- """Resolve the canonical playbook.schema.json bundled with the repo.
+ """Resolve the one canonical ``schemas/playbook.schema.json``.
- CI uses ``schemas/playbook.schema.json`` (the path baked into
- ``scripts/lint_playbooks.py`` and the ``validate-playbooks``
- workflow), so we prefer that location and fall back to the
- legacy root-level copy only if it is missing.
+ There used to be a second copy at the repo root declaring a different
+ step vocabulary, and this function fell back to it whenever the primary
+ was missing — so a packaging slip did not fail, it silently validated
+ against a different contract. There is now exactly one schema (a parity
+ gate enforces that), and a missing one is an error rather than a cue to
+ go looking for something else to trust.
+
+ The walk stops at the first ancestor that has the file so a checkout at
+ any depth works, but it never accepts a differently-shaped substitute.
"""
here = Path(__file__).resolve()
for parent in here.parents:
- primary = parent / "schemas" / "playbook.schema.json"
- if primary.exists():
- return primary
- fallback = parent / "playbook.schema.json"
- if fallback.exists():
- return fallback
- raise FileNotFoundError("playbook.schema.json not found alongside the repo root")
-
-
-# Schema-allowed step types (from ``schemas/playbook.schema.json``). The
-# Pydantic StepType enum is broader than the JSON Schema today; for the
-# drafter we collapse the extended types down to their nearest
-# schema-compatible neighbour so the resulting playbook ALSO passes
-# ``scripts/lint_playbooks.py`` without the user having to touch the
-# schema.
-_SCHEMA_STEP_TYPES: frozenset[str] = frozenset(
- {
- "enrich",
- "investigate",
- "notify",
- "block_ip",
- "isolate_host",
- "create_ticket",
- "close_case",
- "http",
- "condition",
- }
-)
+ candidate = parent / "schemas" / "playbook.schema.json"
+ if candidate.exists():
+ return candidate
+ raise FileNotFoundError("schemas/playbook.schema.json not found in any parent of services/agents/app/playbook/")
-# Pydantic-only StepType → schema-allowed StepType. Anything not in the
-# schema enum collapses to ``investigate`` because the analyst can still
-# upgrade the step in the editor without losing the prompt-derived name.
-_PYDANTIC_TO_SCHEMA_TYPE: dict[str, str] = {
- "block_ioc": "block_ip",
- "disable_user": "investigate",
- "reset_password": "investigate",
- "revoke_session": "investigate",
- "force_mfa": "investigate",
- "kill_process": "investigate",
- "quarantine_file": "investigate",
- "run_av_scan": "investigate",
- "run_script": "http",
- "search_siem": "investigate",
- "create_notable_event": "create_ticket",
- "osquery_live_query": "investigate",
- "approval": "condition",
-}
+
+# The step-type collapse that used to live here is gone.
+#
+# The JSON Schema declared 9 step types against a StepType enum of 22, so the
+# drafter rewrote the 13 it could not express onto their "nearest neighbour"
+# — ``disable_user`` and ``revoke_session`` both became ``investigate``,
+# ``approval`` became ``condition`` — purely so the output would pass
+# ``scripts/lint_playbooks.py``. The original was preserved in
+# ``params.original_type``, but the step that shipped said it would
+# investigate when the analyst had asked to disable an account, and the
+# approval gate an author wrote came out as an ungated branch.
+#
+# The schema now declares all 22, so there is nothing to collapse.
def _strip_nulls(obj: Any) -> Any:
@@ -469,57 +446,22 @@ def _strip_nulls(obj: Any) -> Any:
return obj
-def _collapse_step_types_for_schema(payload: dict[str, Any]) -> dict[str, Any]:
- """Return a copy of ``payload`` with each step ``type`` collapsed
- to the JSON-Schema-allowed enum AND with ``null`` fields stripped.
- Preserves the original type as ``params.original_type`` so the
- editor can recover it.
-
- Pydantic remains the source of truth at runtime — this projection
- exists only so the drafter's output passes ``lint_playbooks.py``.
- """
-
- proj = _strip_nulls(json.loads(json.dumps(payload))) # deep copy + null-strip
- steps = proj.get("steps", [])
- if not isinstance(steps, list):
- return proj
- for step in steps:
- if not isinstance(step, dict):
- continue
- st = step.get("type")
- if not isinstance(st, str) or st in _SCHEMA_STEP_TYPES:
- continue
- mapped = _PYDANTIC_TO_SCHEMA_TYPE.get(st, "investigate")
- params = step.get("params")
- if not isinstance(params, dict):
- params = {}
- params.setdefault("original_type", st)
- step["params"] = params
- step["type"] = mapped
- return proj
-
-
def _load_schema() -> dict[str, Any]:
return json.loads(_schema_path().read_text(encoding="utf-8"))
def _validate_against_schema(payload: dict[str, Any]) -> tuple[bool, str | None]:
- """Validate ``payload`` against playbook.schema.json.
-
- We try two passes: the raw payload first, then the
- step-type-collapsed projection (see
- :func:`_collapse_step_types_for_schema`). Returning ``True`` from
- pass-one means the playbook is **strictly** schema-conformant;
- falling through to pass-two means the drafter is producing a
- Pydantic-richer step set that the JSON Schema can't yet describe
- — that's a known gap tracked in the v8.0 plan, and we widen the
- Pydantic model first because the React Flow editor consumes the
- Pydantic shape directly.
-
- Returns ``(True, None)`` on success of either pass,
- ``(False, error_message)`` on failure of both. ``jsonschema`` is
- imported lazily so the module is importable in environments
- without it.
+ """Validate ``payload`` against ``schemas/playbook.schema.json``.
+
+ One pass. There used to be a second that rewrote any step type the
+ schema could not express onto a "nearest neighbour" and re-validated;
+ it made the drafter's output lint-clean by changing what the playbook
+ said it would do. The schema now covers the whole ``StepType`` range,
+ so a failure here is a real failure.
+
+ Returns ``(True, None)`` on success, ``(False, error_message)`` on
+ failure. ``jsonschema`` is imported lazily so the module stays
+ importable in environments without it.
"""
try:
@@ -534,24 +476,14 @@ def _validate_against_schema(payload: dict[str, Any]) -> tuple[bool, str | None]
logger.warning("playbook.schema.json missing: %s", exc)
return True, None
- # Pass 1 — null-stripped payload (Pydantic emits ``null`` keys that
- # the schema rejects under ``additionalProperties: false``).
+ # Null-stripped: Pydantic emits ``null`` for unset optional keys and the
+ # schema rejects those under ``additionalProperties: false``.
cleaned = _strip_nulls(payload)
try:
jsonschema.validate(cleaned, schema) # type: ignore[attr-defined]
return True, None
except jsonschema.ValidationError as exc: # type: ignore[attr-defined]
- first_err = exc.message
-
- # Pass 2 — also collapse Pydantic-only step types to schema-allowed
- # equivalents. This lets the drafter use the full StepType range
- # internally while still emitting a CI-clean draft.
- try:
- projection = _collapse_step_types_for_schema(payload)
- jsonschema.validate(projection, schema) # type: ignore[attr-defined]
- return True, None
- except jsonschema.ValidationError as exc: # type: ignore[attr-defined]
- return False, f"{first_err} (also failed after step-type collapse: {exc.message})"
+ return False, exc.message
# ---------------------------------------------------------------------------
@@ -571,24 +503,34 @@ Rules — follow exactly:
``^[a-z0-9][a-z0-9-]{2,62}$``. The ``version`` must be semantic,
e.g. ``"1.0.0"``.
3. ``trigger`` must contain ``on`` (one of ``alert`` / ``case`` /
- ``schedule`` / ``manual`` / ``webhook``). Severities, when present,
+ ``schedule`` / ``manual``). Severities, when present,
must be an array of any of: ``info`` / ``low`` / ``medium`` /
``high`` / ``critical``.
-4. Each step must declare a ``type``. Allowed types include
- ``enrich``, ``investigate``, ``notify``, ``block_ip``, ``block_ioc``,
- ``isolate_host``, ``create_ticket``, ``close_case``, ``http``,
- ``condition``, ``approval``, ``disable_user``, ``reset_password``,
- ``revoke_session``, ``force_mfa``, ``kill_process``,
- ``quarantine_file``, ``run_av_scan``, ``run_script``,
- ``search_siem``, ``create_notable_event``, ``osquery_live_query``.
+4. Each step must declare a ``type``, one of exactly:
+ __STEP_TYPES__.
5. Each step must carry a short, action-oriented ``name``, an ``id``
- (8-32 char hex), an ``on_failure`` ∈ ``{abort, continue, retry}``,
- ``retry_max`` (0-5), and ``timeout_seconds`` (1-3600).
+ (8-32 char hex), an ``on_failure`` ∈ ``abort`` / ``continue`` /
+ ``retry``, ``retry_max`` (0-__MAX_RETRIES__), and
+ ``timeout_seconds`` (1-__MAX_TIMEOUT__).
6. The output's ``enabled`` MUST be ``false``. A human reviews before
enabling.
7. Do NOT invent fields not in the schema. Do NOT emit prose. JSON only.
"""
+# Filled from the enum and the bounds module rather than restated, because a
+# hand-written copy of the vocabulary in a prompt is one more place for it to
+# drift — this list previously offered ``webhook`` as a trigger, which no
+# validator in the repo accepts, and capped ``retry_max`` at 5 against a
+# model that allows 25.
+#
+# Token substitution rather than ``str.format`` because the prompt contains
+# a regex with brace quantifiers that format would try to interpret.
+_SYSTEM_PROMPT = (
+ _SYSTEM_PROMPT.replace("__STEP_TYPES__", ", ".join(f"``{st.value}``" for st in StepType))
+ .replace("__MAX_RETRIES__", str(ABSOLUTE_MAX_RETRIES))
+ .replace("__MAX_TIMEOUT__", str(ABSOLUTE_MAX_TIMEOUT_SECONDS))
+)
+
_USER_TEMPLATE = """\
Analyst prompt:
diff --git a/services/agents/tests/test_playbook_engine_correctness.py b/services/agents/tests/test_playbook_engine_correctness.py
index 63614b16d..6c91e3794 100644
--- a/services/agents/tests/test_playbook_engine_correctness.py
+++ b/services/agents/tests/test_playbook_engine_correctness.py
@@ -403,3 +403,102 @@ class TestEngineConditionGate:
assert called["hit"] is True
assert run.step_results[0]["status"] == StepStatus.SUCCESS
+
+
+# ---------------------------------------------------------------------------
+# PlaybookEngine.run — step types the engine cannot run must not report success
+# ---------------------------------------------------------------------------
+#
+# Twelve of the twenty-two StepType members had no entry in ``_HANDLERS``.
+# The run loop answered those with ``{"skipped": True}`` and left
+# ``step_status`` at its SUCCESS default, so a playbook containing them ran to
+# RunStatus.COMPLETED having done nothing that it said it did.
+#
+# ``approval`` is the one that matters most: it is a human decision point, it
+# appears in 14 steps across the shipped packs, and it passed on its own —
+# the run continued straight into the action an analyst was meant to
+# authorise. "Unverifiable means not autonomous" applies to a gate that
+# cannot be evaluated as much as to an action that cannot be verified.
+
+
+class TestUnimplementedStepTypesFailClosed:
+ @pytest.mark.asyncio
+ async def test_an_approval_gate_the_engine_cannot_honour_halts_the_run(self) -> None:
+ """Fails against the pre-change tree, which reported SUCCESS/COMPLETED."""
+ pb = _make_playbook(
+ [
+ PlaybookStep(id="gate", name="Analyst approves isolation", type=StepType.APPROVAL),
+ PlaybookStep(id="act", name="Isolate the host", type=StepType.ISOLATE_HOST),
+ ]
+ )
+
+ run = await PlaybookEngine().run(pb, trigger_context={})
+
+ assert run.step_results[0]["status"] == StepStatus.FAILED
+ assert run.step_results[0]["result"]["unimplemented"] is True
+ assert "approval" in run.step_results[0]["result"]["error"]
+ assert run.status == RunStatus.FAILED
+ # The whole point: the step behind the gate must not have run.
+ assert len(run.step_results) == 1, "execution continued past a gate that was never granted"
+
+ @pytest.mark.asyncio
+ async def test_no_step_type_reports_success_without_a_handler(self) -> None:
+ """Every StepType with no handler, not just the one that hurt most."""
+ unimplemented = [st for st in StepType if st not in engine_mod._HANDLERS and st != StepType.CONDITION]
+ assert unimplemented, "fixture assumption broken: expected some StepType to lack a handler"
+
+ for step_type in unimplemented:
+ pb = _make_playbook([PlaybookStep(id="s1", name=f"step {step_type.value}", type=step_type)])
+ run = await PlaybookEngine().run(pb, trigger_context={})
+
+ result = run.step_results[0]
+ assert result["status"] == StepStatus.FAILED, f"{step_type.value} reported {result['status']} with no handler"
+ assert result["result"].get("unimplemented") is True
+ assert run.status == RunStatus.FAILED
+
+ @pytest.mark.asyncio
+ async def test_a_missing_handler_is_not_retried(self) -> None:
+ """A handler that does not exist will not exist next attempt either;
+ retrying just delays the failure by up to 2**retry_max seconds."""
+ pb = _make_playbook([PlaybookStep(id="s1", name="scan", type=StepType.RUN_AV_SCAN, retry_max=3)])
+
+ run = await PlaybookEngine().run(pb, trigger_context={})
+
+ assert run.step_results[0]["status"] == StepStatus.FAILED
+ assert run.step_results[0]["result"]["_elapsed_ms"] == 0
+
+ @pytest.mark.asyncio
+ async def test_on_failure_continue_is_still_honoured(self) -> None:
+ """Failing closed must not mean ignoring the author's policy: a step
+ explicitly marked ``continue`` still lets the run proceed."""
+ pb = _make_playbook(
+ [
+ PlaybookStep(id="s1", name="best-effort scan", type=StepType.RUN_AV_SCAN, on_failure="continue"),
+ PlaybookStep(id="s2", name="second step", type=StepType.RUN_SCRIPT, on_failure="continue"),
+ ]
+ )
+
+ run = await PlaybookEngine().run(pb, trigger_context={})
+
+ assert len(run.step_results) == 2
+ assert all(r["status"] == StepStatus.FAILED for r in run.step_results)
+ assert run.status == RunStatus.COMPLETED
+
+ @pytest.mark.asyncio
+ async def test_a_dry_run_says_which_steps_would_fail(self) -> None:
+ """A dry run exists to tell the author what would happen. Reporting a
+ bare ``dry_run: true`` for a step that cannot execute would imply it
+ is fine."""
+ pb = _make_playbook(
+ [
+ PlaybookStep(id="s1", name="notify", type=StepType.NOTIFY),
+ PlaybookStep(id="s2", name="gate", type=StepType.APPROVAL),
+ ]
+ )
+
+ run = await PlaybookEngine().run(pb, trigger_context={}, dry_run=True)
+
+ assert run.status == RunStatus.COMPLETED
+ assert run.step_results[0]["result"].get("unimplemented") is None
+ assert run.step_results[1]["result"]["unimplemented"] is True
+ assert run.step_results[1]["result"]["would_fail"] is True
diff --git a/services/agents/tests/test_playbook_nl_drafter.py b/services/agents/tests/test_playbook_nl_drafter.py
index d39ea81eb..251a2cab9 100644
--- a/services/agents/tests/test_playbook_nl_drafter.py
+++ b/services/agents/tests/test_playbook_nl_drafter.py
@@ -39,6 +39,12 @@ from app.playbook import (
# Helpers
# ---------------------------------------------------------------------------
+#: Materialised once. Iterating the enum class directly is correct Python,
+#: but CodeQL resolves ``StepType`` through the ``app.playbook`` package
+#: re-export, does not see the ``Enum`` base, and reads ``for x in StepType``
+#: as iterating a plain class (``py/non-iterable-in-for-loop``).
+_ALL_STEP_TYPES = tuple(StepType.__members__.values())
+
def _run(coro: Any) -> Any:
"""Run an awaitable synchronously, with one event loop per call."""
@@ -153,55 +159,59 @@ class TestSchemaValidity:
assert rehydrated.id == pb.id
assert _step_types(rehydrated) == _step_types(pb)
- def test_step_type_collapse_preserves_original_type(self) -> None:
- # ``run_av_scan`` is Pydantic-only — schema collapse must map it
- # to a schema-allowed type and stash the original in params.
- payload = {
- "id": "test-id",
- "name": "Test",
- "version": "1.0.0",
- "trigger": {"on": "alert"},
- "steps": [
- {
- "id": "s1",
- "name": "Run AV Scan",
- "type": "run_av_scan",
- "params": {},
- }
- ],
- }
- collapsed = nl_drafter._collapse_step_types_for_schema(payload)
- assert collapsed["steps"][0]["type"] in nl_drafter._SCHEMA_STEP_TYPES
- assert collapsed["steps"][0]["params"]["original_type"] == "run_av_scan"
-
- def test_schema_collapse_is_pure(self) -> None:
- # The collapse helper must NOT mutate its input.
- payload = {
- "id": "test-id",
- "name": "Test",
- "version": "1.0.0",
- "trigger": {"on": "alert"},
- "steps": [{"id": "s1", "name": "x", "type": "block_ioc", "params": {}}],
- }
- before = json.dumps(payload, sort_keys=True)
- nl_drafter._collapse_step_types_for_schema(payload)
- after = json.dumps(payload, sort_keys=True)
- assert before == after, "collapse must not mutate its input"
-
- def test_schema_allowed_types_unchanged_by_collapse(self) -> None:
- payload = {
- "id": "test-id",
- "name": "Test",
- "version": "1.0.0",
- "trigger": {"on": "alert"},
- "steps": [
- {"id": "s1", "name": "x", "type": "enrich", "params": {}},
- {"id": "s2", "name": "y", "type": "notify", "params": {}},
- ],
- }
- collapsed = nl_drafter._collapse_step_types_for_schema(payload)
- assert collapsed["steps"][0]["type"] == "enrich"
- assert collapsed["steps"][1]["type"] == "notify"
+ def test_every_step_type_validates_without_being_rewritten(self) -> None:
+ """The three tests that used to live here asserted the collapse.
+
+ The drafter could not express 13 of the 22 ``StepType`` values in the
+ published schema, so it rewrote them onto a "nearest neighbour" to
+ make its output lint-clean: ``run_av_scan`` became ``investigate``,
+ ``approval`` became ``condition``. The original was stashed in
+ ``params.original_type``, but the playbook that shipped said it would
+ investigate when the author had asked to scan, and an approval gate
+ came out as an ungated branch.
+
+ This is the property that replaces it: every step type the model
+ accepts is declared by the published schema. Fails against the
+ pre-change tree, where the schema declared 9 of 22.
+
+ Asserted against the schema document directly, not through
+ ``_validate_against_schema``. Going through the helper is what hid
+ this for so long — its second pass rewrote the step and reported
+ success, so "the drafter's output validates" was true of a playbook
+ the drafter had altered to make it true.
+ """
+ jsonschema = pytest.importorskip("jsonschema")
+ schema = nl_drafter._load_schema()
+ declared = set(schema["definitions"]["PlaybookStep"]["properties"]["type"]["enum"])
+
+ missing = sorted(st.value for st in _ALL_STEP_TYPES if st.value not in declared)
+ assert missing == [], f"StepType members the published schema does not declare: {missing}"
+
+ for step_type in _ALL_STEP_TYPES:
+ payload = {
+ "id": "test-id",
+ "name": "Test playbook",
+ "version": "1.0.0",
+ "trigger": {"on": "alert"},
+ "steps": [{"id": "s1", "name": f"Step {step_type.value}", "type": step_type.value, "params": {}}],
+ }
+ jsonschema.validate(payload, schema)
+
+ def test_the_collapse_helper_is_gone(self) -> None:
+ """Keeping the schema honest is only durable if the workaround cannot
+ quietly come back — a lossy projection would once again let the
+ drafter emit a step whose declared type is not what it asked for."""
+ for removed in ("_collapse_step_types_for_schema", "_PYDANTIC_TO_SCHEMA_TYPE", "_SCHEMA_STEP_TYPES"):
+ assert not hasattr(nl_drafter, removed), f"{removed} is back; the schema should cover the full StepType range instead"
+
+ def test_the_system_prompt_offers_only_real_vocabulary(self) -> None:
+ """The prompt used to restate the vocabulary by hand and had drifted:
+ it offered ``webhook`` as a trigger, which no validator in the repo
+ accepts, and capped ``retry_max`` at 5 against a model allowing 25."""
+ prompt = nl_drafter._SYSTEM_PROMPT
+ for step_type in _ALL_STEP_TYPES:
+ assert f"``{step_type.value}``" in prompt, f"{step_type.value} missing from the drafter prompt"
+ assert "webhook" not in prompt, "prompt offers a trigger no validator accepts"
def test_draft_result_to_dict_round_trip(self) -> None:
async def _go() -> DraftResult:
diff --git a/tests/test_playbook_schema_parity_gate.py b/tests/test_playbook_schema_parity_gate.py
new file mode 100644
index 000000000..6af80ec08
--- /dev/null
+++ b/tests/test_playbook_schema_parity_gate.py
@@ -0,0 +1,217 @@
+"""Tests for the playbook schema/engine parity gate.
+
+The property worth asserting is not "does the tree happen to agree today" but
+"would this gate still notice if it stopped agreeing" — and, specifically,
+would it notice in *both* directions. A check that only asks whether the
+schema declares something the engine lacks passes forever while the engine
+grows verbs nobody tells the schema about, which is exactly how this drift
+reached 13 undeclared step types. A sibling gate in this repo compared a YAML
+file declaring 17 graph labels against Go code with 28 and printed "OK".
+
+So the assertions are paired: the gate must catch a schema-only step type
+*and* an engine-only one; a bound raised on one side *and* the other; a
+shipped playbook that stops matching *and* a clean tree passing. The
+`--self-test` flag exists so the same proof runs in CI as part of the gate
+rather than only here.
+
+The last test is about the gate's own footing: it must refuse to grade a tree
+it cannot read rather than print a confident OK about files it never opened.
+"""
+
+from __future__ import annotations
+
+import dataclasses
+import importlib.util
+import json
+import subprocess
+import sys
+from pathlib import Path
+
+import pytest
+
+_REPO = Path(__file__).resolve().parents[1]
+_SCRIPT = _REPO / "scripts" / "check_playbook_schema_parity.py"
+
+pytest.importorskip("jsonschema")
+pytest.importorskip("httpx")
+
+
+def _load_gate():
+ spec = importlib.util.spec_from_file_location("check_playbook_schema_parity", _SCRIPT)
+ module = importlib.util.module_from_spec(spec)
+ sys.modules[spec.name] = module
+ spec.loader.exec_module(module)
+ return module
+
+
[email protected](scope="module")
+def gate():
+ return _load_gate()
+
+
[email protected](scope="module")
+def registries(gate):
+ return gate.collect(gate._find_repo_root(None))
+
+
+# ---------------------------------------------------------------------------
+# The tree is actually clean
+# ---------------------------------------------------------------------------
+
+
+def test_the_repository_passes(registries, gate):
+ assert gate.compare(registries) == []
+
+
+def test_every_shipped_playbook_matches_the_published_schema(gate):
+ errors, scanned = gate.validate_shipped_playbooks(gate._find_repo_root(None))
+ assert errors == []
+ # The lint job used to scan two files and report "2/2 passed" while 62
+ # playbooks under playbooks/packs/v1 went unvalidated, 32 of which did
+ # not match. A count assertion is the cheapest guard against that
+ # regressing into a gate that passes because it looked at nothing.
+ assert len(scanned) > 60, f"only {len(scanned)} playbooks scanned; the glob has probably stopped matching"
+
+
+# ---------------------------------------------------------------------------
+# ...and the gate would notice if it stopped being clean, in each direction
+# ---------------------------------------------------------------------------
+
+
+def test_the_bundled_self_test_passes(registries, gate):
+ """The gate ships its own drift injection. If this fails, one of the
+ directions below has stopped being checked."""
+ assert gate.self_test(registries) == []
+
+
+def _perturb(registries, **kwargs):
+ return dataclasses.replace(registries, **kwargs)
+
+
+def test_catches_a_step_type_the_schema_declares_and_the_engine_lacks(registries, gate):
+ """The `run_playbook` shape: valid on paper, unparseable at runtime."""
+ broken = _perturb(
+ registries,
+ schema_step_types=registries.schema_step_types | {"run_playbook"},
+ execution={**registries.execution, "run_playbook": "unimplemented"},
+ )
+ assert any("does not implement" in e for e in gate.compare(broken))
+
+
+def test_catches_a_step_type_the_engine_accepts_and_the_schema_lacks(registries, gate):
+ """The direction things actually drift, and the one a single-sided check
+ misses: someone adds a verb to StepType and the published contract never
+ hears about it."""
+ broken = _perturb(registries, model_step_types=registries.model_step_types | {"quarantine_mailbox"})
+ assert any("the schema does not declare it" in e for e in gate.compare(broken))
+
+
+def test_catches_an_execution_claim_the_engine_cannot_deliver(registries, gate):
+ """`x-aisoc-execution` is the schema telling an author what will happen.
+ It has to be checked against the handler table or it is just a promise."""
+ running = sorted(k for k, v in registries.execution.items() if v in gate._RUNS)
+ victim = running[0]
+ broken = _perturb(
+ registries,
+ handler_step_types=registries.handler_step_types - {victim},
+ inline_step_types=registries.inline_step_types - {victim},
+ )
+ assert any("the engine has no handler for it" in e for e in gate.compare(broken))
+
+
+def test_catches_the_schema_understating_what_the_engine_does(registries, gate):
+ """The mirror: a handler exists and the annotation says unimplemented."""
+ running = sorted(k for k, v in registries.execution.items() if v in gate._RUNS)
+ broken = _perturb(registries, execution={**registries.execution, running[0]: "unimplemented"})
+ assert any("understating what the product does" in e for e in gate.compare(broken))
+
+
+ ("field", "needle"),
+ [
+ ("schema_timeout_max", "timeout ceiling disagrees"),
+ ("schema_retry_max", "retry ceiling disagrees"),
+ ],
+)
+def test_catches_bound_drift(registries, gate, field: str, needle: str):
+ """A schema ceiling below the engine's rejects playbooks that would run;
+ above it promises headroom Pydantic refuses. Five shipped playbooks
+ declared a 620s timeout against a schema maximum of 600."""
+ broken = _perturb(registries, **{field: getattr(registries, field) + 1})
+ assert any(needle in e for e in gate.compare(broken))
+
+
+def test_catches_trigger_vocabulary_drift_in_both_directions(registries, gate):
+ schema_wider = _perturb(registries, schema_triggers=registries.schema_triggers | {"webhook"})
+ assert any("validate_playbooks.py rejects" in e for e in gate.compare(schema_wider))
+
+ validator_wider = _perturb(registries, validator_triggers=registries.validator_triggers | {"webhook"})
+ assert any("the schema rejects" in e for e in gate.compare(validator_wider))
+
+
+def test_catches_a_second_schema_file(registries, gate):
+ """Two schemas is how the vocabularies diverged to 15 and 9 against an
+ engine with 22, and the drafter silently fell back from one to the other."""
+ broken = _perturb(registries, schema_files=(*registries.schema_files, "playbook.schema.json"))
+ assert any("second source of truth" in e for e in gate.compare(broken))
+
+
+def test_catches_playbook_level_key_drift_in_both_directions(registries, gate):
+ schema_only = _perturb(registries, schema_playbook_keys=registries.schema_playbook_keys | {"ghost_key"})
+ assert any("drops on load" in e for e in gate.compare(schema_only))
+
+ model_only = _perturb(registries, model_playbook_keys=registries.model_playbook_keys | {"ghost_field"})
+ assert any("does not declare" in e for e in gate.compare(model_only))
+
+
+# ---------------------------------------------------------------------------
+# The gate's own footing
+# ---------------------------------------------------------------------------
+
+
+def test_refuses_to_grade_a_tree_it_cannot_read(gate, tmp_path: Path):
+ """Resolving the repo from `__file__` alone is how a gate ends up
+ reporting OK about a tree it never opened. Every marker file must be
+ present or it declines to run."""
+ with pytest.raises(gate.GateError) as exc:
+ gate._find_repo_root(str(tmp_path))
+ assert "does not look like the AiSOC repository" in str(exc.value)
+
+
+def test_refuses_an_empty_step_type_enum(gate, tmp_path: Path):
+ """A future refactor emptying the enum must fail loudly, not compare two
+ empty sets and pass."""
+ with pytest.raises(gate.GateError):
+ gate._schema_step_types({"definitions": {"PlaybookStep": {"properties": {"type": {"enum": []}}}}})
+
+
+def test_refuses_a_schema_with_no_execution_map(gate):
+ with pytest.raises(gate.GateError):
+ gate._schema_execution({})
+
+
+def test_names_what_it_scanned(gate):
+ """A gate that does not say what it read cannot be told apart from one
+ that read nothing."""
+ proc = subprocess.run(
+ [sys.executable, str(_SCRIPT)],
+ capture_output=True,
+ text=True,
+ cwd=str(_REPO),
+ )
+ assert proc.returncode == 0, proc.stdout + proc.stderr
+ for expected in ("repo root", "schema", "step types", "execution", "playbooks scanned"):
+ assert expected in proc.stdout, f"gate output does not report {expected!r}"
+ assert str(_REPO) in proc.stdout
+
+
+def test_the_execution_map_is_not_all_one_value(registries):
+ """If every step type were annotated the same way the map would carry no
+ information, and the handler cross-check would be trivially satisfiable."""
+ assert len(set(registries.execution.values())) > 1
+
+
+def test_the_schema_on_disk_is_valid_json_schema(gate):
+ jsonschema = pytest.importorskip("jsonschema")
+ schema = json.loads((_REPO / "schemas" / "playbook.schema.json").read_text())
+ jsonschema.Draft7Validator.check_schema(schema)