Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
Make the published playbook schema describe the engine, and contract the last ActionType (#807)
Beenu Arora20fd2b3ebd63
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex c5adf2ff6..66e796b2a 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -987,8 +987,19 @@ jobs:git push origin eval-results || echo "push skipped"# ──────────────────────────────────────────────- # Pillar-2: Playbook JSON Schema lint- # Validates all playbook JSON files against schemas/playbook.schema.json+ # Pillar-2: Playbook JSON Schema lint + schema/engine parity+ #+ # The lint validates every *.playbook.json in the tree against+ # schemas/playbook.schema.json. It used to scan only the two files under+ # services/agents/data/playbooks and report "2/2 passed" while the 62+ # playbooks in playbooks/packs/v1 went unchecked — 32 of which did not match.+ #+ # The parity check asks the other question, which nothing asked before: does+ # the published schema describe the engine that runs these files. It compares+ # the schema, the StepType enum, the engine's handler table and the pack+ # validator's trigger list in both directions, and carries a --self-test that+ # injects drift each way so a passing run cannot mean "the comparison is+ # broken".# ──────────────────────────────────────────────p2-playbook-schema-lint:name: P2 — Playbook Schema Lint@@ -998,10 +1009,17 @@ jobs:- uses: actions/setup-python@5fda3b95a4ea91299a34e894583c3862153e4b97 # v7.0.0with:python-version: '3.12'- - name: Install jsonschema- run: pip install jsonschema+ - name: Install deps+ # pydantic + httpx let the parity check import the real engine rather+ # than parse it; a parser can disagree with the handler table that is+ # actually built at import time.+ run: pip install jsonschema pydantic httpx pytest- name: Lint playbook filesrun: python3 scripts/lint_playbooks.py+ - name: Schema/engine parity (both directions, with self-test)+ run: python3 scripts/check_playbook_schema_parity.py --self-test+ - name: Parity gate tests+ run: python3 -m pytest tests/test_playbook_schema_parity_gate.py -q# ──────────────────────────────────────────────# Docker Compose smoke testdiff --git a/CHANGELOG.md b/CHANGELOG.mdindex 4bacee901..dd7e70d06 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -98,6 +98,77 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0or parses empty is a hard error rather than a quiet pass — a gate thatreports OK about a tree it never opened is worse than no gate.+- **The published playbook schema now describes the engine that runs+ playbooks, and a gate keeps it that way in both directions.**+ `schemas/playbook.schema.json` is the contract authors are told to trust,+ and it had drifted from `services/agents/app/playbook/` in every available+ direction at once. Two schema files existed with different step+ vocabularies — 15 types at the repo root, 9 under `schemas/` — against a+ `StepType` enum of 22, and the NL drafter silently fell back from one to+ the other if the primary was missing. Eleven step types were declared by a+ schema and implemented nowhere (`trigger`, `action`, `loop`, `parallel`,+ `human_approval`, `wait`, `isolate`, `block`, `create_case`,+ `run_playbook`, `script`); thirteen were accepted by the engine and+ declared by neither schema. Six step fields (`blast_radius`, `depends_on`,+ `output_key`, `retry.max_attempts`, `retry.backoff_seconds`,+ `retry.backoff_multiplier`) were declared and never read — `blast_radius`+ carried the description "engine enforces analyst approval for destructive+ steps", and the engine could not see the field at all.++ Resolved by making `schemas/playbook.schema.json` the only schema, widened+ to the full `StepType` range with the condition-as-string form the engine+ already accepted, bounds matched to `bounds.py` (3600s / 25 retries, not+ 600s / 5), and the two authored-but-inert playbook keys (`inputs`,+ `dry_run_support`) declared as the documentation they are. The root+ duplicate is deleted. The schema also carries `x-aisoc-execution`, a+ machine-checked map recording whether each step type is `executed`,+ `simulated`, or vocabulary with no handler — so an author can tell what+ will happen before writing the playbook rather than after running it.++ `run_playbook` is deliberately **not** implemented as a step, on its own+ merits rather than by inheriting the argument that removed it as an action.+ A nested playbook's steps are not visible where the parent declares its+ step-level policy, so the parent cannot bound them; the engine reads no+ step-level approval or blast-radius field today, so nesting would let one+ ungated parent pull in an arbitrary tree; and twelve of the engine's own+ step types have no handler, so a verb whose purpose is to execute more+ steps would multiply that. It can return when playbook steps are graded+ individually — recursion depth and an ancestor set are the easy part.++ `scripts/check_playbook_schema_parity.py` compares the schema enum, the+ `StepType` enum, the engine's handler table, the execution map, the bounds+ module and the pack validator's trigger list — every pair in both+ directions, because the characteristic failure here is a check that asks+ only whether the schema declares something the engine lacks and never the+ reverse, which is the direction things actually drift. It carries a+ `--self-test` that injects drift each way and fails if any goes undetected,+ refuses to run at all on a tree missing its marker files rather than+ printing OK about files it never opened, and names the root, schema, step+ counts and playbook count it inspected.++- **Every `ActionType` now resolves a capability contract.** `notify_slack`+ was the last one without, and it was a naming gap rather than a missing+ capability: the verb is `notify`, it has had a contract throughout, and the+ `SlackNotify` adapter already bridged the two names. Nothing connected a+ lookup *by `ActionType` value* to that bridge, so `approval_gate` found no+ contract and skipped the confidence matrix — a 10%-confidence+ `notify_slack` was approved for auto-execution, and the verb most likely to+ auto-execute was the one graded without reference to confidence. It now+ requires an analyst under the default L1 tier.++ Closed with a one-entry alias (`ACTION_TYPE_CAPABILITY_ALIASES`) rather+ than a rename, because `action_type` is persisted operator intent:+ `remediation_whitelist` (migration 015) stores per-tenant pre-approvals+ keyed `UNIQUE (tenant_id, action_type)`, so renaming the member silently+ orphans every row an operator created for `notify_slack`. It is also a+ documented request field and a member of the `ActionType` union in+ `packages/types`. The retirement condition is recorded rather than left+ open-ended: the map goes when `ActionType` does. `check_action_contract.py`+ gains two directions — every `ActionType` must resolve a contract, and+ every alias must name a real `ActionType`, point at a real contract, not+ shadow a capability of the same name, and describe a bridge some adapter+ actually implements.+- **Competitor product names removed from the docs portal, the benchmark pageand the archived plan subtree, and a CI gate added to keep them out.** AiSOCnames no competitor product, but two published comparison tables and the@@ -519,6 +590,59 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0cover. The one MITRE figure the project does publish — 97.0% in`BenchmarkBand` — is labelled "substrate" and is unchanged.+- **A playbook step the engine could not run reported success.** Twelve of+ the twenty-two `StepType` members had no entry in the engine's handler+ table. The run loop answered those with `{"skipped": true}` and left the+ step's status at its `SUCCESS` default, so a playbook containing them ran+ to `COMPLETED` having done nothing it said it did. `approval` is the one+ that mattered: it is a human decision point, it appears in 14 steps across+ the shipped packs, and it passed on its own — the run continued straight+ into the action an analyst was meant to authorise. An unimplemented step+ type now fails closed with `unimplemented: true` and an error naming the+ verb, is not retried (a missing handler will still be missing next+ attempt), and halts the run under the default `on_failure: abort` while+ still honouring an explicit `continue`. A dry run reports `would_fail` for+ such a step instead of a bare `dry_run: true`.++ `apps/docs/docs/concepts/playbooks.md` had documented the safe behaviour+ all along — "recorded as `SKIPPED` … so unknown actions never silently+ succeed" — and has been corrected to describe what the code now does. The+ same page described a manual approval gate backed by+ `POST /v1/playbook-runs/{id}/approve`, where "the engine pauses on the+ condition until the field flips, then resumes". No such endpoint exists and+ the engine has no pause or resume; that section now says so and points at+ the actions service, which does hold an action for an analyst.++- **The playbook lint job validated two files and reported "2/2 passed".**+ `scripts/lint_playbooks.py` claimed in its own docstring to check "any+ `*.playbook.json` files anywhere in the repo" and only ever scanned the two+ under `services/agents/data/playbooks/`. Pointed at the whole tree, 32 of+ the 62 playbooks in `playbooks/packs/v1/` did not match the published+ schema. It now scans recursively, treats finding no files as a broken scan+ rather than a clean bill of health, and no longer crashes in its own error+ path when a file passed on argv sits outside the repo.++- **20 shipped playbooks carried a duplicated tag.**+ `scripts/generate_playbooks.py` emitted `[category, *tags]` where several+ categories already lead their tag list with the category name, producing+ e.g. `["supply-chain", "supply-chain", "npm", …]`. Fixed in the generator,+ which is what the reproducibility gate diffs, rather than in the 20+ generated files.++- **The NL drafter rewrote steps to make its own output pass lint.** Because+ the schema declared 9 of 22 step types, `_collapse_step_types_for_schema`+ mapped the other 13 onto a "nearest neighbour" — `run_av_scan` and+ `revoke_session` both became `investigate`, `approval` became `condition` —+ keeping the original in `params.original_type`. The playbook that shipped+ said it would investigate when the author had asked to disable an account,+ and an approval gate came out as an ungated branch. The projection and its+ second validation pass are removed; the schema covers the full range, so a+ validation failure is now a real failure. The drafter's system prompt also+ restated the vocabulary by hand and had drifted from it — offering+ `webhook` as a trigger, which no validator in the repo accepts, and capping+ `retry_max` at 5 against a model allowing 25 — and is now generated from+ `StepType` and `bounds.py`.+- **The agent recommended evidence acquisition the platform could notperform.** `capture_forensics` was an `ActionType` with no executoranywhere, and `services/agents/app/agents/investigation_agent.py` proposesdiff --git a/ROADMAP.md b/ROADMAP.mdindex e2d877375..df60431cf 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -56,9 +56,18 @@ The claim-to-gate matrix stands at **108 rows — 99 GATED / 9 PARTIAL / 0 NO GA- [x] Eval harness: 20 synthetic incidents, ≥80% MITRE-tactic accuracy CI gate### Visual SOAR studio-- [x] React Flow playbook editor with full node palette (Trigger, Condition, Action, Loop, Parallel, Human Approval, Wait, Notify)-- [x] DAG playbook engine with retries, idempotency, blast-radius checks-- [x] `playbook.schema.json` (JSON Schema 2020-12) for portability and CI linting+- [x] React Flow playbook editor over the step types the engine implements+ (enrich, investigate, notify, block_ip, isolate_host, create_ticket,+ close_case, http, condition). Loop, Parallel, Wait and Human Approval+ were listed here and exist in neither the palette nor the engine.+- [x] Sequential playbook engine with conditions, branching, retries and+ cycle detection. Not a DAG: there is no `depends_on` and no parallel+ execution. No idempotency keys and no step-level blast-radius check —+ `blast_radius` was declared by a schema the engine could not read, and+ that schema has been removed rather than left as a promise.+- [x] `schemas/playbook.schema.json` (JSON Schema draft-07) for portability+ and CI linting, held to the engine in both directions by+ `scripts/check_playbook_schema_parity.py`- [x] Detection-as-Code: `detections/` directory with Sigma + AiSOC YAML, GitHub Action deploy-on-merge- [x] 12 starter playbook templates- [x] Community playbook marketplace (static index v4.0; publishing flow v4.1)diff --git a/apps/docs/docs/compliance/evidence-pack.md b/apps/docs/docs/compliance/evidence-pack.mdindex 148a9416b..4f7d9c042 100644--- a/apps/docs/docs/compliance/evidence-pack.md+++ b/apps/docs/docs/compliance/evidence-pack.md@@ -33,7 +33,7 @@ and says so.| Prompt-injection resistance | 32-payload adversarial corpus with a recall floor and a false-positive ceiling | `services/agents/tests/adversarial/` || Claim-to-gate traceability | Every product claim mapped to the CI job that fails when it stops being true | `docs/audit/CLAIM_TO_GATE_MATRIX.md` |-The last row is the one worth reading first. It is the honest index: 109 rows+The last row is the one worth reading first. It is the honest index: 112 rows`GATED`, 9 `PARTIAL` with the specific gap named, and a ratchet that refusesto let a row sit at `NO GATE`.diff --git a/apps/docs/docs/concepts/live-actions.md b/apps/docs/docs/concepts/live-actions.mdindex 3e3345a4a..5f92182ee 100644--- a/apps/docs/docs/concepts/live-actions.md+++ b/apps/docs/docs/concepts/live-actions.md@@ -235,15 +235,15 @@ dispatched under governance:| `KNOWN_CAPABILITIES` | The vocabulary everything validates against || `ProposedAction(action_type=...)` | What the agent puts in front of an analyst |-A verb can be complete in three of the four and unreachable, and nothing in+A verb can be complete in four of the five and unreachable, and nothing inthe failure message says which one is missing. `ack_alert` and`suppress_alert` were exactly that for several releases: real executors withSplunk, Elastic and Defender arms, wired into `EXECUTOR_REGISTRY`, and absent-from the other three. Governed dispatch answered `executor_not_found` for code+from the other four. Governed dispatch answered `executor_not_found` for codethat worked, which reads as a misconfigured integration rather than acapability nobody connected.-`scripts/check_action_contract.py` now compares all four, **in both+`scripts/check_action_contract.py` now compares all five, **in bothdirections**, on every pull request. Both directions is the part that matters.The characteristic failure of a drift check is that it compares A against Band never B against A, so drift in the direction things actually change passes@@ -261,7 +261,13 @@ The gate reports:- an adapter pointing at an `ActionType` with no executor, which would fail atexecution rather than at registration;- an `ActionType` with no executor at all;-- a verb the agent proposes by name that no executor implements.+- a verb the agent proposes by name that no executor implements;+- an `ActionType` that resolves no capability contract, so the approval+ matrix is skipped for it and blast radius decides alone;+- an alias in `ACTION_TYPE_CAPABILITY_ALIASES` that points at a capability+ with no contract, keys an `ActionType` that does not exist, shadows a real+ capability of the same name, or claims two names are one verb when no+ adapter pairs them.That last one is the check that spans two services, and it is the one thatmattered most. `investigation_agent` proposes `capture_forensics` whenever an@@ -290,6 +296,34 @@ the `ActionType` surface rather than sit on it dead. The API accepts anymember of the enum, so a member with nothing behind it is a request that isaccepted and then fails with a message that reads like a broken deployment.+### The one alias, and when it goes++`notify_slack` was the last `ActionType` with no capability contract, and it+was never a missing capability: the verb is `notify`, it has had a contract+throughout, and the `SlackNotify` adapter already bridges the two names. What+was missing was anything connecting a lookup *by `ActionType` value* to that+bridge — so `approval_gate` found no contract, skipped the confidence matrix,+and the verb most likely to auto-execute was the one verb graded without+reference to confidence. A 10%-confidence `notify_slack` was approved for+auto-execution; it now needs an analyst under the default tier.++It is closed with an alias rather than a rename. `action_type` is persisted+operator intent: `remediation_whitelist` (migration 015) stores per-tenant+pre-approvals keyed `UNIQUE (tenant_id, action_type)`, so renaming the member+silently orphans every row an operator created for `notify_slack` — their+pre-approval stops matching and nobody is told. It is also a documented+request field and a member of the `ActionType` union in `packages/types`.++A permanent alias nothing ever retires is debt, so the retirement condition+is stated: this map goes when `ActionType` does. The enum is already the+legacy half of the registry — every adapter carries a `_legacy_action_type`+pointing back at it, and governed dispatch works in capability strings. When+`POST /actions` takes a capability and the `remediation_whitelist` rows have+been migrated, the map and the enum go together. Until then a new entry needs+the same justification: a *naming* difference for a verb that already has a+contract, never a stand-in for a capability nobody wrote. The gate enforces+each of those conditions rather than trusting the note.+### Actions that are not finished when they returnMost executors are synchronous: the vendor call returns and the action isdiff --git a/apps/docs/docs/concepts/playbooks.md b/apps/docs/docs/concepts/playbooks.mdindex 681638ac6..c3eb6ae93 100644--- a/apps/docs/docs/concepts/playbooks.md+++ b/apps/docs/docs/concepts/playbooks.md@@ -83,20 +83,51 @@ collapsed into this set in each connector's `normalize()`.## Steps-Each step has a `type` that maps to a handler in-[`engine.py`](https://github.com/beenuar/AiSOC/tree/main/services/agents/app/playbook/engine.py).+Each step has a `type`. The authoring contract is+[`schemas/playbook.schema.json`](https://github.com/beenuar/AiSOC/tree/main/schemas/playbook.schema.json),+and its `x-aisoc-execution` map records what+[`engine.py`](https://github.com/beenuar/AiSOC/tree/main/services/agents/app/playbook/engine.py)+actually does with each one. `scripts/check_playbook_schema_parity.py` holds+the two together in both directions, so the table below cannot quietly stop+being true.++**Executed** — a handler runs and has a real effect:| `type` | What it does ||--------|--------------|| `enrich` | Calls the enrichment service for IOC reputation, geo, ASN, GreyNoise, VT, OTX. || `investigate` | Triggers the AI investigator agent with focus areas (`forensics`, `lateral_movement`, etc.). |-| `notify` | Sends a webhook, Slack, PagerDuty, or email notification. |-| `block_ip` | Calls a firewall/EDR connector to block an IP (currently simulated for safety in OSS builds). |-| `isolate_host` | Calls EDR to isolate a host (simulated by default). |-| `create_ticket` | Opens a ticket in Jira / ServiceNow / Linear via connector. |+| `notify` | Sends a webhook notification. SSRF-guarded. |+| `http` | Generic outbound HTTP request — `method`, `url`, `body`, `headers`. SSRF-guarded. || `close_case` | Marks the AiSOC case as closed via the API service. |-| `http` | Generic outbound HTTP request — `method`, `url`, `body`, `headers`. || `condition` | Pure branching node. Evaluates `condition` and routes to `next_true` / `next_false`. |+| `osquery_live_query` | Distributed osquery via osctrl / FleetDM / aisoc-direct, against an allow-listed template. |++**Simulated** — a handler runs and returns `{"simulated": true, ...}` without+touching a vendor. Wire them to real connectors by editing the handler, or+replace the step with an `http` step that calls your connector's enforcement+endpoint:++| `type` | What it does |+|--------|--------------|+| `block_ip` | Reports the IP it would block. |+| `isolate_host` | Reports the host it would isolate. |+| `create_ticket` | Reports the ticket it would open. |++**Vocabulary without an engine handler** — these parse, validate and are used+by the shipped packs, but no handler is registered, so the engine **fails the+step closed** rather than reporting a success it did not achieve. With the+default `on_failure: abort` that halts the run:++`approval`, `block_ioc`, `disable_user`, `reset_password`, `revoke_session`,+`force_mfa`, `kill_process`, `quarantine_file`, `run_av_scan`, `run_script`,+`search_siem`, `create_notable_event`.++Most of these verbs *do* have working executors in the actions service, which+grades each one against its own capability contract before it runs. What does+not exist yet is a bridge from a playbook step to that governed dispatch, so+today they are authoring vocabulary rather than execution. A run that reaches+one stops and says so.Common step fields:@@ -143,29 +174,47 @@ handling.backoff before falling back to whatever you set as the next-failure mode.- Cycle detection — if the engine revisits the same `step.id`, it aborts with`error: "cycle at step <id>"` rather than looping forever.-- Unknown step type — the step is recorded as `SKIPPED` with- `reason: "no handler for <type>"` so unknown actions never silently succeed.+- Step type with no handler — the step is recorded as `FAILED` with+ `unimplemented: true` and an error naming the verb, so a step the engine+ cannot run never reports success. It is not retried: a handler that is+ missing now will still be missing on the next attempt.++ This page previously said such a step was recorded as `SKIPPED` "so unknown+ actions never silently succeed". The engine did put `skipped: true` in the+ result, but left the step's *status* at `SUCCESS` and the run completed —+ so the documented safety property was the opposite of the behaviour. It now+ does what this section always claimed.Each step result includes `_elapsed_ms` so you can see per-step latency in theUI and in run history.## Approvals and dry-runs-Two patterns are supported today for human-in-the-loop steps:--1. **`dry_run` flag** — pass `dry_run: true` when calling- `PlaybookEngine.run(...)` (or set `dry_run` in step params for individual- destructive actions). Handlers short-circuit with- `{"dry_run": true, "step": <name>}` and emit the same realtime events, so- you can preview an entire run without touching production.-2. **Manual approval gate** — model the gate as a `condition` step backed by a- field that an operator sets via the API (`POST /v1/playbook-runs/{id}/approve`).- The engine pauses on the condition until the field flips, then resumes.--`block_ip`, `isolate_host`, and `create_ticket` are simulated by default in OSS-builds (they return `{"simulated": true, ...}`). Wire them to real connectors-by editing the corresponding handler in `engine.py` or by replacing the step-with an `http` step that calls your connector's enforcement endpoint.+**`dry_run` flag** — pass `dry_run: true` when calling+`PlaybookEngine.run(...)`. Handlers short-circuit with+`{"dry_run": true, "step": <name>}` and emit the same realtime events, so you+can preview an entire run without touching production. A step whose type has+no handler is additionally reported as `unimplemented: true, would_fail: true`,+so a preview tells you which steps a real run would stop on.++**Human-in-the-loop is not implemented.** The engine is a single pass with no+pause or resume, and an `approval` step fails closed (see+[Steps](#steps) above), which halts the run under the default+`on_failure: abort`.++This section previously described a second supported pattern: model the gate+as a `condition` step backed by a field an operator sets via+`POST /v1/playbook-runs/{id}/approve`, and "the engine pauses on the condition+until the field flips, then resumes". No such endpoint exists, and the engine+has no pause or resume — the loop evaluates each condition once against the+run context and moves on. Worse, until recently an `approval` step reported+`SUCCESS` without doing anything, so a playbook that modelled a gate ran+straight through it into whatever it was gating.++If you need an approval gate today, split the playbook: run the+investigative half, and submit the response verb to the actions service+(`POST /actions`), which grades it against its capability contract and holds+it at `awaiting_approval` for an analyst. That path is real and audited.## Realtime eventsdiff --git a/apps/docs/docs/operations/notifications.md b/apps/docs/docs/operations/notifications.mdindex cd84b6624..c214091ca 100644--- a/apps/docs/docs/operations/notifications.md+++ b/apps/docs/docs/operations/notifications.md@@ -122,18 +122,29 @@ Slack tokens **must** come from your secret store (Doppler / Vault / k8s Secret)The `notify_slack` action executor is the simplest possible Slack surface: a one-shot incoming-webhook POST. It does not require the Slack bot to be running.+The playbook step type is `notify` — `notify_slack` is the `action_type` on+the actions API, and the two names are bridged by an alias (see+[Live Actions](../concepts/live-actions)). A step keys its arguments under+`params`, not `parameters`:+```yaml-# playbooks/auto-isolate-on-p0.yamlsteps:- id: notify- type: notify_slack- parameters:- webhook_url: "https://hooks.slack.com/services/..."- channel: "#security-alerts"+ name: Post to the SOC channel+ type: notify+ params:+ channel: webhook+ url: "https://hooks.slack.com/services/..."message: |- AiSOC isolated host {{ context.host }} after P0 alert {{ context.alert_id }}.+ AiSOC isolated host {{ host }} after alert {{ alert_id }}.```+`schemas/playbook.schema.json` is the authoring contract, and+`scripts/check_playbook_schema_parity.py` keeps it matched to the engine in+both directions. Its `x-aisoc-execution` map records what the engine does+with each step type, so you can tell before authoring whether a verb+executes, simulates, or is vocabulary the engine will fail closed on.+Implementation: [`services/actions/app/executors/notification.py`](https://github.com/beenuar/AiSOC/blob/main/services/actions/app/executors/notification.py).Failure mode is non-fatal but visible: if the webhook returns non-2xx the executor returns `ActionStatus.FAILED` with the underlying error string, the playbook step is marked failed, and the rest of the playbook runs as authored (typically with the failure path branching to `escalate_to_human`).diff --git a/apps/web/public/marketplace/index.json b/apps/web/public/marketplace/index.jsonindex e1bd31249..2a3cccfe1 100644--- a/apps/web/public/marketplace/index.json+++ b/apps/web/public/marketplace/index.json@@ -1,7 +1,7 @@{"$schema": "https://example.com/schemas/marketplace/v1.json","version": "1.0.0",- "generated": "2026-09-23T14:14:28Z",+ "generated": "2026-09-24T07:22:47Z","categories": [{"id": "playbooks",@@ -222980,7 +222980,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "bec","bec","identity","conditional-access"@@ -223004,7 +223003,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "bec","bec","email"],@@ -223029,7 +223027,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "bec","bec","email"],@@ -223054,7 +223051,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "bec","bec","identity","token-theft"@@ -223080,7 +223076,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "bec","bec","finance"],@@ -223445,7 +223440,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ddos","ddos","dns"],@@ -223470,7 +223464,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ddos","ddos","waf"],@@ -223495,7 +223488,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ddos","ddos","ato"],@@ -223520,7 +223512,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ddos","ddos","network"],@@ -223545,7 +223536,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ddos","ddos","network"],@@ -224108,7 +224098,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ransomware","ransomware","c2"],@@ -224133,7 +224122,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ransomware","ransomware","fileserver"],@@ -224158,7 +224146,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ransomware","ransomware","endpoint"],@@ -224183,7 +224170,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ransomware","ransomware","exposure"],@@ -224208,7 +224194,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ransomware","ransomware","endpoint"],@@ -224233,7 +224218,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "supply-chain","supply-chain","github"],@@ -224258,7 +224242,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "supply-chain","supply-chain","iac","drift"@@ -224282,7 +224265,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "supply-chain","supply-chain","npm"],@@ -224307,7 +224289,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "supply-chain","supply-chain","pypi"],@@ -224332,7 +224313,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "supply-chain","supply-chain","vendor"],diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.mdindex b6173058b..bcff3ff22 100644--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md@@ -126,10 +126,13 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`| No OSS default resolves to a specific deployment's hostname, so a self-hosted install never presents another deployment's host as its own | self-host contract; `apps/docs/docs/deployment/env-vars.md` (CORS default) | `hosted-hostname.yml` (`scripts/check_hosted_hostname.py` — an exact occurrence count per path, checked in **both** directions: a new or grown occurrence and a stale exemption whose occurrence is gone both fail, so an allow-list entry cannot outlive what justified it; the root is resolved from the working directory rather than `__file__` and verified against sentinels, a zero-match scan is reported as broken rather than clean, and `--self-test` runs first proving the detector separates known-bad from known-good and that the comparison rejects a vacuous pass; it also pins the six vendored `cors.py` copies byte-identical) + `tests/test_hosted_hostname_gate.py` (39 cases, including the ten runtime defaults that leaked and a check that the CORS-tuple extractor itself would notice the origin coming back) | GATED | - || A dry run never reaches a customer's vendor | live-action dry-run contract | `ci.yml` actions job (`test_dry_run_credential_strip.py` — each client factory's read set is re-derived from its own source and compared with the adapter's strip list, stripping that set is asserted to make the factory return `None`, and an end-to-end dry run with full credentials trips a wire if any factory returns a client) | GATED | the wire covers the four SIEM factories the writeback path uses; other vendors' adapters are covered by the strip-list equality check only || Documentation and marketing name no competitor product, while vendors AiSOC integrates with stay named | AGENTS.md (naming rule); docs portal + benchmark comparison tables | `competitor-names.yml` (`scripts/check_competitor_names.py` — an explicit name list plus a path allow-list for integration surfaces, both checked in both directions: a dead pattern, a stale allow entry that no longer excuses anything, and a drifted comparison-table marker each fail rather than reporting a clean tree; the scan prints the absolute root and file count so it cannot report on a tree it never read) + `tests/test_competitor_names_gate.py` (42 cases, including `Torq` asserted as a competitor in prose *and* left untouched across its connector module, manifest, docs page and tests) | GATED | the name list is AI-SOC/SOAR rivals; vendors that are purely integration targets are deliberately absent from it and are instead held out of the pinned comparison-table regions, which are checked against a wider vendor list |+| The published playbook schema describes the engine that runs playbooks | concepts/playbooks.md (step table + `x-aisoc-execution`); `schemas/playbook.schema.json` | `ci.yml` P2 job (`scripts/check_playbook_schema_parity.py --self-test` — compares the schema enum, the `StepType` enum, the engine's `_HANDLERS` table, the `x-aisoc-execution` map, `bounds.py` and the pack validator's trigger list, **every pair in both directions**; refuses to run on a tree missing its marker files instead of printing OK about files it never opened, and names the root, schema, step counts and playbook count it inspected) + `scripts/lint_playbooks.py` (every `*.playbook.json` in the tree, not the two it used to scan while reporting "2/2 passed") + `tests/test_playbook_schema_parity_gate.py` (18 cases; drift injected in each direction and the gate asserted to name it) | GATED | twelve step types are declared vocabulary with no engine handler; the schema says so per type and the engine fails them closed, so the gap is published rather than hidden. Closes when playbook steps dispatch through the actions service's per-capability contract |+| A step a playbook engine cannot run never reports that it ran | concepts/playbooks.md (error handling) | `ci.yml` agents job (`test_playbook_engine_correctness.py::TestUnimplementedStepTypesFailClosed` — an `approval` gate the engine cannot honour halts the run instead of passing it, every handler-less `StepType` is asserted to fail rather than succeed, a missing handler is not retried, an explicit `on_failure: continue` is still honoured, and a dry run reports `would_fail` rather than a bare `dry_run: true`) | GATED | - |+| Every `ActionType` resolves a capability contract, so the confidence matrix is never skipped | concepts/live-actions.md (the one alias, and when it goes) | `ci.yml` actions job (`scripts/check_action_contract.py` directions 7 and 8 — every `ActionType` must resolve a contract by name or alias, and every alias must key a real `ActionType`, point at a real contract, not shadow a capability of the same name, and describe a bridge some adapter actually implements) + `test_approval_gate_wiring.py` (`notify_slack` at 10% confidence was approved for auto-execution and now requires an analyst; the alias is asserted unable to *lower* a requirement) | GATED | one alias entry (`notify_slack` -> `notify`), with its retirement condition recorded: it goes when `ActionType` does |## Summary-- GATED: 108+- GATED: 112- PARTIAL: 9- NO GATE: 0 (**every claim is backed by a failing test.** The last NO GATE — the weekly benchmark scoreboard running live against `main` — closed in Phase E1: `scripts/check_scoreboard.py` ties the published scoreboard to a deterministic per-PR live-agent MITRE-accuracy run, while the funded weekly `wet-eval.yml` appends the LLM-tier rows. The remaining PARTIAL rows are honest, named deferrals — each states the specific gap and what closes it — not unproven claims.)diff --git a/docs/upgrade/MIGRATION.md b/docs/upgrade/MIGRATION.mdindex 113d79703..245501c8d 100644--- a/docs/upgrade/MIGRATION.md+++ b/docs/upgrade/MIGRATION.md@@ -28,7 +28,7 @@ Read it before running `docker compose pull` or deploying to production.| Area | v3 | v4 ||------|----|----|| Core engine | Rule-based SOAR engine | LangGraph multi-agent investigator |-| Playbook format | Custom JSON (no schema) | JSON Schema 2020-12 (`playbook.schema.json`) |+| Playbook format | Custom JSON (no schema) | JSON Schema draft-07 (`schemas/playbook.schema.json`) || Plugin manifest | `aisoc-plugin.json` only | `plugin.yaml` (preferred) + `aisoc-plugin.json` (legacy) || Plugin types | `enricher`, `action`, `connector` | + `responder`, `detection`, `widget` || Plugin distribution | Local directory only | Local directory + OCI images (via `oras`) |@@ -171,7 +171,7 @@ config_schema:### Step 5 — Migrate playbooks-Playbooks are now validated against `playbook.schema.json` (JSON Schema 2020-12).+Playbooks are now validated against `schemas/playbook.schema.json` (JSON Schema draft-07).**Check your existing playbooks:**diff --git a/marketplace/index.json b/marketplace/index.jsonindex e1bd31249..2a3cccfe1 100644--- a/marketplace/index.json+++ b/marketplace/index.json@@ -1,7 +1,7 @@{"$schema": "https://example.com/schemas/marketplace/v1.json","version": "1.0.0",- "generated": "2026-09-23T14:14:28Z",+ "generated": "2026-09-24T07:22:47Z","categories": [{"id": "playbooks",@@ -222980,7 +222980,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "bec","bec","identity","conditional-access"@@ -223004,7 +223003,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "bec","bec","email"],@@ -223029,7 +223027,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "bec","bec","email"],@@ -223054,7 +223051,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "bec","bec","identity","token-theft"@@ -223080,7 +223076,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "bec","bec","finance"],@@ -223445,7 +223440,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ddos","ddos","dns"],@@ -223470,7 +223464,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ddos","ddos","waf"],@@ -223495,7 +223488,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ddos","ddos","ato"],@@ -223520,7 +223512,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ddos","ddos","network"],@@ -223545,7 +223536,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ddos","ddos","network"],@@ -224108,7 +224098,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ransomware","ransomware","c2"],@@ -224133,7 +224122,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ransomware","ransomware","fileserver"],@@ -224158,7 +224146,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ransomware","ransomware","endpoint"],@@ -224183,7 +224170,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ransomware","ransomware","exposure"],@@ -224208,7 +224194,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "ransomware","ransomware","endpoint"],@@ -224233,7 +224218,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "supply-chain","supply-chain","github"],@@ -224258,7 +224242,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "supply-chain","supply-chain","iac","drift"@@ -224282,7 +224265,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "supply-chain","supply-chain","npm"],@@ -224307,7 +224289,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "supply-chain","supply-chain","pypi"],@@ -224332,7 +224313,6 @@"version": "1.0.0","author": "AiSOC","tags": [- "supply-chain","supply-chain","vendor"],diff --git a/playbook.schema.json b/playbook.schema.jsondeleted file mode 100644index c396d19a5..000000000--- a/playbook.schema.json+++ /dev/null@@ -1,249 +0,0 @@-{- "$schema": "https://json-schema.org/draft/2020-12/schema",- "$id": "https://github.com/beenuar/aisoc/blob/main/playbook.schema.json",- "title": "AiSOC Playbook",- "description": "JSON Schema 2020-12 for AiSOC playbook definitions (Pillar 2, v4.0). Portable across file format (YAML / JSON) and CI-lintable.",- "type": "object",- "required": ["id", "name", "version", "trigger", "steps"],- "additionalProperties": false,- "properties": {- "id": {- "type": "string",- "pattern": "^[a-z0-9][a-z0-9-]{2,62}$",- "description": "Stable, globally-unique kebab-case identifier."- },- "name": { "type": "string", "minLength": 3, "maxLength": 120 },- "description": { "type": "string", "maxLength": 1000 },- "version": {- "type": "string",- "pattern": "^\\d+\\.\\d+\\.\\d+$",- "description": "Semantic version string."- },- "author": { "type": "string" },- "tags": {- "type": "array",- "items": { "type": "string", "pattern": "^[a-z0-9-]+$" },- "uniqueItems": true- },- "enabled": { "type": "boolean", "default": true },- "trigger": {- "type": "object",- "required": ["on"],- "additionalProperties": false,- "properties": {- "on": {- "type": "string",- "enum": ["alert", "case", "schedule", "manual", "webhook"]- },- "severity": {- "type": "array",- "items": { "type": "string", "enum": ["info", "low", "medium", "high", "critical"] }- },- "alert_types": { "type": "array", "items": { "type": "string" } },- "cron": {- "type": "string",- "description": "Cron expression; only valid when on=schedule."- },- "filters": {- "type": "object",- "description": "Key/value filters applied to the trigger event.",- "additionalProperties": { "type": "string" }- }- }- },- "steps": {- "type": "array",- "minItems": 1,- "items": { "$ref": "#/$defs/step" }- },- "on_failure": { "$ref": "#/$defs/failure_policy" },- "timeout_seconds": {- "type": "integer",- "minimum": 1,- "maximum": 86400,- "default": 3600- },- "metadata": {- "type": "object",- "description": "Arbitrary key/value metadata; not processed by the engine.",- "additionalProperties": true- }- },-- "$defs": {- "step": {- "type": "object",- "required": ["id", "name", "type"],- "additionalProperties": false,- "properties": {- "id": {- "type": "string",- "pattern": "^[a-z0-9][a-z0-9-]{1,62}$"- },- "name": { "type": "string", "minLength": 1, "maxLength": 120 },- "type": {- "type": "string",- "enum": [- "trigger",- "condition",- "action",- "loop",- "parallel",- "human_approval",- "wait",- "notify",- "enrich",- "isolate",- "block",- "close_case",- "create_case",- "run_playbook",- "script"- ]- },- "description": { "type": "string" },- "params": {- "type": "object",- "description": "Step-type-specific parameters; validated at runtime.",- "additionalProperties": true- },- "condition": { "$ref": "#/$defs/condition_expr" },- "on_failure": { "$ref": "#/$defs/failure_policy" },- "retry": { "$ref": "#/$defs/retry_policy" },- "timeout_seconds": { "type": "integer", "minimum": 1, "maximum": 3600 },- "depends_on": {- "type": "array",- "items": { "type": "string" },- "description": "Step IDs that must complete before this step starts (DAG edges)."- },- "blast_radius": {- "type": "string",- "enum": ["dry_run", "reversible", "destructive"],- "default": "dry_run",- "description": "Declared impact level; engine enforces analyst approval for destructive steps."- },- "output_key": {- "type": "string",- "description": "Variable name under which this step's output is stored in the run context."- }- }- },-- "condition_expr": {- "oneOf": [- {- "type": "object",- "required": ["field", "op", "value"],- "additionalProperties": false,- "properties": {- "field": { "type": "string" },- "op": {- "type": "string",- "enum": ["eq", "ne", "lt", "lte", "gt", "gte", "contains", "startswith", "endswith", "in", "not_in", "exists", "not_exists"]- },- "value": {}- }- },- {- "type": "object",- "required": ["and"],- "additionalProperties": false,- "properties": {- "and": { "type": "array", "items": { "$ref": "#/$defs/condition_expr" }, "minItems": 2 }- }- },- {- "type": "object",- "required": ["or"],- "additionalProperties": false,- "properties": {- "or": { "type": "array", "items": { "$ref": "#/$defs/condition_expr" }, "minItems": 2 }- }- },- {- "type": "object",- "required": ["not"],- "additionalProperties": false,- "properties": {- "not": { "$ref": "#/$defs/condition_expr" }- }- }- ]- },-- "retry_policy": {- "type": "object",- "additionalProperties": false,- "properties": {- "max_attempts": { "type": "integer", "minimum": 1, "maximum": 10, "default": 3 },- "backoff_seconds": { "type": "number", "minimum": 0, "default": 2 },- "backoff_multiplier": { "type": "number", "minimum": 1, "default": 2 },- "retry_on": {- "type": "array",- "items": { "type": "string" },- "description": "Exception class names that trigger a retry; empty = retry on any error."- }- }- },-- "failure_policy": {- "type": "object",- "additionalProperties": false,- "properties": {- "action": {- "type": "string",- "enum": ["abort", "continue", "fallback"],- "default": "abort"- },- "fallback_step": {- "type": "string",- "description": "Step ID to jump to when action=fallback."- },- "notify": {- "type": "object",- "additionalProperties": false,- "properties": {- "channel": { "type": "string" },- "message": { "type": "string" }- }- }- }- }- },-- "examples": [- {- "id": "phishing-triage-v1",- "name": "Phishing Email Triage",- "description": "Enrich sender, notify SOC, auto-close low-risk.",- "version": "1.0.0",- "author": "AiSOC Core Team",- "tags": ["phishing", "email", "triage"],- "trigger": { "on": "alert", "severity": ["low", "medium"] },- "steps": [- {- "id": "enrich-ip",- "name": "Enrich sender IP",- "type": "enrich",- "params": { "ioc_type": "ip", "providers": ["virustotal", "greynoise"] },- "blast_radius": "dry_run"- },- {- "id": "notify-soc",- "name": "Notify SOC channel",- "type": "notify",- "params": { "channel": "webhook", "template": "alert_summary" },- "depends_on": ["enrich-ip"]- },- {- "id": "close-if-clean",- "name": "Auto-close clean alerts",- "type": "close_case",- "condition": { "field": "enrich-ip.malicious_score", "op": "lte", "value": 20 },- "depends_on": ["notify-soc"],- "blast_radius": "reversible"- }- ]- }- ]-}diff --git a/playbooks/packs/v1/bec/bec-conditional-access-bypass.playbook.json b/playbooks/packs/v1/bec/bec-conditional-access-bypass.playbook.jsonindex 6f337075f..f07c10572 100644--- a/playbooks/packs/v1/bec/bec-conditional-access-bypass.playbook.json+++ b/playbooks/packs/v1/bec/bec-conditional-access-bypass.playbook.json@@ -4,7 +4,6 @@"description": "Triggered when a BEC-related sign-in succeeds from a region that should be blocked by Conditional Access. Restores policy.","version": "1.0.0","tags": [- "bec","bec","identity","conditional-access"diff --git a/playbooks/packs/v1/bec/bec-impersonation-domain.playbook.json b/playbooks/packs/v1/bec/bec-impersonation-domain.playbook.jsonindex d4865246f..0a18b4c2f 100644--- a/playbooks/packs/v1/bec/bec-impersonation-domain.playbook.json+++ b/playbooks/packs/v1/bec/bec-impersonation-domain.playbook.json@@ -4,7 +4,6 @@"description": "Triggered when an inbound email originates from a domain that look-alikes an executive or vendor. Quarantines and blocks.","version": "1.0.0","tags": [- "bec","bec","email","mitre.t1656"diff --git a/playbooks/packs/v1/bec/bec-inbox-rule-malicious.playbook.json b/playbooks/packs/v1/bec/bec-inbox-rule-malicious.playbook.jsonindex 3e3774a0b..3559b8668 100644--- a/playbooks/packs/v1/bec/bec-inbox-rule-malicious.playbook.json+++ b/playbooks/packs/v1/bec/bec-inbox-rule-malicious.playbook.json@@ -4,7 +4,6 @@"description": "Triggered when a forwarding/filter rule is created that hides vendor or finance keywords. Removes the rule and resets creds.","version": "1.0.0","tags": [- "bec","bec","email","mitre.t1564.008"diff --git a/playbooks/packs/v1/bec/bec-token-theft.playbook.json b/playbooks/packs/v1/bec/bec-token-theft.playbook.jsonindex c774ea207..5f1d4c767 100644--- a/playbooks/packs/v1/bec/bec-token-theft.playbook.json+++ b/playbooks/packs/v1/bec/bec-token-theft.playbook.json@@ -4,7 +4,6 @@"description": "Triggered when a refresh token is observed from a new ASN with no interactive sign-in. Revokes refresh tokens and forces MFA.","version": "1.0.0","tags": [- "bec","bec","identity","token-theft",diff --git a/playbooks/packs/v1/bec/bec-vendor-payment-redirect.playbook.json b/playbooks/packs/v1/bec/bec-vendor-payment-redirect.playbook.jsonindex 912cbb424..a2fdbc26d 100644--- a/playbooks/packs/v1/bec/bec-vendor-payment-redirect.playbook.json+++ b/playbooks/packs/v1/bec/bec-vendor-payment-redirect.playbook.json@@ -4,7 +4,6 @@"description": "Triggered when a vendor banking detail change happens via email. Freezes pending payments and pages CFO.","version": "1.0.0","tags": [- "bec","bec","finance","mitre.t1565.001"diff --git a/playbooks/packs/v1/ddos/ddos-amplification-dns.playbook.json b/playbooks/packs/v1/ddos/ddos-amplification-dns.playbook.jsonindex 8112a786e..aedc32ca7 100644--- a/playbooks/packs/v1/ddos/ddos-amplification-dns.playbook.json+++ b/playbooks/packs/v1/ddos/ddos-amplification-dns.playbook.json@@ -4,7 +4,6 @@"description": "Triggered when an inbound spike of crafted DNS queries hits an authoritative server. Sinkholes spoofed sources.","version": "1.0.0","tags": [- "ddos","ddos","dns","mitre.t1498.002"diff --git a/playbooks/packs/v1/ddos/ddos-app-layer-l7.playbook.json b/playbooks/packs/v1/ddos/ddos-app-layer-l7.playbook.jsonindex a7612e1ad..9a58a6a2e 100644--- a/playbooks/packs/v1/ddos/ddos-app-layer-l7.playbook.json+++ b/playbooks/packs/v1/ddos/ddos-app-layer-l7.playbook.json@@ -4,7 +4,6 @@"description": "Triggered by anomalous L7 request rate or pattern. Engages WAF rate-limit rule and challenge-page.","version": "1.0.0","tags": [- "ddos","ddos","waf","mitre.t1498.001"diff --git a/playbooks/packs/v1/ddos/ddos-credential-stuffing-fraud.playbook.json b/playbooks/packs/v1/ddos/ddos-credential-stuffing-fraud.playbook.jsonindex 287c81131..3b417e578 100644--- a/playbooks/packs/v1/ddos/ddos-credential-stuffing-fraud.playbook.json+++ b/playbooks/packs/v1/ddos/ddos-credential-stuffing-fraud.playbook.json@@ -4,7 +4,6 @@"description": "Triggered when /login takes a sustained spike of failed attempts. Combines DDoS + credential-stuffing response: WAF challenge, rate-limit, source-IP block.","version": "1.0.0","tags": [- "ddos","ddos","ato","mitre.t1110.004"diff --git a/playbooks/packs/v1/ddos/ddos-syn-flood.playbook.json b/playbooks/packs/v1/ddos/ddos-syn-flood.playbook.jsonindex 5e1d517ca..fc0ff3843 100644--- a/playbooks/packs/v1/ddos/ddos-syn-flood.playbook.json+++ b/playbooks/packs/v1/ddos/ddos-syn-flood.playbook.json@@ -4,7 +4,6 @@"description": "Triggered by SYN flood pattern at edge. Enables SYN cookies and increases backend capacity.","version": "1.0.0","tags": [- "ddos","ddos","network","mitre.t1498"diff --git a/playbooks/packs/v1/ddos/ddos-volumetric-l3.playbook.json b/playbooks/packs/v1/ddos/ddos-volumetric-l3.playbook.jsonindex 2995bef38..050caa724 100644--- a/playbooks/packs/v1/ddos/ddos-volumetric-l3.playbook.json+++ b/playbooks/packs/v1/ddos/ddos-volumetric-l3.playbook.json@@ -4,7 +4,6 @@"description": "Triggered by volumetric flood detection at edge. Engages scrubbing provider and pages on-call.","version": "1.0.0","tags": [- "ddos","ddos","network","mitre.t1498"diff --git a/playbooks/packs/v1/ransomware/ransomware-c2-block.playbook.json b/playbooks/packs/v1/ransomware/ransomware-c2-block.playbook.jsonindex 9a3881391..d508802e9 100644--- a/playbooks/packs/v1/ransomware/ransomware-c2-block.playbook.json+++ b/playbooks/packs/v1/ransomware/ransomware-c2-block.playbook.json@@ -4,7 +4,6 @@"description": "Triggered by C2 beaconing detections that match ransomware-family TTPs. Blocks C2 destinations at edge, isolates source host.","version": "1.0.0","tags": [- "ransomware","ransomware","c2","mitre.t1071"diff --git a/playbooks/packs/v1/ransomware/ransomware-fileserver-encrypt.playbook.json b/playbooks/packs/v1/ransomware/ransomware-fileserver-encrypt.playbook.jsonindex 7f7b4657c..ecf5079a5 100644--- a/playbooks/packs/v1/ransomware/ransomware-fileserver-encrypt.playbook.json+++ b/playbooks/packs/v1/ransomware/ransomware-fileserver-encrypt.playbook.json@@ -4,7 +4,6 @@"description": "Triggered by mass-rename / mass-extension-change on a file server. Network-segments the host to halt SMB lateral encryption.","version": "1.0.0","tags": [- "ransomware","ransomware","fileserver","mitre.t1486"diff --git a/playbooks/packs/v1/ransomware/ransomware-host-isolate.playbook.json b/playbooks/packs/v1/ransomware/ransomware-host-isolate.playbook.jsonindex eb9414a7c..4e0b01704 100644--- a/playbooks/packs/v1/ransomware/ransomware-host-isolate.playbook.json+++ b/playbooks/packs/v1/ransomware/ransomware-host-isolate.playbook.json@@ -4,7 +4,6 @@"description": "Immediate host isolation, disk snapshot for forensics, page on-call. Standard first-response for endpoint ransomware.","version": "1.0.0","tags": [- "ransomware","ransomware","endpoint","mitre.t1486"diff --git a/playbooks/packs/v1/ransomware/ransomware-public-exposure.playbook.json b/playbooks/packs/v1/ransomware/ransomware-public-exposure.playbook.jsonindex 763c55414..6bb6c9b56 100644--- a/playbooks/packs/v1/ransomware/ransomware-public-exposure.playbook.json+++ b/playbooks/packs/v1/ransomware/ransomware-public-exposure.playbook.json@@ -4,7 +4,6 @@"description": "Triggered when a host with public exposure shows ransomware indicators. Pulls the asset off the internet immediately.","version": "1.0.0","tags": [- "ransomware","ransomware","exposure","mitre.t1486"diff --git a/playbooks/packs/v1/ransomware/ransomware-shadow-copy-deletion.playbook.json b/playbooks/packs/v1/ransomware/ransomware-shadow-copy-deletion.playbook.jsonindex 83c3aba3b..a1a18d8b9 100644--- a/playbooks/packs/v1/ransomware/ransomware-shadow-copy-deletion.playbook.json+++ b/playbooks/packs/v1/ransomware/ransomware-shadow-copy-deletion.playbook.json@@ -4,7 +4,6 @@"description": "Triggered on `vssadmin delete shadows` or wmic shadowcopy delete. Pre-encryption signal \u2014 isolate fast and verify backups.","version": "1.0.0","tags": [- "ransomware","ransomware","endpoint","mitre.t1490"diff --git a/playbooks/packs/v1/supply-chain/supply-github-action-compromise.playbook.json b/playbooks/packs/v1/supply-chain/supply-github-action-compromise.playbook.jsonindex 3f22beb22..a6e36c86c 100644--- a/playbooks/packs/v1/supply-chain/supply-github-action-compromise.playbook.json+++ b/playbooks/packs/v1/supply-chain/supply-github-action-compromise.playbook.json@@ -4,7 +4,6 @@"description": "Triggered when a referenced GitHub Action SHA changes unexpectedly or matches a compromise advisory. Pin to last-known-good and audit.","version": "1.0.0","tags": [- "supply-chain","supply-chain","github","mitre.t1195.002"diff --git a/playbooks/packs/v1/supply-chain/supply-iac-drift.playbook.json b/playbooks/packs/v1/supply-chain/supply-iac-drift.playbook.jsonindex 8c878a129..2f82707d8 100644--- a/playbooks/packs/v1/supply-chain/supply-iac-drift.playbook.json+++ b/playbooks/packs/v1/supply-chain/supply-iac-drift.playbook.json@@ -4,7 +4,6 @@"description": "Triggered when production infra deviates from IaC source of truth in a security-relevant way. Reverts to declared state.","version": "1.0.0","tags": [- "supply-chain","supply-chain","iac","drift"diff --git a/playbooks/packs/v1/supply-chain/supply-malicious-npm.playbook.json b/playbooks/packs/v1/supply-chain/supply-malicious-npm.playbook.jsonindex ed7487f77..e00d494e2 100644--- a/playbooks/packs/v1/supply-chain/supply-malicious-npm.playbook.json+++ b/playbooks/packs/v1/supply-chain/supply-malicious-npm.playbook.json@@ -4,7 +4,6 @@"description": "Triggered when a published npm package matches a known-bad list. Blocks installs at registry-mirror, audits org installs.","version": "1.0.0","tags": [- "supply-chain","supply-chain","npm","mitre.t1195"diff --git a/playbooks/packs/v1/supply-chain/supply-pypi-typosquat.playbook.json b/playbooks/packs/v1/supply-chain/supply-pypi-typosquat.playbook.jsonindex 6eb0bc12e..745c07b30 100644--- a/playbooks/packs/v1/supply-chain/supply-pypi-typosquat.playbook.json+++ b/playbooks/packs/v1/supply-chain/supply-pypi-typosquat.playbook.json@@ -4,7 +4,6 @@"description": "Triggered when a typosquat package is observed in CI installs. Pulls install lock to canonical name and audits.","version": "1.0.0","tags": [- "supply-chain","supply-chain","pypi","mitre.t1195"diff --git a/playbooks/packs/v1/supply-chain/supply-vendor-breach.playbook.json b/playbooks/packs/v1/supply-chain/supply-vendor-breach.playbook.jsonindex 5f6793c06..ce4c34760 100644--- a/playbooks/packs/v1/supply-chain/supply-vendor-breach.playbook.json+++ b/playbooks/packs/v1/supply-chain/supply-vendor-breach.playbook.json@@ -4,7 +4,6 @@"description": "Triggered when a vendor publishes a breach advisory matching internal vendor inventory. Blocks vendor IPs, opens procurement review.","version": "1.0.0","tags": [- "supply-chain","supply-chain","vendor","mitre.t1199"diff --git a/schemas/playbook.schema.json b/schemas/playbook.schema.jsonindex 8ccfed06e..18e19c3fe 100644--- a/schemas/playbook.schema.json+++ b/schemas/playbook.schema.json@@ -2,10 +2,37 @@"$schema": "http://json-schema.org/draft-07/schema#","$id": "https://github.com/beenuar/AiSOC/schemas/playbook.schema.json","title": "AiSOC Playbook",- "description": "Schema for AiSOC v4 Pillar-2 playbook definitions.",+ "description": "The authoring contract for AiSOC playbooks. Every step type listed in `properties.steps.items.properties.type.enum` is accepted by the runtime model (`services/agents/app/playbook/models.py::StepType`), and `x-aisoc-execution` records what the engine actually does with it. Kept in lockstep with the engine by `scripts/check_playbook_schema_parity.py`.","type": "object","required": ["name", "steps"],"additionalProperties": false,++ "x-aisoc-execution": {+ "description": "What `services/agents/app/playbook/engine.py` does with each step type. `executed` = a handler runs and has a real effect (an outbound call to a service, or engine-internal control flow). `simulated` = a handler runs and returns `simulated: true` without touching a vendor. `unimplemented` = no handler is registered; the engine fails the step closed rather than reporting a success it did not achieve. This map is machine-checked against the engine's handler table, so it cannot drift into an aspirational claim.",+ "enrich": "executed",+ "investigate": "executed",+ "notify": "executed",+ "http": "executed",+ "close_case": "executed",+ "condition": "executed",+ "osquery_live_query": "executed",+ "block_ip": "simulated",+ "isolate_host": "simulated",+ "create_ticket": "simulated",+ "approval": "unimplemented",+ "block_ioc": "unimplemented",+ "disable_user": "unimplemented",+ "reset_password": "unimplemented",+ "revoke_session": "unimplemented",+ "force_mfa": "unimplemented",+ "kill_process": "unimplemented",+ "quarantine_file": "unimplemented",+ "run_av_scan": "unimplemented",+ "run_script": "unimplemented",+ "search_siem": "unimplemented",+ "create_notable_event": "unimplemented"+ },+"properties": {"id": {"type": "string",@@ -79,12 +106,33 @@"updated_at": {"type": "string","format": "date-time"+ },+ "inputs": {+ "type": "object",+ "description": "Authored documentation of the trigger-context fields this playbook expects. The engine does NOT bind, inject or validate these — a step reads its own values from `params` and from the run context. Present so a pack can describe its interface to a reader; do not treat it as a parameter-binding mechanism.",+ "additionalProperties": {+ "type": "object",+ "properties": {+ "description": { "type": "string" },+ "field": {+ "type": "string",+ "description": "Dot-path on the trigger event where this input is expected to be found."+ },+ "required": { "type": "boolean" }+ },+ "additionalProperties": true+ }+ },+ "dry_run_support": {+ "type": "boolean",+ "description": "Authored assertion that this playbook is safe to run with the engine's `dry_run` flag. The engine's `dry_run` applies uniformly to every step regardless of this value, so it is catalogue metadata rather than a runtime switch."}},+"definitions": {"StepCondition": {"type": "object",- "required": ["field", "operator"],+ "description": "Structured condition form. Every field has a runtime default, so none is required; an empty object is a condition that always passes.","additionalProperties": false,"properties": {"field": {@@ -97,6 +145,10 @@},"value": {"description": "Expected value (not required for 'exists' operator)."+ },+ "expression": {+ "type": "string",+ "description": "Expression form, e.g. \"inputs.source_ip != null\". Mutually exclusive with field/operator/value; evaluated at runtime by the engine."}}},@@ -115,16 +167,30 @@},"type": {"type": "string",+ "description": "The step verb. See `x-aisoc-execution` at the root of this schema for what the engine does with each one — a type listed there as `unimplemented` is a valid authoring vocabulary word that the engine will fail closed on rather than execute.","enum": ["enrich","investigate","notify",+ "http",+ "close_case",+ "condition",+ "osquery_live_query","block_ip","isolate_host","create_ticket",- "close_case",- "http",- "condition"+ "approval",+ "block_ioc",+ "disable_user",+ "reset_password",+ "revoke_session",+ "force_mfa",+ "kill_process",+ "quarantine_file",+ "run_av_scan",+ "run_script",+ "search_siem",+ "create_notable_event"]},"params": {@@ -133,7 +199,14 @@"additionalProperties": true},"condition": {- "$ref": "#/definitions/StepCondition"+ "description": "Guard that must pass before the step runs. Accepts either an expression string or the structured form; the engine coerces a bare string into `{\"expression\": ...}`.",+ "oneOf": [+ {+ "type": "string",+ "description": "Expression form, e.g. \"steps.approval_isolate.outcome == 'approved'\"."+ },+ { "$ref": "#/definitions/StepCondition" }+ ]},"on_failure": {"type": "string",@@ -143,14 +216,16 @@"retry_max": {"type": "integer","minimum": 0,- "maximum": 5,- "default": 0+ "maximum": 25,+ "default": 0,+ "description": "Retry attempts. Ceiling mirrors `bounds.ABSOLUTE_MAX_RETRIES`."},"timeout_seconds": {"type": "integer","minimum": 1,- "maximum": 600,- "default": 30+ "maximum": 3600,+ "default": 30,+ "description": "Per-step timeout. Ceiling mirrors `bounds.ABSOLUTE_MAX_TIMEOUT_SECONDS`; long values exist for `approval` steps that wait on a human. Note that handlers reading `params.timeout_seconds` are separately clamped to `bounds.ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS`."},"next_true": {"type": "string",diff --git a/scripts/check_action_contract.py b/scripts/check_action_contract.pyindex 92715e30a..11e31a243 100644--- a/scripts/check_action_contract.py+++ b/scripts/check_action_contract.py@@ -466,6 +466,70 @@ def check_capability_reachability() -> list[str]:f"deployment rather than a verb that was never built.")+ # ── Direction 7: ActionType → capability contract ──────────────────────+ # An ActionType with no reachable contract is not a cosmetic gap. The+ # submit path looks the contract up *by ActionType value* to run the+ # confidence matrix; a miss means the matrix is skipped and blast radius+ # decides alone, so a 40%-confidence guess is graded like a corroborated+ # finding. That is what happened to `notify_slack` for as long as the+ # capability was called `notify` and nothing connected the two.+ from app.live_actions.capability_contracts import (+ ACTION_TYPE_CAPABILITY_ALIASES,+ CAPABILITY_CONTRACTS,+ contract_for_action_type,+ )++ for action_type in sorted(ActionType, key=lambda a: a.value):+ if contract_for_action_type(action_type.value) is None:+ errors.append(+ f"{action_type.value}: no capability contract resolves for this "+ f"ActionType. The approval matrix has no impact to reason about, "+ f"so it is skipped and blast radius decides alone. Give the verb a "+ f"contract, or — if this is only a naming difference for a verb "+ f"that already has one — add it to ACTION_TYPE_CAPABILITY_ALIASES."+ )++ # ── Direction 8: alias map → the rest of the world, both ways ──────────+ # An alias is a claim that two names mean one verb. Three ways it can rot,+ # each checked here, because an alias nothing validates is just a lookup+ # that happens to succeed.+ known_action_type_values = {a.value for a in ActionType}+ adapter_bridges = {+ (getattr(cls, "capability", ""), getattr(cls, "_legacy_action_type", None))+ for cls in adapters+ if getattr(cls, "_legacy_action_type", None) is not None+ }+ for legacy_value, capability in sorted(ACTION_TYPE_CAPABILITY_ALIASES.items()):+ if legacy_value not in known_action_type_values:+ errors.append(+ f"ACTION_TYPE_CAPABILITY_ALIASES maps {legacy_value!r}, which is not an "+ f"ActionType. The alias can never fire; delete it or fix the key."+ )+ if capability not in CAPABILITY_CONTRACTS:+ errors.append(+ f"ACTION_TYPE_CAPABILITY_ALIASES points {legacy_value!r} at capability "+ f"{capability!r}, which has no contract — the alias resolves to nothing."+ )+ # The alias must describe a bridge that actually exists. If no adapter+ # pairs this capability with this ActionType, the two names are not+ # two names for one verb and the alias is asserting something false.+ if not any(cap == capability and lat is not None and lat.value == legacy_value for cap, lat in adapter_bridges):+ errors.append(+ f"ACTION_TYPE_CAPABILITY_ALIASES claims {legacy_value!r} and {capability!r} "+ f"are the same verb, but no adapter pairs them (no executor with "+ f"capability={capability!r} and _legacy_action_type={legacy_value!r}). "+ f"An alias asserting a bridge nobody built is worse than no alias."+ )++ # And the other way: an alias must not shadow a real capability of the+ # same name, which would silently redirect a verb that has its own contract.+ for legacy_value in sorted(ACTION_TYPE_CAPABILITY_ALIASES):+ if legacy_value in CAPABILITY_CONTRACTS:+ errors.append(+ f"{legacy_value!r} has its own capability contract AND an alias entry. "+ f"The alias is dead code at best and a silent redirect at worst; remove it."+ )+errors.extend(_ratchet("KNOWN_UNGOVERNED_EXECUTORS", KNOWN_UNGOVERNED_EXECUTORS, {a.value for a in adapter_action_types}))errors.extend(_ratchet(diff --git a/scripts/check_playbook_schema_parity.py b/scripts/check_playbook_schema_parity.pynew file mode 100644index 000000000..b08dbdd42--- /dev/null+++ b/scripts/check_playbook_schema_parity.py@@ -0,0 +1,515 @@+#!/usr/bin/env python3+"""Gate: the published playbook schema and the engine that runs playbooks+must describe the same product.++``schemas/playbook.schema.json`` is the contract authors are told to trust.+When it drifts from ``services/agents/app/playbook/``, a playbook validates+and then fails at runtime — the worst shape for a schema to have, because the+failure lands on the person who did exactly what the documentation said.++Before this gate the two had drifted in every available direction at once:++ * two schema files existed with different step vocabularies (15 types and+ 9 types) against an engine with 22, and the drafter silently fell back+ from one to the other;+ * 11 step types were declared by a schema and implemented nowhere;+ * 13 step types were accepted by the engine and declared by no schema;+ * 12 of those had no handler and the engine reported them ``SUCCESS``,+ including ``approval`` — a human gate that auto-passed;+ * 32 of 64 shipped playbooks failed the published schema, while the lint+ job printed ``2/2 passed`` because it only ever looked at two files.++Every comparison here runs in **both** directions. A check that only asks+"does the schema declare something the engine lacks" passes forever while+drift accumulates in the direction things actually change — the engine grows+a verb and nobody tells the schema. That is how a sibling gate in this repo+reported "OK" on a YAML file declaring 17 labels against Go code with 28.++Run:+ python3 scripts/check_playbook_schema_parity.py+ python3 scripts/check_playbook_schema_parity.py --self-test+"""++from __future__ import annotations++import argparse+import inspect+import json+import sys+from dataclasses import dataclass, field, replace+from pathlib import Path++#: Playbook-level keys the schema declares that the runtime model does not+#: bind. Both are authored documentation rather than engine inputs, and the+#: schema says so in their ``description``. This list is a named exemption,+#: not a silent pass: anything else appearing in one and not the other is an+#: error. Keep it empty-by-default in spirit — an entry here is a promise+#: that the key is inert, and the schema text must say the same.+INERT_AUTHORED_KEYS: frozenset[str] = frozenset({"inputs", "dry_run_support"})++#: Execution classes usable in the schema's ``x-aisoc-execution`` map.+_RUNS = frozenset({"executed", "simulated"})+_EXECUTION_CLASSES = _RUNS | {"unimplemented"}+++class GateError(RuntimeError):+ """The gate could not inspect what it is supposed to inspect.++ Raised instead of returning "no problems found", because a gate that+ cannot read its inputs and prints OK is worse than no gate at all.+ """+++@dataclass(frozen=True)+class Registries:+ """Every declaration of the playbook vocabulary, read from its source."""++ schema_step_types: frozenset[str]+ model_step_types: frozenset[str]+ handler_step_types: frozenset[str]+ inline_step_types: frozenset[str]+ execution: dict[str, str]+ schema_timeout_max: int+ schema_timeout_min: int+ schema_retry_max: int+ engine_timeout_max: int+ engine_timeout_min: int+ engine_retry_max: int+ schema_triggers: frozenset[str]+ validator_triggers: frozenset[str]+ schema_playbook_keys: frozenset[str]+ model_playbook_keys: frozenset[str]+ schema_files: tuple[str, ...] = ()+ scanned_playbooks: tuple[str, ...] = field(default=())++ @property+ def runnable(self) -> frozenset[str]:+ """Step types the engine can actually act on."""+ return self.handler_step_types | self.inline_step_types+++# ---------------------------------------------------------------------------+# Collection — read each registry from the file that owns it+# ---------------------------------------------------------------------------+++def _find_repo_root(explicit: str | None) -> Path:+ """Resolve the tree to inspect, and prove it is the right one.++ Resolving from ``__file__`` alone is how a gate ends up confidently+ reporting OK about a tree it never opened: copy the script somewhere, or+ run it from an installed package, and it grades whatever happens to sit+ two directories up. Every marker below must exist or we refuse to run.+ """+ root = Path(explicit).resolve() if explicit else Path(__file__).resolve().parent.parent+ markers = (+ Path("schemas/playbook.schema.json"),+ Path("services/agents/app/playbook/engine.py"),+ Path("services/agents/app/playbook/models.py"),+ Path("scripts/validate_playbooks.py"),+ )+ missing = [str(m) for m in markers if not (root / m).exists()]+ if missing:+ raise GateError(+ f"{root} does not look like the AiSOC repository — missing: "+ f"{', '.join(missing)}. Pass --repo-root to point at the tree to check."+ )+ return root+++def _load_schema(root: Path) -> dict:+ return json.loads((root / "schemas" / "playbook.schema.json").read_text())+++def _schema_step_types(schema: dict) -> frozenset[str]:+ try:+ enum = schema["definitions"]["PlaybookStep"]["properties"]["type"]["enum"]+ except (KeyError, TypeError) as exc:+ raise GateError(f"schema step-type enum not found at the expected path: {exc}") from exc+ if not enum:+ raise GateError("schema step-type enum is empty; refusing to compare against nothing")+ return frozenset(enum)+++def _schema_execution(schema: dict) -> dict[str, str]:+ raw = schema.get("x-aisoc-execution")+ if not isinstance(raw, dict):+ raise GateError("schema has no `x-aisoc-execution` map; the engine's behaviour would be undeclared")+ mapping = {k: v for k, v in raw.items() if k != "description"}+ if not mapping:+ raise GateError("`x-aisoc-execution` is empty; refusing to compare against nothing")+ return mapping+++def _schema_step_bounds(schema: dict) -> tuple[int, int, int]:+ try:+ props = schema["definitions"]["PlaybookStep"]["properties"]+ return (+ int(props["timeout_seconds"]["maximum"]),+ int(props["timeout_seconds"]["minimum"]),+ int(props["retry_max"]["maximum"]),+ )+ except (KeyError, TypeError, ValueError) as exc:+ raise GateError(f"schema step bounds not found at the expected path: {exc}") from exc+++def _import_engine(root: Path):+ """Import the real engine module rather than parsing it.++ Parsing would let a handler table that is built at import time disagree+ with what the parser sees. The import is cheap: the engine documents+ "zero external dependencies beyond httpx + stdlib".+ """+ agents = str(root / "services" / "agents")+ if agents not in sys.path:+ sys.path.insert(0, agents)+ try:+ from app.playbook import bounds, engine, models # type: ignore[import-not-found]+ except Exception as exc: # noqa: BLE001 — surfaced, never swallowed+ raise GateError(+ f"could not import the playbook engine from {agents}: {type(exc).__name__}: {exc}. "+ "Install httpx and pydantic, or the gate is grading nothing."+ ) from exc+ return engine, models, bounds+++def _inline_step_types(engine_mod, models_mod) -> frozenset[str]:+ """Step types handled inside the run loop rather than via ``_HANDLERS``.++ ``condition`` never reaches the handler table — it branches in the loop+ body. Reading the source keeps that visible to the gate; if someone+ deletes the branch, the type stops counting as runnable here too.+ """+ try:+ source = inspect.getsource(engine_mod.PlaybookEngine.run)+ except (OSError, TypeError, AttributeError) as exc:+ raise GateError(f"could not read PlaybookEngine.run source to find inline step handling: {exc}") from exc+ found = {st.value for st in models_mod.StepType if f"StepType.{st.name}" in source}+ return frozenset(found)+++def _model_playbook_keys(models_mod) -> frozenset[str]:+ return frozenset(models_mod.Playbook.model_fields.keys())+++def _validator_triggers(root: Path) -> frozenset[str]:+ """``scripts/validate_playbooks.py`` keeps its own trigger allow-list."""+ scripts = str(root / "scripts")+ if scripts not in sys.path:+ sys.path.insert(0, scripts)+ try:+ import validate_playbooks # type: ignore[import-not-found]+ except Exception as exc: # noqa: BLE001+ raise GateError(f"could not import scripts/validate_playbooks.py: {type(exc).__name__}: {exc}") from exc+ triggers = getattr(validate_playbooks, "SUPPORTED_TRIGGERS", None)+ if not triggers:+ raise GateError("validate_playbooks.SUPPORTED_TRIGGERS is missing or empty")+ return frozenset(triggers)+++def _schema_files(root: Path) -> tuple[str, ...]:+ hits = [+ p+ for p in root.rglob("playbook.schema.json")+ if "node_modules" not in p.parts and ".git" not in p.parts and ".venv-gate" not in p.parts+ ]+ return tuple(sorted(str(p.relative_to(root)) for p in hits))+++def _playbook_files(root: Path) -> list[Path]:+ return sorted(p for p in root.rglob("*.playbook.json") if "node_modules" not in p.parts and ".git" not in p.parts)+++def collect(root: Path) -> Registries:+ schema = _load_schema(root)+ engine_mod, models_mod, bounds_mod = _import_engine(root)+ s_tmax, s_tmin, s_rmax = _schema_step_bounds(schema)+ return Registries(+ schema_step_types=_schema_step_types(schema),+ model_step_types=frozenset(st.value for st in models_mod.StepType),+ handler_step_types=frozenset(st.value for st in engine_mod._HANDLERS),+ inline_step_types=_inline_step_types(engine_mod, models_mod),+ execution=_schema_execution(schema),+ schema_timeout_max=s_tmax,+ schema_timeout_min=s_tmin,+ schema_retry_max=s_rmax,+ engine_timeout_max=bounds_mod.ABSOLUTE_MAX_TIMEOUT_SECONDS,+ engine_timeout_min=bounds_mod.MIN_TIMEOUT_SECONDS,+ engine_retry_max=bounds_mod.ABSOLUTE_MAX_RETRIES,+ schema_triggers=frozenset(schema["properties"]["trigger"]["properties"]["on"]["enum"]),+ validator_triggers=_validator_triggers(root),+ schema_playbook_keys=frozenset(schema["properties"].keys()),+ model_playbook_keys=_model_playbook_keys(models_mod),+ schema_files=_schema_files(root),+ )+++# ---------------------------------------------------------------------------+# Comparison — pure, so the self-test can perturb it+# ---------------------------------------------------------------------------+++def compare(reg: Registries) -> list[str]:+ """Every check, in both directions. Returns human-readable failures."""+ errors: list[str] = []++ if not reg.schema_step_types or not reg.model_step_types:+ # Defence against a future refactor making the whole gate vacuous.+ return ["one of the step-type registries is empty; the comparison would pass on nothing"]++ # 1. schema enum <-> StepType, both ways.+ for missing in sorted(reg.schema_step_types - reg.model_step_types):+ errors.append(+ f"schema declares step type {missing!r} that StepType does not implement — "+ f"a playbook using it validates and then fails to parse at runtime. "+ f"Build it or remove it from the schema."+ )+ for missing in sorted(reg.model_step_types - reg.schema_step_types):+ errors.append(+ f"StepType implements {missing!r} and the schema does not declare it — "+ f"an author following the published contract cannot use it, and the "+ f"drafter has to collapse it onto some other verb. Declare it or delete it."+ )++ # 2. execution map <-> schema enum, both ways.+ exec_keys = frozenset(reg.execution)+ for missing in sorted(reg.schema_step_types - exec_keys):+ errors.append(f"step type {missing!r} is declared but `x-aisoc-execution` does not say what the engine does with it")+ for extra in sorted(exec_keys - reg.schema_step_types):+ errors.append(f"`x-aisoc-execution` describes {extra!r}, which the schema does not declare as a step type")++ bad = {k: v for k, v in reg.execution.items() if v not in _EXECUTION_CLASSES}+ for k, v in sorted(bad.items()):+ errors.append(f"`x-aisoc-execution[{k}]` is {v!r}; expected one of {sorted(_EXECUTION_CLASSES)}")++ # 3. execution claims <-> the engine's real handler table, both ways.+ claims_to_run = frozenset(k for k, v in reg.execution.items() if v in _RUNS)+ for k in sorted(claims_to_run - reg.runnable):+ errors.append(+ f"`x-aisoc-execution` claims {k!r} runs, but the engine has no handler for it — "+ f"this is the claim the schema exists to make true."+ )+ for k in sorted(reg.runnable - claims_to_run):+ errors.append(+ f"the engine has a handler for {k!r} but `x-aisoc-execution` marks it "+ f"{reg.execution.get(k, '(absent)')!r} — the schema is understating what the product does."+ )++ # 4. bounds, both ways (a schema ceiling below the engine's rejects valid+ # playbooks; above it promises headroom Pydantic will refuse).+ if reg.schema_timeout_max != reg.engine_timeout_max:+ errors.append(+ f"step timeout ceiling disagrees: schema {reg.schema_timeout_max} vs "+ f"bounds.ABSOLUTE_MAX_TIMEOUT_SECONDS {reg.engine_timeout_max}"+ )+ if reg.schema_timeout_min != reg.engine_timeout_min:+ errors.append(+ f"step timeout floor disagrees: schema {reg.schema_timeout_min} vs bounds.MIN_TIMEOUT_SECONDS {reg.engine_timeout_min}"+ )+ if reg.schema_retry_max != reg.engine_retry_max:+ errors.append(f"retry ceiling disagrees: schema {reg.schema_retry_max} vs bounds.ABSOLUTE_MAX_RETRIES {reg.engine_retry_max}")++ # 5. trigger vocabulary <-> the pack validator's allow-list, both ways.+ for missing in sorted(reg.schema_triggers - reg.validator_triggers):+ errors.append(f"schema allows trigger.on={missing!r} that validate_playbooks.py rejects")+ for missing in sorted(reg.validator_triggers - reg.schema_triggers):+ errors.append(f"validate_playbooks.py allows trigger.on={missing!r} that the schema rejects")++ # 6. playbook-level keys <-> Playbook model fields, both ways.+ for missing in sorted(reg.schema_playbook_keys - reg.model_playbook_keys - INERT_AUTHORED_KEYS):+ errors.append(+ f"schema declares playbook key {missing!r} that the Playbook model drops on load — "+ f"add the field, remove the key, or list it in INERT_AUTHORED_KEYS and say so in its description"+ )+ for missing in sorted(reg.model_playbook_keys - reg.schema_playbook_keys):+ errors.append(f"Playbook model has field {missing!r} that the schema does not declare, so `additionalProperties: false` rejects it")++ # 7. exactly one schema file. Two is how the vocabularies diverged.+ if len(reg.schema_files) > 1:+ errors.append(+ f"more than one playbook.schema.json in the tree ({', '.join(reg.schema_files)}) — "+ f"a second copy is a second source of truth and the drafter silently fell back to it once already"+ )++ return errors+++# ---------------------------------------------------------------------------+# Content check — the contract must be true of what ships+# ---------------------------------------------------------------------------+++def validate_shipped_playbooks(root: Path) -> tuple[list[str], list[Path]]:+ try:+ import jsonschema+ except ImportError as exc:+ raise GateError("jsonschema is not installed; the content half of this gate cannot run") from exc++ schema = _load_schema(root)+ validator = jsonschema.Draft7Validator(schema)+ files = _playbook_files(root)+ if not files:+ raise GateError("no *.playbook.json found; refusing to report success having validated nothing")++ errors: list[str] = []+ for path in files:+ rel = path.relative_to(root)+ try:+ data = json.loads(path.read_text())+ except json.JSONDecodeError as exc:+ errors.append(f"{rel}: JSON parse error: {exc}")+ continue+ for err in sorted(validator.iter_errors(data), key=lambda e: list(e.path)):+ loc = ".".join(str(p) for p in err.path) or "(root)"+ errors.append(f"{rel}: {loc}: {err.message}")+ return errors, files+++# ---------------------------------------------------------------------------+# Self-test — prove the gate fires, in each direction+# ---------------------------------------------------------------------------+++def self_test(reg: Registries) -> list[str]:+ """Inject drift in each direction and assert the gate notices.++ A gate nobody has seen fail is indistinguishable from a gate that cannot.+ """+ failures: list[str] = []++ def expect(label: str, perturbed: Registries, needle: str) -> None:+ errs = compare(perturbed)+ if not any(needle in e for e in errs):+ failures.append(f"self-test: {label}: injected drift was NOT detected (needle {needle!r}); got {errs or 'no errors'}")++ baseline = compare(reg)+ if baseline:+ failures.append(f"self-test: the unperturbed tree already fails, so a pass proves nothing: {baseline}")++ victim = sorted(reg.schema_step_types)[0]+ absent = "step_type_that_exists_nowhere"++ expect(+ "schema declares a type the engine lacks",+ replace(+ reg,+ schema_step_types=reg.schema_step_types | {absent},+ execution={**reg.execution, absent: "unimplemented"},+ ),+ "does not implement",+ )+ expect(+ "engine implements a type the schema lacks",+ replace(reg, model_step_types=reg.model_step_types | {absent}),+ "the schema does not declare it",+ )+ expect(+ "declared type with no execution annotation",+ replace(reg, execution={k: v for k, v in reg.execution.items() if k != victim}),+ "does not say what the engine does with it",+ )+ expect(+ "execution annotation for an undeclared type",+ replace(reg, execution={**reg.execution, absent: "executed"}),+ "which the schema does not declare",+ )+ runs_now = sorted(k for k, v in reg.execution.items() if v in _RUNS)+ if not runs_now:+ failures.append("self-test: no step type is annotated as running; cannot test the handler directions")+ else:+ running = runs_now[0]+ expect(+ "annotation claims execution the engine cannot deliver",+ replace(reg, handler_step_types=reg.handler_step_types - {running}, inline_step_types=reg.inline_step_types - {running}),+ "the engine has no handler for it",+ )+ expect(+ "engine gained a handler the annotation calls unimplemented",+ replace(reg, execution={**reg.execution, running: "unimplemented"}),+ "understating what the product does",+ )+ expect("timeout ceiling drift", replace(reg, schema_timeout_max=reg.engine_timeout_max + 1), "timeout ceiling disagrees")+ expect("retry ceiling drift", replace(reg, schema_retry_max=reg.engine_retry_max + 1), "retry ceiling disagrees")+ expect(+ "schema allows a trigger the validator rejects",+ replace(reg, schema_triggers=reg.schema_triggers | {"webhook"}),+ "that validate_playbooks.py rejects",+ )+ expect(+ "validator allows a trigger the schema rejects",+ replace(reg, validator_triggers=reg.validator_triggers | {"webhook"}),+ "that the schema rejects",+ )+ expect("schema key the model drops", replace(reg, schema_playbook_keys=reg.schema_playbook_keys | {"ghost_key"}), "drops on load")+ expect(+ "model field the schema rejects", replace(reg, model_playbook_keys=reg.model_playbook_keys | {"ghost_field"}), "does not declare"+ )+ expect(+ "a second schema file reappears", replace(reg, schema_files=(*reg.schema_files, "playbook.schema.json")), "second source of truth"+ )++ return failures+++# ---------------------------------------------------------------------------+++def main() -> int:+ parser = argparse.ArgumentParser(description=__doc__, formatter_class=argparse.RawDescriptionHelpFormatter)+ parser.add_argument("--repo-root", default=None, help="Tree to inspect (default: the repo this script lives in)")+ parser.add_argument("--self-test", action="store_true", help="Prove the gate detects injected drift in each direction")+ args = parser.parse_args()++ try:+ root = _find_repo_root(args.repo_root)+ reg = collect(root)+ content_errors, scanned = validate_shipped_playbooks(root)+ except GateError as exc:+ print(f"FATAL: {exc}", file=sys.stderr)+ return 2++ # Name what was inspected. A gate that does not say what it read cannot+ # be distinguished from one that read nothing.+ print(f"repo root {root}")+ print(f"schema {', '.join(reg.schema_files) or '(none)'}")+ print("engine services/agents/app/playbook/engine.py")+ print(f"step types schema {len(reg.schema_step_types)} | StepType {len(reg.model_step_types)} | runnable {len(reg.runnable)}")+ counts = {c: sum(1 for v in reg.execution.values() if v == c) for c in sorted(_EXECUTION_CLASSES)}+ print("execution " + " | ".join(f"{k} {v}" for k, v in counts.items()))+ print(f"bounds timeout {reg.schema_timeout_min}..{reg.schema_timeout_max}s | retries <={reg.schema_retry_max}")+ print(f"playbooks scanned {len(scanned)}")+ print()++ errors = compare(reg)++ if args.self_test:+ st = self_test(reg)+ if st:+ print("SELF-TEST FAILED")+ for f in st:+ print(f" {f}")+ return 1+ print("self-test OK — every direction detected its injected drift, and the clean tree passed")+ print()++ if content_errors:+ print(f"{len(content_errors)} shipped playbook(s) do not match the published schema:")+ for e in content_errors[:40]:+ print(f" {e}")+ if len(content_errors) > 40:+ print(f" ... and {len(content_errors) - 40} more")++ if errors:+ print(f"{len(errors)} schema/engine disagreement(s):")+ for e in errors:+ print(f" {e}")++ if errors or content_errors:+ return 1++ print("OK — schema and engine agree in both directions, and every shipped playbook matches the schema.")+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/scripts/generate_playbooks.py b/scripts/generate_playbooks.pyindex 38f0f2ee5..504fce9b8 100755--- a/scripts/generate_playbooks.py+++ b/scripts/generate_playbooks.py@@ -204,7 +204,12 @@ def make_playbook("name": name,"description": description,"version": version,- "tags": [category, *tags],+ # dict.fromkeys de-duplicates while preserving order. Several+ # categories already lead their tag list with the category name, so+ # the plain `[category, *tags]` emitted e.g.+ # ["supply-chain", "supply-chain", "npm", ...] in 20 of the 62 packs+ # — which the schema rejects under `uniqueItems`.+ "tags": list(dict.fromkeys([category, *tags])),"trigger": {"on": trigger_on,"severity": severity,diff --git a/scripts/lint_playbooks.py b/scripts/lint_playbooks.pyindex 856a4c9c8..d2df40876 100755--- a/scripts/lint_playbooks.py+++ b/scripts/lint_playbooks.py@@ -13,6 +13,7 @@ Usage:python3 scripts/lint_playbooks.pypython3 scripts/lint_playbooks.py path/to/my-playbook.json"""+from __future__ import annotationsimport json@@ -28,12 +29,20 @@ except ImportError:REPO_ROOT = Path(__file__).resolve().parent.parentSCHEMA_PATH = REPO_ROOT / "schemas" / "playbook.schema.json"-# Directories / glob patterns to scan automatically+# Directories scanned for loose ``*.json`` playbooks.SCAN_DIRS = [REPO_ROOT / "services" / "agents" / "data" / "playbooks",]SCAN_GLOB = "*.json"+# ...plus every ``*.playbook.json`` in the tree, which is what the docstring+# above always claimed. It did not: only the two files in SCAN_DIRS were ever+# checked, so the job printed "2/2 playbook files passed" while the 62+# playbooks under ``playbooks/packs/v1/`` were never looked at — 32 of them+# did not match the schema at all.+RECURSIVE_GLOB = "*.playbook.json"+SKIP_DIR_PARTS = {"node_modules", ".git", ".venv", "venv", "dist", "build"}+def load_schema() -> dict:if not SCHEMA_PATH.exists():@@ -47,9 +56,23 @@ def collect_files(extra: list[str]) -> list[Path]:for d in SCAN_DIRS:if d.is_dir():files.extend(sorted(d.glob(SCAN_GLOB)))+ for p in REPO_ROOT.rglob(RECURSIVE_GLOB):+ if SKIP_DIR_PARTS.isdisjoint(p.parts):+ files.append(p)for p in extra:- files.append(Path(p))- return files+ files.append(Path(p).resolve())+ # A file can match both SCAN_DIRS and the recursive glob.+ return sorted(set(files))+++def _display(path: Path) -> str:+ """Repo-relative where possible. ``Path.relative_to`` raises for a path+ outside the repo, which used to crash the reporter on the first failure+ of a file passed on argv."""+ try:+ return str(path.relative_to(REPO_ROOT))+ except ValueError:+ return str(path)def validate_file(path: Path, schema: dict, validator_cls) -> list[str]:@@ -76,19 +99,21 @@ def main() -> None:files = collect_files(extra_args)if not files:- print("No playbook files found — skipping lint (exit 0).")- sys.exit(0)+ # Previously exit 0. A scan that finds nothing in a repo that ships+ # 64 playbooks is a broken scan, not a clean bill of health.+ print("ERROR: no playbook files found — the scan is broken, not the tree.", file=sys.stderr)+ sys.exit(1)fail_count = 0for path in files:errs = validate_file(path, schema, validator_cls)if errs:- print(f"FAIL {path.relative_to(REPO_ROOT)}")+ print(f"FAIL {_display(path)}")for e in errs:print(e)fail_count += 1else:- print(f"OK {path.relative_to(REPO_ROOT)}")+ print(f"OK {_display(path)}")total = len(files)print(f"\n{total - fail_count}/{total} playbook files passed schema validation.")diff --git a/services/actions/app/live_actions/capability_contracts.py b/services/actions/app/live_actions/capability_contracts.pyindex 843bbda06..16757fad8 100644--- a/services/actions/app/live_actions/capability_contracts.py+++ b/services/actions/app/live_actions/capability_contracts.py@@ -526,6 +526,56 @@ CAPABILITY_CONTRACTS: dict[str, CapabilityContract] = {}+#: Legacy ``ActionType`` value -> capability name, for the cases where the+#: two vocabularies chose different words for the same verb.+#:+#: Only one entry, and it is a naming gap rather than a missing capability:+#: the public API verb is ``notify_slack`` (it predates the capability+#: registry and names the transport), the capability is ``notify`` (it names+#: the verb, which is the whole point of a per-capability contract), and+#: ``SlackNotify`` in ``builtins.py`` already bridges them. What was missing+#: was anything that told a *lookup by ActionType value* about the bridge, so+#: ``approval_gate`` found no contract, skipped the confidence matrix+#: entirely and let blast radius decide alone.+#:+#: Why an alias and not a rename. ``action_type`` is persisted as free text:+#: ``remediation_whitelist`` (migration 015) stores operator pre-approvals+#: keyed ``UNIQUE (tenant_id, action_type)``, so renaming the enum member+#: silently orphans every row an operator created for ``notify_slack`` —+#: their pre-approval stops matching and nobody is told. It is also a+#: documented request field (``{"action_type": "notify_slack"}`` in+#: ``apps/docs/docs/operations/notifications.md``) and a member of the+#: ``ActionType`` union in ``packages/types``.+#:+#: When to delete this. It retires with ``ActionType`` itself: the enum is+#: already the legacy half of the registry (every adapter carries a+#: ``_legacy_action_type`` pointing back at it) and the live-actions path+#: dispatches on capability strings. When the ``POST /actions`` body takes a+#: capability and the ``remediation_whitelist`` rows have been migrated,+#: this map and the enum go together. Until then a new entry here needs the+#: same justification: a *naming* difference for a verb that already has a+#: contract, never a stand-in for a capability nobody wrote.+ACTION_TYPE_CAPABILITY_ALIASES: dict[str, str] = {+ "notify_slack": "notify",+}+++def contract_for_action_type(action_type_value: str) -> CapabilityContract | None:+ """Resolve a legacy ``ActionType`` value to its capability contract.++ Tries the value as a capability name first, then the alias map. Returns+ ``None`` when the verb genuinely has no contract, so callers can say so+ rather than invent an impact for it.+ """+ contract = CAPABILITY_CONTRACTS.get(action_type_value)+ if contract is not None:+ return contract+ aliased = ACTION_TYPE_CAPABILITY_ALIASES.get(action_type_value)+ if aliased is None:+ return None+ return CAPABILITY_CONTRACTS.get(aliased)++def apply_contract(cls: type) -> type:"""Class decorator: stamp the capability's contract onto an executor.diff --git a/services/actions/app/services/approval_gate.py b/services/actions/app/services/approval_gate.pyindex 2277093d5..cc530ea4d 100644--- a/services/actions/app/services/approval_gate.py+++ b/services/actions/app/services/approval_gate.py@@ -16,18 +16,22 @@ the matrix already states for its own three inputs: each can raise arequirement, none can lower it. So switching it on cannot make anythingauto-execute that did not before.-Two honest limits, both logged rather than papered over:--One of the 24 ``ActionType`` members has no capability contract, down from-five: ``capture_forensics`` and ``chatops_verify`` gained one, and-``add_ioc_to_blocklist`` and ``run_playbook`` left the enum because neither-had an implementation path. The one left is ``notify_slack``, and it is a-naming gap rather than an undeclared verb — the lookup below is by-``ActionType`` value and the matching capability is called ``notify``, which-does have a contract. For an unmapped verb the matrix has no impact to reason-about and blast radius decides alone. That is recorded at debug rather than-guessed at, because inventing an impact for an unmapped verb is how a gate-starts certifying things it never examined.+Every ``ActionType`` member now resolves to a capability contract, and+``scripts/check_action_contract.py`` fails if one stops doing so.++The last hold-out was ``notify_slack``. It was never a missing capability:+the verb is ``notify``, it has a contract, and ``SlackNotify`` already+bridged the two. What was missing was anything connecting a lookup *by+ActionType value* to that bridge, so this gate found no contract, skipped+the confidence matrix and let blast radius decide alone — a 40%-confidence+guess and a corroborated finding got the same answer for the one verb most+likely to be auto-executed. ``contract_for_action_type`` closes it by alias+rather than by rename, because ``action_type`` is persisted operator intent+(see the note on ``ACTION_TYPE_CAPABILITY_ALIASES``).++An unmapped verb is still possible in principle, and is still recorded at+debug rather than guessed at: inventing an impact for a verb nobody declared+is how a gate starts certifying things it never examined.A request with no ``confidence`` is treated as the lowest band. For anythingabove READ_ONLY impact that means analyst approval, which is a real tightening@@ -38,7 +42,7 @@ from __future__ import annotationsimport structlog-from app.live_actions.capability_contracts import CAPABILITY_CONTRACTS+from app.live_actions.capability_contracts import contract_for_action_typefrom app.live_actions.contract import ApprovalRequirementfrom app.models.action import ActionRequest, ActionStatus, BlastRadiusfrom app.services.approval_matrix import evaluate as evaluate_matrix@@ -90,7 +94,7 @@ async def apply_matrix(Never lowers: a blast-radius gate that already demands approval keepsdemanding it whatever the confidence says."""- contract = CAPABILITY_CONTRACTS.get(request.action_type.value)+ contract = contract_for_action_type(request.action_type.value)if contract is None:logger.debug("approval_gate.no_capability_contract",diff --git a/services/actions/tests/test_approval_gate_wiring.py b/services/actions/tests/test_approval_gate_wiring.pyindex 869279773..70933298d 100644--- a/services/actions/tests/test_approval_gate_wiring.py+++ b/services/actions/tests/test_approval_gate_wiring.py@@ -17,7 +17,8 @@ from __future__ import annotationsfrom uuid import uuid4import pytest-from app.live_actions.capability_contracts import CAPABILITY_CONTRACTS+from app.live_actions import capability_contracts+from app.live_actions.capability_contracts import CAPABILITY_CONTRACTS, contract_for_action_typefrom app.models.action import ActionRequest, ActionStatus, ActionType, BlastRadiusfrom app.services.approval_gate import apply_matrixfrom app.services.blast_radius import BlastRadiusGate@@ -143,13 +144,68 @@ class TestTheTierIsHonoured:assert "L1" in reason+class TestTheAliasedVerbIsGatedToo:+ """``notify_slack`` was the last ActionType the matrix never saw.++ It was never a missing capability — the verb is ``notify``, it has had a+ contract throughout, and ``SlackNotify`` bridges the two names. What was+ missing was anything telling a lookup *by ActionType value* about that+ bridge, so ``apply_matrix`` found nothing, returned the blast-radius+ verdict unchanged, and the one verb most likely to auto-execute was the+ one verb graded without reference to confidence.+ """++ def test_the_two_names_are_not_the_same_string(self):+ """The premise. If these ever converge the alias becomes dead code,+ and the contract gate says so rather than leaving it to rot."""+ assert ActionType.NOTIFY_SLACK.value == "notify_slack"+ assert ActionType.NOTIFY_SLACK.value not in CAPABILITY_CONTRACTS+ assert "notify" in CAPABILITY_CONTRACTS++ def test_the_alias_resolves_to_the_notify_contract(self):+ assert contract_for_action_type(ActionType.NOTIFY_SLACK.value) is CAPABILITY_CONTRACTS["notify"]++ @pytest.mark.asyncio+ async def test_a_low_confidence_notify_no_longer_auto_executes(self):+ """Fails against the pre-change tree, which approved this outright.++ ``notify_slack`` is MINIMAL blast radius, so ``BlastRadiusGate``+ approves it on its own. With the contract now reachable the matrix+ gets a say, and at 10% confidence under the default L1 tier it+ demands an analyst.+ """+ request = _request(ActionType.NOTIFY_SLACK, confidence=0.1)+ before, br, before_reason = BlastRadiusGate().evaluate(request)+ after, reason = await apply_matrix(request, before, br, before_reason)++ assert before == ActionStatus.APPROVED+ assert after == ActionStatus.AWAITING_APPROVAL+ assert reason != before_reason++ @pytest.mark.asyncio+ async def test_the_alias_cannot_lower_a_requirement(self):+ """An alias route into the matrix must obey the same composition rule+ as a direct one: each input can raise a requirement, none can lower+ it. A LOW-impact contract must not talk a blocked action into running.+ """+ request = _request(ActionType.NOTIFY_SLACK, confidence=1.0)+ after, _ = await apply_matrix(request, ActionStatus.AWAITING_APPROVAL, BlastRadius.MINIMAL, "held by policy")++ assert after == ActionStatus.AWAITING_APPROVAL++class TestVerbsWithNoContract:@pytest.mark.asyncio- async def test_an_unmapped_verb_is_left_to_blast_radius(self):- """Seven ActionType members have no capability contract. Inventing an- impact for them is how a gate starts certifying things it never- examined, so blast radius decides alone and the gap is logged."""- assert ActionType.NOTIFY_SLACK.value not in CAPABILITY_CONTRACTS+ async def test_an_unmapped_verb_is_still_left_to_blast_radius(self, monkeypatch: pytest.MonkeyPatch):+ """Every ActionType resolves a contract today and a gate keeps it that+ way, so this path is no longer reachable through the enum. It stays+ tested because the fallback is what stops the gate inventing an impact+ for a verb nobody declared — a gate that certifies what it never+ examined is the failure this whole module exists to avoid.++ Pulling the alias out is the smallest way to reach the branch.+ """+ monkeypatch.setattr(capability_contracts, "ACTION_TYPE_CAPABILITY_ALIASES", {})request = _request(ActionType.NOTIFY_SLACK, confidence=0.1)before, br, before_reason = BlastRadiusGate().evaluate(request)@@ -158,6 +214,12 @@ class TestVerbsWithNoContract:assert after == beforeassert reason == before_reason+ def test_no_action_type_is_left_unmapped(self):+ """The property the contract gate enforces, asserted here too so it+ fails in the service's own suite and not only in a CI script."""+ unmapped = [a.value for a in ActionType if contract_for_action_type(a.value) is None]+ assert unmapped == [], f"ActionType members with no reachable capability contract: {unmapped}"+class TestTierLabelMapping:@pytest.mark.parametrize(diff --git a/services/agents/app/playbook/engine.py b/services/agents/app/playbook/engine.pyindex 45faf835b..134613440 100644--- a/services/agents/app/playbook/engine.py+++ b/services/agents/app/playbook/engine.py@@ -627,28 +627,56 @@ class PlaybookEngine:attempt = 0handler = _HANDLERS.get(step.type)- while True:- attempt += 1- t0 = time.perf_counter()- try:- if dry_run:- result = {"dry_run": True, "step": step.name}- elif handler:- result = await handler(step, pr.context, http)- else:- result = {"skipped": True, "reason": f"no handler for {step.type}"}- elapsed = time.perf_counter() - t0- result["_elapsed_ms"] = round(elapsed * 1000)- break # success- except Exception as exc: # noqa: BLE001- elapsed = time.perf_counter() - t0- logger.error("Step %s attempt %d failed: %s", step.name, attempt, exc)- if attempt <= step.retry_max:- await asyncio.sleep(min(2**attempt, 30))- else:- step_status = StepStatus.FAILED- result = {"error": str(exc), "_elapsed_ms": round(elapsed * 1000)}- break+ if handler is None and not dry_run:+ # Fail closed, and skip the retry loop — a missing handler+ # will still be missing on the next attempt.+ #+ # This branch used to return ``{"skipped": True}`` while+ # leaving step_status at SUCCESS, so twelve of the+ # twenty-two declared step types reported that they had run+ # when nothing had. The worst of them was ``approval``: a+ # human decision point that passed on its own and let the+ # run continue into the very action an analyst was supposed+ # to authorise. Falling through to the shared tail below+ # means the default ``on_failure: abort`` halts the run.+ step_status = StepStatus.FAILED+ result = {+ "error": f"step type {step.type.value!r} has no handler in this engine",+ "unimplemented": True,+ "_elapsed_ms": 0,+ }+ logger.error(+ "Step %s (%s) has no handler; failing closed rather than reporting success",+ step.name,+ step.type.value,+ )+ else:+ while True:+ attempt += 1+ t0 = time.perf_counter()+ try:+ if dry_run:+ result = {"dry_run": True, "step": step.name}+ if handler is None:+ # A dry run exists to tell the author what+ # would happen. "dry_run: true" alone would+ # imply this step is fine.+ result["unimplemented"] = True+ result["would_fail"] = True+ else:+ result = await handler(step, pr.context, http)+ elapsed = time.perf_counter() - t0+ result["_elapsed_ms"] = round(elapsed * 1000)+ break # success+ except Exception as exc: # noqa: BLE001+ elapsed = time.perf_counter() - t0+ logger.error("Step %s attempt %d failed: %s", step.name, attempt, exc)+ if attempt <= step.retry_max:+ await asyncio.sleep(min(2**attempt, 30))+ else:+ step_status = StepStatus.FAILED+ result = {"error": str(exc), "_elapsed_ms": round(elapsed * 1000)}+ breakpr.step_results.append({"step_id": step.id, "name": step.name, "status": step_status, "result": result})# Merge result into context for downstream steps. The namespaceddiff --git a/services/agents/app/playbook/nl_drafter.py b/services/agents/app/playbook/nl_drafter.pyindex 53053e098..8cee751f9 100644--- a/services/agents/app/playbook/nl_drafter.py+++ b/services/agents/app/playbook/nl_drafter.py@@ -23,9 +23,9 @@ Two execution paths::func:`services.agents.app.llm.safe_ainvoke` and asks for aJSON playbook back. The output is validated against the canonical:class:`Playbook` Pydantic model AND against the project's- `playbook.schema.json` (JSON Schema 2020-12) before being- returned. Any validation failure falls back to the substrate- draft so the UX never hangs.+ `schemas/playbook.schema.json` (JSON Schema draft-07) before+ being returned. Any validation failure falls back to the+ substrate draft so the UX never hangs.The drafter is the single Pydantic-aware Python entry point forNL → playbook drafting; the API route at@@ -44,6 +44,7 @@ from datetime import UTC, datetimefrom pathlib import Pathfrom typing import Any+from .bounds import ABSOLUTE_MAX_RETRIES, ABSOLUTE_MAX_TIMEOUT_SECONDSfrom .models import Playbook, PlaybookStep, StepTypelogger = logging.getLogger("aisoc.playbook.nl_drafter")@@ -392,63 +393,39 @@ def _summarise_prompt_for_name(prompt: str) -> str:def _schema_path() -> Path:- """Resolve the canonical playbook.schema.json bundled with the repo.+ """Resolve the one canonical ``schemas/playbook.schema.json``.- CI uses ``schemas/playbook.schema.json`` (the path baked into- ``scripts/lint_playbooks.py`` and the ``validate-playbooks``- workflow), so we prefer that location and fall back to the- legacy root-level copy only if it is missing.+ There used to be a second copy at the repo root declaring a different+ step vocabulary, and this function fell back to it whenever the primary+ was missing — so a packaging slip did not fail, it silently validated+ against a different contract. There is now exactly one schema (a parity+ gate enforces that), and a missing one is an error rather than a cue to+ go looking for something else to trust.++ The walk stops at the first ancestor that has the file so a checkout at+ any depth works, but it never accepts a differently-shaped substitute."""here = Path(__file__).resolve()for parent in here.parents:- primary = parent / "schemas" / "playbook.schema.json"- if primary.exists():- return primary- fallback = parent / "playbook.schema.json"- if fallback.exists():- return fallback- raise FileNotFoundError("playbook.schema.json not found alongside the repo root")---# Schema-allowed step types (from ``schemas/playbook.schema.json``). The-# Pydantic StepType enum is broader than the JSON Schema today; for the-# drafter we collapse the extended types down to their nearest-# schema-compatible neighbour so the resulting playbook ALSO passes-# ``scripts/lint_playbooks.py`` without the user having to touch the-# schema.-_SCHEMA_STEP_TYPES: frozenset[str] = frozenset(- {- "enrich",- "investigate",- "notify",- "block_ip",- "isolate_host",- "create_ticket",- "close_case",- "http",- "condition",- }-)+ candidate = parent / "schemas" / "playbook.schema.json"+ if candidate.exists():+ return candidate+ raise FileNotFoundError("schemas/playbook.schema.json not found in any parent of services/agents/app/playbook/")-# Pydantic-only StepType → schema-allowed StepType. Anything not in the-# schema enum collapses to ``investigate`` because the analyst can still-# upgrade the step in the editor without losing the prompt-derived name.-_PYDANTIC_TO_SCHEMA_TYPE: dict[str, str] = {- "block_ioc": "block_ip",- "disable_user": "investigate",- "reset_password": "investigate",- "revoke_session": "investigate",- "force_mfa": "investigate",- "kill_process": "investigate",- "quarantine_file": "investigate",- "run_av_scan": "investigate",- "run_script": "http",- "search_siem": "investigate",- "create_notable_event": "create_ticket",- "osquery_live_query": "investigate",- "approval": "condition",-}++# The step-type collapse that used to live here is gone.+#+# The JSON Schema declared 9 step types against a StepType enum of 22, so the+# drafter rewrote the 13 it could not express onto their "nearest neighbour"+# — ``disable_user`` and ``revoke_session`` both became ``investigate``,+# ``approval`` became ``condition`` — purely so the output would pass+# ``scripts/lint_playbooks.py``. The original was preserved in+# ``params.original_type``, but the step that shipped said it would+# investigate when the analyst had asked to disable an account, and the+# approval gate an author wrote came out as an ungated branch.+#+# The schema now declares all 22, so there is nothing to collapse.def _strip_nulls(obj: Any) -> Any:@@ -469,57 +446,22 @@ def _strip_nulls(obj: Any) -> Any:return obj-def _collapse_step_types_for_schema(payload: dict[str, Any]) -> dict[str, Any]:- """Return a copy of ``payload`` with each step ``type`` collapsed- to the JSON-Schema-allowed enum AND with ``null`` fields stripped.- Preserves the original type as ``params.original_type`` so the- editor can recover it.-- Pydantic remains the source of truth at runtime — this projection- exists only so the drafter's output passes ``lint_playbooks.py``.- """-- proj = _strip_nulls(json.loads(json.dumps(payload))) # deep copy + null-strip- steps = proj.get("steps", [])- if not isinstance(steps, list):- return proj- for step in steps:- if not isinstance(step, dict):- continue- st = step.get("type")- if not isinstance(st, str) or st in _SCHEMA_STEP_TYPES:- continue- mapped = _PYDANTIC_TO_SCHEMA_TYPE.get(st, "investigate")- params = step.get("params")- if not isinstance(params, dict):- params = {}- params.setdefault("original_type", st)- step["params"] = params- step["type"] = mapped- return proj--def _load_schema() -> dict[str, Any]:return json.loads(_schema_path().read_text(encoding="utf-8"))def _validate_against_schema(payload: dict[str, Any]) -> tuple[bool, str | None]:- """Validate ``payload`` against playbook.schema.json.-- We try two passes: the raw payload first, then the- step-type-collapsed projection (see- :func:`_collapse_step_types_for_schema`). Returning ``True`` from- pass-one means the playbook is **strictly** schema-conformant;- falling through to pass-two means the drafter is producing a- Pydantic-richer step set that the JSON Schema can't yet describe- — that's a known gap tracked in the v8.0 plan, and we widen the- Pydantic model first because the React Flow editor consumes the- Pydantic shape directly.-- Returns ``(True, None)`` on success of either pass,- ``(False, error_message)`` on failure of both. ``jsonschema`` is- imported lazily so the module is importable in environments- without it.+ """Validate ``payload`` against ``schemas/playbook.schema.json``.++ One pass. There used to be a second that rewrote any step type the+ schema could not express onto a "nearest neighbour" and re-validated;+ it made the drafter's output lint-clean by changing what the playbook+ said it would do. The schema now covers the whole ``StepType`` range,+ so a failure here is a real failure.++ Returns ``(True, None)`` on success, ``(False, error_message)`` on+ failure. ``jsonschema`` is imported lazily so the module stays+ importable in environments without it."""try:@@ -534,24 +476,14 @@ def _validate_against_schema(payload: dict[str, Any]) -> tuple[bool, str | None]logger.warning("playbook.schema.json missing: %s", exc)return True, None- # Pass 1 — null-stripped payload (Pydantic emits ``null`` keys that- # the schema rejects under ``additionalProperties: false``).+ # Null-stripped: Pydantic emits ``null`` for unset optional keys and the+ # schema rejects those under ``additionalProperties: false``.cleaned = _strip_nulls(payload)try:jsonschema.validate(cleaned, schema) # type: ignore[attr-defined]return True, Noneexcept jsonschema.ValidationError as exc: # type: ignore[attr-defined]- first_err = exc.message-- # Pass 2 — also collapse Pydantic-only step types to schema-allowed- # equivalents. This lets the drafter use the full StepType range- # internally while still emitting a CI-clean draft.- try:- projection = _collapse_step_types_for_schema(payload)- jsonschema.validate(projection, schema) # type: ignore[attr-defined]- return True, None- except jsonschema.ValidationError as exc: # type: ignore[attr-defined]- return False, f"{first_err} (also failed after step-type collapse: {exc.message})"+ return False, exc.message# ---------------------------------------------------------------------------@@ -571,24 +503,34 @@ Rules — follow exactly:``^[a-z0-9][a-z0-9-]{2,62}$``. The ``version`` must be semantic,e.g. ``"1.0.0"``.3. ``trigger`` must contain ``on`` (one of ``alert`` / ``case`` /- ``schedule`` / ``manual`` / ``webhook``). Severities, when present,+ ``schedule`` / ``manual``). Severities, when present,must be an array of any of: ``info`` / ``low`` / ``medium`` /``high`` / ``critical``.-4. Each step must declare a ``type``. Allowed types include- ``enrich``, ``investigate``, ``notify``, ``block_ip``, ``block_ioc``,- ``isolate_host``, ``create_ticket``, ``close_case``, ``http``,- ``condition``, ``approval``, ``disable_user``, ``reset_password``,- ``revoke_session``, ``force_mfa``, ``kill_process``,- ``quarantine_file``, ``run_av_scan``, ``run_script``,- ``search_siem``, ``create_notable_event``, ``osquery_live_query``.+4. Each step must declare a ``type``, one of exactly:+ __STEP_TYPES__.5. Each step must carry a short, action-oriented ``name``, an ``id``- (8-32 char hex), an ``on_failure`` ∈ ``{abort, continue, retry}``,- ``retry_max`` (0-5), and ``timeout_seconds`` (1-3600).+ (8-32 char hex), an ``on_failure`` ∈ ``abort`` / ``continue`` /+ ``retry``, ``retry_max`` (0-__MAX_RETRIES__), and+ ``timeout_seconds`` (1-__MAX_TIMEOUT__).6. The output's ``enabled`` MUST be ``false``. A human reviews beforeenabling.7. Do NOT invent fields not in the schema. Do NOT emit prose. JSON only."""+# Filled from the enum and the bounds module rather than restated, because a+# hand-written copy of the vocabulary in a prompt is one more place for it to+# drift — this list previously offered ``webhook`` as a trigger, which no+# validator in the repo accepts, and capped ``retry_max`` at 5 against a+# model that allows 25.+#+# Token substitution rather than ``str.format`` because the prompt contains+# a regex with brace quantifiers that format would try to interpret.+_SYSTEM_PROMPT = (+ _SYSTEM_PROMPT.replace("__STEP_TYPES__", ", ".join(f"``{st.value}``" for st in StepType))+ .replace("__MAX_RETRIES__", str(ABSOLUTE_MAX_RETRIES))+ .replace("__MAX_TIMEOUT__", str(ABSOLUTE_MAX_TIMEOUT_SECONDS))+)+_USER_TEMPLATE = """\Analyst prompt:diff --git a/services/agents/tests/test_playbook_engine_correctness.py b/services/agents/tests/test_playbook_engine_correctness.pyindex 63614b16d..6c91e3794 100644--- a/services/agents/tests/test_playbook_engine_correctness.py+++ b/services/agents/tests/test_playbook_engine_correctness.py@@ -403,3 +403,102 @@ class TestEngineConditionGate:assert called["hit"] is Trueassert run.step_results[0]["status"] == StepStatus.SUCCESS+++# ---------------------------------------------------------------------------+# PlaybookEngine.run — step types the engine cannot run must not report success+# ---------------------------------------------------------------------------+#+# Twelve of the twenty-two StepType members had no entry in ``_HANDLERS``.+# The run loop answered those with ``{"skipped": True}`` and left+# ``step_status`` at its SUCCESS default, so a playbook containing them ran to+# RunStatus.COMPLETED having done nothing that it said it did.+#+# ``approval`` is the one that matters most: it is a human decision point, it+# appears in 14 steps across the shipped packs, and it passed on its own —+# the run continued straight into the action an analyst was meant to+# authorise. "Unverifiable means not autonomous" applies to a gate that+# cannot be evaluated as much as to an action that cannot be verified.+++class TestUnimplementedStepTypesFailClosed:+ @pytest.mark.asyncio+ async def test_an_approval_gate_the_engine_cannot_honour_halts_the_run(self) -> None:+ """Fails against the pre-change tree, which reported SUCCESS/COMPLETED."""+ pb = _make_playbook(+ [+ PlaybookStep(id="gate", name="Analyst approves isolation", type=StepType.APPROVAL),+ PlaybookStep(id="act", name="Isolate the host", type=StepType.ISOLATE_HOST),+ ]+ )++ run = await PlaybookEngine().run(pb, trigger_context={})++ assert run.step_results[0]["status"] == StepStatus.FAILED+ assert run.step_results[0]["result"]["unimplemented"] is True+ assert "approval" in run.step_results[0]["result"]["error"]+ assert run.status == RunStatus.FAILED+ # The whole point: the step behind the gate must not have run.+ assert len(run.step_results) == 1, "execution continued past a gate that was never granted"++ @pytest.mark.asyncio+ async def test_no_step_type_reports_success_without_a_handler(self) -> None:+ """Every StepType with no handler, not just the one that hurt most."""+ unimplemented = [st for st in StepType if st not in engine_mod._HANDLERS and st != StepType.CONDITION]+ assert unimplemented, "fixture assumption broken: expected some StepType to lack a handler"++ for step_type in unimplemented:+ pb = _make_playbook([PlaybookStep(id="s1", name=f"step {step_type.value}", type=step_type)])+ run = await PlaybookEngine().run(pb, trigger_context={})++ result = run.step_results[0]+ assert result["status"] == StepStatus.FAILED, f"{step_type.value} reported {result['status']} with no handler"+ assert result["result"].get("unimplemented") is True+ assert run.status == RunStatus.FAILED++ @pytest.mark.asyncio+ async def test_a_missing_handler_is_not_retried(self) -> None:+ """A handler that does not exist will not exist next attempt either;+ retrying just delays the failure by up to 2**retry_max seconds."""+ pb = _make_playbook([PlaybookStep(id="s1", name="scan", type=StepType.RUN_AV_SCAN, retry_max=3)])++ run = await PlaybookEngine().run(pb, trigger_context={})++ assert run.step_results[0]["status"] == StepStatus.FAILED+ assert run.step_results[0]["result"]["_elapsed_ms"] == 0++ @pytest.mark.asyncio+ async def test_on_failure_continue_is_still_honoured(self) -> None:+ """Failing closed must not mean ignoring the author's policy: a step+ explicitly marked ``continue`` still lets the run proceed."""+ pb = _make_playbook(+ [+ PlaybookStep(id="s1", name="best-effort scan", type=StepType.RUN_AV_SCAN, on_failure="continue"),+ PlaybookStep(id="s2", name="second step", type=StepType.RUN_SCRIPT, on_failure="continue"),+ ]+ )++ run = await PlaybookEngine().run(pb, trigger_context={})++ assert len(run.step_results) == 2+ assert all(r["status"] == StepStatus.FAILED for r in run.step_results)+ assert run.status == RunStatus.COMPLETED++ @pytest.mark.asyncio+ async def test_a_dry_run_says_which_steps_would_fail(self) -> None:+ """A dry run exists to tell the author what would happen. Reporting a+ bare ``dry_run: true`` for a step that cannot execute would imply it+ is fine."""+ pb = _make_playbook(+ [+ PlaybookStep(id="s1", name="notify", type=StepType.NOTIFY),+ PlaybookStep(id="s2", name="gate", type=StepType.APPROVAL),+ ]+ )++ run = await PlaybookEngine().run(pb, trigger_context={}, dry_run=True)++ assert run.status == RunStatus.COMPLETED+ assert run.step_results[0]["result"].get("unimplemented") is None+ assert run.step_results[1]["result"]["unimplemented"] is True+ assert run.step_results[1]["result"]["would_fail"] is Truediff --git a/services/agents/tests/test_playbook_nl_drafter.py b/services/agents/tests/test_playbook_nl_drafter.pyindex d39ea81eb..251a2cab9 100644--- a/services/agents/tests/test_playbook_nl_drafter.py+++ b/services/agents/tests/test_playbook_nl_drafter.py@@ -39,6 +39,12 @@ from app.playbook import (# Helpers# ---------------------------------------------------------------------------+#: Materialised once. Iterating the enum class directly is correct Python,+#: but CodeQL resolves ``StepType`` through the ``app.playbook`` package+#: re-export, does not see the ``Enum`` base, and reads ``for x in StepType``+#: as iterating a plain class (``py/non-iterable-in-for-loop``).+_ALL_STEP_TYPES = tuple(StepType.__members__.values())+def _run(coro: Any) -> Any:"""Run an awaitable synchronously, with one event loop per call."""@@ -153,55 +159,59 @@ class TestSchemaValidity:assert rehydrated.id == pb.idassert _step_types(rehydrated) == _step_types(pb)- def test_step_type_collapse_preserves_original_type(self) -> None:- # ``run_av_scan`` is Pydantic-only — schema collapse must map it- # to a schema-allowed type and stash the original in params.- payload = {- "id": "test-id",- "name": "Test",- "version": "1.0.0",- "trigger": {"on": "alert"},- "steps": [- {- "id": "s1",- "name": "Run AV Scan",- "type": "run_av_scan",- "params": {},- }- ],- }- collapsed = nl_drafter._collapse_step_types_for_schema(payload)- assert collapsed["steps"][0]["type"] in nl_drafter._SCHEMA_STEP_TYPES- assert collapsed["steps"][0]["params"]["original_type"] == "run_av_scan"-- def test_schema_collapse_is_pure(self) -> None:- # The collapse helper must NOT mutate its input.- payload = {- "id": "test-id",- "name": "Test",- "version": "1.0.0",- "trigger": {"on": "alert"},- "steps": [{"id": "s1", "name": "x", "type": "block_ioc", "params": {}}],- }- before = json.dumps(payload, sort_keys=True)- nl_drafter._collapse_step_types_for_schema(payload)- after = json.dumps(payload, sort_keys=True)- assert before == after, "collapse must not mutate its input"-- def test_schema_allowed_types_unchanged_by_collapse(self) -> None:- payload = {- "id": "test-id",- "name": "Test",- "version": "1.0.0",- "trigger": {"on": "alert"},- "steps": [- {"id": "s1", "name": "x", "type": "enrich", "params": {}},- {"id": "s2", "name": "y", "type": "notify", "params": {}},- ],- }- collapsed = nl_drafter._collapse_step_types_for_schema(payload)- assert collapsed["steps"][0]["type"] == "enrich"- assert collapsed["steps"][1]["type"] == "notify"+ def test_every_step_type_validates_without_being_rewritten(self) -> None:+ """The three tests that used to live here asserted the collapse.++ The drafter could not express 13 of the 22 ``StepType`` values in the+ published schema, so it rewrote them onto a "nearest neighbour" to+ make its output lint-clean: ``run_av_scan`` became ``investigate``,+ ``approval`` became ``condition``. The original was stashed in+ ``params.original_type``, but the playbook that shipped said it would+ investigate when the author had asked to scan, and an approval gate+ came out as an ungated branch.++ This is the property that replaces it: every step type the model+ accepts is declared by the published schema. Fails against the+ pre-change tree, where the schema declared 9 of 22.++ Asserted against the schema document directly, not through+ ``_validate_against_schema``. Going through the helper is what hid+ this for so long — its second pass rewrote the step and reported+ success, so "the drafter's output validates" was true of a playbook+ the drafter had altered to make it true.+ """+ jsonschema = pytest.importorskip("jsonschema")+ schema = nl_drafter._load_schema()+ declared = set(schema["definitions"]["PlaybookStep"]["properties"]["type"]["enum"])++ missing = sorted(st.value for st in _ALL_STEP_TYPES if st.value not in declared)+ assert missing == [], f"StepType members the published schema does not declare: {missing}"++ for step_type in _ALL_STEP_TYPES:+ payload = {+ "id": "test-id",+ "name": "Test playbook",+ "version": "1.0.0",+ "trigger": {"on": "alert"},+ "steps": [{"id": "s1", "name": f"Step {step_type.value}", "type": step_type.value, "params": {}}],+ }+ jsonschema.validate(payload, schema)++ def test_the_collapse_helper_is_gone(self) -> None:+ """Keeping the schema honest is only durable if the workaround cannot+ quietly come back — a lossy projection would once again let the+ drafter emit a step whose declared type is not what it asked for."""+ for removed in ("_collapse_step_types_for_schema", "_PYDANTIC_TO_SCHEMA_TYPE", "_SCHEMA_STEP_TYPES"):+ assert not hasattr(nl_drafter, removed), f"{removed} is back; the schema should cover the full StepType range instead"++ def test_the_system_prompt_offers_only_real_vocabulary(self) -> None:+ """The prompt used to restate the vocabulary by hand and had drifted:+ it offered ``webhook`` as a trigger, which no validator in the repo+ accepts, and capped ``retry_max`` at 5 against a model allowing 25."""+ prompt = nl_drafter._SYSTEM_PROMPT+ for step_type in _ALL_STEP_TYPES:+ assert f"``{step_type.value}``" in prompt, f"{step_type.value} missing from the drafter prompt"+ assert "webhook" not in prompt, "prompt offers a trigger no validator accepts"def test_draft_result_to_dict_round_trip(self) -> None:async def _go() -> DraftResult:diff --git a/tests/test_playbook_schema_parity_gate.py b/tests/test_playbook_schema_parity_gate.pynew file mode 100644index 000000000..6af80ec08--- /dev/null+++ b/tests/test_playbook_schema_parity_gate.py@@ -0,0 +1,217 @@+"""Tests for the playbook schema/engine parity gate.++The property worth asserting is not "does the tree happen to agree today" but+"would this gate still notice if it stopped agreeing" — and, specifically,+would it notice in *both* directions. A check that only asks whether the+schema declares something the engine lacks passes forever while the engine+grows verbs nobody tells the schema about, which is exactly how this drift+reached 13 undeclared step types. A sibling gate in this repo compared a YAML+file declaring 17 graph labels against Go code with 28 and printed "OK".++So the assertions are paired: the gate must catch a schema-only step type+*and* an engine-only one; a bound raised on one side *and* the other; a+shipped playbook that stops matching *and* a clean tree passing. The+`--self-test` flag exists so the same proof runs in CI as part of the gate+rather than only here.++The last test is about the gate's own footing: it must refuse to grade a tree+it cannot read rather than print a confident OK about files it never opened.+"""++from __future__ import annotations++import dataclasses+import importlib.util+import json+import subprocess+import sys+from pathlib import Path++import pytest++_REPO = Path(__file__).resolve().parents[1]+_SCRIPT = _REPO / "scripts" / "check_playbook_schema_parity.py"++pytest.importorskip("jsonschema")+pytest.importorskip("httpx")+++def _load_gate():+ spec = importlib.util.spec_from_file_location("check_playbook_schema_parity", _SCRIPT)+ module = importlib.util.module_from_spec(spec)+ sys.modules[spec.name] = module+ spec.loader.exec_module(module)+ return module++[email protected](scope="module")+def gate():+ return _load_gate()++[email protected](scope="module")+def registries(gate):+ return gate.collect(gate._find_repo_root(None))+++# ---------------------------------------------------------------------------+# The tree is actually clean+# ---------------------------------------------------------------------------+++def test_the_repository_passes(registries, gate):+ assert gate.compare(registries) == []+++def test_every_shipped_playbook_matches_the_published_schema(gate):+ errors, scanned = gate.validate_shipped_playbooks(gate._find_repo_root(None))+ assert errors == []+ # The lint job used to scan two files and report "2/2 passed" while 62+ # playbooks under playbooks/packs/v1 went unvalidated, 32 of which did+ # not match. A count assertion is the cheapest guard against that+ # regressing into a gate that passes because it looked at nothing.+ assert len(scanned) > 60, f"only {len(scanned)} playbooks scanned; the glob has probably stopped matching"+++# ---------------------------------------------------------------------------+# ...and the gate would notice if it stopped being clean, in each direction+# ---------------------------------------------------------------------------+++def test_the_bundled_self_test_passes(registries, gate):+ """The gate ships its own drift injection. If this fails, one of the+ directions below has stopped being checked."""+ assert gate.self_test(registries) == []+++def _perturb(registries, **kwargs):+ return dataclasses.replace(registries, **kwargs)+++def test_catches_a_step_type_the_schema_declares_and_the_engine_lacks(registries, gate):+ """The `run_playbook` shape: valid on paper, unparseable at runtime."""+ broken = _perturb(+ registries,+ schema_step_types=registries.schema_step_types | {"run_playbook"},+ execution={**registries.execution, "run_playbook": "unimplemented"},+ )+ assert any("does not implement" in e for e in gate.compare(broken))+++def test_catches_a_step_type_the_engine_accepts_and_the_schema_lacks(registries, gate):+ """The direction things actually drift, and the one a single-sided check+ misses: someone adds a verb to StepType and the published contract never+ hears about it."""+ broken = _perturb(registries, model_step_types=registries.model_step_types | {"quarantine_mailbox"})+ assert any("the schema does not declare it" in e for e in gate.compare(broken))+++def test_catches_an_execution_claim_the_engine_cannot_deliver(registries, gate):+ """`x-aisoc-execution` is the schema telling an author what will happen.+ It has to be checked against the handler table or it is just a promise."""+ running = sorted(k for k, v in registries.execution.items() if v in gate._RUNS)+ victim = running[0]+ broken = _perturb(+ registries,+ handler_step_types=registries.handler_step_types - {victim},+ inline_step_types=registries.inline_step_types - {victim},+ )+ assert any("the engine has no handler for it" in e for e in gate.compare(broken))+++def test_catches_the_schema_understating_what_the_engine_does(registries, gate):+ """The mirror: a handler exists and the annotation says unimplemented."""+ running = sorted(k for k, v in registries.execution.items() if v in gate._RUNS)+ broken = _perturb(registries, execution={**registries.execution, running[0]: "unimplemented"})+ assert any("understating what the product does" in e for e in gate.compare(broken))+++ ("field", "needle"),+ [+ ("schema_timeout_max", "timeout ceiling disagrees"),+ ("schema_retry_max", "retry ceiling disagrees"),+ ],+)+def test_catches_bound_drift(registries, gate, field: str, needle: str):+ """A schema ceiling below the engine's rejects playbooks that would run;+ above it promises headroom Pydantic refuses. Five shipped playbooks+ declared a 620s timeout against a schema maximum of 600."""+ broken = _perturb(registries, **{field: getattr(registries, field) + 1})+ assert any(needle in e for e in gate.compare(broken))+++def test_catches_trigger_vocabulary_drift_in_both_directions(registries, gate):+ schema_wider = _perturb(registries, schema_triggers=registries.schema_triggers | {"webhook"})+ assert any("validate_playbooks.py rejects" in e for e in gate.compare(schema_wider))++ validator_wider = _perturb(registries, validator_triggers=registries.validator_triggers | {"webhook"})+ assert any("the schema rejects" in e for e in gate.compare(validator_wider))+++def test_catches_a_second_schema_file(registries, gate):+ """Two schemas is how the vocabularies diverged to 15 and 9 against an+ engine with 22, and the drafter silently fell back from one to the other."""+ broken = _perturb(registries, schema_files=(*registries.schema_files, "playbook.schema.json"))+ assert any("second source of truth" in e for e in gate.compare(broken))+++def test_catches_playbook_level_key_drift_in_both_directions(registries, gate):+ schema_only = _perturb(registries, schema_playbook_keys=registries.schema_playbook_keys | {"ghost_key"})+ assert any("drops on load" in e for e in gate.compare(schema_only))++ model_only = _perturb(registries, model_playbook_keys=registries.model_playbook_keys | {"ghost_field"})+ assert any("does not declare" in e for e in gate.compare(model_only))+++# ---------------------------------------------------------------------------+# The gate's own footing+# ---------------------------------------------------------------------------+++def test_refuses_to_grade_a_tree_it_cannot_read(gate, tmp_path: Path):+ """Resolving the repo from `__file__` alone is how a gate ends up+ reporting OK about a tree it never opened. Every marker file must be+ present or it declines to run."""+ with pytest.raises(gate.GateError) as exc:+ gate._find_repo_root(str(tmp_path))+ assert "does not look like the AiSOC repository" in str(exc.value)+++def test_refuses_an_empty_step_type_enum(gate, tmp_path: Path):+ """A future refactor emptying the enum must fail loudly, not compare two+ empty sets and pass."""+ with pytest.raises(gate.GateError):+ gate._schema_step_types({"definitions": {"PlaybookStep": {"properties": {"type": {"enum": []}}}}})+++def test_refuses_a_schema_with_no_execution_map(gate):+ with pytest.raises(gate.GateError):+ gate._schema_execution({})+++def test_names_what_it_scanned(gate):+ """A gate that does not say what it read cannot be told apart from one+ that read nothing."""+ proc = subprocess.run(+ [sys.executable, str(_SCRIPT)],+ capture_output=True,+ text=True,+ cwd=str(_REPO),+ )+ assert proc.returncode == 0, proc.stdout + proc.stderr+ for expected in ("repo root", "schema", "step types", "execution", "playbooks scanned"):+ assert expected in proc.stdout, f"gate output does not report {expected!r}"+ assert str(_REPO) in proc.stdout+++def test_the_execution_map_is_not_all_one_value(registries):+ """If every step type were annotated the same way the map would carry no+ information, and the handler cross-check would be trivially satisfiable."""+ assert len(set(registries.execution.values())) > 1+++def test_the_schema_on_disk_is_valid_json_schema(gate):+ jsonschema = pytest.importorskip("jsonschema")+ schema = json.loads((_REPO / "schemas" / "playbook.schema.json").read_text())+ jsonschema.Draft7Validator.check_schema(schema)