Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat(playbooks): add wait, parallel and loop steps with idempotency keys (depth 5.3) (#1207)
Beenu Arora9e5e6ee75a85
diff --git a/apps/docs/docs/concepts/playbooks.md b/apps/docs/docs/concepts/playbooks.mdindex 95be4841a..76f77a076 100644--- a/apps/docs/docs/concepts/playbooks.md+++ b/apps/docs/docs/concepts/playbooks.md@@ -100,6 +100,10 @@ being true.| `http` | Calls one path on an integration this tenant has configured — see [Integration references](#integration-references). SSRF-guarded, and **off by default**. || `close_case` | Marks the AiSOC case as closed via the API service. || `condition` | Pure branching node. Evaluates `condition` and routes to `next_true` / `next_false`. |+| `wait` | Hold for a timer or a callback — see [Control flow](#control-flow). |+| `parallel` | Run the child steps concurrently, then join. |+| `loop` | Run the child steps once per item in a list from the run context, bounded. |+| `osquery_live_query` | Distributed osquery via osctrl / FleetDM / aisoc-direct, against an allow-listed template. |**Governed** — the step is dispatched to the action registry in the actionsservice, which grades it against the verb's own capability contract and the@@ -122,6 +126,33 @@ per verb, per step, at the moment that step runs. See[Live actions](./live-actions.md) for what each verb declares about its impact,reversibility, approval requirement and verification probe.+### Control flow++A playbook could branch and it could not wait, fan out or repeat, so+"contain, then re-check in five minutes" had to be hand-unrolled into a fixed+chain of steps. `wait`, `parallel` and `loop` close that, and each carries a+ceiling rather than trusting the author:++| Step | Shape | Bound |+|------|-------|-------|+| `wait` | `{"until": "timer", "seconds": 300}` or `{"until": "callback"}` | Up to 60s runs in place; longer suspends the run to Postgres and a sweeper resumes it. A callback resumes through `POST /api/v1/playbook-waits/{token}/resume`. Maximum 7 days. |+| `parallel` | `steps: [...]`, `{"join": "all"}` or `{"join": "any"}` | At most 20 branches. Each branch gets its own copy of the run context, so two branches cannot overwrite each other and leave whichever finished last to win. |+| `loop` | `{"over": "sessions"}` with `steps: [...]` | At most 100 iterations, 25 by default. Each iteration binds `item` and `index`. |++Three refusals worth knowing, because a control-flow step that quietly does+less than it says is worse than one that is absent:++- an empty `parallel` or `loop` **fails**, rather than reporting a success+ for work it did not do;+- a `loop` that hits its ceiling **says so**, rather than truncating —+ stopping at 25 of 300 sessions leaves 275 live and a clean-looking record;+- a `wait` whose pause cannot be written **fails**, because continuing past+ it would run the steps the wait exists to delay.++Every step execution carries an **idempotency key** derived from the run, the+step and the position it was reached at, so the third loop iteration is+distinguishable from the fourth and a resumed run reproduces the keys its+first attempt used. It is a name, not a distributed deduplication store.### Integration referencesAn `http` step addresses its target by **name**, never by host:diff --git a/apps/web/src/components/playbooks/packHelpers.ts b/apps/web/src/components/playbooks/packHelpers.tsindex 5abe4c67b..92773f10e 100644--- a/apps/web/src/components/playbooks/packHelpers.ts+++ b/apps/web/src/components/playbooks/packHelpers.ts@@ -255,6 +255,12 @@ const STEP_TYPE_TO_INTEGRATION: Record<StepType, IntegrationType | null> = {// showing a vendor badge for them would overstate what a pack integrates.close_case: null,condition: null,+ // Control flow. They orchestrate other steps and reach nothing of their+ // own, so a vendor badge on them would attribute a pack's integrations+ // to the step that scheduled the work rather than the one that did it.+ wait: null,+ parallel: null,+ loop: null,// Not runnable at all — see `STEP_SCHEMAS.approval`.approval: null,};diff --git a/apps/web/src/components/playbooks/stepSchemas.ts b/apps/web/src/components/playbooks/stepSchemas.tsindex 4b63bcf2d..360fe010c 100644--- a/apps/web/src/components/playbooks/stepSchemas.ts+++ b/apps/web/src/components/playbooks/stepSchemas.ts@@ -164,6 +164,16 @@ const SEVERITIES: readonly FieldOption[] = [{ value: 'critical', label: 'Critical' },];+const WAIT_UNTIL: readonly FieldOption[] = [+ { value: 'timer', label: 'A timer' },+ { value: 'callback', label: 'A callback' },+];++const PARALLEL_JOINS: readonly FieldOption[] = [+ { value: 'all', label: 'All branches must succeed' },+ { value: 'any', label: 'Any one branch is enough' },+];+const OSQUERY_BACKENDS: readonly FieldOption[] = [{ value: 'osctrl', label: 'osctrl' },{ value: 'fleetdm', label: 'FleetDM' },@@ -491,6 +501,83 @@ export const STEP_SCHEMAS: Record<StepType, StepSchema> = {// condition has no params — its predicate lives on `step.condition`.fields: [],},+ wait: {+ type: 'wait',+ execution: 'executed',+ label: 'Wait',+ description:+ 'Hold the run for a timer or until something calls back. A short timer sleeps in place; anything longer becomes a durable pause that survives a restart, on the same mechanism the approval step uses.',+ accent: '#a78bfa',+ bgColor: '#241a40',+ icon: '\u23F3',+ shortCode: 'wait',+ fields: [+ {+ key: 'until',+ label: 'Wait for',+ kind: 'select',+ required: true,+ options: WAIT_UNTIL,+ defaultValue: 'timer',+ },+ {+ key: 'seconds',+ label: 'Seconds',+ kind: 'number',+ placeholder: '300',+ help: 'Required for a timer. Up to 60 seconds runs in place; longer suspends the run and resumes it on a sweep.',+ },+ ],+ },+ parallel: {+ type: 'parallel',+ execution: 'executed',+ label: 'Parallel',+ description:+ 'Run the child steps at the same time and join. Each branch gets its own copy of the run context, so two branches cannot overwrite each other and leave whichever finished last to win.',+ accent: '#38bdf8',+ bgColor: '#102a3a',+ icon: '\u21C9',+ shortCode: 'par',+ fields: [+ {+ key: 'join',+ label: 'Join',+ kind: 'select',+ required: true,+ options: PARALLEL_JOINS,+ defaultValue: 'all',+ },+ ],+ },+ loop: {+ type: 'loop',+ execution: 'executed',+ label: 'Loop',+ description:+ 'Run the child steps once per item in a list from the run context. Bounded: the iteration ceiling is reported in the result rather than silently truncating, because a loop that quietly stopped at 25 of 300 sessions leaves 275 live.',+ accent: '#34d399',+ bgColor: '#10332a',+ icon: '\u21BB',+ shortCode: 'loop',+ fields: [+ {+ key: 'over',+ label: 'Over',+ kind: 'string',+ required: true,+ placeholder: 'sessions',+ help: 'A dot-path into the run context that resolves to a list. Each iteration binds `item` and `index`.',+ },+ {+ key: 'max_iterations',+ label: 'Max iterations',+ kind: 'number',+ placeholder: '25',+ help: 'Capped at 100 whatever is set here.',+ },+ ],+ },osquery_live_query: {type: 'osquery_live_query',execution: 'governed',diff --git a/apps/web/src/components/playbooks/stepVocabulary.test.ts b/apps/web/src/components/playbooks/stepVocabulary.test.tsindex 12e223f98..5783ce604 100644--- a/apps/web/src/components/playbooks/stepVocabulary.test.ts+++ b/apps/web/src/components/playbooks/stepVocabulary.test.ts@@ -61,6 +61,9 @@ const VALID_PARAMS: Record<StepType, Record<string, unknown>> = {run_script: { host: 'web-prod-04', script_name: 'collect-persistence' },search_siem: { query: 'index=proxy dest_ip=203.0.113.10' },create_notable_event: { title: 'Credential access on web-prod-04' },+ wait: { until: 'timer', seconds: 300 },+ parallel: { join: 'all' },+ loop: { over: 'sessions' },};/**@@ -103,6 +106,11 @@ const TARGET_KEY: Record<StepType, string | null> = {force_mfa: 'user',search_siem: 'query',create_notable_event: 'title',+ // Control flow acts on child steps, not on a host or a principal, so+ // there is no scalar target for `_resolve_target` to look for.+ wait: null,+ parallel: null,+ loop: null,};/**diff --git a/apps/web/src/components/playbooks/types.ts b/apps/web/src/components/playbooks/types.tsindex 2db449e10..e7308af0c 100644--- a/apps/web/src/components/playbooks/types.ts+++ b/apps/web/src/components/playbooks/types.ts@@ -39,6 +39,8 @@ export interface PlaybookStep {timeout_seconds: number;next_true?: string;next_false?: string;+ /** Child steps for `parallel` and `loop`. See `@aisoc/types`. */+ steps?: PlaybookStep[];}export interface PlaybookTrigger {diff --git a/packages/types/src/playbook.ts b/packages/types/src/playbook.tsindex d97e67a2a..6f3c56c49 100644--- a/packages/types/src/playbook.ts+++ b/packages/types/src/playbook.ts@@ -39,6 +39,9 @@ export type StepType =| "close_case"| "http"| "condition"+ | "wait"+ | "parallel"+ | "loop"| "osquery_live_query"| "approval"| "disable_user"@@ -93,6 +96,13 @@ export interface PlaybookStep {/** Step id to jump to. The engine branches; it does not build a DAG. */next_true?: string;next_false?: string;+ /**+ * Child steps, for `parallel` (run concurrently, then joined) and `loop`+ * (run in order, once per item). Empty for every other type: the engine+ * fails a `parallel` or `loop` that declares none rather than reporting a+ * success for work it did not do.+ */+ steps?: PlaybookStep[];}export interface Playbook {diff --git a/schemas/playbook.schema.json b/schemas/playbook.schema.jsonindex b23b05fe4..6cd0f0db3 100644--- a/schemas/playbook.schema.json+++ b/schemas/playbook.schema.json@@ -8,7 +8,7 @@"additionalProperties": false,"x-aisoc-execution": {- "description": "What `services/agents/app/playbook/engine.py` does with each step type. `executed` = a handler runs and has a real effect (an outbound call to a service, or engine-internal control flow). `governed` = the step is dispatched through the action registry in `services/actions`, where the capability contract and the tenant autonomy policy decide whether a vendor is touched; the step reports which happened and `executed` is the single field that means one was, so a preview, an approval queue and a missing integration are each named rather than reported as a success. `unimplemented` = no handler is registered; the engine fails the step closed rather than reporting a success it did not achieve. `approval` became `executed` in parity 5.2: it suspends the run to Postgres and resumes on the decision, where before it had nowhere to pause to and failed closed while 12 shipped playbooks aborted on it. This map is machine-checked against the engine's handler table, so it cannot drift into an aspirational claim.",+ "description": "What `services/agents/app/playbook/engine.py` does with each step type. `executed` = a handler runs and has a real effect (an outbound call to a service, or engine-internal control flow). `governed` = the step is dispatched through the action registry in `services/actions`, where the capability contract and the tenant autonomy policy decide whether a vendor is touched; the step reports which happened and `executed` is the single field that means one was, so a preview, an approval queue and a missing integration are each named rather than reported as a success. `unimplemented` = no handler is registered; the engine fails the step closed rather than reporting a success it did not achieve. `approval` became `executed` in parity 5.2: it suspends the run to Postgres and resumes on the decision, where before it had nowhere to pause to and failed closed while 12 shipped playbooks aborted on it. `wait`, `parallel` and `loop` are `executed` and are answered in the run loop rather than from the handler table: they need the run, the HTTP client and the dry-run flag, none of which a handler receives. This map is machine-checked against the engine's handler table *and* its control-flow table, so it cannot drift into an aspirational claim.","enrich": "executed","investigate": "executed","notify": "governed",@@ -30,7 +30,10 @@"run_av_scan": "governed","run_script": "governed","search_siem": "governed",- "create_notable_event": "governed"+ "create_notable_event": "governed",+ "wait": "executed",+ "parallel": "executed",+ "loop": "executed"},"properties": {@@ -175,6 +178,9 @@"http","close_case","condition",+ "wait",+ "parallel",+ "loop","osquery_live_query","block_ip","isolate_host",@@ -231,6 +237,12 @@"type": "string","description": "Step ID to jump to when condition is true (branching)."},+ "steps": {+ "type": "array",+ "items": { "$ref": "#/definitions/PlaybookStep" },+ "default": [],+ "description": "Child steps, for `parallel` (run concurrently, then joined) and `loop` (run in order, once per item). Empty for every other type; the engine fails a `parallel` or `loop` that declares none rather than reporting a success for work it did not do. Nesting is capped at `bounds.MAX_CONTROL_FLOW_DEPTH`."+ },"next_false": {"type": "string","description": "Step ID to jump to when condition is false (branching)."diff --git a/scripts/check_playbook_schema_parity.py b/scripts/check_playbook_schema_parity.pyindex db6c6da49..ac6dc5b06 100644--- a/scripts/check_playbook_schema_parity.py+++ b/scripts/check_playbook_schema_parity.py@@ -284,16 +284,36 @@ def _import_engine(root: Path):def _inline_step_types(engine_mod, models_mod) -> frozenset[str]:"""Step types handled inside the run loop rather than via ``_HANDLERS``.- ``condition`` never reaches the handler table — it branches in the loop- body. Reading the source keeps that visible to the gate; if someone- deletes the branch, the type stops counting as runnable here too.+ Two sources, because the engine has two ways of not using the handler+ table and crediting only one would under-report what the product runs:++ * ``condition`` and ``approval`` branch in the body of+ ``PlaybookEngine.run``. Read from the source, so deleting the branch+ stops the type counting as runnable here too.+ * ``wait``, ``parallel`` and ``loop`` are in ``_CONTROL_FLOW``, a real+ dict read directly. They take the run, the HTTP client and the+ dry-run flag, none of which a ``_HANDLERS`` entry receives — widening+ that signature would hand a mutable run object to fifteen response+ verbs that have no business touching it.++ ``_CONTROL_FLOW`` is required rather than optional. Treating a missing+ table as "no control flow" is how a gate comes to report agreement+ about a registry that was renamed out from under it."""try:source = inspect.getsource(engine_mod.PlaybookEngine.run)except (OSError, TypeError, AttributeError) as exc:raise GateError(f"could not read PlaybookEngine.run source to find inline step handling: {exc}") from excfound = {st.value for st in models_mod.StepType if f"StepType.{st.name}" in source}- return frozenset(found)++ control_flow = getattr(engine_mod, "_CONTROL_FLOW", None)+ if not isinstance(control_flow, dict) or not control_flow:+ raise GateError(+ "engine._CONTROL_FLOW is missing or empty. The control-flow step types are runnable and are not in "+ "_HANDLERS, so without this table the gate would report them as unimplemented — or, if the schema "+ "also lost them, report agreement about a vocabulary three verbs short."+ )+ return frozenset(found | {st.value for st in control_flow})def _model_playbook_keys(models_mod) -> frozenset[str]:diff --git a/scripts/check_route_auth.py b/scripts/check_route_auth.pyindex 269e3d483..f4287972d 100644--- a/scripts/check_route_auth.py+++ b/scripts/check_route_auth.py@@ -200,6 +200,13 @@ IN_BAND_CREDENTIAL_ROUTES: dict[str, tuple[str, str]] = {"vendor webhook authenticated by a per-tenant inbox token in the path plus an HMAC secret, with the connector's tenant cross- ""checked against the token's",),+ "services/agents/app/api/playbooks.py::resume_wait": (+ "find_wait_by_token",+ "a playbook `wait` callback; the caller is whatever the playbook was waiting for — a vendor webhook, a scanner, a ticket "+ "closing — and holds no AiSOC credential. The 32-byte token in the path is the whole credential: unique-indexed so a guess "+ "has one target, minted separately from the pause id (which appears in run records an analyst can read), scoped to "+ "`kind = 'wait'` so it can never resume an approval, and single-use because resuming resolves the pause",+ ),"services/agents/app/api/metrics.py::metrics": ("_authorise","Prometheus scrape; _authorise compares METRICS_TOKEN in constant time and refuses outside a development environment when it is "diff --git a/scripts/check_tenant_query_predicates.py b/scripts/check_tenant_query_predicates.pyindex 32a152b97..4e8ba6d16 100644--- a/scripts/check_tenant_query_predicates.py+++ b/scripts/check_tenant_query_predicates.py@@ -244,6 +244,19 @@ RATCHET: dict[str, str] = {"never sent while the dead-letter view reads empty. The row carries the tenant it belongs to ""and every read surface filters on it"),+ "services/agents/app/playbook/pause.py::find_wait_by_token::aisoc_playbook_pauses": (+ "the resume token is the whole credential, so a caller presenting one has no tenant to filter "+ "by — the same shape as the SCIM-token and WebAuthn-challenge entries above. It is minted "+ "separately from the row id precisely so it cannot be guessed from anything an analyst can "+ "read, and `kind = 'wait'` is in the predicate so an approval can never be resumed this way"+ ),+ "services/agents/app/playbook/pause.py::due_waits::aisoc_playbook_pauses": (+ "the wait sweeper asks 'which waits are due', which cannot presuppose a tenant — the same "+ "discover-a-working-set shape as due_deliveries and report_scheduler.run_once below. It is "+ "read-only; `resolve` claims one row at a time under a tenant, and the two scoped reads "+ "beside it (find_waiting, resolve) do filter, so resuming another tenant's run takes two "+ "mistakes rather than one"+ ),"services/api/app/workers/report_scheduler.py::run_once::ReportTemplate": ("the report scheduler asks 'whose schedule is due', which cannot presuppose a tenant — the same ""reasoning recorded for hunt_scheduler.run_once and oauth_refresh._select_due_connectors, both of "@@ -400,7 +413,22 @@ RATCHET: dict[str, str] = {# "what is due" and `report_scheduler.run_once` is "whose schedule fired" —# both are the discover-a-working-set shape already recorded three times# above, and both act per tenant once they have the set.-MAX_RATCHET = 42+# 42 -> 43 (depth 5.3): the `wait` step's two reads, one of each shape that+# is already recorded here. `find_wait_by_token` is a credential lookup —+# the resume token is the whole credential, minted separately from the row+# id precisely so it is not guessable from anything an analyst can read, and+# a caller presenting one has no tenant to filter by; it is the same shape+# as the SCIM token and WebAuthn challenge entries above. `due_waits` is a+# sweeper discovering its working set, read-only, with `resolve` claiming+# one row at a time under a tenant. The two scoped reads beside them+# (`find_waiting`, `resolve`) do take a tenant and filter on it, so resuming+# another tenant's run still takes two mistakes rather than one.+#+# Two entries added against a ceiling that rises by one, because 42 was a+# ceiling with a spare slot in it and the list held 41. The ceiling now+# equals the list, so the next unscoped statement has to be argued for+# rather than absorbed — which is what this number is for.+MAX_RATCHET = 43# ---------------------------------------------------------------------------diff --git a/scripts/upgrade_playbooks.py b/scripts/upgrade_playbooks.pyindex 7e7fdc570..c0820dac5 100644--- a/scripts/upgrade_playbooks.py+++ b/scripts/upgrade_playbooks.py@@ -55,10 +55,15 @@ STEP_TYPE_RENAMES: dict[str, str] = {}#: v3 step types with no v4 equivalent, and why. Reported, never rewritten.+#+# `wait`, `parallel` and `loop` were here, each with a reason that was true+# when it was written and is not now: depth 5.3 gave the engine a durable+# pause, a fan-out with a join and a bounded iteration construct, and they+# are in `engine._CONTROL_FLOW` and the schema's enum. An entry claiming a+# v3 spelling has no v4 equivalent, for a verb v4 runs, would block an+# upgrade that should succeed — which is what the test beside this table+# checks, and what caught it.UNMAPPABLE: dict[str, str] = {- "loop": "the engine is a single-threaded index walk with no iteration construct",- "parallel": "the engine runs one step at a time; there is no fan-out",- "wait": "the engine has no timer and no pause","run_playbook": ("nested playbooks are deliberately not implemented: a nested playbook's steps are ""not visible where the parent declares its policy, so the parent cannot bound them"diff --git a/services/agents/app/api/playbooks.py b/services/agents/app/api/playbooks.pyindex c5f0839fa..0e1394cf0 100644--- a/services/agents/app/api/playbooks.py+++ b/services/agents/app/api/playbooks.py@@ -26,6 +26,8 @@ from app.playbook import (PlaybookStore,draft_from_nl,)+from app.playbook import engine as engine_module+from app.playbook import pause as playbook_pausefrom app.security.tenant_scope import require_console_or_service_authlogger = logging.getLogger("aisoc.api.playbooks")@@ -35,6 +37,15 @@ logger = logging.getLogger("aisoc.api.playbooks")#: for — a bearer-token-only scheme would lock the browser out.router = APIRouter(prefix="/api/v1/playbooks", tags=["playbooks"], dependencies=[Depends(require_console_or_service_auth)])+#: A second router with no auth dependency, for the one route that cannot+#: have one. See `resume_wait` at the bottom of this file for why, and note+#: that it is a separate router rather than an exemption on the one above:+#: an `Annotated[..., Depends(...)]`-less route inside an authenticated+#: router reads as authenticated to everyone who skims it, and this repo has+#: already shipped eleven routes whose authorization looked present and+#: never ran.+waits_router = APIRouter(prefix="/api/v1/playbook-waits", tags=["playbooks"])+# In-memory run store for Pillar-2 (swap for Redis/DB in production)_runs: dict[str, PlaybookRun] = {}@@ -205,3 +216,46 @@ async def _execute_and_update(playbook: Playbook, context: dict[str, Any], dry_r# Preserve the pre-allocated run_idpr.run_id = run_id_runs[run_id] = pr+++# ---------------------------------------------------------------------------+# Wait callbacks+# ---------------------------------------------------------------------------+++@waits_router.post("/{resume_token}/resume", summary="Resume a playbook run waiting on a callback", status_code=202)+async def resume_wait(resume_token: str, background_tasks: BackgroundTasks) -> dict[str, Any]:+ """Wake a run suspended at a ``wait`` step with ``until: "callback"``.++ Deliberately on its own router with **no** auth dependency, and the+ reason is the shape of the thing: a callback comes from whatever the+ playbook was waiting for — a vendor webhook, a scanner finishing, a+ ticket closing — none of which holds an AiSOC session or a service+ token. The token in the path is the credential. It is 32 random bytes,+ it is unique-indexed so a guess has one target, it is minted separately+ from the pause id (which appears in run records an analyst can read),+ and it is single-use: resuming resolves the pause, so a second delivery+ of the same callback matches no waiting row.++ Returns 202 and resumes in the background. A vendor's webhook sender+ wants a fast 2xx, and holding the connection open for the remainder of+ a playbook is how a sender decides the delivery failed and retries it.+ """+ if not await playbook_pause.find_wait_by_token(resume_token=resume_token):+ # One answer for "no such token" and "already resumed", on purpose.+ # Distinguishing them would turn this route into an oracle for+ # whether a token was ever valid.+ raise HTTPException(status_code=404, detail="no run is waiting on this token")++ background_tasks.add_task(_resume_wait_in_background, resume_token)+ return {"status": "resuming"}+++async def _resume_wait_in_background(resume_token: str) -> None:+ run = await engine_module.resume_wait(resume_token=resume_token)+ if run is None:+ # Not an error: another delivery of the same callback, or the+ # sweeper's timer, got there first.+ logger.info("playbook wait callback matched no waiting run (already resumed)")+ return+ _runs[run.run_id] = rundiff --git a/services/agents/app/main.py b/services/agents/app/main.pyindex fe0162133..4fc0f628a 100644--- a/services/agents/app/main.py+++ b/services/agents/app/main.py@@ -16,6 +16,7 @@ from app.api.hunts import router as hunts_routerfrom app.api.investigate import router as investigate_routerfrom app.api.metrics import router as metrics_routerfrom app.api.playbooks import router as playbook_router+from app.api.playbooks import waits_router as playbook_waits_routerfrom app.api.replay_router import router as replay_routerfrom app.api.router import routerfrom app.api.triage import router as triage_router@@ -35,6 +36,7 @@ from app.hunt import store as hunt_storefrom app.investigator import ledger as investigation_ledgerfrom app.llm.factory import preflight_llmfrom app.playbook import PlaybookStore+from app.playbook import sweeper as playbook_sweeperfrom app.tools.mitre_full import embed_techniques_into_qdrant, load_attck_corpusfrom app.workers.business_context import BusinessContextApplierfrom app.workers.business_context import is_enabled as business_context_enabled@@ -136,6 +138,16 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:except Exception as exc: # noqa: BLE001 — never block API startuplogger.warning("auto_triage_worker.start_failed", error=str(exc))+ # Depth 5.3 — wake timer waits and expire pauses nobody decided.+ # `pause.expire_due` had been written and never called from production,+ # so an undecided approval sat `waiting` forever; a `wait` step inherits+ # that and turns it into a run that stops and never continues.+ app.state.playbook_sweeper_task = None+ try:+ playbook_sweeper.start(app)+ except Exception as exc: # noqa: BLE001 — never block API startup+ logger.warning("playbook_sweeper.start_failed", error=str(exc))+# Phase 2.6 — flip /readyz to 200 once startup work is done.app.state.mark_ready()@@ -153,6 +165,9 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:except Exception as exc: # noqa: BLE001logger.warning("auto_triage_worker.stop_failed", error=str(exc))+ if getattr(app.state, "playbook_sweeper_task", None) is not None:+ app.state.playbook_sweeper_task.cancel()+# Stop the hunt scheduler before draining DB pools so in-flight runs# can flush their writes.try:@@ -209,7 +224,8 @@ app.include_router(metrics_router)app.include_router(router, prefix="/api/v1")app.include_router(investigate_router) # prefix already set in investigate.pyapp.include_router(triage_router) # prefix: /api/v1 (POST /cases/{id}/triage — router topology, T2.2)-app.include_router(playbook_router) # prefix: /api/v1/playbooks+app.include_router(playbook_router)+app.include_router(playbook_waits_router) # prefix: /api/v1/playbook-waitsapp.include_router(contextual_router) # prefix: /api/v1/contextualapp.include_router(hunts_router) # prefix: /api/v1/huntsapp.include_router(hunt_search_router) # prefix: /api/v1/hunt (search + saved)diff --git a/services/agents/app/playbook/bounds.py b/services/agents/app/playbook/bounds.pyindex 4978c2024..d473438d7 100644--- a/services/agents/app/playbook/bounds.py+++ b/services/agents/app/playbook/bounds.py@@ -70,6 +70,50 @@ ABSOLUTE_MAX_RETRIES: Final[int] = 25DEFAULT_MAX_TIMEOUT_SECONDS: Final[int] = 300 # 5 minutesDEFAULT_MAX_RETRIES: Final[int] = 10+# ── Control-flow bounds (`parallel`, `loop`, `wait`) ───────────────────────+# A playbook is author-supplied and the three control-flow steps are the+# first ones that can multiply work rather than do a fixed amount of it, so+# each needs a ceiling that does not depend on the author being reasonable.+#+# - Nesting is capped because a `parallel` of `loop`s of `parallel`s is+# exponential in the depth, and three levels covers every shape the+# packs and the drafter produce.+# - The iteration cap bounds a `loop` whose `over` list came out of an+# enrichment response — i.e. out of attacker-influenced data.+# - The fan-out cap bounds how many outbound calls one step can start at+# once; the branches run concurrently, so this is a concurrency limit on+# somebody else's API as much as on ours.+# - A `wait` longer than the inline ceiling becomes a durable pause instead+# of holding a worker, so the ceiling is about worker occupancy, not+# about how long an author may wait.+MAX_CONTROL_FLOW_DEPTH: Final[int] = 3+ABSOLUTE_MAX_LOOP_ITERATIONS: Final[int] = 100+DEFAULT_MAX_LOOP_ITERATIONS: Final[int] = 25+ABSOLUTE_MAX_PARALLEL_BRANCHES: Final[int] = 20+MAX_INLINE_WAIT_SECONDS: Final[int] = 60+# The longest a `wait` may ask for. Not `clamp_timeout`'s ceiling: that one+# bounds how long a worker may be held on one outbound call, and a durable+# wait holds nothing. Capping a six-hour wait at five minutes would be the+# silent truncation the loop ceiling is explicitly written to avoid.+ABSOLUTE_MAX_WAIT_SECONDS: Final[int] = 7 * 24 * 60 * 60 # 7 days+++def clamp_wait_seconds(value: Any) -> int:+ """Parse and bound a wait duration. 0 means "no timer was asked for".++ `bool` is rejected explicitly rather than coerced: `seconds: true` would+ otherwise become a one-second wait, which is a wait that did not happen+ reported as one that did.+ """+ if value is None or isinstance(value, bool):+ return 0+ try:+ candidate = int(value)+ except (TypeError, ValueError):+ logger.warning("Invalid wait seconds=%r, treating as no timer", value)+ return 0+ return _clamp_int(candidate, 0, ABSOLUTE_MAX_WAIT_SECONDS)+def _clamp_int(value: int, lo: int, hi: int) -> int:if value < lo:@@ -167,13 +211,20 @@ def clamp_retries(value: Any, *, default: int = 0) -> int:__all__ = [+ "ABSOLUTE_MAX_LOOP_ITERATIONS",+ "ABSOLUTE_MAX_PARALLEL_BRANCHES",+ "ABSOLUTE_MAX_WAIT_SECONDS","ABSOLUTE_MAX_PARAM_TIMEOUT_SECONDS","ABSOLUTE_MAX_RETRIES","ABSOLUTE_MAX_TIMEOUT_SECONDS",+ "DEFAULT_MAX_LOOP_ITERATIONS","DEFAULT_MAX_RETRIES","DEFAULT_MAX_TIMEOUT_SECONDS",+ "MAX_CONTROL_FLOW_DEPTH",+ "MAX_INLINE_WAIT_SECONDS","MIN_TIMEOUT_SECONDS","clamp_retries",+ "clamp_wait_seconds","clamp_timeout","max_retries","max_timeout_seconds",diff --git a/services/agents/app/playbook/engine.py b/services/agents/app/playbook/engine.pyindex 77a65d8dd..dad65ed5c 100644--- a/services/agents/app/playbook/engine.py+++ b/services/agents/app/playbook/engine.py@@ -17,9 +17,10 @@ import jsonimport loggingimport osimport re+import secretsimport timeimport uuid-from datetime import UTC, datetime+from datetime import UTC, datetime, timedeltafrom enum import Enumfrom typing import Any@@ -27,8 +28,15 @@ import httpxfrom app.playbook import pause as playbook_pause-from . import action_bridge, references-from .bounds import clamp_timeout+from . import action_bridge, idempotency, references+from .bounds import (+ ABSOLUTE_MAX_LOOP_ITERATIONS,+ ABSOLUTE_MAX_PARALLEL_BRANCHES,+ DEFAULT_MAX_LOOP_ITERATIONS,+ MAX_INLINE_WAIT_SECONDS,+ clamp_timeout,+ clamp_wait_seconds,+)from .errors import PermanentStepFailurefrom .models import Playbook, PlaybookStep, StepCondition, StepTypefrom .ssrf_guard import SSRFError, validate_outbound_url@@ -95,6 +103,10 @@ class PlaybookRun:self.run_id: str = str(uuid.uuid4())self.playbook_id: str = playbook.idself.playbook_name: str = playbook.name+ #: The top-level step list, so a `wait` can find the index it must+ #: resume from. A nested wait has no such index and is refused+ #: rather than written as a pause nothing can use.+ self.steps: list[PlaybookStep] = list(playbook.steps)self.status: RunStatus = RunStatus.PENDINGself.trigger_context: dict[str, Any] = trigger_context# Accumulated output from previous steps — available to later steps as {{prev.*}}@@ -826,6 +838,70 @@ async def _suspend_for_approval(pr: PlaybookRun, step: PlaybookStep, step_idx: i)+async def _suspend_for_wait(+ pr: PlaybookRun,+ step: PlaybookStep,+ http: Any,+ *,+ seconds: int,+ until_callback: bool,+) -> Any:+ """Persist where a long wait stopped, and how it will be woken.++ The step index is found by identity rather than passed in, because a+ wait can sit inside a ``loop`` or a ``parallel`` branch where there is+ no top-level index to resume from. A nested wait that cannot name its+ resume point is refused here rather than written as a pause nothing+ can use — a row that looks resumable and is not is worse than no row.+ """+ tenant_id = str(pr.context.get("tenant_id") or pr.trigger_context.get("tenant_id") or "")+ if not tenant_id:+ logger.warning("Wait step %s has no tenant in context; cannot suspend", step.name)+ return None++ step_index = next((i for i, candidate in enumerate(pr.steps) if candidate.id == step.id), -1)+ if step_index < 0:+ logger.error(+ "Wait step %s is nested inside control flow, so the run has no index to resume from; failing the step closed",+ step.name,+ )+ return None++ return await playbook_pause.suspend(+ kind="wait",+ resume_at=(datetime.now(UTC) + timedelta(seconds=seconds)) if seconds > 0 and not until_callback else None,+ resume_token=secrets.token_urlsafe(32),+ tenant_id=tenant_id,+ run_id=pr.run_id,+ playbook_id=pr.playbook_id,+ playbook_name=pr.playbook_name,+ step_index=step_index,+ step_id=step.id,+ run_context=pr.context,+ step_results=pr.step_results,+ ttl_hours=_wait_ttl(step, seconds),+ )+++def _wait_ttl(step: PlaybookStep, seconds: int) -> float:+ """How long the pause row may sit before it expires with an outcome.++ Comfortably past the wait itself, so a timer that fires a little late+ is still resumable; an author who sets a deadline gets theirs instead.+ """+ raw = step.params.get("expires_in_hours")+ try:+ if raw is not None:+ return float(raw)+ except (TypeError, ValueError):+ # An unparseable deadline falls through to the computed default+ # below rather than failing the step. The value is author-supplied+ # in playbook YAML, and refusing to run a containment playbook over+ # a malformed TTL would be the worse outcome of the two.+ pass+ return max(playbook_pause.DEFAULT_WAIT_TTL_HOURS, (seconds / 3600.0) * 2)++def _approval_ttl(step: PlaybookStep) -> float | None:"""A per-step deadline, if the author set one."""raw = step.params.get("expires_in_hours") or step.params.get("timeout_hours")@@ -929,6 +1005,521 @@ async def resume_after_approval(return await PlaybookEngine().resume(playbook, pause)+async def resume_wait(*, resume_token: str) -> PlaybookRun | None:+ """Continue a run suspended at a ``wait`` step.++ Returns the finished run, or None when the token matches no waiting+ pause. The pause is resolved **before** the run continues, so a+ callback delivered twice — which is the normal behaviour of every+ webhook sender that does not get a 2xx quickly enough — resumes once.+ """+ pause = await playbook_pause.find_wait_by_token(resume_token=resume_token)+ if pause is None:+ return None+ return await _resume_pause(pause, resolution="resumed by callback")+++async def resume_due_waits(*, now: Any = None, limit: int = 50) -> list[str]:+ """Resume every timer wait that has come due. Returns the run ids.++ Driven by ``app.playbook.sweeper``. Without a caller this function and+ the whole durable half of ``wait`` would be the shape this plan keeps+ finding: a mechanism that exists, passes its tests and is reached by+ nothing, so every long wait hangs and the run record says ``paused``+ forever.+ """+ resumed: list[str] = []+ for pause in await playbook_pause.due_waits(now=now, limit=limit):+ run = await _resume_pause(pause, resolution="resumed on its timer")+ if run is not None:+ resumed.append(pause.run_id)+ return resumed+++async def _resume_pause(pause: Any, *, resolution: str) -> PlaybookRun | None:+ """Claim a pause and continue its run. Shared by both wait resumers."""+ claimed = await playbook_pause.resolve(+ pause_id=pause.id,+ tenant_id=pause.tenant_id,+ status="resumed",+ resolution=resolution,+ )+ if not claimed:+ # Another replica, or a second delivery of the same callback, got+ # there first. Not an error.+ return None++ from app.playbook.store import PlaybookStore++ playbook = PlaybookStore.default().get(pause.playbook_id)+ if playbook is None:+ logger.warning(+ "Cannot resume run %s: playbook %s is no longer in the store",+ pause.run_id,+ pause.playbook_id,+ )+ return None+ return await PlaybookEngine().resume(playbook, pause)+++# ---------------------------------------------------------------------------+# Control flow — wait, parallel, loop+# ---------------------------------------------------------------------------+#+# A playbook could branch and it could not wait, fan out or repeat. So+# "contain, then re-check in five minutes", "revoke every session this user+# holds" and "ask three vendors at once" each had to be hand-unrolled into a+# fixed chain of steps, which is why none of the 62 shipped packs attempts+# any of them.+#+# Three properties the three steps share, each of which has an opposite that+# would be worse than not having the step at all:+#+# * **Bounded.** A playbook is author-supplied and these are the first steps+# that multiply work. Iterations, fan-out and nesting all have ceilings in+# `bounds.py`, and the ceiling is reported in the result rather than+# silently truncating.+# * **Honest when empty.** A `parallel` or `loop` with no children, or a+# `loop` over a path that resolved to nothing, fails rather than reporting+# a success for work it did not do. That is the same rule the missing-+# handler branch below exists to enforce.+# * **Idempotent per execution.** Every child carries a key derived from its+# coordinates, so the third iteration is distinguishable from the fourth+# and a resumed run reproduces the same keys. See `idempotency.py`.+++async def _run_children(+ step: PlaybookStep,+ pr: PlaybookRun,+ http: httpx.AsyncClient,+ *,+ dry_run: bool,+ path: tuple[str, ...],+ context: dict[str, Any],+) -> tuple[bool, list[dict]]:+ """Run a child list in order. Returns ``(all_succeeded, results)``.++ Children are executed through ``_invoke_step``, so a child gets the same+ retry policy and the same ``executed`` check a top-level step gets.++ Each child's result is merged into the *local* context rather than the+ run's, so two parallel branches cannot overwrite each other's values and+ then be read by whichever happened to finish last. The branch results+ are attached to the parent step's result, which is what a later step+ reads through ``_step_<parent id>``.+ """+ results: list[dict] = []+ succeeded = True+ for child in step.steps:+ if child.condition and not _evaluate_condition(child.condition, context):+ results.append({"step_id": child.id, "name": child.name, "status": StepStatus.SKIPPED})+ continue+ child_pr = _scoped_run(pr, context)+ status, result = await _invoke_step(child, child_pr, http, dry_run=dry_run, path=path)+ results.append({"step_id": child.id, "name": child.name, "status": status, "result": result})+ for key, value in result.items():+ if not key.startswith("_"):+ context[key] = value+ if status == StepStatus.FAILED:+ succeeded = False+ if child.on_failure == "abort":+ break+ return succeeded, results+++def _scoped_run(pr: PlaybookRun, context: dict[str, Any]) -> PlaybookRun:+ """A view of the run whose context is the child's.++ ``_invoke_step`` reads ``pr.context`` (for the dispatch tenant and the+ response verbs' targets) and ``pr.run_id`` (for the audit trail). A+ child needs the second unchanged and the first scoped, so it gets a+ shallow stand-in rather than a copy of the whole run — copying would+ give a loop iteration its own ``step_results`` and lose them.+ """+ scoped = object.__new__(PlaybookRun)+ scoped.__dict__.update(pr.__dict__)+ scoped.context = context+ return scoped+++async def _control_wait(+ step: PlaybookStep,+ pr: PlaybookRun,+ http: httpx.AsyncClient,+ *,+ dry_run: bool,+ path: tuple[str, ...],+) -> tuple[StepStatus, dict]:+ """Hold the run for a timer or until something calls back.++ Two forms, and the split is about worker occupancy rather than about+ how long an author may wait:++ * a short timer sleeps in process, because suspending to Postgres and+ waking again costs more than the wait;+ * anything longer, or ``until: callback``, becomes a **durable pause**+ on the same table the approval step uses, so it survives a restart.+ A ``wait`` that lived in a process would be lost by a deploy, and the+ run would hang with no record of why.++ A dry run neither sleeps nor suspends: a preview that took five minutes+ to tell an author their playbook is sound is a preview nobody runs.+ """+ until = str(step.params.get("until") or "").strip().lower()+ seconds = clamp_wait_seconds(step.params.get("seconds", step.params.get("duration_seconds")))++ if dry_run:+ return StepStatus.SUCCESS, {+ "dry_run": True,+ "executed": False,+ "would_wait_seconds": seconds,+ "would_wait_for_callback": until == "callback",+ }++ if until != "callback" and 0 < seconds <= MAX_INLINE_WAIT_SECONDS:+ await asyncio.sleep(seconds)+ return StepStatus.SUCCESS, {"waited_seconds": seconds, "durable": False}++ if until != "callback" and seconds <= 0:+ # Refused rather than treated as "no wait". A `wait` step with no+ # duration and no callback is an authoring mistake, and passing it+ # through would make the step a no-op that reports success — which+ # is indistinguishable from a wait that happened.+ return StepStatus.FAILED, {+ "executed": False,+ "error": 'a wait step needs `seconds` or `until: "callback"`; it waited for nothing and did not report that it had',+ }++ pause = await _suspend_for_wait(pr, step, http, seconds=seconds, until_callback=until == "callback")+ if pause is None:+ # Nothing can resume this run, so continuing past the wait would+ # run the steps the wait exists to delay. Same reasoning as the+ # approval step failing closed when its pause cannot be written.+ return StepStatus.FAILED, {+ "executed": False,+ "error": "the wait could not be persisted, so nothing could resume this run; it was failed rather than skipped",+ }+ return StepStatus.PENDING, {+ "paused": True,+ "pause_id": pause.id,+ "resume_token": pause.resume_token,+ "resume_at": pause.resume_at.isoformat() if pause.resume_at else None,+ "durable": True,+ }+++async def _control_parallel(+ step: PlaybookStep,+ pr: PlaybookRun,+ http: httpx.AsyncClient,+ *,+ dry_run: bool,+ path: tuple[str, ...],+) -> tuple[StepStatus, dict]:+ """Run each child concurrently, then join.++ ``join`` is ``all`` (default) or ``any``. ``all`` fails the parallel+ step when any branch failed; ``any`` succeeds when at least one did.+ There is deliberately no "ignore failures" join — that is what+ ``on_failure: continue`` on the parallel step itself already means, and+ spelling it twice would let a playbook say two different things.++ Each branch gets its **own context**, snapshotted from the run before+ the branches start. Sharing one dict would make the merged result+ depend on which branch finished first, and two branches enriching the+ same indicator would race to overwrite each other.+ """+ branches = step.steps+ if not branches:+ return StepStatus.FAILED, {+ "executed": False,+ "error": "a parallel step with no branches runs nothing; declare its `steps` or remove it",+ }+ if len(branches) > ABSOLUTE_MAX_PARALLEL_BRANCHES:+ return StepStatus.FAILED, {+ "executed": False,+ "error": f"a parallel step may fan out to at most {ABSOLUTE_MAX_PARALLEL_BRANCHES} branches; this one declares {len(branches)}",+ }++ join = str(step.params.get("join") or "all").strip().lower()+ if join not in {"all", "any"}:+ return StepStatus.FAILED, {"executed": False, "error": f"unknown join {join!r}; expected 'all' or 'any'"}++ async def _branch(index: int, child: PlaybookStep) -> tuple[bool, dict]:+ branch_path = idempotency.child_path(path, "b", index)+ context = dict(pr.context)+ if child.condition and not _evaluate_condition(child.condition, context):+ return True, {"step_id": child.id, "name": child.name, "status": StepStatus.SKIPPED}+ status, result = await _invoke_step(child, _scoped_run(pr, context), http, dry_run=dry_run, path=branch_path)+ return status != StepStatus.FAILED, {"step_id": child.id, "name": child.name, "status": status, "result": result}++ outcomes = await asyncio.gather(*(_branch(i, child) for i, child in enumerate(branches)))+ succeeded = [ok for ok, _ in outcomes]+ results = [record for _, record in outcomes]++ joined = all(succeeded) if join == "all" else any(succeeded)+ payload: dict[str, Any] = {+ "join": join,+ "branches": results,+ "branches_succeeded": sum(1 for ok in succeeded if ok),+ "branches_total": len(results),+ }+ if not joined:+ payload["error"] = f"{len(results) - sum(succeeded)} of {len(results)} parallel branches failed under join {join!r}"+ return (StepStatus.SUCCESS if joined else StepStatus.FAILED), payload+++async def _control_loop(+ step: PlaybookStep,+ pr: PlaybookRun,+ http: httpx.AsyncClient,+ *,+ dry_run: bool,+ path: tuple[str, ...],+) -> tuple[StepStatus, dict]:+ """Run the child steps once per item, bounded.++ ``over`` is a dot-path into the run context — never a literal list in+ the playbook, which would make the loop a copy-paste of its body. The+ list it resolves to routinely came out of an enrichment response, which+ is to say out of attacker-influenced data, so the iteration count is+ capped and the cap is reported rather than silently truncating.++ Each iteration binds ``item`` and ``index`` into a context of its own,+ so a later iteration cannot read a value an earlier one left behind+ unless the author asked for it through the loop's own result.+ """+ if not step.steps:+ return StepStatus.FAILED, {+ "executed": False,+ "error": "a loop step with no body runs nothing; declare its `steps` or remove it",+ }++ over = str(step.params.get("over") or "").strip()+ if not over:+ return StepStatus.FAILED, {"executed": False, "error": "a loop step needs `over`, a dot-path into the run context"}++ items = _resolve_field(pr.context, over)+ if items is None:+ # Distinct from an empty list below: "the path resolved to nothing"+ # is usually a typo in the path, and reporting it as "zero items"+ # sends the author looking at their data instead of their playbook.+ return StepStatus.FAILED, {"executed": False, "error": f"loop `over` path {over!r} resolved to nothing in the run context"}+ if not isinstance(items, list | tuple):+ return StepStatus.FAILED, {+ "executed": False,+ "error": f"loop `over` path {over!r} resolved to a {type(items).__name__}, not a list",+ }++ requested = len(items)+ ceiling = min(+ int(step.params.get("max_iterations") or DEFAULT_MAX_LOOP_ITERATIONS),+ ABSOLUTE_MAX_LOOP_ITERATIONS,+ )+ truncated = requested > ceiling+ items = list(items)[:ceiling]++ iterations: list[dict] = []+ failures = 0+ for index, item in enumerate(items):+ context = dict(pr.context)+ context["item"] = item+ context["index"] = index+ ok, results = await _run_children(+ step,+ pr,+ http,+ dry_run=dry_run,+ path=idempotency.child_path(path, "i", index),+ context=context,+ )+ iterations.append({"index": index, "item": item, "steps": results})+ if not ok:+ failures += 1+ if step.params.get("on_item_failure", "continue") == "abort":+ break++ payload: dict[str, Any] = {+ "over": over,+ "items_seen": requested,+ "iterations_run": len(iterations),+ "iterations": iterations,+ "failed_iterations": failures,+ }+ if truncated:+ # Said out loud. A loop that quietly stopped at 25 of 300 sessions+ # leaves 275 live and a run record that reads as a clean sweep.+ payload["truncated"] = True+ payload["error"] = (+ f"loop over {over!r} had {requested} items and the ceiling is {ceiling}; {requested - ceiling} were not processed"+ )+ return StepStatus.FAILED, payload+ if failures:+ payload["error"] = f"{failures} of {len(iterations)} loop iterations failed"+ return StepStatus.FAILED, payload+ return StepStatus.SUCCESS, payload+++#: Step types the run loop hands to a control-flow coroutine rather than to+#: ``_HANDLERS``. Keyed here rather than tested with an ``in`` against a set+#: so a type added to one and not the other cannot silently fall through to+#: "no handler".+_CONTROL_FLOW: dict[StepType, Any] = {+ StepType.WAIT: _control_wait,+ StepType.PARALLEL: _control_parallel,+ StepType.LOOP: _control_loop,+}+++async def _invoke_step(+ step: PlaybookStep,+ pr: PlaybookRun,+ http: httpx.AsyncClient,+ *,+ dry_run: bool,+ path: tuple[str, ...] = (),+) -> tuple[StepStatus, dict]:+ """Run one step and report ``(status, result)``.++ Lifted out of ``PlaybookEngine.run`` unchanged so ``parallel`` and+ ``loop`` execute their children through exactly the same path as a+ top-level step — the retry policy, the permanent-failure rule and the+ ``executed`` check are properties of a step, not of where it sits in a+ playbook. A second copy for children would be a second place for "a+ handler that returned is not a step that ran" to drift out of.++ What stays in the caller: branching, cycle detection and the approval+ suspension, all of which are about a step's *position* in a run rather+ than about running it.++ ``path`` is the control-flow coordinates (``("b1", "i3")``). It reaches+ only the idempotency key, which is the one thing that has to tell the+ third loop iteration from the fourth.+ """+ result: dict = {}+ step_status = StepStatus.SUCCESS+ attempt = 0+ handler = _HANDLERS.get(step.type)+ unbridgeable = _UNBRIDGEABLE.get(step.type)+ idempotency_key = idempotency.step_key(run_id=pr.run_id, step_id=step.id, path=path)++ if step.type in _CONTROL_FLOW:+ # Control flow needs the run, the client and the dry-run flag, none+ # of which a `_HANDLERS` entry receives. Routed here rather than by+ # widening the handler signature, so the fifteen response verbs are+ # not handed a mutable run object they have no business touching.+ step_status, result = await _CONTROL_FLOW[step.type](step, pr, http, dry_run=dry_run, path=path)+ result["idempotency_key"] = idempotency_key+ return step_status, result++ if handler is None and not dry_run:+ # Fail closed, and skip the retry loop — a missing handler+ # will still be missing on the next attempt.+ #+ # This branch used to return ``{"skipped": True}`` while+ # leaving step_status at SUCCESS, so twelve of the+ # twenty-two declared step types reported that they had run+ # when nothing had. The worst of them was ``approval``: a+ # human decision point that passed on its own and let the+ # run continue into the very action an analyst was supposed+ # to authorise. Falling through to the shared tail below+ # means the default ``on_failure: abort`` halts the run.+ step_status = StepStatus.FAILED+ result = {+ "error": (+ f"step type {step.type.value!r} is not runnable: {unbridgeable}"+ if unbridgeable+ else f"step type {step.type.value!r} has no handler in this engine"+ ),+ "unimplemented": True,+ "executed": False,+ "_elapsed_ms": 0,+ }+ logger.error(+ "Step %s (%s) has no handler; failing closed rather than reporting success",+ step.name,+ step.type.value,+ )+ else:+ while True:+ attempt += 1+ t0 = time.perf_counter()+ try:+ if dry_run:+ result = {"dry_run": True, "executed": False, "step": step.name}+ if handler is None:+ # A dry run exists to tell the author what+ # would happen. "dry_run: true" alone would+ # imply this step is fine.+ result["unimplemented"] = True+ result["would_fail"] = True+ if unbridgeable:+ result["reason"] = unbridgeable+ elif step.type in RESPONSE_STEP_TYPES:+ # Name the verb a live run would dispatch,+ # so a preview of a containment playbook+ # reads as a containment playbook.+ result["would_dispatch"] = step.type.value+ result["target"] = _resolve_target(step, pr.context)+ else:+ result = await handler(step, pr.context, http)+ elapsed = time.perf_counter() - t0+ result["_elapsed_ms"] = round(elapsed * 1000)+ # A handler that returned is not a step that ran.+ # Every response verb reports `executed`, and a+ # False there means the action was previewed, held+ # for an analyst, blocked, unconfigured or refused+ # — none of which is a step that did what it says.+ # Reporting those as SUCCESS is the defect this+ # whole path exists to remove, so they fail closed+ # and the default `on_failure: abort` halts the run.+ # `not dry_run` carried over from main: a preview reports+ # `executed: false` by construction, so without this the+ # engine called its own preview a failure and all 62 packs+ # previewed as broken. The refactor would have dropped it.+ if not dry_run and step.type in RESPONSE_STEP_TYPES and not result.get("executed"):+ step_status = StepStatus.FAILED+ result.setdefault(+ "error",+ f"{step.type.value} did not execute ({result.get('status', 'unknown')}): "+ f"{result.get('summary') or result.get('detail') or 'no vendor was touched'}",+ )+ break # the handler answered; status is set above+ except Exception as exc: # noqa: BLE001+ elapsed = time.perf_counter() - t0+ # A permanent failure will not become a different+ # failure by being asked again. Sleeping 2s, 4s+ # then 8s before repeating "this run has no+ # tenant" costs an operator fourteen seconds of an+ # incident and, worse, dresses a misconfiguration+ # up as flakiness — so they wait for it to settle+ # instead of going and fixing it.+ permanent = isinstance(exc, PermanentStepFailure)+ logger.error(+ "Step %s attempt %d failed (%s): %s",+ step.name,+ attempt,+ "permanent, not retried" if permanent else "retryable",+ exc,+ )+ if not permanent and attempt <= step.retry_max:+ await asyncio.sleep(min(2**attempt, 30))+ else:+ step_status = StepStatus.FAILED+ result = {+ "error": str(exc),+ # Says, in the run record, why there was+ # one attempt and not four.+ "permanent": permanent,+ "attempts": attempt,+ "_elapsed_ms": round(elapsed * 1000),+ }+ break++ result["idempotency_key"] = idempotency_key+ return step_status, result++class PlaybookEngine:"""Executes playbooks step-by-step, emitting realtime events."""@@ -1019,13 +1610,6 @@ class PlaybookEngine:await _emit(pr.run_id, "step.started", {"step": step.name, "type": step.type}, http)- result: dict = {}- step_status = StepStatus.SUCCESS- attempt = 0- handler = _HANDLERS.get(step.type)-- unbridgeable = _UNBRIDGEABLE.get(step.type)-# Parity 5.2. An approval step is a pause, and this engine# had nowhere to pause to, so it failed closed and 12# shipped playbooks aborted here. The position and the@@ -1063,116 +1647,7 @@ class PlaybookEngine:step.name,)- if handler is None and not dry_run:- # Fail closed, and skip the retry loop — a missing handler- # will still be missing on the next attempt.- #- # This branch used to return ``{"skipped": True}`` while- # leaving step_status at SUCCESS, so twelve of the- # twenty-two declared step types reported that they had run- # when nothing had. The worst of them was ``approval``: a- # human decision point that passed on its own and let the- # run continue into the very action an analyst was supposed- # to authorise. Falling through to the shared tail below- # means the default ``on_failure: abort`` halts the run.- step_status = StepStatus.FAILED- result = {- "error": (- f"step type {step.type.value!r} is not runnable: {unbridgeable}"- if unbridgeable- else f"step type {step.type.value!r} has no handler in this engine"- ),- "unimplemented": True,- "executed": False,- "_elapsed_ms": 0,- }- logger.error(- "Step %s (%s) has no handler; failing closed rather than reporting success",- step.name,- step.type.value,- )- else:- while True:- attempt += 1- t0 = time.perf_counter()- try:- if dry_run:- result = {"dry_run": True, "executed": False, "step": step.name}- if handler is None:- # A dry run exists to tell the author what- # would happen. "dry_run: true" alone would- # imply this step is fine.- result["unimplemented"] = True- result["would_fail"] = True- if unbridgeable:- result["reason"] = unbridgeable- elif step.type in RESPONSE_STEP_TYPES:- # Name the verb a live run would dispatch,- # so a preview of a containment playbook- # reads as a containment playbook.- result["would_dispatch"] = step.type.value- result["target"] = _resolve_target(step, pr.context)- else:- result = await handler(step, pr.context, http)- elapsed = time.perf_counter() - t0- result["_elapsed_ms"] = round(elapsed * 1000)- # A handler that returned is not a step that ran.- # Every response verb reports `executed`, and a- # False there means the action was previewed, held- # for an analyst, blocked, unconfigured or refused- # — none of which is a step that did what it says.- # Reporting those as SUCCESS is the defect this- # whole path exists to remove, so they fail closed- # and the default `on_failure: abort` halts the run.- #- # Except under `dry_run`, where no handler was- # called and `executed: False` is the answer the- # caller asked for. Applying it there made a- # preview of *any* containment playbook report- # FAILED — so the one mode an author uses to check- # a playbook before trusting it could not tell a- # sound playbook from a broken one. The result- # still says `executed: false`; what changes is- # that the engine stops calling its own preview a- # failure.- if not dry_run and step.type in RESPONSE_STEP_TYPES and not result.get("executed"):- step_status = StepStatus.FAILED- result.setdefault(- "error",- f"{step.type.value} did not execute ({result.get('status', 'unknown')}): "- f"{result.get('summary') or result.get('detail') or 'no vendor was touched'}",- )- break # the handler answered; status is set above- except Exception as exc: # noqa: BLE001- elapsed = time.perf_counter() - t0- # A permanent failure will not become a different- # failure by being asked again. Sleeping 2s, 4s- # then 8s before repeating "this run has no- # tenant" costs an operator fourteen seconds of an- # incident and, worse, dresses a misconfiguration- # up as flakiness — so they wait for it to settle- # instead of going and fixing it.- permanent = isinstance(exc, PermanentStepFailure)- logger.error(- "Step %s attempt %d failed (%s): %s",- step.name,- attempt,- "permanent, not retried" if permanent else "retryable",- exc,- )- if not permanent and attempt <= step.retry_max:- await asyncio.sleep(min(2**attempt, 30))- else:- step_status = StepStatus.FAILED- result = {- "error": str(exc),- # Says, in the run record, why there was- # one attempt and not four.- "permanent": permanent,- "attempts": attempt,- "_elapsed_ms": round(elapsed * 1000),- }- break+ step_status, result = await _invoke_step(step, pr, http, dry_run=dry_run)pr.step_results.append({"step_id": step.id, "name": step.name, "status": step_status, "result": result})# Merge result into context for downstream steps. The namespaceddiff --git a/services/agents/app/playbook/idempotency.py b/services/agents/app/playbook/idempotency.pynew file mode 100644index 000000000..03e1a7ea4--- /dev/null+++ b/services/agents/app/playbook/idempotency.py@@ -0,0 +1,67 @@+"""A stable name for one step execution, so repeating it is detectable.++Why a key at all+----------------+Three of the engine's behaviours can run the same step object twice:++* a resumed run replays from a stored index, and a decision that arrives+ twice is routine (a double-tap in the responder app, a retried webhook);+* ``loop`` runs its children once per item;+* ``parallel`` runs its branches concurrently, and a branch that is retried+ must not be confused with the branch beside it.++"The same step" is therefore not the same thing as "the same step id". The+key below is derived from the run, the step, and the *position* the step was+reached at — ``i2`` for the third loop iteration, ``b1`` for the second+parallel branch — so two executions collide exactly when they are the same+logical unit of work and never otherwise.++What it is not+--------------+This is not a distributed deduplication store. Nothing here remembers keys+across runs or across replicas; the key is a name, and the dedupe that uses+it is the engine's own "this step already has a result in this run" check+plus whatever the vendor does with it downstream (PagerDuty's ``dedup_key``+is the obvious one). Claiming more than that would be the kind of+exactly-once promise that needs a transaction log to keep.++Deterministic on purpose: the same step reached the same way in a resumed+run produces the same key, which is what makes a resume safe. That means it+must not read the clock, a UUID, or anything else that changes between the+first attempt and the retry.+"""++from __future__ import annotations++import hashlib++#: Separator that cannot appear in a run id, a step id or a coordinate, so+#: ``("a", "b:c")`` and ``("a:b", "c")`` cannot hash to the same key.+_SEP = "\x1f"+++def step_key(*, run_id: str, step_id: str, path: tuple[str, ...] = ()) -> str:+ """The idempotency key for one step execution.++ ``path`` is the coordinates that got here: ``()`` at the top level,+ ``("i3",)`` inside the fourth loop iteration, ``("b1", "i0")`` for the+ first iteration of a loop inside the second parallel branch.++ Hashed rather than concatenated because the key travels to vendors in+ request bodies, and a raw join would leak the playbook's internal step+ ids and the run id into a third party's incident record. A hash is also+ a fixed length, which matters for the vendors that cap theirs —+ PagerDuty's ``dedup_key`` is 255 characters.+ """+ material = _SEP.join((run_id, step_id, *path))+ return "aisoc-" + hashlib.sha256(material.encode("utf-8")).hexdigest()[:32]+++def child_path(path: tuple[str, ...], kind: str, index: int) -> tuple[str, ...]:+ """Extend a path by one coordinate.++ ``kind`` is a single letter — ``i`` for a loop iteration, ``b`` for a+ parallel branch — so a path reads as ``b1/i3`` in a log line and stays+ short in the hash material.+ """+ return (*path, f"{kind}{index}")diff --git a/services/agents/app/playbook/models.py b/services/agents/app/playbook/models.pyindex dda894594..b3ef7d347 100644--- a/services/agents/app/playbook/models.py+++ b/services/agents/app/playbook/models.py@@ -29,6 +29,14 @@ class StepType(str, Enum):HTTP = "http" # Generic outbound HTTP callCONDITION = "condition" # Branching / gateOSQUERY_LIVE_QUERY = "osquery_live_query" # Distributed osquery via osctrl/FleetDM/aisoc-direct+ # Control flow. A playbook could branch and it could not wait, fan out+ # or repeat, so anything shaped "contain, then check again in five+ # minutes" or "revoke every session this user has" had to be written as+ # a hand-unrolled chain of steps — which is why none of the 62 shipped+ # packs attempts either.+ WAIT = "wait" # Hold for a timer or a callback+ PARALLEL = "parallel" # Run child steps concurrently, then join+ LOOP = "loop" # Run child steps once per item, bounded# Human-in-the-loopAPPROVAL = "approval" # Require analyst approval before proceeding# Identity response@@ -95,6 +103,16 @@ class PlaybookStep(BaseModel):# For branching: step IDs to jump to on true / falsenext_true: str | None = Nonenext_false: str | None = None+ #: Child steps, for ``parallel`` (run concurrently, then joined) and+ #: ``loop`` (run in order, once per item). Empty for every other type,+ #: and the engine fails a ``parallel`` or ``loop`` that declares none+ #: rather than treating it as a no-op that reports success.+ #:+ #: One field for both rather than ``branches`` and ``body``: they are+ #: the same thing — an ordered list of steps the engine owns — and two+ #: fields would be two shapes for the editor, the schema and the+ #: validator to agree about.+ steps: list[PlaybookStep] = Field(default_factory=list)class Playbook(BaseModel):diff --git a/services/agents/app/playbook/pause.py b/services/agents/app/playbook/pause.pyindex 4b2e9dfcf..af192a385 100644--- a/services/agents/app/playbook/pause.py+++ b/services/agents/app/playbook/pause.py@@ -53,6 +53,13 @@ logger = logging.getLogger("aisoc.playbook.pause")DEFAULT_TTL_HOURS = float(os.getenv("AISOC_PLAYBOOK_APPROVAL_TTL_HOURS", "72"))+#: How long a `wait` pause may sit before it expires with a recorded+#: outcome. Longer than an approval's because a wait is often deliberate+#: ("re-check in six hours"), and short enough that a callback nobody ever+#: fires does not leave a run open forever.+DEFAULT_WAIT_TTL_HOURS = float(os.getenv("AISOC_PLAYBOOK_WAIT_TTL_HOURS", "168"))++@dataclass(frozen=True)class Pause:"""A suspended run, as stored."""@@ -67,6 +74,19 @@ class Pause:step_results: list[dict[str, Any]]approval_id: str | Noneexpires_at: datetime+ #: ``approval`` (waiting on a person) or ``wait`` (a clock or a+ #: callback). Defaults to ``approval`` for the same reason the column+ #: does: every row that predates migration 096 is one, and the other+ #: default would have the sweeper resume live approvals with no+ #: decision.+ kind: str = "approval"+ #: When a timer wait becomes due. ``None`` for an approval and for a+ #: callback wait, which means "not on a clock", not "due now".+ resume_at: datetime | None = None+ #: The handle a callback presents. Never the row id: the id appears in+ #: run records an analyst can read, and a callback is unauthenticated+ #: by construction.+ resume_token: str | None = None@propertydef resume_index(self) -> int:@@ -92,6 +112,9 @@ async def _pool() -> Any | None:async def suspend(*,+ kind: str = "approval",+ resume_at: datetime | None = None,+ resume_token: str | None = None,tenant_id: str,run_id: str,playbook_id: str,@@ -123,9 +146,10 @@ async def suspend("""INSERT INTO aisoc_playbook_pauses(id, tenant_id, run_id, playbook_id, playbook_name, step_index,- step_id, run_context, step_results, approval_id, expires_at)+ step_id, run_context, step_results, approval_id, expires_at,+ kind, resume_at, resume_token)VALUES ($1::uuid, $2::uuid, $3, $4, $5, $6, $7, $8::jsonb, $9::jsonb,- $10::uuid, $11)+ $10::uuid, $11, $12, $13, $14)""",pause_id,tenant_id,@@ -138,6 +162,9 @@ async def suspend(json.dumps(step_results, default=str),approval_id,expires_at,+ kind,+ resume_at,+ resume_token,)except Exception as exc: # noqa: BLE001logger.warning("playbook_pause: write failed for run %s: %s", run_id, exc)@@ -161,6 +188,9 @@ async def suspend(step_results=step_results,approval_id=approval_id,expires_at=expires_at,+ kind=kind,+ resume_at=resume_at,+ resume_token=resume_token,)@@ -210,6 +240,15 @@ def _row_to_pause(row: Any) -> Pause:return defaultreturn value+ # `.get` rather than subscripting: `find_waiting` predates migration+ # 096 and does not select the three new columns, and a KeyError there+ # would turn an approval resume into a crash.+ def _column(name: str) -> Any:+ try:+ return row[name]+ except (KeyError, IndexError):+ return None+return Pause(id=str(row["id"]),tenant_id=str(row["tenant_id"]),@@ -221,9 +260,89 @@ def _row_to_pause(row: Any) -> Pause:step_results=_json(row["step_results"], []),approval_id=str(row["approval_id"]) if row["approval_id"] else None,expires_at=row["expires_at"],+ kind=str(_column("kind") or "approval"),+ resume_at=_column("resume_at"),+ resume_token=_column("resume_token"),)+#: The columns a wait lookup needs. Named rather than ``SELECT *`` so an+#: added column cannot change what a row means to ``_row_to_pause``.+_PAUSE_COLUMNS = (+ "id, tenant_id, run_id, playbook_id, step_index, step_id, run_context, "+ "step_results, approval_id, expires_at, kind, resume_at, resume_token"+)+++async def find_wait_by_token(*, resume_token: str) -> Pause | None:+ """The run waiting on this callback token, if any.++ Not scoped on the tenant, and that is deliberate rather than an+ oversight: a callback is unauthenticated by construction, so the caller+ has no tenant to present. The token is the whole credential, which is+ why it is minted separately from the row id (the id appears in run+ records an analyst can read) and is never returned to anything outside+ this service. ``kind = 'wait'`` is in the predicate so an approval can+ never be resumed this way even if a token were somehow written onto one.+ """+ pool = await _pool()+ if pool is None:+ return None+ try:+ async with pool.acquire() as conn:+ row = await conn.fetchrow(+ f"""+ SELECT {_PAUSE_COLUMNS}+ FROM aisoc_playbook_pauses+ WHERE resume_token = $1+ AND kind = 'wait'+ AND status = 'waiting'+ LIMIT 1+ """,+ resume_token,+ )+ except Exception as exc: # noqa: BLE001+ logger.warning("playbook_pause: token lookup failed: %s", exc)+ return None+ return _row_to_pause(row) if row is not None else None+++async def due_waits(*, now: datetime | None = None, limit: int = 50) -> list[Pause]:+ """Timer waits whose deadline has passed.++ Cross-tenant for the same reason ``expire_due`` is, and recorded in+ ``scripts/check_tenant_query_predicates.py`` alongside it: a per-tenant+ sweep needs a list of tenants, and a tenant missing from that list has+ waits that never wake — which is the silent hang this exists to stop.++ Read-only. Claiming a row is ``resolve``'s job and happens one at a+ time, so two replicas sweeping at once resume each wait once.+ """+ pool = await _pool()+ if pool is None:+ return []+ try:+ async with pool.acquire() as conn:+ rows = await conn.fetch(+ f"""+ SELECT {_PAUSE_COLUMNS}+ FROM aisoc_playbook_pauses+ WHERE status = 'waiting'+ AND kind = 'wait'+ AND resume_at IS NOT NULL+ AND resume_at <= $1+ ORDER BY resume_at+ LIMIT $2+ """,+ now or datetime.now(UTC),+ limit,+ )+ except Exception as exc: # noqa: BLE001+ logger.warning("playbook_pause: due-wait sweep failed: %s", exc)+ return []+ return [_row_to_pause(row) for row in rows]++async def resolve(*, pause_id: str, status: str, resolution: str, tenant_id: str) -> bool:"""Mark a pause resumed, denied, expired or cancelled.diff --git a/services/agents/app/playbook/sweeper.py b/services/agents/app/playbook/sweeper.pynew file mode 100644index 000000000..4ebc3a0eb--- /dev/null+++ b/services/agents/app/playbook/sweeper.py@@ -0,0 +1,128 @@+"""The loop that wakes timer waits and expires pauses that nobody decided.++Why this file exists at all+---------------------------+``pause.expire_due`` was written with the approval pause and has had no+production caller since: its only reference in the tree is a test. So an+approval that nobody decided stayed ``waiting`` forever, and the `expired`+outcome the design calls "a decision, not an absence of one" was never+recorded by anything.++A ``wait`` step inherits that hole and makes it worse, because an approval+at least has a human who might come back. A timer wait with no sweeper is a+run that stops and never continues, reporting ``paused`` indefinitely.++Three properties, each the opposite of a way this could go wrong+-----------------------------------------------------------------+**Tells permanent from transient.** An unreachable database is transient and+is retried with a capped backoff. A missing playbook is permanent for that+pause, and the resume path records it rather than retrying forever.++**Says so when it cannot do its job.** A sweeper that silently stops is+indistinguishable from one with nothing to do. Every error is logged at+``warning`` with the reason, and consecutive failures escalate the interval+rather than hammering a store that is down.++**Safe to run on several replicas.** Claiming a pause is a conditional+``UPDATE ... WHERE status = 'waiting'``, so two sweepers racing the same row+resume it once; the loser sees zero rows changed and moves on.+"""++from __future__ import annotations++import asyncio+import contextlib+import logging+import os++logger = logging.getLogger("aisoc.playbook.sweeper")++#: How often to look. A wait is a coarse instrument — the shortest one that+#: reaches this loop is already longer than the inline ceiling — so a minute+#: of latency on a six-hour wait costs nothing and a tight poll would be a+#: query per second against a table that is empty most of the time.+DEFAULT_INTERVAL_SECONDS = float(os.getenv("AISOC_PLAYBOOK_SWEEP_INTERVAL_SECONDS", "60"))++#: Ceiling for the backoff a failing store earns. Past this the sweeper is+#: still trying, just not pretending the next attempt is urgent.+_MAX_BACKOFF_SECONDS = 600.0+++def enabled() -> bool:+ """Off only by explicit opt-out.++ On by default, unlike everything in this wave that reaches a vendor:+ this reaches nothing outward. Its job is to finish work the product has+ already started, and the failure mode of not running it is a run that+ hangs — which is the defect, not the safe state.+ """+ return os.getenv("AISOC_PLAYBOOK_SWEEPER_DISABLE", "").strip().lower() not in {"1", "true", "yes", "on"}+++async def sweep_once() -> dict[str, int]:+ """One pass: resume what is due, expire what is past its deadline.++ Resume first. A pause whose timer came due in the same tick that its+ TTL expired should run, not be cancelled — the run did what it was+ asked to, slightly late, and expiring it would discard work for a+ scheduling detail.+ """+ from app.playbook import engine+ from app.playbook import pause as playbook_pause++ resumed = await engine.resume_due_waits()+ expired = await playbook_pause.expire_due()+ if resumed or expired:+ logger.info("playbook_sweeper: resumed %d wait(s), expired %d pause(s)", len(resumed), len(expired))+ return {"resumed": len(resumed), "expired": len(expired)}+++async def run_forever(*, interval_seconds: float | None = None) -> None:+ """Sweep on a fixed cadence until cancelled."""+ interval = interval_seconds or DEFAULT_INTERVAL_SECONDS+ consecutive_failures = 0+ while True:+ try:+ await sweep_once()+ consecutive_failures = 0+ delay = interval+ except asyncio.CancelledError:+ raise+ except Exception as exc: # noqa: BLE001 — a failed sweep must not end the loop+ consecutive_failures += 1+ delay = min(interval * (2**consecutive_failures), _MAX_BACKOFF_SECONDS)+ logger.warning(+ "playbook_sweeper: pass %d failed (%s); next attempt in %.0fs",+ consecutive_failures,+ exc,+ delay,+ )+ await asyncio.sleep(delay)+++def start(app: object) -> asyncio.Task | None:+ """Start the sweeper as a background task on the app, or say why not."""+ if not enabled():+ logger.info("playbook_sweeper: disabled by AISOC_PLAYBOOK_SWEEPER_DISABLE")+ return None+ task = asyncio.create_task(run_forever())+ task.add_done_callback(_log_exit)+ with contextlib.suppress(AttributeError):+ app.state.playbook_sweeper_task = task # type: ignore[attr-defined]+ logger.info("playbook_sweeper: started, every %.0fs", DEFAULT_INTERVAL_SECONDS)+ return task+++def _log_exit(task: asyncio.Task) -> None:+ """Retrieve the task's outcome so a dead sweeper is in the log.++ The same pairing the triage worker and the fusion consumer carry. A+ background task whose exception is never retrieved dies quietly, and+ the only symptom is runs that stop resuming — which looks like a+ playbook problem rather than a missing sweeper.+ """+ if task.cancelled():+ return+ exc = task.exception()+ if exc is not None:+ logger.error("playbook_sweeper: exited on %s; waits will no longer resume", exc)diff --git a/services/agents/tests/test_playbook_control_flow.py b/services/agents/tests/test_playbook_control_flow.pynew file mode 100644index 000000000..8784bd624--- /dev/null+++ b/services/agents/tests/test_playbook_control_flow.py@@ -0,0 +1,650 @@+"""Depth 5.3 — `wait`, `parallel` and `loop`, and the keys that make them safe.++Reproduce, on the tree this started from: ``StepType`` had 22 members and+none of them was any of these three. ``Playbook.model_validate`` rejected+``{"type": "wait"}`` at ``steps.0.type``, the published union in+``packages/types`` had 22 members and the console palette could not offer+one. So "contain, then re-check in five minutes", "revoke every session this+user holds" and "ask three vendors at once" each had to be hand-unrolled+into a fixed chain, which is why none of the 62 shipped packs attempts any+of them.++What is asserted here is mostly the *refusals*, because a control-flow step+that quietly does less than it says is worse than one that is absent:++* an empty ``parallel`` or ``loop`` fails rather than reporting a success+ for work it did not do;+* a ``loop`` that hits its ceiling says so instead of silently truncating —+ stopping at 25 of 300 sessions leaves 275 live and a clean-looking record;+* a ``wait`` with neither a duration nor a callback fails rather than+ becoming a no-op that reads exactly like a wait that happened;+* a ``wait`` whose pause cannot be written fails, because continuing past it+ runs the steps the wait exists to delay.+"""++from __future__ import annotations++import asyncio+from datetime import UTC, datetime+from typing import Any++import pytest+from app.playbook import engine as engine_mod+from app.playbook import idempotency+from app.playbook.bounds import ABSOLUTE_MAX_LOOP_ITERATIONS, ABSOLUTE_MAX_WAIT_SECONDS, MAX_INLINE_WAIT_SECONDS+from app.playbook.engine import PlaybookEngine, RunStatus, StepStatus+from app.playbook.models import Playbook, PlaybookStep, StepCondition, StepType++pytestmark = pytest.mark.asyncio+++def _playbook(*steps: PlaybookStep) -> Playbook:+ return Playbook(id="pb-cf", name="control flow", steps=list(steps))+++def _child(step_id: str, **kwargs: Any) -> PlaybookStep:+ """A child step whose handler is replaced per test."""+ return PlaybookStep(id=step_id, name=step_id, type=StepType.ENRICH, **kwargs)+++def enrich(monkeypatch: pytest.MonkeyPatch) -> list[dict[str, Any]]:+ """Record every enrich call, with the context it saw."""+ calls: list[dict[str, Any]] = []++ async def _handler(step: PlaybookStep, context: dict[str, Any], http: Any) -> dict:+ calls.append({"step_id": step.id, "item": context.get("item"), "index": context.get("index")})+ return {"ok": True, "seen": step.id}++ monkeypatch.setitem(engine_mod._HANDLERS, StepType.ENRICH, _handler)+ return calls+++# ---------------------------------------------------------------------------+# The vocabulary+# ---------------------------------------------------------------------------+++def test_the_three_types_exist_and_are_runnable() -> None:+ for name in ("wait", "parallel", "loop"):+ assert name in {s.value for s in StepType}+ assert set(engine_mod._CONTROL_FLOW) == {StepType.WAIT, StepType.PARALLEL, StepType.LOOP}+++def test_a_control_flow_step_is_not_in_the_handler_table() -> None:+ """They need the run, the client and the dry-run flag.++ Pinned because the obvious "fix" for the gate is to put them in+ `_HANDLERS` with a widened signature, which would hand a mutable run+ object to fifteen response verbs that have no business touching it.+ """+ for step_type in engine_mod._CONTROL_FLOW:+ assert step_type not in engine_mod._HANDLERS+++# ---------------------------------------------------------------------------+# wait+# ---------------------------------------------------------------------------+++class TestWait:+ async def test_a_short_timer_sleeps_in_place(self, monkeypatch: pytest.MonkeyPatch) -> None:+ slept: list[float] = []+ real_sleep = asyncio.sleep++ async def _recording(delay: float, *args: Any, **kwargs: Any) -> Any:+ slept.append(delay)+ return await real_sleep(0, *args, **kwargs)++ monkeypatch.setattr(engine_mod.asyncio, "sleep", _recording)+ run = await PlaybookEngine().run(+ _playbook(PlaybookStep(id="w", name="hold", type=StepType.WAIT, params={"seconds": 5})),+ {"tenant_id": "t"},+ )+ assert run.status is RunStatus.COMPLETED+ assert slept == [5]+ assert run.step_results[0]["result"]["durable"] is False++ async def test_a_long_timer_becomes_a_durable_pause(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """Past the inline ceiling it suspends, because a wait held in a+ process is lost by the deploy most likely to interrupt it."""+ suspended: dict[str, Any] = {}++ async def _suspend(**kwargs: Any) -> Any:+ suspended.update(kwargs)++ class _Pause:+ id = "pause-1"+ resume_token = "tok"+ resume_at = None++ return _Pause()++ monkeypatch.setattr(engine_mod.playbook_pause, "suspend", _suspend)+ run = await PlaybookEngine().run(+ _playbook(PlaybookStep(id="w", name="hold", type=StepType.WAIT, params={"seconds": MAX_INLINE_WAIT_SECONDS + 1})),+ {"tenant_id": "t"},+ )+ assert suspended["kind"] == "wait"+ assert suspended["resume_at"] is not None+ assert suspended["resume_token"]+ assert run.step_results[0]["status"] is StepStatus.PENDING++ async def test_a_callback_wait_has_no_timer(self, monkeypatch: pytest.MonkeyPatch) -> None:+ suspended: dict[str, Any] = {}++ async def _suspend(**kwargs: Any) -> Any:+ suspended.update(kwargs)+ return type("P", (), {"id": "p", "resume_token": "tok", "resume_at": None})()++ monkeypatch.setattr(engine_mod.playbook_pause, "suspend", _suspend)+ await PlaybookEngine().run(+ _playbook(PlaybookStep(id="w", name="hold", type=StepType.WAIT, params={"until": "callback"})),+ {"tenant_id": "t"},+ )+ assert suspended["resume_at"] is None, "a callback wait must not also be on a clock"++ @pytest.mark.parametrize("params", [{}, {"seconds": 0}, {"seconds": "soon"}, {"seconds": True}])+ async def test_a_wait_with_nothing_to_wait_for_fails(self, params: dict[str, Any]) -> None:+ """A no-op that reports success reads exactly like a wait that happened.++ ``seconds: true`` is in here because ``int(True)`` is 1: coercing it+ would turn "this field is malformed" into a one-second wait that the+ run record calls a wait.+ """+ run = await PlaybookEngine().run(+ _playbook(PlaybookStep(id="w", name="hold", type=StepType.WAIT, params=params)),+ {"tenant_id": "t"},+ )+ assert run.status is RunStatus.FAILED+ assert "waited for nothing" in run.step_results[0]["result"]["error"]++ async def test_a_long_wait_is_not_clamped_to_the_step_timeout(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """A six-hour wait must stay six hours.++ ``clamp_timeout`` bounds how long a worker may be held on one+ outbound call and tops out at five minutes. Reusing it here — which+ the first version of this did — silently turned every long wait into+ a five-minute one, which is the same silent truncation the loop+ ceiling is explicitly written to avoid.+ """+ suspended: dict[str, Any] = {}++ async def _suspend(**kwargs: Any) -> Any:+ suspended.update(kwargs)+ return type("P", (), {"id": "p", "resume_token": "tok", "resume_at": None})()++ monkeypatch.setattr(engine_mod.playbook_pause, "suspend", _suspend)+ await PlaybookEngine().run(+ _playbook(PlaybookStep(id="w", name="hold", type=StepType.WAIT, params={"seconds": 6 * 3600})),+ {"tenant_id": "t"},+ )+ waited = (suspended["resume_at"] - datetime.now(UTC)).total_seconds()+ assert 6 * 3600 - 60 < waited <= 6 * 3600++ async def test_a_wait_beyond_the_absolute_ceiling_is_bounded(self, monkeypatch: pytest.MonkeyPatch) -> None:+ async def _suspend(**kwargs: Any) -> Any:+ return type("P", (), {"id": "p", "resume_token": "tok", "resume_at": None})()++ monkeypatch.setattr(engine_mod.playbook_pause, "suspend", _suspend)+ run = await PlaybookEngine().run(+ _playbook(PlaybookStep(id="w", name="hold", type=StepType.WAIT, params={"seconds": 10**9})),+ {"tenant_id": "t"},+ dry_run=True,+ )+ assert run.step_results[0]["result"]["would_wait_seconds"] == ABSOLUTE_MAX_WAIT_SECONDS++ async def test_a_pause_that_cannot_be_written_fails_closed(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """Continuing would run the steps the wait exists to delay."""++ async def _no_pause(**kwargs: Any) -> Any:+ return None++ monkeypatch.setattr(engine_mod.playbook_pause, "suspend", _no_pause)+ ran: list[str] = []++ async def _after(step: PlaybookStep, context: dict[str, Any], http: Any) -> dict:+ ran.append(step.id)+ return {"ok": True}++ monkeypatch.setitem(engine_mod._HANDLERS, StepType.ENRICH, _after)+ run = await PlaybookEngine().run(+ _playbook(+ PlaybookStep(id="w", name="hold", type=StepType.WAIT, params={"until": "callback"}),+ _child("after"),+ ),+ {"tenant_id": "t"},+ )+ assert run.status is RunStatus.FAILED+ assert ran == [], "the run continued past a wait nothing could resume"++ async def test_a_preview_neither_sleeps_nor_suspends(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """A preview that took five minutes is a preview nobody runs."""++ async def _boom(**kwargs: Any) -> Any:+ raise AssertionError("a dry run suspended")++ monkeypatch.setattr(engine_mod.playbook_pause, "suspend", _boom)+ run = await PlaybookEngine().run(+ _playbook(PlaybookStep(id="w", name="hold", type=StepType.WAIT, params={"seconds": 3600})),+ {"tenant_id": "t"},+ dry_run=True,+ )+ assert run.status is RunStatus.COMPLETED+ assert run.step_results[0]["result"]["would_wait_seconds"] == 3600+++# ---------------------------------------------------------------------------+# parallel+# ---------------------------------------------------------------------------+++class TestParallel:+ async def test_branches_all_run_and_the_step_joins(self, enrich: list[dict[str, Any]]) -> None:+ run = await PlaybookEngine().run(+ _playbook(+ PlaybookStep(+ id="p",+ name="fan out",+ type=StepType.PARALLEL,+ steps=[_child("a"), _child("b"), _child("c")],+ )+ ),+ {"tenant_id": "t"},+ )+ assert run.status is RunStatus.COMPLETED+ assert {call["step_id"] for call in enrich} == {"a", "b", "c"}+ assert run.step_results[0]["result"]["branches_succeeded"] == 3++ async def test_branches_run_concurrently(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """Sequential execution would satisfy every other assertion here."""+ running = 0+ peak = 0++ async def _handler(step: PlaybookStep, context: dict[str, Any], http: Any) -> dict:+ nonlocal running, peak+ running += 1+ peak = max(peak, running)+ await asyncio.sleep(0)+ running -= 1+ return {"ok": True}++ monkeypatch.setitem(engine_mod._HANDLERS, StepType.ENRICH, _handler)+ await PlaybookEngine().run(+ _playbook(PlaybookStep(id="p", name="fan out", type=StepType.PARALLEL, steps=[_child("a"), _child("b"), _child("c")])),+ {"tenant_id": "t"},+ )+ assert peak > 1, "the branches ran one after another"++ async def test_a_branch_cannot_overwrite_another_branchs_context(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """Each branch gets its own copy, so the merged result does not+ depend on which finished first."""+ seen: list[Any] = []++ async def _handler(step: PlaybookStep, context: dict[str, Any], http: Any) -> dict:+ seen.append(context.get("verdict"))+ return {"verdict": step.id}++ monkeypatch.setitem(engine_mod._HANDLERS, StepType.ENRICH, _handler)+ await PlaybookEngine().run(+ _playbook(PlaybookStep(id="p", name="fan out", type=StepType.PARALLEL, steps=[_child("a"), _child("b")])),+ {"tenant_id": "t", "verdict": "initial"},+ )+ assert seen == ["initial", "initial"]++ async def test_join_all_fails_when_one_branch_fails(self, monkeypatch: pytest.MonkeyPatch) -> None:+ async def _handler(step: PlaybookStep, context: dict[str, Any], http: Any) -> dict:+ if step.id == "b":+ raise RuntimeError("vendor said no")+ return {"ok": True}++ monkeypatch.setitem(engine_mod._HANDLERS, StepType.ENRICH, _handler)+ run = await PlaybookEngine().run(+ _playbook(PlaybookStep(id="p", name="fan out", type=StepType.PARALLEL, steps=[_child("a"), _child("b")])),+ {"tenant_id": "t"},+ )+ assert run.status is RunStatus.FAILED+ assert "1 of 2 parallel branches failed" in run.step_results[0]["result"]["error"]++ async def test_join_any_succeeds_when_one_branch_succeeds(self, monkeypatch: pytest.MonkeyPatch) -> None:+ async def _handler(step: PlaybookStep, context: dict[str, Any], http: Any) -> dict:+ if step.id == "b":+ raise RuntimeError("vendor said no")+ return {"ok": True}++ monkeypatch.setitem(engine_mod._HANDLERS, StepType.ENRICH, _handler)+ run = await PlaybookEngine().run(+ _playbook(+ PlaybookStep(id="p", name="fan out", type=StepType.PARALLEL, params={"join": "any"}, steps=[_child("a"), _child("b")])+ ),+ {"tenant_id": "t"},+ )+ assert run.status is RunStatus.COMPLETED++ async def test_a_parallel_with_no_branches_fails(self) -> None:+ run = await PlaybookEngine().run(+ _playbook(PlaybookStep(id="p", name="fan out", type=StepType.PARALLEL)),+ {"tenant_id": "t"},+ )+ assert run.status is RunStatus.FAILED+ assert "runs nothing" in run.step_results[0]["result"]["error"]++ async def test_an_unknown_join_is_refused_rather_than_defaulted(self) -> None:+ run = await PlaybookEngine().run(+ _playbook(PlaybookStep(id="p", name="fan out", type=StepType.PARALLEL, params={"join": "most"}, steps=[_child("a")])),+ {"tenant_id": "t"},+ )+ assert run.status is RunStatus.FAILED+ assert "unknown join" in run.step_results[0]["result"]["error"]++ async def test_a_branch_condition_is_honoured(self, enrich: list[dict[str, Any]]) -> None:+ await PlaybookEngine().run(+ _playbook(+ PlaybookStep(+ id="p",+ name="fan out",+ type=StepType.PARALLEL,+ steps=[_child("a"), _child("b", condition=StepCondition(expression="severity == 'critical'"))],+ )+ ),+ {"tenant_id": "t", "severity": "low"},+ )+ assert {call["step_id"] for call in enrich} == {"a"}+++# ---------------------------------------------------------------------------+# loop+# ---------------------------------------------------------------------------+++class TestLoop:+ async def test_the_body_runs_once_per_item(self, enrich: list[dict[str, Any]]) -> None:+ run = await PlaybookEngine().run(+ _playbook(PlaybookStep(id="l", name="each", type=StepType.LOOP, params={"over": "sessions"}, steps=[_child("revoke")])),+ {"tenant_id": "t", "sessions": ["s1", "s2", "s3"]},+ )+ assert run.status is RunStatus.COMPLETED+ assert [call["item"] for call in enrich] == ["s1", "s2", "s3"]+ assert [call["index"] for call in enrich] == [0, 1, 2]++ async def test_a_path_that_resolves_to_nothing_is_not_zero_items(self) -> None:+ """A typo in the path and an empty list are different facts.++ Reporting "zero items" for a bad path sends the author to look at+ their data instead of their playbook.+ """+ run = await PlaybookEngine().run(+ _playbook(PlaybookStep(id="l", name="each", type=StepType.LOOP, params={"over": "sessionz"}, steps=[_child("revoke")])),+ {"tenant_id": "t", "sessions": ["s1"]},+ )+ assert run.status is RunStatus.FAILED+ assert "resolved to nothing" in run.step_results[0]["result"]["error"]++ async def test_a_non_list_is_refused(self) -> None:+ run = await PlaybookEngine().run(+ _playbook(PlaybookStep(id="l", name="each", type=StepType.LOOP, params={"over": "user"}, steps=[_child("revoke")])),+ {"tenant_id": "t", "user": "alice"},+ )+ assert run.status is RunStatus.FAILED+ assert "not a list" in run.step_results[0]["result"]["error"]++ async def test_the_ceiling_is_reported_not_silently_applied(self, enrich: list[dict[str, Any]]) -> None:+ """A loop that stopped at 25 of 300 sessions leaves 275 live."""+ run = await PlaybookEngine().run(+ _playbook(+ PlaybookStep(+ id="l",+ name="each",+ type=StepType.LOOP,+ params={"over": "sessions", "max_iterations": 2},+ steps=[_child("revoke")],+ )+ ),+ {"tenant_id": "t", "sessions": ["s1", "s2", "s3", "s4"]},+ )+ assert run.status is RunStatus.FAILED+ result = run.step_results[0]["result"]+ assert result["truncated"] is True+ assert result["items_seen"] == 4+ assert result["iterations_run"] == 2+ assert "2 were not processed" in result["error"]++ async def test_the_absolute_ceiling_cannot_be_raised_by_a_playbook(self, enrich: list[dict[str, Any]]) -> None:+ """`over` routinely resolves to an enrichment response, which is to+ say to attacker-influenced data."""+ run = await PlaybookEngine().run(+ _playbook(+ PlaybookStep(+ id="l",+ name="each",+ type=StepType.LOOP,+ params={"over": "sessions", "max_iterations": 10_000},+ steps=[_child("revoke")],+ )+ ),+ {"tenant_id": "t", "sessions": [f"s{i}" for i in range(ABSOLUTE_MAX_LOOP_ITERATIONS + 5)]},+ )+ assert run.status is RunStatus.FAILED+ assert run.step_results[0]["result"]["iterations_run"] == ABSOLUTE_MAX_LOOP_ITERATIONS++ async def test_a_loop_with_no_body_fails(self) -> None:+ run = await PlaybookEngine().run(+ _playbook(PlaybookStep(id="l", name="each", type=StepType.LOOP, params={"over": "sessions"})),+ {"tenant_id": "t", "sessions": ["s1"]},+ )+ assert run.status is RunStatus.FAILED+ assert "runs nothing" in run.step_results[0]["result"]["error"]++ async def test_a_failed_iteration_is_counted_and_reported(self, monkeypatch: pytest.MonkeyPatch) -> None:+ async def _handler(step: PlaybookStep, context: dict[str, Any], http: Any) -> dict:+ if context.get("index") == 1:+ raise RuntimeError("vendor said no")+ return {"ok": True}++ monkeypatch.setitem(engine_mod._HANDLERS, StepType.ENRICH, _handler)+ run = await PlaybookEngine().run(+ _playbook(PlaybookStep(id="l", name="each", type=StepType.LOOP, params={"over": "sessions"}, steps=[_child("revoke")])),+ {"tenant_id": "t", "sessions": ["s1", "s2", "s3"]},+ )+ assert run.status is RunStatus.FAILED+ result = run.step_results[0]["result"]+ assert result["failed_iterations"] == 1+ assert result["iterations_run"] == 3, "the default is to carry on through a failed item"+++# ---------------------------------------------------------------------------+# idempotency keys+# ---------------------------------------------------------------------------+++class TestIdempotencyKeys:+ def test_a_key_is_stable_for_the_same_step_in_the_same_run(self) -> None:+ """This is what makes a resume safe: the retry carries the key the+ first attempt did."""+ first = idempotency.step_key(run_id="r1", step_id="s1")+ assert first == idempotency.step_key(run_id="r1", step_id="s1")++ def test_keys_differ_by_run_step_and_position(self) -> None:+ keys = {+ idempotency.step_key(run_id="r1", step_id="s1"),+ idempotency.step_key(run_id="r2", step_id="s1"),+ idempotency.step_key(run_id="r1", step_id="s2"),+ idempotency.step_key(run_id="r1", step_id="s1", path=("i0",)),+ idempotency.step_key(run_id="r1", step_id="s1", path=("i1",)),+ idempotency.step_key(run_id="r1", step_id="s1", path=("b0", "i1")),+ }+ assert len(keys) == 6++ def test_the_separator_cannot_be_forged_by_an_id(self) -> None:+ """``("a", "b:c")`` and ``("a:b", "c")`` must not collide.++ Ids reach this from playbook content, so a separator an author can+ type would let two different steps share a key — and a shared key+ is a deduplicated action somebody expected to happen twice.+ """+ assert idempotency.step_key(run_id="a", step_id="b:c") != idempotency.step_key(run_id="a:b", step_id="c")+ assert idempotency.step_key(run_id="a", step_id="b-c") != idempotency.step_key(run_id="a-b", step_id="c")++ def test_a_key_does_not_leak_the_run_or_step_id(self) -> None:+ """Keys travel to vendors in request bodies."""+ key = idempotency.step_key(run_id="run-secret", step_id="step-secret")+ assert "run-secret" not in key+ assert "step-secret" not in key++ async def test_every_step_result_carries_its_key(self, enrich: list[dict[str, Any]]) -> None:+ run = await PlaybookEngine().run(_playbook(_child("a"), _child("b")), {"tenant_id": "t"})+ keys = [r["result"]["idempotency_key"] for r in run.step_results]+ assert all(keys)+ assert len(set(keys)) == 2++ async def test_loop_iterations_get_distinct_keys(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """Without the iteration coordinate every revoke in a loop would+ carry one key, and a vendor deduplicating on it would act once."""+ run = await PlaybookEngine().run(+ _playbook(PlaybookStep(id="l", name="each", type=StepType.LOOP, params={"over": "sessions"}, steps=[_child("revoke")])),+ {"tenant_id": "t", "sessions": ["s1", "s2", "s3"]},+ )+ iterations = run.step_results[0]["result"]["iterations"]+ keys = [it["steps"][0]["result"]["idempotency_key"] for it in iterations]+ assert len(set(keys)) == 3++ async def test_parallel_branches_get_distinct_keys(self, enrich: list[dict[str, Any]]) -> None:+ run = await PlaybookEngine().run(+ _playbook(PlaybookStep(id="p", name="fan out", type=StepType.PARALLEL, steps=[_child("a"), _child("b")])),+ {"tenant_id": "t"},+ )+ branches = run.step_results[0]["result"]["branches"]+ keys = [b["result"]["idempotency_key"] for b in branches]+ assert len(set(keys)) == 2++ async def test_a_resumed_run_reproduces_its_keys(self, enrich: list[dict[str, Any]]) -> None:+ """A resume must not look like new work to a vendor dedupe window."""+ first = await PlaybookEngine().run(_playbook(_child("a")), {"tenant_id": "t"})+ again = await PlaybookEngine().run(+ _playbook(_child("a")),+ {"tenant_id": "t"},+ resume_run_id=first.run_id,+ )+ assert first.step_results[0]["result"]["idempotency_key"] == again.step_results[0]["result"]["idempotency_key"]+++# ---------------------------------------------------------------------------+# The resumers — a durable pause with no caller is a run that never continues+# ---------------------------------------------------------------------------+++class TestWaitsAreActuallyResumed:+ """The half that a passing unit test cannot see.++ `pause.expire_due` has existed since parity 5.2 and its only reference+ in the tree is a test, so an approval nobody decided stayed `waiting`+ forever and the `expired` outcome was never recorded by anything. A+ `wait` inherits that and makes it worse: there is no human who might+ come back, so the run simply stops.+ """++ async def test_the_sweeper_drives_both_resume_and_expiry(self, monkeypatch: pytest.MonkeyPatch) -> None:+ from app.playbook import sweeper++ called: list[str] = []++ async def _resume(**kwargs: Any) -> list[str]:+ called.append("resume")+ return ["run-1"]++ async def _expire(**kwargs: Any) -> list[str]:+ called.append("expire")+ return []++ monkeypatch.setattr(engine_mod, "resume_due_waits", _resume)+ monkeypatch.setattr(engine_mod.playbook_pause, "expire_due", _expire)++ counts = await sweeper.sweep_once()+ # Resume before expire: a pause whose timer came due in the same+ # tick that its TTL lapsed should run, not be cancelled.+ assert called == ["resume", "expire"]+ assert counts == {"resumed": 1, "expired": 0}++ async def test_the_sweeper_survives_a_failing_pass(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """A sweeper that dies on one bad pass stops resuming everything."""+ from app.playbook import sweeper++ attempts = 0++ async def _boom() -> dict[str, int]:+ nonlocal attempts+ attempts += 1+ if attempts == 1:+ raise RuntimeError("database away")+ raise asyncio.CancelledError++ monkeypatch.setattr(sweeper, "sweep_once", _boom)++ async def _no_delay(_delay: float) -> None:+ return None++ monkeypatch.setattr(sweeper.asyncio, "sleep", _no_delay)+ with pytest.raises(asyncio.CancelledError):+ await sweeper.run_forever(interval_seconds=0.01)+ assert attempts == 2, "the loop ended on the first failure"++ async def test_a_callback_resumes_once_however_often_it_is_delivered(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """Every webhook sender retries when a 2xx is slow.++ The pause is claimed before the run continues, so the second+ delivery matches no waiting row and resumes nothing.+ """+ claims: list[str] = []++ class _Pause:+ id = "p1"+ tenant_id = "t"+ run_id = "r1"+ playbook_id = "pb-cf"+ resume_index = 1+ run_context: dict[str, Any] = {"tenant_id": "t"}+ step_results: list[dict[str, Any]] = []++ async def _find(**kwargs: Any) -> Any:+ return _Pause()++ async def _resolve(**kwargs: Any) -> bool:+ claims.append(kwargs["pause_id"])+ return len(claims) == 1++ monkeypatch.setattr(engine_mod.playbook_pause, "find_wait_by_token", _find)+ monkeypatch.setattr(engine_mod.playbook_pause, "resolve", _resolve)+ from app.playbook.store import PlaybookStore++ monkeypatch.setattr(PlaybookStore, "default", classmethod(lambda cls: _StoreStub()))++ first = await engine_mod.resume_wait(resume_token="tok")+ second = await engine_mod.resume_wait(resume_token="tok")+ assert first is not None+ assert second is None, "a replayed callback resumed the run twice"++ async def test_an_unknown_token_resumes_nothing(self, monkeypatch: pytest.MonkeyPatch) -> None:+ async def _none(**kwargs: Any) -> Any:+ return None++ monkeypatch.setattr(engine_mod.playbook_pause, "find_wait_by_token", _none)+ assert await engine_mod.resume_wait(resume_token="nope") is None++ def test_the_callback_route_is_registered_and_is_deliberately_unauthenticated(self) -> None:+ """A callback comes from a vendor that holds no AiSOC credential.++ Asserted rather than assumed because the route lives on a second+ router, and the shape this repo has shipped before is an+ authorization that reads as present and never runs.+ """+ from app.api.playbooks import router, waits_router++ paths = {route.path for route in waits_router.routes}+ assert "/api/v1/playbook-waits/{resume_token}/resume" in paths+ assert waits_router.dependencies == []+ assert router.dependencies, "the main playbooks router must stay default-deny"+++class _StoreStub:+ def get(self, playbook_id: str) -> Playbook:+ return _playbook(_child("a"), _child("b"))diff --git a/services/agents/tests/test_playbook_engine_correctness.py b/services/agents/tests/test_playbook_engine_correctness.pyindex 556166f7b..fbcddb9d9 100644--- a/services/agents/tests/test_playbook_engine_correctness.py+++ b/services/agents/tests/test_playbook_engine_correctness.py@@ -459,7 +459,14 @@ class TestUnimplementedStepTypesFailClosed:@pytest.mark.asyncioasync def test_no_step_type_reports_success_without_a_handler(self) -> None:"""Every StepType with no handler, not just the one that hurt most."""- unimplemented = [st for st in StepType if st not in engine_mod._HANDLERS and st != StepType.CONDITION]+ # Not in the handler table is not the same as not runnable. Three+ # step types are answered in the run loop because they need the run+ # itself (`_CONTROL_FLOW`), and `condition` branches in the loop+ # body. Deriving from `_HANDLERS` alone made this test pick `wait`+ # and assert that a working step reports `unimplemented`.+ unimplemented = [+ st for st in StepType if st not in engine_mod._HANDLERS and st not in engine_mod._CONTROL_FLOW and st != StepType.CONDITION+ ]assert unimplemented, "fixture assumption broken: expected some StepType to lack a handler"for step_type in unimplemented:@@ -494,7 +501,14 @@ class TestUnimplementedStepTypesFailClosed:So: take the step type from the registry rather than naming one, andassert the property directly by recording the backoff sleeps."""- unimplemented = [st for st in StepType if st not in engine_mod._HANDLERS and st != StepType.CONDITION]+ # Not in the handler table is not the same as not runnable. Three+ # step types are answered in the run loop because they need the run+ # itself (`_CONTROL_FLOW`), and `condition` branches in the loop+ # body. Deriving from `_HANDLERS` alone made this test pick `wait`+ # and assert that a working step reports `unimplemented`.+ unimplemented = [+ st for st in StepType if st not in engine_mod._HANDLERS and st not in engine_mod._CONTROL_FLOW and st != StepType.CONDITION+ ]assert unimplemented, "fixture assumption broken: expected some StepType to lack a handler"step_type = unimplemented[0]diff --git a/services/api/migrations/096_playbook_wait_pauses.sql b/services/api/migrations/096_playbook_wait_pauses.sqlnew file mode 100644index 000000000..a1891973b--- /dev/null+++ b/services/api/migrations/096_playbook_wait_pauses.sql@@ -0,0 +1,61 @@+-- 096_playbook_wait_pauses.sql — let a pause be a timer or a callback, not+-- only an approval.+--+-- Depth plan 5.3. `aisoc_playbook_pauses` (migration 081) already stores the+-- position, the context and the step results a suspended run needs, and does+-- so on disk precisely because "survives restarts" is the requirement. A+-- `wait` step needs all of that and none of the approval part: there is no+-- human to ask, there is a clock or a callback.+--+-- Rather than a second table with the same five columns and a second set of+-- RLS policies, the existing one grows three:+--+-- kind what this pause is waiting for+-- resume_at when a timer pause becomes due+-- resume_token the unguessable handle a callback presents+--+-- Why `kind` defaults to 'approval'+-- ---------------------------------+-- Every existing row is one. Defaulting the other way would relabel live+-- approvals as waits and the sweeper would resume them without a decision,+-- which is the single worst thing this table can do.+--+-- Why the token is not the id+-- ---------------------------+-- A caller presenting a resume token is unauthenticated by construction —+-- that is what a callback is. The row id appears in the run record, which+-- an analyst can read, so using it as the handle would let anyone who can+-- see a run resume it. The token is minted separately, is never put in a+-- step result that leaves the service, and is indexed unique so a guess has+-- one target rather than a range.+--+-- Reversible: DROP COLUMN IF EXISTS on each.++ALTER TABLE aisoc_playbook_pauses+ ADD COLUMN IF NOT EXISTS kind TEXT NOT NULL DEFAULT 'approval'+ CHECK (kind IN ('approval', 'wait'));++-- When a timer pause becomes due. NULL for an approval (which waits on a+-- person) and for a callback wait (which waits on an event). The sweeper+-- reads it, so a NULL means "not on a clock" rather than "due now".+ALTER TABLE aisoc_playbook_pauses+ ADD COLUMN IF NOT EXISTS resume_at TIMESTAMPTZ;++ALTER TABLE aisoc_playbook_pauses+ ADD COLUMN IF NOT EXISTS resume_token TEXT;++CREATE UNIQUE INDEX IF NOT EXISTS aisoc_playbook_pauses_resume_token_idx+ ON aisoc_playbook_pauses (resume_token)+ WHERE resume_token IS NOT NULL;++-- The sweeper's query: waiting timer pauses that are due. Partial, because+-- the resolved rows accumulate and a full-table scan every tick is a cost+-- that grows with history rather than with work.+CREATE INDEX IF NOT EXISTS aisoc_playbook_pauses_due_idx+ ON aisoc_playbook_pauses (resume_at)+ WHERE status = 'waiting' AND resume_at IS NOT NULL;++COMMENT ON COLUMN aisoc_playbook_pauses.kind IS+ 'approval = waiting on a person; wait = waiting on a clock or a callback.';+COMMENT ON COLUMN aisoc_playbook_pauses.resume_token IS+ 'Unguessable handle a callback presents to resume a wait. Not the row id, which appears in run records.';diff --git a/tests/test_upgrade_playbooks.py b/tests/test_upgrade_playbooks.pyindex 190f30949..40ff90ee2 100644--- a/tests/test_upgrade_playbooks.py+++ b/tests/test_upgrade_playbooks.py@@ -126,8 +126,12 @@ class TestTheCommandLine:def test_a_blocked_file_exits_two_and_is_not_rewritten(self, tmp_path: Path) -> None:pack = tmp_path / "playbooks"pack.mkdir()- target = pack / "loopy.playbook.json"- original = json.dumps(_v3(type="loop"))+ # `run_playbook`, not `loop`. Depth 5.3 gave the engine a bounded+ # iteration construct, so `loop` is no longer blocked and this case+ # needs a type that still is — nested playbooks are deliberately+ # unimplemented, so that one is not going to lapse.+ target = pack / "nested.playbook.json"+ original = json.dumps(_v3(type="run_playbook"))target.write_text(original)assert up.main(["--repo-root", str(ROOT), "--dir", str(pack), "--write"]) == 2