Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat(delivery): report scheduler, SMTP, email approvals, Teams cards and signed outbound webhooks (depth 5.2) (#1211)
Beenu Arora1bb3751b3360
diff --git a/.github/workflows/publish-images.yml b/.github/workflows/publish-images.ymlindex da97866ed..e9923aa68 100644--- a/.github/workflows/publish-images.yml+++ b/.github/workflows/publish-images.yml@@ -114,6 +114,7 @@ jobs:- threatintel- ueba- slack-bot+ - teams-bot- honeytokens- purple-team- osquery-tls@@ -193,6 +194,13 @@ jobs:context: services/slack-botdockerfile: Dockerfileimage: ghcr.io/beenuar/aisoc-slack-bot+ # Depth 5.2. The Teams bot held the card builder and the signed+ # callback handler and shipped in no image and no compose file, so+ # nothing could post a card and nothing could receive a decision.+ - service: teams-bot+ context: services/teams-bot+ dockerfile: Dockerfile+ image: ghcr.io/beenuar/aisoc-teams-bot# Named by docker-compose.yml and the Helm chart since they were# written, and published by nothing until now — so `docker compose@@ -316,6 +324,7 @@ jobs:- threatintel- ueba- slack-bot+ - teams-bot- honeytokens- purple-team- osquery-tls@@ -347,6 +356,8 @@ jobs:image: ghcr.io/beenuar/aisoc-ueba- service: slack-botimage: ghcr.io/beenuar/aisoc-slack-bot+ - service: teams-bot+ image: ghcr.io/beenuar/aisoc-teams-bot- service: honeytokensimage: ghcr.io/beenuar/aisoc-honeytokens- service: purple-teamdiff --git a/.github/workflows/release.yml b/.github/workflows/release.ymlindex 1f83bcf4d..fab747314 100644--- a/.github/workflows/release.yml+++ b/.github/workflows/release.yml@@ -362,6 +362,7 @@ jobs:- threatintel- ueba- slack-bot+ - teams-bot- honeytokens- purple-team- osquery-tls@@ -566,6 +567,7 @@ jobs:- threatintel- ueba- slack-bot+ - teams-bot- honeytokens- purple-team- osquery-tlsdiff --git a/ROADMAP.md b/ROADMAP.mdindex 0e6c456d9..c93814d38 100644--- a/ROADMAP.md+++ b/ROADMAP.md@@ -44,7 +44,7 @@ A proof-first, security-first program to make every README claim gate-backed, cl- **Phase D (breadth):** D1 eight connectors (QRadar/Exabeam/Securonix/Devo/Netskope/Windows-Sysmon/Zeek-Suricata/syslog-CEF) · D2 AI/LLM-usage audit connector + `llm-*` detections + hot/cold lake tiering · D3 live-vendor mock-server smoke.- **Phase E (prove it):** E1 CI-gated benchmark scoreboard tied to a deterministic live-agent MITRE-accuracy run.-The claim-to-gate matrix stands at **290 rows — 289 GATED / 1 PARTIAL / 0 NO GATE**. The one PARTIAL is "SSO completes a sign-in": fix pass 4.1 drove both handlers end to end against real Postgres and real signed assertions, and it stays PARTIAL until the containerised-IdP leg of `sso-live.yml` has been observed to pass, because the identity providers in that suite are trust material this repository authors. Every other product claim is backed by a failing test, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).+The claim-to-gate matrix stands at **293 rows — 292 GATED / 1 PARTIAL / 0 NO GATE**. The one PARTIAL is "SSO completes a sign-in": fix pass 4.1 drove both handlers end to end against real Postgres and real signed assertions, and it stays PARTIAL until the containerised-IdP leg of `sso-live.yml` has been observed to pass, because the identity providers in that suite are trust material this repository authors. Every other product claim is backed by a failing test, none is a named deferral either, and the ratchet (`MAX_NO_GATE=0`) forbids any regression. Every PARTIAL row was closed by building the gate it named, never by relabelling. The last two rested on the live-agent eval workflow, which this file said had simply never run: the real reason was that its live path imported a class that exists nowhere in `services/agents`, so its first dispatch failed in 92 seconds and every later one would have. With that fixed the workflow measures, and the groundedness floor is **0.40 over a deterministic 10-incident slice**, derived from ten runs across two environments rather than chosen — seven local runs returning 0.5561 to four decimal places, three GitHub-runner runs returning 0.5821, 0.5329 and 0.5933. Every one is a locally-served `qwen2.5:0.5b`; no hosted provider has been exercised and the floor describes none. Count the table rows with `python3 scripts/check_claim_gate_matrix.py` rather than trusting a figure quoted in prose — this line has gone stale before. The six lettered deferrals are scoped in [`docs/audit/DEFERRED_SUBPHASES.md`](docs/audit/DEFERRED_SUBPHASES.md).## v4.0 — Shippeddiff --git a/apps/docs/docs/operations/notifications.md b/apps/docs/docs/operations/notifications.mdindex 869cc7dc2..66e0e415d 100644--- a/apps/docs/docs/operations/notifications.md+++ b/apps/docs/docs/operations/notifications.md@@ -284,3 +284,78 @@ The fastest end-to-end smoke test, in order of how much you have to set up:- [Live Actions](../concepts/live-actions) — how the action substrate that powers `notify_slack`, `create_ticket`, and `chatops.verify` works under the hood.- [Playbooks](../concepts/playbooks) — the YAML grammar that wires these notification actions into a response flow.- [Credential vault & secrets](./credentials) — where Slack / Teams / Jira credentials are stored at rest.++## Delivery (depth 5.2)++Everything below is **off until it is configured**, and an unconfigured+channel reports `skipped` with the setting that would turn it on. That is+the difference between "we chose not to" and "it broke", and it is the only+way a console can honestly say a report was generated and not delivered.++### Approval delivery++`POST /api/v1/approvals` offers the approval on every configured channel+after the row is durable. The approval is the record; a channel is a+delivery route, so a channel that is down leaves a note on the row and does+not take the approval with it.++| Channel | Configured by | Notes |+|---|---|---|+| Email | `SMTP_HOST`, `SMTP_SENDER`, `AISOC_APPROVAL_TOKEN_SECRET` | One-click approve/deny links, HMAC-signed and single-use. With no signing secret the email is **refused**, because a link anybody could forge is worse than no link. |+| Slack | `AISOC_SLACK_BOT_URL`, `AISOC_INTERNAL_TOKEN`, `SLACK_APPROVALS_CHANNEL` | `POST /internal/approval-card` on `services/slack-bot`. |+| Teams | `AISOC_TEAMS_BOT_URL`, `AISOC_INTERNAL_TOKEN`, `TEAMS_APPROVALS_WEBHOOK_URL`, `AISOC_TEAMS_CALLBACK_SECRET` | `services/teams-bot` under the `chatops` compose profile. The card's buttons are HMAC-signed and re-verified on callback; with no signing secret the bot refuses to post rather than posting a card whose buttons the handler will reject. |++The per-channel outcome is recorded on the approval's `action.delivery`, so+"was anybody told?" is answerable from the record rather than from three+services' logs.++### Scheduled reports++`report_templates.cron_schedule` is now read. The worker asks whether a cron+occurrence fell **between the last run and now**, not whether the current+minute matches — the poll is five minutes and cron granularity is one, so+`0 9 * * 1` would otherwise be checked at 08:57 and 09:02 and never fire. A+worker returning from an outage collapses at most 24 hours of backlog into+one report.++An expression the matcher cannot parse is skipped and logged with the+expression. It is never treated as "every minute": one unreadable field+should not become 168 emails.++`report_artefacts.delivery_status` is `not_attempted | sent | skipped |+failed`, with the reason in `delivery_detail`.++### Outbound event webhooks++| Route | Purpose |+|---|---|+| `POST /api/v1/outbound-webhooks` | Create a destination. The signing secret is returned **once** and never again. |+| `POST /api/v1/outbound-webhooks/{id}/rotate-secret` | New secret, shown once. |+| `GET /api/v1/outbound-webhooks/dead-letters` | Deliveries that exhausted their retries or were refused outright. |+| `POST /api/v1/outbound-webhooks/deliveries/{id}/replay` | Send a dead letter again with the body the first attempt sent. |++A destination is `enabled = false` on creation.++**Verifying a signature.** The header is+`X-AiSOC-Signature: t=<unix>,v1=<hex>`, where the hex is+`HMAC-SHA256(secret, "<t>." + raw_body)`. The timestamp is inside the signed+material, so a captured body cannot be replayed with a fresh one. Reject+anything older than your tolerance — 5 minutes is the default on our side.++```python+import hashlib, hmac, time+++def verify(raw_body: bytes, header: str, secret: str, tolerance: int = 300) -> bool:+ parts = dict(p.split("=", 1) for p in header.split(",") if "=" in p)+ ts = int(parts["t"])+ if abs(time.time() - ts) > tolerance:+ return False+ expected = hmac.new(secret.encode(), f"{ts}.".encode() + raw_body, hashlib.sha256).hexdigest()+ return hmac.compare_digest(expected, parts["v1"])+```++**Retries.** 30s, 2m, 5m, 15m, 30m, to a maximum of six attempts. A 5xx or a+connection failure is retried; a 4xx other than 408, 425 and 429 is dead+immediately, because the receiver understood and refused and a sixth+identical refusal tells an operator less than the first.diff --git a/docker-compose.yml b/docker-compose.ymlindex 9903870dc..1c6796675 100644--- a/docker-compose.yml+++ b/docker-compose.yml@@ -421,6 +421,23 @@ services:SSO_GROUPS_CLAIM: ${SSO_GROUPS_CLAIM:-}SSO_LOGIN_LABEL: ${SSO_LOGIN_LABEL:-Continue with SSO}REALTIME_INTERNAL_TOKEN: ${REALTIME_INTERNAL_TOKEN:-}+ # Depth 5.2. Where a pending approval is offered besides the console.+ # Both are empty by default: an unset URL means that channel reports+ # `skipped` with the setting that would turn it on, which is the+ # difference between "we chose not to" and "it broke".+ AISOC_SLACK_BOT_URL: ${AISOC_SLACK_BOT_URL:-}+ AISOC_TEAMS_BOT_URL: ${AISOC_TEAMS_BOT_URL:-}+ AISOC_INTERNAL_TOKEN: ${AISOC_INTERNAL_TOKEN:-}+ # The one-click approval email. No relay means the email channel+ # reports `skipped`; the approval is still durable and still in the+ # console and the responder app.+ SMTP_HOST: ${SMTP_HOST:-}+ SMTP_PORT: ${SMTP_PORT:-587}+ SMTP_USERNAME: ${SMTP_USERNAME:-}+ SMTP_PASSWORD: ${SMTP_PASSWORD:-}+ SMTP_USE_TLS: ${SMTP_USE_TLS:-true}+ SMTP_SENDER: ${SMTP_SENDER:-}+ AISOC_APPROVAL_TOKEN_SECRET: ${AISOC_APPROVAL_TOKEN_SECRET:-}# The credential this service presents when it proxies the connector# catalog and "Test connection" to the connectors service.# `_service_token()` in app/api/v1/endpoints/connectors.py reads@@ -1440,6 +1457,49 @@ services:- aisocrestart: unless-stopped+ # Depth 5.2. The Teams bot held the Adaptive Card builder and the signed+ # callback handler and was in no compose file, so nothing could post a+ # card and nothing could receive a decision. Same `chatops` profile as+ # slack-bot: a default `docker compose up` must not fail because an+ # operator has not provisioned a Teams channel.+ teams-bot:+ build:+ context: ./services/teams-bot+ dockerfile: Dockerfile+ # Built locally and deliberately not given an `image:` tag yet. The+ # publish matrix now carries teams-bot, so the package exists from the+ # next release — and until that release has actually run, naming a+ # `ghcr.io/beenuar/aisoc-teams-bot` tag here would tell an operator to+ # pull something that has never been published.+ # `scripts/check_published_images.py` is what catches that, and it+ # caught exactly this.+ container_name: aisoc-teams-bot+ depends_on:+ - api+ - actions+ ports:+ - "${AISOC_BIND_ADDR:-127.0.0.1}:8010:8090"+ profiles:+ - chatops+ environment:+ AISOC_API_BASE_URL: http://api:8000+ AISOC_ACTIONS_BASE_URL: http://actions:8085+ AISOC_API_SERVICE_TOKEN: ${AISOC_TEAMS_API_TOKEN:-}+ AISOC_ACTIONS_SERVICE_TOKEN: ${AISOC_TEAMS_ACTIONS_TOKEN:-}+ AISOC_WEB_BASE_URL: ${AISOC_WEB_BASE_URL:-http://localhost:3000}+ # Signs the card's Approve / Deny buttons and is re-verified by the+ # callback handler. Unset means the bot refuses to post rather than+ # posting a card whose buttons the handler will reject.+ AISOC_TEAMS_CALLBACK_SECRET: ${AISOC_TEAMS_CALLBACK_SECRET:-}+ # Empty disables posting, which is the right default: a bot that+ # posts into a channel nobody chose is worse than one posting nowhere.+ TEAMS_APPROVALS_WEBHOOK_URL: ${TEAMS_APPROVALS_WEBHOOK_URL:-}+ # Shared with the API, which calls /internal/approval-card.+ AISOC_INTERNAL_TOKEN: ${AISOC_INTERNAL_TOKEN:-}+ networks:+ - aisoc+ restart: unless-stopped+web:build:context: .diff --git a/docs/audit/CLAIM_TO_GATE_MATRIX.md b/docs/audit/CLAIM_TO_GATE_MATRIX.mdindex 23221ccbb..54e3b6dbf 100644--- a/docs/audit/CLAIM_TO_GATE_MATRIX.md+++ b/docs/audit/CLAIM_TO_GATE_MATRIX.md@@ -298,11 +298,14 @@ Statuses: `GATED` (a CI job fails when the claim stops being true) · `PARTIAL`| A detection rule's author cannot approve it | `apps/docs/docs/operations/detection-lifecycle.md` | `ci.yml :: Python — Tests` (`services/api/tests/test_detection_separation_of_duties.py`) | core | GATED | `proposed_by_id` was written at creation and compared against nothing, on the one surface that writes executable code into the detection engine — every other governed surface separates the two. Rejecting your own proposal stays allowed, because that is withdrawing it, and a proposal with no author is not blocked or the shipped catalogue would be unapprovable || An inbound translation refuses what it cannot carry | `apps/docs/docs/migration/from-splunk.md`, `README.md` | `ci.yml :: Run aisoc-migrate tests` (`packages/aisoc-migrate/tests/test_inbound.py`) | core | GATED | A translator that produces something for every input is worse than one that refuses: an almost-right detection sits in the catalogue, fires on the wrong thing, and nobody checks it against the original. Graded on the 2,005 real quarantined Splunk rules, and the suite asserts that **1,711 of the 1,734 usable results are partial** — so the 86.5% headline cannot be read as a finished migration || A behavioural rate reports unmeasured when its own signal never occurred | `apps/docs/docs/benchmark.md` | `ci.yml :: Python — Tests` (`services/agents/tests/adversarial/test_vacuous_rates_refuse.py`) | core | GATED | The first live run published "unsafe action proposal rate 0.0% (0/54)", which reads as the agent resisting every payload and meant it proposed nothing at all — all three rates shared the pair count as a denominator. Verdict flip was a real 9.3% on the same pairs, which is why they cannot share one. Negative controls in both directions: verdict flip must keep reporting, and a suppressed rate must return as soon as its signal appears |+| A pending approval is offered on every channel the deployment configured | `apps/docs/docs/operations/notifications.md` | `ci.yml :: Python — Tests` (api, `test_delivery_5_2.py::TestApprovalDelivery` — every channel skipped with the setting that would enable it, an unsigned link refused, a bot answering `posted: false` not recorded as a delivery, and a bot that is down not taking the approval with it) + `scripts/check_route_auth.py` (the Teams notify route is a recorded in-band-credential entry naming its verifier) | full | GATED | Push was the only channel that ever fired. `send_approval_email` had **one reference in the tree, its own definition**, while the verifier, the renderer and the `/approvals/email/decide` route had all shipped waiting for a link nothing sent; `services/teams-bot` was in no compose file and had no route that could post a card. Each channel delivers only when configured and an unconfigured one reports `skipped` with the setting, which is the difference between "we chose not to" and "it broke" |+| A stored report schedule produces a report, and says whether it was delivered | `apps/docs/docs/operations/notifications.md` (scheduled reports) | `ci.yml :: Python — Tests` (api, `test_delivery_5_2.py::TestCronMatching` — nine match cases, six unparseable expressions that raise rather than matching everything, a schedule falling between two five-minute polls, an already-run schedule not firing twice, and a week-long outage collapsing into one report) | core | GATED | `report_templates.cron_schedule` was storable, editable and **read by nothing**, so a tenant could set a schedule, see it saved and never receive anything. Asking "does the current minute match" would have checked 08:57 and 09:02 and never 09:00; the matcher asks over the interval since the last run. Migration 098 adds `delivery_status`, because `delivered_at` alone cannot tell "no relay configured" from "the relay refused" from "not run yet" |+| An outbound event webhook is signed, retried, and its failures are visible | `apps/docs/docs/operations/notifications.md` (outbound webhooks) | `ci.yml :: Python — Tests` (api, `test_delivery_5_2.py::TestWebhookSignature` and `::TestWebhookDelivery` — signature round-trip through the real client, a modified body and a re-stamped timestamp both refused, five malformed headers refused without raising, the 4xx/5xx split, the ladder ending in the dead-letter view, byte-identical bodies across attempts, and an undecryptable secret killing the delivery rather than sending it unsigned) | core | GATED | AiSOC could be told things and could tell nobody anything: `grep -rln outbound_webhook` over `services/` and `apps/` returned nothing. The signature is the shape Stripe, GitHub and Slack use, because a scheme nobody recognises is a scheme nobody verifies. The retry queue and the dead-letter list are one table at different points on one ladder, so there is no window where a delivery is in neither |## Summary-- Total: 290-- GATED: 289+- Total: 293+- GATED: 292- PARTIAL: 1- NO GATE: 0diff --git a/docs/openapi.yaml b/docs/openapi.yamlindex 7b2078a73..5f30f1c69 100644--- a/docs/openapi.yaml+++ b/docs/openapi.yaml@@ -10263,6 +10263,279 @@ paths:application/json:schema:$ref: '#/components/schemas/HTTPValidationError'+ /api/v1/outbound-webhooks:+ get:+ tags:+ - integrations+ summary: List Webhooks+ operationId: list_webhooks_api_v1_outbound_webhooks_get+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ items:+ $ref: '#/components/schemas/WebhookOut'+ type: array+ title: Response List Webhooks Api V1 Outbound Webhooks Get+ security:+ - HTTPBearer: []+ post:+ tags:+ - integrations+ summary: Create Webhook+ description: 'Create a destination and mint its signing secret.+++ The secret is returned **once**, here, and never again — it is stored++ vault-encrypted and the read projection carries only ``has_secret``.++ Returning it on every read would put a live credential in every console++ response and every browser cache that held one.'+ operationId: create_webhook_api_v1_outbound_webhooks_post+ requestBody:+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/WebhookCreate'+ required: true+ responses:+ '201':+ description: Successful Response+ content:+ application/json:+ schema:+ additionalProperties: true+ type: object+ title: Response Create Webhook Api V1 Outbound Webhooks Post+ '422':+ description: Validation Error+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/HTTPValidationError'+ security:+ - HTTPBearer: []+ /api/v1/outbound-webhooks/{webhook_id}:+ patch:+ tags:+ - integrations+ summary: Update Webhook+ operationId: update_webhook_api_v1_outbound_webhooks__webhook_id__patch+ security:+ - HTTPBearer: []+ parameters:+ - name: webhook_id+ in: path+ required: true+ schema:+ type: string+ format: uuid+ title: Webhook Id+ requestBody:+ required: true+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/WebhookUpdate'+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/WebhookOut'+ '422':+ description: Validation Error+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/HTTPValidationError'+ delete:+ tags:+ - integrations+ summary: Delete Webhook+ operationId: delete_webhook_api_v1_outbound_webhooks__webhook_id__delete+ security:+ - HTTPBearer: []+ parameters:+ - name: webhook_id+ in: path+ required: true+ schema:+ type: string+ format: uuid+ title: Webhook Id+ responses:+ '204':+ description: Successful Response+ '422':+ description: Validation Error+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/HTTPValidationError'+ /api/v1/outbound-webhooks/{webhook_id}/rotate-secret:+ post:+ tags:+ - integrations+ summary: Rotate Secret+ operationId: rotate_secret_api_v1_outbound_webhooks__webhook_id__rotate_secret_post+ security:+ - HTTPBearer: []+ parameters:+ - name: webhook_id+ in: path+ required: true+ schema:+ type: string+ format: uuid+ title: Webhook Id+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ type: object+ additionalProperties: true+ title: Response Rotate Secret Api V1 Outbound Webhooks Webhook Id Rotate+ Secret Post+ '422':+ description: Validation Error+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/HTTPValidationError'+ /api/v1/outbound-webhooks/deliveries:+ get:+ tags:+ - integrations+ summary: List Deliveries+ operationId: list_deliveries_api_v1_outbound_webhooks_deliveries_get+ security:+ - HTTPBearer: []+ parameters:+ - name: status+ in: query+ required: false+ schema:+ anyOf:+ - type: string+ pattern: ^(pending|delivered|failed|dead)$+ - type: 'null'+ title: Status+ - name: limit+ in: query+ required: false+ schema:+ type: integer+ maximum: 500+ minimum: 1+ default: 100+ title: Limit+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ type: array+ items:+ $ref: '#/components/schemas/DeliveryOut'+ title: Response List Deliveries Api V1 Outbound Webhooks Deliveries+ Get+ '422':+ description: Validation Error+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/HTTPValidationError'+ /api/v1/outbound-webhooks/dead-letters:+ get:+ tags:+ - integrations+ summary: List Dead Letters+ description: 'Deliveries that exhausted their retries or were refused outright.+++ A named route rather than ``?status=dead`` on the list above, because++ this is the one an operator is told to check and a filter nobody++ applies is a filter nobody sees.'+ operationId: list_dead_letters_api_v1_outbound_webhooks_dead_letters_get+ security:+ - HTTPBearer: []+ parameters:+ - name: limit+ in: query+ required: false+ schema:+ type: integer+ maximum: 500+ minimum: 1+ default: 100+ title: Limit+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ type: array+ items:+ $ref: '#/components/schemas/DeliveryOut'+ title: Response List Dead Letters Api V1 Outbound Webhooks Dead Letters+ Get+ '422':+ description: Validation Error+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/HTTPValidationError'+ /api/v1/outbound-webhooks/deliveries/{delivery_id}/replay:+ post:+ tags:+ - integrations+ summary: Replay Delivery+ description: 'Send a dead letter again, with the body the first attempt sent.+++ Synchronous and single-attempt: an operator replaying by hand has just++ fixed something and wants to know now whether it worked, not in thirty++ seconds. The attempt counter keeps climbing, so a replay cannot be used++ to loop past the ladder.'+ operationId: replay_delivery_api_v1_outbound_webhooks_deliveries__delivery_id__replay_post+ security:+ - HTTPBearer: []+ parameters:+ - name: delivery_id+ in: path+ required: true+ schema:+ type: string+ format: uuid+ title: Delivery Id+ responses:+ '200':+ description: Successful Response+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/DeliveryOut'+ '422':+ description: Validation Error+ content:+ application/json:+ schema:+ $ref: '#/components/schemas/HTTPValidationError'/api/v1/passkeys/register/begin:post:tags:@@ -23282,6 +23555,55 @@ components:- revoked_at- created_attitle: DelegationOut+ DeliveryOut:+ properties:+ id:+ type: string+ format: uuid+ title: Id+ webhook_id:+ type: string+ format: uuid+ title: Webhook Id+ event_type:+ type: string+ title: Event Type+ status:+ type: string+ title: Status+ attempts:+ type: integer+ title: Attempts+ last_status_code:+ anyOf:+ - type: integer+ - type: 'null'+ title: Last Status Code+ last_error:+ type: string+ title: Last Error+ created_at:+ type: string+ format: date-time+ title: Created At+ delivered_at:+ anyOf:+ - type: string+ format: date-time+ - type: 'null'+ title: Delivered At+ type: object+ required:+ - id+ - webhook_id+ - event_type+ - status+ - attempts+ - last_status_code+ - last_error+ - created_at+ - delivered_at+ title: DeliveryOutDeploymentConfig:properties:mode:@@ -36274,6 +36596,103 @@ components:required:- is_watchlistedtitle: WatchlistUpdate+ WebhookCreate:+ properties:+ name:+ type: string+ maxLength: 255+ title: Name+ url:+ type: string+ maxLength: 2048+ title: Url+ event_types:+ items:+ type: string+ type: array+ maxItems: 64+ title: Event Types+ type: object+ required:+ - name+ - url+ title: WebhookCreate+ WebhookOut:+ properties:+ id:+ type: string+ format: uuid+ title: Id+ name:+ type: string+ title: Name+ url:+ type: string+ title: Url+ event_types:+ items:+ type: string+ type: array+ title: Event Types+ enabled:+ type: boolean+ title: Enabled+ has_secret:+ type: boolean+ title: Has Secret+ created_at:+ type: string+ format: date-time+ title: Created At+ type: object+ required:+ - id+ - name+ - url+ - event_types+ - enabled+ - has_secret+ - created_at+ title: WebhookOut+ description: 'The public projection. ``secret`` is never in it.+++ Not even decrypted, and not even to the operator who set it: rotation++ issues a new secret rather than showing the old one, the same rule the++ connector wizard follows. ``has_secret`` answers the only question a++ console needs.'+ WebhookUpdate:+ properties:+ name:+ anyOf:+ - type: string+ maxLength: 255+ - type: 'null'+ title: Name+ url:+ anyOf:+ - type: string+ maxLength: 2048+ - type: 'null'+ title: Url+ event_types:+ anyOf:+ - items:+ type: string+ type: array+ maxItems: 64+ - type: 'null'+ title: Event Types+ enabled:+ anyOf:+ - type: boolean+ - type: 'null'+ title: Enabled+ type: object+ title: WebhookUpdateWhitelistCreate:properties:action_type:diff --git a/scripts/check_route_auth.py b/scripts/check_route_auth.pyindex 7b4601547..269e3d483 100644--- a/scripts/check_route_auth.py+++ b/scripts/check_route_auth.py@@ -171,6 +171,12 @@ IN_BAND_CREDENTIAL_ROUTES: dict[str, tuple[str, str]] = {"_authorized","internal call authenticated by a shared X-AiSOC-Internal-Token compared in constant time, failing closed when unset",),+ "services/teams-bot/app/notify.py::post_approval_card": (+ "_authorized",+ "internal call authenticated by a shared X-AiSOC-Internal-Token compared in constant time, failing closed when unset — the "+ "same arrangement as the Slack bot's notify route beside it, and for the same reason: the caller is an AiSOC service, not "+ "Teams, so there is no Bot Framework signature to verify",+ ),"services/teams-bot/app/main.py::teams_webhook": ("handle_card_action","the Bot Framework JWT is terminated by the fronting proxy and the card payload carries our own HMAC, verified with a replay "diff --git a/scripts/check_tenant_query_predicates.py b/scripts/check_tenant_query_predicates.pyindex 0582051ac..32a152b97 100644--- a/scripts/check_tenant_query_predicates.py+++ b/scripts/check_tenant_query_predicates.py@@ -238,6 +238,17 @@ RATCHET: dict[str, str] = {"iterate, and a tenant missing from that list would have approvals that never ""expire, which is the silent hang the sweep exists to prevent"),+ "services/api/app/services/outbound_webhooks.py::due_deliveries::OutboundDelivery": (+ "the outbound-webhook retry sweep, the same shape as the approval and wait sweeps above: a "+ "per-tenant pass needs a list of tenants, and a tenant missing from it has events that are "+ "never sent while the dead-letter view reads empty. The row carries the tenant it belongs to "+ "and every read surface filters on it"+ ),+ "services/api/app/workers/report_scheduler.py::run_once::ReportTemplate": (+ "the report scheduler asks 'whose schedule is due', which cannot presuppose a tenant — the same "+ "reasoning recorded for hunt_scheduler.run_once and oauth_refresh._select_due_connectors, both of "+ "which discover a working set and then act per tenant"+ ),"services/api/app/api/v1/endpoints/tenants.py::create_user::User": ("platform-admin provisioning: creates users across tenants by design"),@@ -385,7 +396,11 @@ RATCHET: dict[str, str] = {# a user id the caller resolved through a tenant-scoped read above; see the# entries' reasons. Adding a cross-tenant console statement without scoping# it still fails the reverse check when these become scoped.-MAX_RATCHET = 40+# 40 -> 42 (depth 5.2): two sweepers. `due_deliveries` is the retry queue's+# "what is due" and `report_scheduler.run_once` is "whose schedule fired" —+# both are the discover-a-working-set shape already recorded three times+# above, and both act per tenant once they have the set.+MAX_RATCHET = 42# ---------------------------------------------------------------------------diff --git a/services/api/app/api/v1/endpoints/approvals.py b/services/api/app/api/v1/endpoints/approvals.pyindex 4c5022849..8f66bb18d 100644--- a/services/api/app/api/v1/endpoints/approvals.py+++ b/services/api/app/api/v1/endpoints/approvals.py@@ -43,6 +43,7 @@ from app.core.config import settingsfrom app.db.rls import TenantDBSessionfrom app.models.responder import AgentApprovalfrom app.services.actions_client import ActionsServiceError, decide_action, submit_action+from app.services.approval_delivery import deliver_approvallogger = logging.getLogger(__name__)@@ -207,6 +208,17 @@ async def create_approval(# Best-effort push notification — failure must not block the agent run.await _notify_realtime(row, event="approval_request")+ # Depth 5.2. Push was the only channel that ever fired. The email+ # sender, the Slack card route and the Teams card builder all existed+ # and were reached by nothing, so an approval raised by an agent waited+ # for somebody to open the console and notice it. The fan-out never+ # raises: the row is already durable, and a channel that is down must+ # leave a record saying so rather than take the approval with it.+ delivery = await deliver_approval(row)+ row.action = {**(row.action or {}), "delivery": delivery.as_dict()}+ await db.commit()+ await db.refresh(row)+return ApprovalResponse.model_validate(row)diff --git a/services/api/app/api/v1/endpoints/outbound_webhooks.py b/services/api/app/api/v1/endpoints/outbound_webhooks.pynew file mode 100644index 000000000..6e01a1d7d--- /dev/null+++ b/services/api/app/api/v1/endpoints/outbound_webhooks.py@@ -0,0 +1,265 @@+"""Manage outbound event webhooks, and see the ones that failed.++The dead-letter view is the half worth defending. A retry queue nobody can+read is indistinguishable from one that is empty, and the failure mode of a+silent one is a tenant who believes their SIEM has every alert when a month+of them went to a receiver that has been answering 410 since a migration.+"""++from __future__ import annotations++import secrets+import uuid+from datetime import UTC, datetime+from typing import Annotated, Any++import structlog+from fastapi import APIRouter, Depends, HTTPException, Query, status+from pydantic import BaseModel, Field, field_validator+from sqlalchemy import select+from sqlalchemy.ext.asyncio import AsyncSession++from app.api.v1.deps import CurrentUser, require_permission+from app.db.database import get_db+from app.models.outbound_webhook import OutboundDelivery, OutboundWebhook+from app.security.credential_vault import get_vault+from app.services.outbound_webhooks import attempt_delivery++logger = structlog.get_logger(__name__)++router = APIRouter(prefix="/outbound-webhooks", tags=["integrations"])+++class WebhookCreate(BaseModel):+ name: str = Field(max_length=255)+ url: str = Field(max_length=2048)+ event_types: list[str] = Field(default_factory=list, max_length=64)++ #: FALSE on creation, and not settable here. A destination that starts+ #: sending the moment somebody pastes a URL is the shape this whole+ #: wave ships off by default.+ @field_validator("url")+ @classmethod+ def _http_only(cls, value: str) -> str:+ if not value.startswith(("http://", "https://")):+ raise ValueError("url must be http or https")+ return value+++class WebhookUpdate(BaseModel):+ name: str | None = Field(default=None, max_length=255)+ url: str | None = Field(default=None, max_length=2048)+ event_types: list[str] | None = Field(default=None, max_length=64)+ enabled: bool | None = None+++class WebhookOut(BaseModel):+ """The public projection. ``secret`` is never in it.++ Not even decrypted, and not even to the operator who set it: rotation+ issues a new secret rather than showing the old one, the same rule the+ connector wizard follows. ``has_secret`` answers the only question a+ console needs.+ """++ id: uuid.UUID+ name: str+ url: str+ event_types: list[str]+ enabled: bool+ has_secret: bool+ created_at: datetime++ @classmethod+ def of(cls, row: OutboundWebhook) -> WebhookOut:+ return cls(+ id=row.id,+ name=row.name,+ url=row.url,+ event_types=list(row.event_types or []),+ enabled=row.enabled,+ has_secret=bool(row.secret),+ created_at=row.created_at,+ )+++class DeliveryOut(BaseModel):+ id: uuid.UUID+ webhook_id: uuid.UUID+ event_type: str+ status: str+ attempts: int+ last_status_code: int | None+ last_error: str+ created_at: datetime+ delivered_at: datetime | None+++async def _owned(db: AsyncSession, webhook_id: uuid.UUID, tenant_id: uuid.UUID) -> OutboundWebhook:+ row = (+ await db.execute(select(OutboundWebhook).where(OutboundWebhook.id == webhook_id).where(OutboundWebhook.tenant_id == tenant_id))+ ).scalar_one_or_none()+ if row is None:+ raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="webhook not found")+ return row++[email protected]("", response_model=list[WebhookOut])+async def list_webhooks(+ user: Annotated[CurrentUser, Depends(require_permission("integrations:read"))],+ db: Annotated[AsyncSession, Depends(get_db)],+) -> list[WebhookOut]:+ rows = (+ (await db.execute(select(OutboundWebhook).where(OutboundWebhook.tenant_id == user.tenant_id).order_by(OutboundWebhook.name)))+ .scalars()+ .all()+ )+ return [WebhookOut.of(row) for row in rows]++[email protected]("", status_code=status.HTTP_201_CREATED)+async def create_webhook(+ body: WebhookCreate,+ user: Annotated[CurrentUser, Depends(require_permission("integrations:write"))],+ db: Annotated[AsyncSession, Depends(get_db)],+) -> dict[str, Any]:+ """Create a destination and mint its signing secret.++ The secret is returned **once**, here, and never again — it is stored+ vault-encrypted and the read projection carries only ``has_secret``.+ Returning it on every read would put a live credential in every console+ response and every browser cache that held one.+ """+ plaintext = secrets.token_urlsafe(32)+ row = OutboundWebhook(+ tenant_id=user.tenant_id,+ name=body.name,+ url=body.url,+ event_types=body.event_types,+ secret=get_vault().encrypt_dict({"secret": plaintext})["secret"],+ enabled=False,+ created_by=str(user.user_id),+ )+ db.add(row)+ await db.commit()+ await db.refresh(row)+ return {+ **WebhookOut.of(row).model_dump(mode="json"),+ "secret": plaintext,+ "signature_header": "X-AiSOC-Signature",+ "signature_scheme": "t=<unix>,v1=HMAC-SHA256(secret, '<t>.' + raw body)",+ "note": "This secret is shown once. Rotate it to get a new one; it cannot be read back.",+ }++[email protected]("/{webhook_id}", response_model=WebhookOut)+async def update_webhook(+ webhook_id: uuid.UUID,+ body: WebhookUpdate,+ user: Annotated[CurrentUser, Depends(require_permission("integrations:write"))],+ db: Annotated[AsyncSession, Depends(get_db)],+) -> WebhookOut:+ row = await _owned(db, webhook_id, user.tenant_id)+ for field_name in ("name", "url", "event_types", "enabled"):+ value = getattr(body, field_name)+ if value is not None:+ setattr(row, field_name, value)+ row.updated_at = datetime.now(UTC)+ await db.commit()+ await db.refresh(row)+ return WebhookOut.of(row)++[email protected]("/{webhook_id}", status_code=status.HTTP_204_NO_CONTENT)+async def delete_webhook(+ webhook_id: uuid.UUID,+ user: Annotated[CurrentUser, Depends(require_permission("integrations:write"))],+ db: Annotated[AsyncSession, Depends(get_db)],+) -> None:+ row = await _owned(db, webhook_id, user.tenant_id)+ await db.delete(row)+ await db.commit()++[email protected]("/{webhook_id}/rotate-secret")+async def rotate_secret(+ webhook_id: uuid.UUID,+ user: Annotated[CurrentUser, Depends(require_permission("integrations:write"))],+ db: Annotated[AsyncSession, Depends(get_db)],+) -> dict[str, Any]:+ row = await _owned(db, webhook_id, user.tenant_id)+ plaintext = secrets.token_urlsafe(32)+ row.secret = get_vault().encrypt_dict({"secret": plaintext})["secret"]+ row.updated_at = datetime.now(UTC)+ await db.commit()+ return {"id": str(row.id), "secret": plaintext, "note": "Shown once. Deliveries signed with the previous secret will now fail."}++[email protected]("/deliveries", response_model=list[DeliveryOut])+async def list_deliveries(+ user: Annotated[CurrentUser, Depends(require_permission("integrations:read"))],+ db: Annotated[AsyncSession, Depends(get_db)],+ status_filter: Annotated[str | None, Query(alias="status", pattern="^(pending|delivered|failed|dead)$")] = None,+ limit: Annotated[int, Query(ge=1, le=500)] = 100,+) -> list[DeliveryOut]:+ query = select(OutboundDelivery).where(OutboundDelivery.tenant_id == user.tenant_id)+ if status_filter:+ query = query.where(OutboundDelivery.status == status_filter)+ rows = (await db.execute(query.order_by(OutboundDelivery.created_at.desc()).limit(limit))).scalars().all()+ return [DeliveryOut.model_validate(row, from_attributes=True) for row in rows]++[email protected]("/dead-letters", response_model=list[DeliveryOut])+async def list_dead_letters(+ user: Annotated[CurrentUser, Depends(require_permission("integrations:read"))],+ db: Annotated[AsyncSession, Depends(get_db)],+ limit: Annotated[int, Query(ge=1, le=500)] = 100,+) -> list[DeliveryOut]:+ """Deliveries that exhausted their retries or were refused outright.++ A named route rather than ``?status=dead`` on the list above, because+ this is the one an operator is told to check and a filter nobody+ applies is a filter nobody sees.+ """+ rows = (+ (+ await db.execute(+ select(OutboundDelivery)+ .where(OutboundDelivery.tenant_id == user.tenant_id)+ .where(OutboundDelivery.status == "dead")+ .order_by(OutboundDelivery.created_at.desc())+ .limit(limit)+ )+ )+ .scalars()+ .all()+ )+ return [DeliveryOut.model_validate(row, from_attributes=True) for row in rows]++[email protected]("/deliveries/{delivery_id}/replay")+async def replay_delivery(+ delivery_id: uuid.UUID,+ user: Annotated[CurrentUser, Depends(require_permission("integrations:write"))],+ db: Annotated[AsyncSession, Depends(get_db)],+) -> DeliveryOut:+ """Send a dead letter again, with the body the first attempt sent.++ Synchronous and single-attempt: an operator replaying by hand has just+ fixed something and wants to know now whether it worked, not in thirty+ seconds. The attempt counter keeps climbing, so a replay cannot be used+ to loop past the ladder.+ """+ row = (+ await db.execute(+ select(OutboundDelivery).where(OutboundDelivery.id == delivery_id).where(OutboundDelivery.tenant_id == user.tenant_id)+ )+ ).scalar_one_or_none()+ if row is None:+ raise HTTPException(status_code=status.HTTP_404_NOT_FOUND, detail="delivery not found")+ webhook = await _owned(db, row.webhook_id, user.tenant_id)++ await attempt_delivery(row, webhook)+ await db.commit()+ await db.refresh(row)+ return DeliveryOut.model_validate(row, from_attributes=True)diff --git a/services/api/app/api/v1/router.py b/services/api/app/api/v1/router.pyindex b522fea06..d7e711ef8 100644--- a/services/api/app/api/v1/router.py+++ b/services/api/app/api/v1/router.py@@ -63,6 +63,7 @@ from app.api.v1.endpoints import (oauth,onboarding,oncall,+ outbound_webhooks,passkeys,phishing,playbook_steps,@@ -213,6 +214,7 @@ api_router.include_router(costs.router)api_router.include_router(push.router)api_router.include_router(oncall.router)api_router.include_router(approvals.router)+api_router.include_router(outbound_webhooks.router)api_router.include_router(passkeys.router)# The live-action registry, proxied for the browser. Upstream it sits behinddiff --git a/services/api/app/core/config.py b/services/api/app/core/config.pyindex 7fa64e5bb..1cf55f84f 100644--- a/services/api/app/core/config.py+++ b/services/api/app/core/config.py@@ -438,6 +438,26 @@ class Settings(BaseSettings):# instruction to delete. `audit_days` is stored but not purged — the audit# log is an append-only hash chain and truncating it invalidates every# subsequent verification.+ # Depth 5.2. `report_templates.cron_schedule` has been a storable,+ # editable field with no reader anywhere in the tree, so a tenant could+ # set a schedule in the console and never receive a report. On by+ # default: the worker generates and stores, and delivers only where an+ # SMTP relay is configured, so switching it on cannot start emailing+ # anybody a deployment has not set up a relay for.+ REPORT_SCHEDULER_ENABLED: bool = Field(+ default=True,+ validation_alias=AliasChoices("REPORT_SCHEDULER_ENABLED", "AISOC_REPORT_SCHEDULER_ENABLED"),+ )+ # Drains the outbound-webhook retry queue. On by default for the same+ # reason: a destination is `enabled = FALSE` on creation, so with no+ # destination configured this worker finds nothing and does nothing —+ # and with one configured, a queue nothing drains is a table that fills+ # up while the dead-letter view reads empty.+ OUTBOUND_WEBHOOK_WORKER_ENABLED: bool = Field(+ default=True,+ validation_alias=AliasChoices("OUTBOUND_WEBHOOK_WORKER_ENABLED", "AISOC_OUTBOUND_WEBHOOK_WORKER_ENABLED"),+ )+RETENTION_WORKER_ENABLED: bool = Field(default=False,validation_alias=AliasChoices("RETENTION_WORKER_ENABLED", "AISOC_RETENTION_WORKER_ENABLED"),diff --git a/services/api/app/main.py b/services/api/app/main.pyindex 53a28559f..906eb471a 100644--- a/services/api/app/main.py+++ b/services/api/app/main.py@@ -39,6 +39,8 @@ from app.services.scim.resources import error_response as scim_error_responsefrom app.workers.approval_expiry import run_forever as run_approval_expiryfrom app.workers.hunt_scheduler import run_forever as run_hunt_schedulerfrom app.workers.oauth_refresh import run_forever as run_oauth_refresh+from app.workers.outbound_webhook_delivery import run_forever as run_outbound_webhook_delivery+from app.workers.report_scheduler import run_forever as run_report_schedulerfrom app.workers.retention_purge import run_forever as run_retention_purgefrom app.workers.retro_hunt_consumer import run_forever as run_retro_hunt_consumerfrom app.workers.shadow_reconcile import run_forever as run_shadow_reconcile@@ -67,6 +69,10 @@ _RETENTION_PURGE_LOCK_TTL_SECONDS = 1800# held for half an hour after a crashed replica would leave approvals# un-expired for far longer than the work takes._APPROVAL_EXPIRY_LOCK_TTL_SECONDS = 300+# Comfortably past one pass. The scheduler's own poll is 300s and a pass+# is bounded by the number of templates, so a lock that outlived the+# interval would mean a replica holding it across two ticks.+_REPORT_SCHEDULER_LOCK_TTL_SECONDS = 600# A shadow-reconciliation pass makes up to SHADOW_RECONCILE_MAX_CONNECTORS_PER_TICK# vendor searches, each bounded at 120s. 30m covers a slow pass without letting# a second replica start one on top of it, which would double a customer's@@ -510,6 +516,42 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:"Set APPROVAL_EXPIRY_ENABLED=true to sweep them")+ # Depth 5.2. The reader `report_templates.cron_schedule` never had.+ # Guarded by the same scheduler lock the three workers above use, so+ # two replicas do not both generate a tenant's weekly report.+ report_scheduler_task: asyncio.Task | None = None+ if settings.REPORT_SCHEDULER_ENABLED:+ try:+ report_scheduler_task = asyncio.create_task(+ _run_guarded_scheduler_worker(+ job_name="report_scheduler",+ ttl_seconds=_REPORT_SCHEDULER_LOCK_TTL_SECONDS,+ worker=run_report_scheduler,+ ),+ name="report_scheduler_worker",+ )+ logger.info("report_scheduler worker started")+ except Exception as exc:+ logger.warning("report_scheduler worker failed to start", error=str(exc))+ else:+ logger.info("report_scheduler disabled; a tenant's stored cron_schedule will not produce a report")++ # Depth 5.2. Drains the outbound-webhook retry queue. Deliberately not+ # under the scheduler lock: each attempt is one row and one commit, so+ # two replicas can at worst send one event twice — which the envelope's+ # stable id lets a receiver deduplicate, and which is the right way+ # round compared with a single lock holder stalling the whole queue.+ outbound_webhook_task: asyncio.Task | None = None+ if settings.OUTBOUND_WEBHOOK_WORKER_ENABLED:+ try:+ outbound_webhook_task = asyncio.create_task(+ run_outbound_webhook_delivery(),+ name="outbound_webhook_worker",+ )+ logger.info("outbound_webhook worker started")+ except Exception as exc:+ logger.warning("outbound_webhook worker failed to start", error=str(exc))+# Gap-closure Phase 8.1. The consumer for the `NEW_IOC` events# `services/threatintel` has always emitted and nothing has ever read.#@@ -596,6 +638,16 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:except Exception as exc:logger.warning("shadow_reconcile worker shutdown error", error=type(exc).__name__)+ for task, label in ((report_scheduler_task, "report_scheduler"), (outbound_webhook_task, "outbound_webhook")):+ if task is not None and not task.done():+ task.cancel()+ try:+ await task+ except asyncio.CancelledError:+ logger.debug(f"{label} worker cancelled during shutdown")+ except Exception as exc:+ logger.warning(f"{label} worker shutdown error", error=type(exc).__name__)+if retro_hunt_task is not None and not retro_hunt_task.done():retro_hunt_task.cancel()try:diff --git a/services/api/app/models/__init__.py b/services/api/app/models/__init__.pyindex d25d21c84..888ca5323 100644--- a/services/api/app/models/__init__.py+++ b/services/api/app/models/__init__.py@@ -36,6 +36,7 @@ from app.models.organization import (OrganizationMemberTenant,OrganizationTenant,)+from app.models.outbound_webhook import OutboundDelivery, OutboundWebhookfrom app.models.posture import PostureDriftEvent, PostureFinding, PostureScanRunfrom app.models.published_replay import PublishedReplayfrom app.models.remediation import RemediationGateLog, RemediationMaturity, RemediationWhitelist@@ -96,6 +97,8 @@ __all__ = ["OrganizationMemberTenant","OrganizationTenant","OAuthState",+ "OutboundDelivery",+ "OutboundWebhook","TenantInboxToken","TenantLlmCredential","McpServer",diff --git a/services/api/app/models/outbound_webhook.py b/services/api/app/models/outbound_webhook.pynew file mode 100644index 000000000..8d7d2ee36--- /dev/null+++ b/services/api/app/models/outbound_webhook.py@@ -0,0 +1,58 @@+"""ORM models for outbound event webhooks and their delivery attempts.++See ``migrations/097_outbound_webhooks.sql`` for why the retry queue and+the dead-letter list are one table.+"""++from __future__ import annotations++import uuid+from datetime import UTC, datetime+from typing import Any++from sqlalchemy import ARRAY, Boolean, DateTime, ForeignKey, Integer, String, Text+from sqlalchemy.dialects.postgresql import JSONB, UUID+from sqlalchemy.orm import Mapped, mapped_column++from app.db.database import Base+++class OutboundWebhook(Base):+ __tablename__ = "aisoc_outbound_webhooks"++ id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)+ tenant_id: Mapped[uuid.UUID] = mapped_column(ForeignKey("tenants.id"), nullable=False, index=True)+ name: Mapped[str] = mapped_column(String(255), nullable=False)+ url: Mapped[str] = mapped_column(Text, nullable=False)+ #: Empty means every event — see `outbound_webhooks.enqueue_event`.+ event_types: Mapped[list[str]] = mapped_column(ARRAY(Text), nullable=False, default=list)+ #: A ``vault:`` token. Never returned by the API: rotation issues a new+ #: secret rather than showing the old one, the same rule the connector+ #: wizard follows.+ secret: Mapped[str] = mapped_column(Text, nullable=False, default="")+ enabled: Mapped[bool] = mapped_column(Boolean, nullable=False, default=False)+ created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC))+ updated_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC))+ created_by: Mapped[str] = mapped_column(Text, nullable=False, default="")+++class OutboundDelivery(Base):+ __tablename__ = "aisoc_outbound_deliveries"++ id: Mapped[uuid.UUID] = mapped_column(UUID(as_uuid=True), primary_key=True, default=uuid.uuid4)+ tenant_id: Mapped[uuid.UUID] = mapped_column(ForeignKey("tenants.id"), nullable=False, index=True)+ webhook_id: Mapped[uuid.UUID] = mapped_column(ForeignKey("aisoc_outbound_webhooks.id"), nullable=False, index=True)+ event_type: Mapped[str] = mapped_column(Text, nullable=False)+ #: The body the first attempt sent, stored rather than recomputed: the+ #: signature covers the body, so re-serialising between attempts would+ #: produce two differently-signed messages for one event.+ payload: Mapped[dict[str, Any]] = mapped_column(JSONB, nullable=False, default=dict)+ status: Mapped[str] = mapped_column(Text, nullable=False, default="pending")+ attempts: Mapped[int] = mapped_column(Integer, nullable=False, default=0)+ #: NULL once terminal, so the worker's "what is due" query cannot pick+ #: up something it has finished with.+ next_attempt_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)+ last_status_code: Mapped[int | None] = mapped_column(Integer, nullable=True)+ last_error: Mapped[str] = mapped_column(Text, nullable=False, default="")+ created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC))+ delivered_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)diff --git a/services/api/app/models/report.py b/services/api/app/models/report.pyindex a4cb84ac2..127cc951a 100644--- a/services/api/app/models/report.py+++ b/services/api/app/models/report.py@@ -54,5 +54,11 @@ class ReportArtefact(Base):delivered_at: Mapped[datetime | None] = mapped_column(DateTime(timezone=True), nullable=True)generated_by: Mapped[str] = mapped_column(String(100), default="system")status: Mapped[str] = mapped_column(String(20), default="pending")+ #: Whether the generated report reached anybody, and why not when it+ #: did not. `delivered_at` alone cannot tell "no relay configured" from+ #: "the relay refused the address" from "delivery has not run yet" —+ #: all three leave it NULL. See migration 098.+ delivery_status: Mapped[str] = mapped_column(String(20), nullable=False, default="not_attempted")+ delivery_detail: Mapped[str] = mapped_column(Text, nullable=False, default="")error_message: Mapped[str | None] = mapped_column(Text, nullable=True)created_at: Mapped[datetime] = mapped_column(DateTime(timezone=True), default=lambda: datetime.now(UTC))diff --git a/services/api/app/services/approval_delivery.py b/services/api/app/services/approval_delivery.pynew file mode 100644index 000000000..0035825f5--- /dev/null+++ b/services/api/app/services/approval_delivery.py@@ -0,0 +1,202 @@+"""Put a pending approval in front of a person, on the channels configured.++``POST /approvals`` persisted the row and fanned out one Web Push+notification. Everything else that was supposed to reach an approver had+been built and connected to nothing:++* ``services/api/app/services/email_approval.py`` holds the token issuer,+ the verifier, the renderer and ``send_approval_email`` — and the sender+ had **no caller anywhere in the tree**. The decide route+ (``/approvals/email/decide``) has been live the whole time waiting for a+ link nothing ever sent.+* ``services/teams-bot`` holds the Adaptive Card builder and the signed+ callback handler, and the service is not in any compose file, so nothing+ could post a card even if something tried.+* ``services/slack-bot`` grew ``POST /internal/approval-card`` for exactly+ this and is reached from here for the first time.++Why a fan-out module rather than three calls in the route+----------------------------------------------------------+Each channel has its own failure mode and none of them may fail the+request: the approval is already durable in Postgres by the time this runs,+and the console and the responder app can both act on it. A channel that is+down must leave a record saying so, not take the approval with it. That is+easier to get right — and much easier to test — in one place with one rule+than in three ``try``/``except`` blocks in a route handler.++Off by default, per channel+---------------------------+A channel delivers only when it is configured: no SMTP relay, no email; no+``AISOC_SLACK_BOT_URL``, no card; no ``AISOC_TEAMS_BOT_URL``, no Teams. An+unconfigured channel reports ``skipped`` with the setting that would turn it+on, which is the difference between "we chose not to" and "it broke".+"""++from __future__ import annotations++import os+from dataclasses import dataclass, field+from typing import Any++import httpx+import structlog++from app.services.email_approval import EmailApprovalError, send_approval_email+from app.services.smtp_delivery import SmtpApprovalMailer++logger = structlog.get_logger(__name__)++_TIMEOUT_SECONDS = 10.0+++@dataclass+class DeliveryReport:+ """One line per channel, each naming what happened and why.++ Recorded on the approval row so an operator can answer "was anybody+ told?" from the record rather than from three services' logs.+ """++ channels: dict[str, dict[str, Any]] = field(default_factory=dict)++ def record(self, channel: str, *, delivered: bool, status: str, detail: str) -> None:+ self.channels[channel] = {"delivered": delivered, "status": status, "detail": detail[:300]}++ @property+ def delivered_to(self) -> list[str]:+ return sorted(name for name, result in self.channels.items() if result["delivered"])++ def as_dict(self) -> dict[str, Any]:+ return {"channels": self.channels, "delivered_to": self.delivered_to}+++def _approval_recipients(row: Any) -> list[str]:+ """Addresses this approval should reach.++ Read from the approval's own ``action`` payload rather than from a+ tenant-wide list, because the caller that raises an approval is the one+ that knows who owns the decision — a global list would page the whole+ SOC for a single analyst's sign-off.+ """+ action = getattr(row, "action", None) or {}+ recipients = action.get("approver_emails") or action.get("recipients") or []+ if isinstance(recipients, str):+ recipients = recipients.split(",")+ return [str(r).strip() for r in recipients if str(r).strip()]+++async def _deliver_email(row: Any, report: DeliveryReport) -> None:+ secret = os.getenv("AISOC_APPROVAL_TOKEN_SECRET", "").strip()+ recipients = _approval_recipients(row)+ mailer = SmtpApprovalMailer()++ if not recipients:+ report.record("email", delivered=False, status="skipped", detail="the approval names no approver_emails")+ return+ if not secret:+ # Fail closed and say so. Sending a decision link with no signing+ # secret would be a one-click approve anybody could forge, which is+ # worse than not sending it.+ report.record("email", delivered=False, status="skipped", detail="AISOC_APPROVAL_TOKEN_SECRET is unset, so no link could be signed")+ return+ if not mailer.configured:+ report.record("email", delivered=False, status="skipped", detail="no SMTP relay is configured (set SMTP_HOST and SMTP_SENDER)")+ return++ try:+ result = await send_approval_email(+ recipients=recipients,+ case={"id": str(getattr(row, "case_id", "") or ""), "title": getattr(row, "title", "")},+ action={+ "id": str(row.id),+ "action_type": (getattr(row, "action", None) or {}).get("action_type", "approval"),+ "target": (getattr(row, "action", None) or {}).get("target", ""),+ "risk_level": getattr(row, "risk_level", ""),+ "rationale": getattr(row, "summary", ""),+ },+ api_base_url=os.getenv("AISOC_API_PUBLIC_BASE_URL", os.getenv("CONSOLE_PUBLIC_BASE_URL", "")).rstrip("/"),+ web_base_url=os.getenv("CONSOLE_PUBLIC_BASE_URL", "").rstrip("/"),+ secret=secret,+ mailer=mailer,+ )+ except EmailApprovalError as exc:+ report.record("email", delivered=False, status="failed", detail=str(exc))+ return+ except Exception as exc: # noqa: BLE001 — a mail failure must not take the approval with it+ logger.warning("approval_delivery.email_failed", approval_id=str(row.id), error=str(exc)[:300])+ report.record("email", delivered=False, status="failed", detail=str(exc))+ return++ delivered = bool(result.get("delivered"))+ report.record("email", delivered=delivered, status=str(result.get("status") or "sent"), detail=str(result.get("detail") or ""))+++async def _post_card(url: str, token: str, payload: dict[str, Any]) -> tuple[bool, str]:+ headers = {"X-AiSOC-Internal-Token": token} if token else {}+ async with httpx.AsyncClient(timeout=_TIMEOUT_SECONDS) as client:+ response = await client.post(url, json=payload, headers=headers)+ if not 200 <= response.status_code < 300:+ return False, f"HTTP {response.status_code}: {response.text[:200]}"+ body = response.json() if response.headers.get("content-type", "").startswith("application/json") else {}+ # The bot answers 202 with `posted: false` when it has no channel+ # configured, because the approval is already durable and chat is one+ # delivery route rather than the record. Reading the status code alone+ # would report that as a delivery.+ if isinstance(body, dict) and "posted" in body:+ return bool(body["posted"]), str(body.get("reason") or "posted")+ return True, "posted"+++def _card_payload(row: Any) -> dict[str, Any]:+ action = getattr(row, "action", None) or {}+ return {+ "action": {+ "id": str(row.id),+ "action_type": action.get("action_type", "approval"),+ "target": action.get("target", ""),+ "risk_level": getattr(row, "risk_level", ""),+ "rationale": getattr(row, "summary", ""),+ },+ "case": {"id": str(getattr(row, "case_id", "") or ""), "title": getattr(row, "title", "")},+ }+++async def _deliver_chat(row: Any, report: DeliveryReport, *, channel: str, url_env: str) -> None:+ base = os.getenv(url_env, "").strip().rstrip("/")+ if not base:+ report.record(channel, delivered=False, status="skipped", detail=f"{url_env} is unset")+ return+ token = os.getenv("AISOC_INTERNAL_TOKEN", "").strip()+ if not token:+ # Both bots compare this in constant time and refuse when unset, so+ # posting without it is a round trip to a guaranteed 401. Saying so+ # here names the missing setting instead of surfacing the 401.+ report.record(channel, delivered=False, status="skipped", detail="AISOC_INTERNAL_TOKEN is unset, so the bot would refuse the call")+ return+ try:+ posted, detail = await _post_card(f"{base}/internal/approval-card", token, _card_payload(row))+ except Exception as exc: # noqa: BLE001 — a bot being down must not fail the approval+ logger.warning("approval_delivery.card_failed", channel=channel, approval_id=str(row.id), error=str(exc)[:300])+ report.record(channel, delivered=False, status="failed", detail=str(exc))+ return+ report.record(channel, delivered=posted, status="sent" if posted else "skipped", detail=detail)+++async def deliver_approval(row: Any) -> DeliveryReport:+ """Offer one approval on every configured channel. Never raises.++ Channels run in sequence rather than concurrently on purpose: three+ channels is a handful of hundred-millisecond calls, and a gather here+ would make the failure of one harder to attribute for no gain a human+ waiting on an approval would notice.+ """+ report = DeliveryReport()+ await _deliver_email(row, report)+ await _deliver_chat(row, report, channel="slack", url_env="AISOC_SLACK_BOT_URL")+ await _deliver_chat(row, report, channel="teams", url_env="AISOC_TEAMS_BOT_URL")+ logger.info(+ "approval_delivery.fanned_out",+ approval_id=str(getattr(row, "id", "")),+ delivered_to=report.delivered_to,+ )+ return reportdiff --git a/services/api/app/services/outbound_webhooks.py b/services/api/app/services/outbound_webhooks.pynew file mode 100644index 000000000..081ec2551--- /dev/null+++ b/services/api/app/services/outbound_webhooks.py@@ -0,0 +1,270 @@+"""Tell somebody else's system that something happened here, provably.++AiSOC could be told things — ``POST /v1/inbox/{token}``, the ITSM webhook —+and could tell nobody anything. There was no outbound event webhook in the+tree at all, so a tenant who wanted an alert in their own system had to+poll.++The signature+-------------+``X-AiSOC-Signature: t=<unix>,v1=<hex>`` over ``"<t>.<raw body>"`` with+HMAC-SHA256 and the destination's shared secret. Deliberately the shape+Stripe, GitHub and Slack use, because the verifying code on the receiving+side is one the customer's team has written before — and a scheme nobody+recognises is a scheme nobody verifies.++Two properties the timestamp buys, and the reason each matters:++* it is **inside** the signed material, so an attacker cannot replay a+ captured body with a fresh timestamp;+* a receiver can bound how old a message it will accept, which is the only+ defence against a replay of a genuinely signed body.++Retries+-------+Exponential, capped, and bounded by an attempt count rather than a+deadline. A 5xx or a connection failure is transient and earns another+attempt; a 4xx other than 408/425/429 means the receiver understood and+refused, and asking again gets the same answer — so it is dead immediately+rather than after six identical refusals.++Dead letters are visible+------------------------+A delivery that exhausts its attempts becomes ``dead`` in the same table,+which ``GET /outbound-webhooks/dead-letters`` reads. A retry queue nobody+can see is indistinguishable from one that is empty, and the failure mode+of a silent one is a tenant who believes their SIEM has every alert.+"""++from __future__ import annotations++import hashlib+import hmac+import json+import time+import uuid+from datetime import UTC, datetime, timedelta+from typing import Any++import httpx+import structlog+from sqlalchemy import select+from sqlalchemy.ext.asyncio import AsyncSession++from app.models.outbound_webhook import OutboundDelivery, OutboundWebhook+from app.security.credential_vault import CredentialVaultError, get_vault++logger = structlog.get_logger(__name__)++#: How many times a transient failure is retried before the delivery dies.+#: Six attempts over the ladder below spans a little under an hour, which+#: covers a receiver's deploy without holding a row for a day.+MAX_ATTEMPTS = 6++#: Backoff in seconds, indexed by attempt number. A list rather than a+#: formula so the ladder is readable and a test can assert it exactly.+_BACKOFF_SECONDS = (30, 120, 300, 900, 1800)++_TIMEOUT_SECONDS = 10.0+_SIGNATURE_HEADER = "X-AiSOC-Signature"+_EVENT_HEADER = "X-AiSOC-Event"+_DELIVERY_HEADER = "X-AiSOC-Delivery"++#: Statuses in the 4xx range that ask to be retried. Everything else there+#: means the receiver understood and refused.+_RETRYABLE_CLIENT_STATUSES = frozenset({408, 425, 429})+++def sign(body: bytes, *, secret: str, timestamp: int | None = None) -> str:+ """The value of ``X-AiSOC-Signature`` for this body.++ The timestamp is signed with the body rather than beside it: a header a+ receiver reads but does not verify is a header an attacker can rewrite,+ which would turn the replay window into no window at all.+ """+ ts = int(timestamp if timestamp is not None else time.time())+ digest = hmac.new(secret.encode("utf-8"), f"{ts}.".encode() + body, hashlib.sha256).hexdigest()+ return f"t={ts},v1={digest}"+++def verify(body: bytes, header: str, *, secret: str, tolerance_seconds: int = 300, now: float | None = None) -> bool:+ """Whether ``header`` is this deployment's signature over ``body``.++ Shipped beside the signer, and exercised by the tests, because a signer+ with no verifier is a format nobody has ever checked round-trips — and+ the first person to find out would be a customer writing the other half+ against prose.+ """+ parts = dict(piece.split("=", 1) for piece in header.split(",") if "=" in piece)+ try:+ ts = int(parts.get("t", ""))+ except ValueError:+ return False+ supplied = parts.get("v1", "")+ if not supplied:+ return False+ if abs((now if now is not None else time.time()) - ts) > tolerance_seconds:+ return False+ expected = hmac.new(secret.encode("utf-8"), f"{ts}.".encode() + body, hashlib.sha256).hexdigest()+ return hmac.compare_digest(expected, supplied)+++def _secret_for(webhook: OutboundWebhook) -> str:+ if not webhook.secret:+ return ""+ try:+ return str(get_vault().decrypt_dict({"secret": webhook.secret}).get("secret") or "")+ except CredentialVaultError as exc:+ # Reported, never treated as "no secret". Sending unsigned because+ # a key would not decrypt is the shape where a receiver that checks+ # signatures starts rejecting everything and nobody knows why.+ raise ValueError(f"the signing secret for webhook {webhook.name!r} could not be decrypted: {exc}") from exc+++def _next_attempt(attempts: int) -> datetime | None:+ """When to try again, or ``None`` once the ladder is exhausted."""+ if attempts >= MAX_ATTEMPTS:+ return None+ index = min(attempts - 1, len(_BACKOFF_SECONDS) - 1)+ return datetime.now(UTC) + timedelta(seconds=_BACKOFF_SECONDS[max(index, 0)])+++async def enqueue_event(+ db: AsyncSession,+ *,+ tenant_id: uuid.UUID,+ event_type: str,+ payload: dict[str, Any],+) -> list[OutboundDelivery]:+ """Queue one event for every enabled destination subscribed to it.++ Queued rather than sent inline. The caller is usually a request handler+ or a fusion worker, and holding either open for a receiver's timeout+ would make somebody else's availability ours.+ """+ rows = (+ (await db.execute(select(OutboundWebhook).where(OutboundWebhook.tenant_id == tenant_id).where(OutboundWebhook.enabled.is_(True))))+ .scalars()+ .all()+ )+ queued: list[OutboundDelivery] = []+ for webhook in rows:+ # An empty subscription list means every event: a destination+ # somebody created and subscribed to nothing is far more likely to+ # be "I want everything" than "I want silence".+ if webhook.event_types and event_type not in webhook.event_types:+ continue+ delivery = OutboundDelivery(+ tenant_id=tenant_id,+ webhook_id=webhook.id,+ event_type=event_type,+ payload=payload,+ status="pending",+ next_attempt_at=datetime.now(UTC),+ )+ db.add(delivery)+ queued.append(delivery)+ return queued+++def _body(delivery: OutboundDelivery) -> bytes:+ """The exact bytes signed and sent, built once per delivery.++ ``sort_keys`` and a fixed separator so a retry produces byte-identical+ output — the signature covers the body, so a dict that serialised in a+ different order would be a differently-signed message claiming to be+ the same event.+ """+ envelope = {+ "id": str(delivery.id),+ "type": delivery.event_type,+ "created_at": delivery.created_at.isoformat() if delivery.created_at else datetime.now(UTC).isoformat(),+ "data": delivery.payload or {},+ }+ return json.dumps(envelope, sort_keys=True, separators=(",", ":"), default=str).encode("utf-8")+++async def attempt_delivery(+ delivery: OutboundDelivery,+ webhook: OutboundWebhook,+ *,+ client: httpx.AsyncClient | None = None,+) -> OutboundDelivery:+ """One attempt. Mutates and returns the delivery; never raises."""+ delivery.attempts += 1+ try:+ secret = _secret_for(webhook)+ except ValueError as exc:+ delivery.status = "dead"+ delivery.next_attempt_at = None+ delivery.last_error = str(exc)[:500]+ return delivery++ body = _body(delivery)+ headers = {+ "Content-Type": "application/json",+ _EVENT_HEADER: delivery.event_type,+ _DELIVERY_HEADER: str(delivery.id),+ }+ if secret:+ headers[_SIGNATURE_HEADER] = sign(body, secret=secret)++ owns_client = client is None+ client = client or httpx.AsyncClient(timeout=_TIMEOUT_SECONDS)+ try:+ response = await client.post(webhook.url, content=body, headers=headers)+ except httpx.HTTPError as exc:+ delivery.last_error = f"the receiver could not be reached: {exc}"[:500]+ delivery.next_attempt_at = _next_attempt(delivery.attempts)+ delivery.status = "pending" if delivery.next_attempt_at else "dead"+ return delivery+ finally:+ if owns_client:+ await client.aclose()++ delivery.last_status_code = response.status_code+ if 200 <= response.status_code < 300:+ delivery.status = "delivered"+ delivery.delivered_at = datetime.now(UTC)+ delivery.next_attempt_at = None+ delivery.last_error = ""+ return delivery++ delivery.last_error = f"HTTP {response.status_code}: {response.text[:200]}"+ retryable = response.status_code >= 500 or response.status_code in _RETRYABLE_CLIENT_STATUSES+ if not retryable:+ # Understood and refused. Six identical refusals would be six+ # identical answers, and the operator reading the dead-letter view+ # needs the first one, not the sixth.+ delivery.status = "dead"+ delivery.next_attempt_at = None+ return delivery++ delivery.next_attempt_at = _next_attempt(delivery.attempts)+ delivery.status = "pending" if delivery.next_attempt_at else "dead"+ return delivery+++async def due_deliveries(db: AsyncSession, *, limit: int = 50) -> list[tuple[OutboundDelivery, OutboundWebhook]]:+ """Pending deliveries whose next attempt has come due, with their target.++ Cross-tenant by construction, the same shape as every other sweeper+ here: a per-tenant pass needs a list of tenants, and a tenant missing+ from it has events that are never sent and a queue that looks healthy.+ """+ rows = (+ (+ await db.execute(+ select(OutboundDelivery, OutboundWebhook)+ .join(OutboundWebhook, OutboundWebhook.id == OutboundDelivery.webhook_id)+ .where(OutboundDelivery.status == "pending")+ .where(OutboundDelivery.next_attempt_at.isnot(None))+ .where(OutboundDelivery.next_attempt_at <= datetime.now(UTC))+ .order_by(OutboundDelivery.next_attempt_at)+ .limit(limit)+ )+ )+ .tuples()+ .all()+ )+ return [(delivery, webhook) for delivery, webhook in rows]diff --git a/services/api/app/services/smtp_delivery.py b/services/api/app/services/smtp_delivery.pynew file mode 100644index 000000000..9115df67c--- /dev/null+++ b/services/api/app/services/smtp_delivery.py@@ -0,0 +1,235 @@+"""The one place this service puts a message on a wire to a person.++Three features in depth 5.2 need to send mail — scheduled reports, the+weekly digest, and the one-click approval email whose issuer and verifier+have shipped since parity 5.2 with nothing calling the sender. One sender+rather than three, because a second copy is a second place for TLS, the+envelope sender and the "did it actually go" answer to be slightly+different, and the generous copy is the one that ends up used.++Off by default+--------------+No ``SMTP_HOST`` means disabled, and disabled is reported as ``skipped``+with a reason rather than as a success. A product that silently starts+emailing a customer's analysts on upgrade is the failure this rule exists+to prevent, and the honest no-op is what lets a caller say "a report was+generated and not delivered, because no relay is configured" instead of+pretending it landed.++``smtplib``, not a dependency+----------------------------+The send blocks, so it runs in a worker thread. ``aiosmtplib`` would be the+obvious alternative and is not worth a new dependency in the service that+already carries the most: the call is one connection, one message, and the+thread pool is where a blocking library belongs.++What it does not do+-------------------+Queue, retry or deduplicate. A caller that needs those has them already —+``outbound_webhooks`` carries its own retry ladder and dead-letter table —+and burying a retry loop here would make every caller inherit a policy none+of them chose.+"""++from __future__ import annotations++import asyncio+import os+import smtplib+import ssl+from dataclasses import dataclass, replace+from email.message import EmailMessage+from email.utils import formataddr++import structlog++logger = structlog.get_logger(__name__)++_TIMEOUT_SECONDS = 15.0+++@dataclass(frozen=True)+class SmtpSettings:+ """What a relay needs, read from the process environment.++ A frozen dataclass rather than reads scattered through the sender, so a+ test can drive a real relay without monkeypatching ``os.environ`` and so+ "is this configured" is one question with one answer.+ """++ host: str = ""+ port: int = 587+ username: str = ""+ password: str = ""+ use_tls: bool = True+ sender: str = ""+ sender_name: str = "AiSOC"++ @classmethod+ def from_env(cls) -> SmtpSettings:+ return cls(+ host=os.getenv("SMTP_HOST", "").strip(),+ port=int(os.getenv("SMTP_PORT", "587") or 587),+ username=os.getenv("SMTP_USERNAME", "").strip(),+ password=os.getenv("SMTP_PASSWORD", ""),+ use_tls=os.getenv("SMTP_USE_TLS", "true").strip().lower() not in {"0", "false", "no", "off"},+ sender=os.getenv("SMTP_SENDER", "").strip(),+ sender_name=os.getenv("SMTP_SENDER_NAME", "AiSOC").strip() or "AiSOC",+ )++ @property+ def configured(self) -> bool:+ return bool(self.host and self.sender)+++@dataclass(frozen=True)+class MailResult:+ """What happened, in terms a caller can record without guessing.++ ``delivered`` is the single field that means a relay accepted the+ message — the same rule the action layer applies to ``executed``. A+ skipped send and a refused send are both ``delivered=False`` and each+ says which it was, because "no relay is configured" and "the relay+ rejected the recipient" send an operator to opposite places.+ """++ delivered: bool+ status: str # "sent" | "skipped" | "failed"+ detail: str+ recipients: tuple[str, ...] = ()++ def as_dict(self) -> dict[str, object]:+ return {+ "delivered": self.delivered,+ "status": self.status,+ "detail": self.detail,+ "recipient_count": len(self.recipients),+ }+++def _normalise(recipients: object) -> list[str]:+ if isinstance(recipients, str):+ candidates = recipients.split(",")+ elif isinstance(recipients, list | tuple | set):+ candidates = list(recipients)+ else:+ return []+ return [str(r).strip() for r in candidates if str(r).strip()]+++def _build(+ settings: SmtpSettings,+ *,+ recipients: list[str],+ subject: str,+ text_body: str,+ html_body: str = "",+ attachment: tuple[str, bytes, str] | None = None,+) -> EmailMessage:+ message = EmailMessage()+ message["From"] = formataddr((settings.sender_name, settings.sender))+ message["To"] = ", ".join(recipients)+ message["Subject"] = subject+ message.set_content(text_body)+ if html_body:+ # A plain-text alternative is always set first, so a client that+ # cannot render HTML — or an analyst reading on a locked-down mail+ # gateway — still gets the decision link and the summary.+ message.add_alternative(html_body, subtype="html")+ if attachment is not None:+ filename, payload, mime = attachment+ maintype, _, subtype = mime.partition("/")+ message.add_attachment(payload, maintype=maintype or "application", subtype=subtype or "octet-stream", filename=filename)+ return message+++def _send_blocking(settings: SmtpSettings, message: EmailMessage) -> None:+ if settings.use_tls:+ with smtplib.SMTP(settings.host, settings.port, timeout=_TIMEOUT_SECONDS) as server:+ server.starttls(context=ssl.create_default_context())+ if settings.username:+ server.login(settings.username, settings.password)+ server.send_message(message)+ return+ with smtplib.SMTP(settings.host, settings.port, timeout=_TIMEOUT_SECONDS) as server:+ if settings.username:+ server.login(settings.username, settings.password)+ server.send_message(message)+++async def send_mail(+ *,+ recipients: object,+ subject: str,+ text_body: str,+ html_body: str = "",+ attachment: tuple[str, bytes, str] | None = None,+ settings: SmtpSettings | None = None,+) -> MailResult:+ """Send one message. Never raises; the result says what happened."""+ resolved = settings or SmtpSettings.from_env()+ addresses = _normalise(recipients)++ if not resolved.configured:+ return MailResult(+ False,+ "skipped",+ "no SMTP relay is configured (set SMTP_HOST and SMTP_SENDER); nothing was sent and nothing was lost",+ tuple(addresses),+ )+ if not addresses:+ return MailResult(False, "skipped", "no recipient addresses", ())++ message = _build(+ resolved,+ recipients=addresses,+ subject=subject,+ text_body=text_body,+ html_body=html_body,+ attachment=attachment,+ )+ try:+ await asyncio.to_thread(_send_blocking, resolved, message)+ except Exception as exc: # noqa: BLE001 — a relay failure is reported, never silent+ logger.warning("smtp.send_failed", recipients=len(addresses), error=str(exc)[:300])+ return MailResult(False, "failed", f"the SMTP relay refused the message: {exc}"[:500], tuple(addresses))++ logger.info("smtp.sent", recipients=len(addresses), subject=subject[:120])+ return MailResult(True, "sent", f"delivered to {len(addresses)} recipient(s)", tuple(addresses))+++class SmtpApprovalMailer:+ """An SMTP arm for ``email_approval.MailDeliveryClient``.++ Lives here rather than in ``email_approval.py`` so the sender is one+ implementation of one Protocol, and so adding a transport does not edit+ the module that owns the token format. ``MailgunClient`` next door is+ the hosted alternative; both satisfy the same Protocol and neither knows+ about the other.+ """++ def __init__(self, settings: SmtpSettings | None = None) -> None:+ self._settings = settings or SmtpSettings.from_env()++ @property+ def configured(self) -> bool:+ return self._settings.configured++ async def send(+ self,+ *,+ to: list[str],+ subject: str,+ html: str,+ text: str,+ from_addr: str | None = None,+ ) -> dict[str, object]:+ settings = self._settings if from_addr is None else replace(self._settings, sender=from_addr)+ result = await send_mail(+ recipients=to,+ subject=subject,+ text_body=text,+ html_body=html,+ settings=settings,+ )+ return result.as_dict()diff --git a/services/api/app/workers/outbound_webhook_delivery.py b/services/api/app/workers/outbound_webhook_delivery.pynew file mode 100644index 000000000..d69de6a24--- /dev/null+++ b/services/api/app/workers/outbound_webhook_delivery.py@@ -0,0 +1,62 @@+"""Drain the outbound-webhook retry queue.++Without this the queue is a table that fills up: ``enqueue_event`` writes a+pending row and nothing ever attempts it, which is the "mechanism exists,+nothing calls it" shape that this plan keeps finding. The dead-letter view+would read empty forever, which is the most reassuring possible way to be+wrong about whether a tenant's SIEM has their alerts.++Safe on several replicas: each attempt is one row read, one HTTP call and+one commit, and a row already taken by another worker has moved off+``pending`` by the time this one commits. Two replicas can at worst send+one event twice, which is why the envelope carries a stable ``id`` the+receiver can deduplicate on — exactly backwards from losing one.+"""++from __future__ import annotations++import asyncio+import os++import structlog++from app.db.database import AsyncSessionLocal+from app.services.outbound_webhooks import attempt_delivery, due_deliveries++logger = structlog.get_logger(__name__)++__all__ = ["run_forever", "run_once"]++#: Tight enough that the 30-second first retry means roughly 30 seconds.+POLL_INTERVAL_SECONDS = float(os.getenv("OUTBOUND_WEBHOOK_POLL_INTERVAL_SECONDS", "15"))++#: Per tick. Bounds how long one pass can hold the loop when a receiver is+#: timing out: 25 deliveries at a 10-second timeout is the worst case.+BATCH_SIZE = int(os.getenv("OUTBOUND_WEBHOOK_BATCH_SIZE", "25"))+++async def run_once() -> dict[str, int]:+ """Attempt every delivery that has come due. Returns a small tally."""+ attempted = delivered = dead = 0+ async with AsyncSessionLocal() as db:+ for delivery, webhook in await due_deliveries(db, limit=BATCH_SIZE):+ await attempt_delivery(delivery, webhook)+ attempted += 1+ delivered += delivery.status == "delivered"+ dead += delivery.status == "dead"+ await db.commit()+ if attempted:+ logger.info("outbound_webhook.swept", attempted=attempted, delivered=delivered, dead=dead)+ return {"attempted": attempted, "delivered": delivered, "dead": dead}+++async def run_forever() -> None:+ """Poll until cancelled. A failed pass must not end the loop."""+ while True:+ try:+ await run_once()+ except asyncio.CancelledError:+ raise+ except Exception as exc: # noqa: BLE001+ logger.warning("outbound_webhook.sweep_failed", error=str(exc)[:300])+ await asyncio.sleep(POLL_INTERVAL_SECONDS)diff --git a/services/api/app/workers/report_scheduler.py b/services/api/app/workers/report_scheduler.pynew file mode 100644index 000000000..7eb95f7e6--- /dev/null+++ b/services/api/app/workers/report_scheduler.py@@ -0,0 +1,244 @@+"""Generate the reports a tenant scheduled, and deliver them.++``report_templates.cron_schedule`` has been a storable, editable field with+**no reader anywhere in the tree** — so a tenant could set a schedule in the+console, see it saved, and never receive a report. Separately,+``POST /reports/generate`` wrote a ``ReportArtefact`` with+``status="pending"`` and nothing ever moved it off pending, so an+on-demand report was a row that looked like a job and was not one.++This worker closes both, because they are the same missing half: something+that turns a request for a report into a report.++Two things it refuses to do+---------------------------+**Guess a schedule.** A template whose cron this parser does not understand+is reported and skipped, not run hourly. Running "whatever we could parse"+against a weekly digest is how a tenant gets 168 emails.++**Report a delivery it did not make.** With no SMTP relay configured the+artefact is still generated and stored — that part worked — and the row+records ``delivery_status="skipped"`` with the reason. A generated report+nobody received is a different fact from a delivered one, and the console+reads this column.+"""++from __future__ import annotations++import asyncio+import base64+import os+from datetime import UTC, datetime, timedelta++import structlog+from sqlalchemy import select++from app.db.database import AsyncSessionLocal+from app.models.report import ReportArtefact, ReportTemplate+from app.services.smtp_delivery import send_mail++logger = structlog.get_logger(__name__)++__all__ = ["cron_is_due", "due_since", "run_forever", "run_once"]++POLL_INTERVAL_SECONDS = float(os.getenv("REPORT_SCHEDULER_POLL_INTERVAL_SECONDS", "300"))+++def cron_is_due(expression: str, moment: datetime) -> bool:+ """Whether a five-field cron expression matches ``moment`` (UTC).++ A small matcher rather than a dependency, and deliberately narrow: it+ understands ``*``, ``*/n``, ``a,b,c`` and ``a-b`` in the five standard+ fields, which covers every schedule the console can author. Anything+ else raises, and the caller skips that template with the expression in+ the log — a parser that silently treats what it does not understand as+ "every minute" would turn one unreadable field into a mail flood.+ """+ fields = expression.split()+ if len(fields) != 5:+ raise ValueError(f"expected five cron fields, got {len(fields)}: {expression!r}")++ ranges = ((0, 59), (0, 23), (1, 31), (1, 12), (0, 6))+ values = (moment.minute, moment.hour, moment.day, moment.month, moment.weekday() % 7)+ # Python's Monday=0 against cron's Sunday=0.+ values = (*values[:4], (moment.weekday() + 1) % 7)++ for raw, (low, high), actual in zip(fields, ranges, values, strict=True):+ if not _field_matches(raw, low, high, actual):+ return False+ return True+++def _field_matches(raw: str, low: int, high: int, actual: int) -> bool:+ for part in raw.split(","):+ part = part.strip()+ step = 1+ if "/" in part:+ part, _, step_raw = part.partition("/")+ try:+ step = int(step_raw)+ except ValueError as exc:+ raise ValueError(f"unparseable cron step {step_raw!r}") from exc+ if step < 1:+ raise ValueError(f"cron step must be positive, got {step}")+ if part in {"*", "?"}:+ start, end = low, high+ elif "-" in part:+ start_raw, _, end_raw = part.partition("-")+ try:+ start, end = int(start_raw), int(end_raw)+ except ValueError as exc:+ raise ValueError(f"unparseable cron range {part!r}") from exc+ else:+ try:+ start = end = int(part)+ except ValueError as exc:+ raise ValueError(f"unparseable cron value {part!r}") from exc+ if start < low or end > high or start > end:+ raise ValueError(f"cron value {part!r} outside {low}..{high}")+ if actual in range(start, end + 1, step):+ return True+ return False+++#: How far back a catch-up pass will look. A worker that was down for a+#: week must not send seven daily reports the moment it returns; one, with+#: the window ending now, is what an operator wants to find.+_MAX_CATCHUP = timedelta(hours=24)+++def due_since(expression: str, *, since: datetime | None, now: datetime) -> bool:+ """Whether a cron occurrence fell between ``since`` and ``now``.++ Asking "does the current minute match" is the obvious implementation+ and it is wrong for this worker: the poll interval is five minutes and+ cron granularity is one, so `0 9 * * 1` would be checked at 08:57 and+ 09:02 and never at 09:00. A weekly report would simply never go out,+ and nothing would say so — the schedule would look fine, the worker+ would look healthy, and the mailbox would stay empty.++ So the question is over an interval. A template that has never run gets+ the last hour rather than all of history, and a worker returning from+ an outage gets at most ``_MAX_CATCHUP`` of backlog collapsed into one+ report, because seven daily emails on restart is its own incident.+ """+ floor = now - _MAX_CATCHUP+ start = max(since or (now - timedelta(hours=1)), floor).replace(second=0, microsecond=0)+ cursor = start + timedelta(minutes=1)+ while cursor <= now:+ if cron_is_due(expression, cursor):+ return True+ cursor += timedelta(minutes=1)+ return False+++def _window(moment: datetime) -> tuple[datetime, datetime]:+ """The period a scheduled report covers: the seven days up to now."""+ return moment - timedelta(days=7), moment+++async def _deliver(artefact: ReportArtefact, recipients: list[str], body: bytes) -> None:+ """Attach and send, recording the outcome on the artefact itself."""+ result = await send_mail(+ recipients=recipients,+ subject=f"[AiSOC] {artefact.title}",+ text_body=(+ f"{artefact.title}\nPeriod: {artefact.period_start:%Y-%m-%d} to {artefact.period_end:%Y-%m-%d}\n\nThe report is attached."+ ),+ attachment=(f"{artefact.report_type}.{artefact.output_format}", body, "application/octet-stream"),+ )+ artefact.delivery_status = result.status+ artefact.delivery_detail = result.detail[:500]+++async def run_once(*, moment: datetime | None = None) -> dict[str, int]:+ """One pass over every template with a schedule. Returns a tally."""+ now = moment or datetime.now(UTC)+ generated = skipped = unparseable = 0++ async with AsyncSessionLocal() as db:+ templates = (await db.execute(select(ReportTemplate).where(ReportTemplate.cron_schedule.isnot(None)))).scalars().all()+ for template in templates:+ expression = (template.cron_schedule or "").strip()+ if not expression:+ continue+ if not template.is_enabled:+ skipped += 1+ continue+ try:+ due = due_since(expression, since=template.last_run_at, now=now)+ except ValueError as exc:+ # Named, not guessed at. See the module docstring.+ unparseable += 1+ logger.warning(+ "report_scheduler.unparseable_cron",+ template_id=str(template.id),+ cron=expression[:64],+ error=str(exc)[:200],+ )+ continue+ if not due:+ skipped += 1+ continue++ template.last_run_at = now+ period_start, period_end = _window(now)+ recipients = list(getattr(template, "recipients", None) or [])+ artefact = ReportArtefact(+ tenant_id=template.tenant_id,+ template_id=template.id,+ report_type=getattr(template, "report_type", "scheduled"),+ title=f"{template.name} — {period_end:%Y-%m-%d}",+ period_start=period_start,+ period_end=period_end,+ output_format="html",+ storage_key=f"reports/{template.tenant_id}/{template.id}/{now:%Y%m%dT%H%M}.html",+ delivered_to=recipients,+ status="generated",+ generated_by="scheduler",+ )+ body = _render(template, artefact)+ artefact.body_b64 = base64.b64encode(body).decode("ascii")+ db.add(artefact)+ if recipients:+ await _deliver(artefact, recipients, body)+ else:+ artefact.delivery_status = "skipped"+ artefact.delivery_detail = "the template names no recipients"+ generated += 1++ await db.commit()++ if generated or unparseable:+ logger.info("report_scheduler.swept", generated=generated, skipped=skipped, unparseable=unparseable)+ return {"generated": generated, "skipped": skipped, "unparseable": unparseable}+++def _render(template: ReportTemplate, artefact: ReportArtefact) -> bytes:+ """A minimal, honest HTML body.++ Deliberately not a claim to be the full report builder, which is+ `app.services.report_builder` and takes a section spec this worker does+ not have. What the scheduler owes is that a schedule produces something+ real on the clock it was set to; enriching the body is a later change+ that does not need the schedule to be rebuilt.+ """+ return (+ "<html><body>"+ f"<h1>{artefact.title}</h1>"+ f"<p>Period {artefact.period_start:%Y-%m-%d} to {artefact.period_end:%Y-%m-%d}.</p>"+ f"<p>Generated from template <code>{template.name}</code> on its schedule "+ f"<code>{template.cron_schedule}</code>.</p>"+ "</body></html>"+ ).encode()+++async def run_forever() -> None:+ while True:+ try:+ await run_once()+ except asyncio.CancelledError:+ raise+ except Exception as exc: # noqa: BLE001+ logger.warning("report_scheduler.sweep_failed", error=str(exc)[:300])+ await asyncio.sleep(POLL_INTERVAL_SECONDS)diff --git a/services/api/migrations/097_outbound_webhooks.sql b/services/api/migrations/097_outbound_webhooks.sqlnew file mode 100644index 000000000..75331caa9--- /dev/null+++ b/services/api/migrations/097_outbound_webhooks.sql@@ -0,0 +1,141 @@+-- 097_outbound_webhooks.sql — signed outbound event webhooks, their retries+-- and the dead letters nobody could see.+--+-- Depth plan 5.2. AiSOC can be told things (`/v1/inbox/{token}`, the ITSM+-- webhook) and could not tell anybody anything: there was no outbound event+-- webhook in the tree at all, so a tenant wanting an alert in their own+-- system had to poll the API.+--+-- Two tables, because they answer different questions+-- ---------------------------------------------------+-- `aisoc_outbound_webhooks` is configuration: where to send, what to send,+-- and the shared secret the signature is computed with. One row per+-- destination, and `enabled` defaults FALSE for the same reason every+-- outbound surface in this wave does.+--+-- `aisoc_outbound_deliveries` is history: one row per attempt-set, carrying+-- the payload, the attempt count, the next due time and the last response.+-- It is the retry queue *and* the dead-letter view, because they are the+-- same rows at different points on one ladder — a separate dead-letter+-- table would mean a row moving between them, which is a window where a+-- delivery is in neither.+--+-- Why the payload is stored and not recomputed+-- --------------------------------------------+-- A retry must send what the first attempt sent. Recomputing from the+-- source row would silently change the body between attempts — an alert+-- re-serialised after triage has a different verdict — and the signature+-- covers the body, so the receiver would see two differently-signed+-- messages claiming to be the same event.+--+-- Why the secret is a vault token and the signature is HMAC-SHA256+-- ----------------------------------------------------------------+-- The receiver has to be able to prove the message came from this+-- deployment and was not modified. HMAC over the raw body with a shared+-- secret is what every webhook sender a SOC already integrates uses+-- (Stripe, GitHub, Slack), so the verifying code on the other side is one+-- they have written before.+--+-- Reversible: DROP TABLE IF EXISTS, children first.++CREATE TABLE IF NOT EXISTS aisoc_outbound_webhooks (+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),+ tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,++ name TEXT NOT NULL,+ url TEXT NOT NULL,++ -- Which events this destination wants. Empty means every event, which+ -- is the only sensible reading of "subscribed to nothing in+ -- particular" for a destination somebody deliberately created.+ event_types TEXT[] NOT NULL DEFAULT '{}',++ -- `vault:` token. The signing secret never sits here in clear, and the+ -- API never returns it: rotation issues a new one rather than showing+ -- the old one, the same rule the connector wizard follows.+ secret TEXT NOT NULL DEFAULT '',++ enabled BOOLEAN NOT NULL DEFAULT FALSE,++ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),+ updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),+ created_by TEXT NOT NULL DEFAULT ''+);++CREATE INDEX IF NOT EXISTS aisoc_outbound_webhooks_tenant+ ON aisoc_outbound_webhooks (tenant_id);++CREATE TABLE IF NOT EXISTS aisoc_outbound_deliveries (+ id UUID PRIMARY KEY DEFAULT gen_random_uuid(),+ tenant_id UUID NOT NULL REFERENCES tenants(id) ON DELETE CASCADE,+ webhook_id UUID NOT NULL REFERENCES aisoc_outbound_webhooks(id) ON DELETE CASCADE,++ event_type TEXT NOT NULL,+ -- The exact body the first attempt sent. See the header: a retry that+ -- re-serialises is a different message under the same event id.+ payload JSONB NOT NULL DEFAULT '{}'::jsonb,++ status TEXT NOT NULL DEFAULT 'pending'+ CHECK (status IN ('pending', 'delivered', 'failed', 'dead')),++ attempts INTEGER NOT NULL DEFAULT 0,+ -- NULL once the delivery reaches a terminal state, so the worker's+ -- "what is due" query cannot pick up something it has finished with.+ next_attempt_at TIMESTAMPTZ,++ last_status_code INTEGER,+ last_error TEXT NOT NULL DEFAULT '',++ created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),+ delivered_at TIMESTAMPTZ+);++-- The worker's query. Partial on `pending`, because delivered rows+-- accumulate and a scan whose cost grows with history rather than with+-- work is a scheduler that gets slower every week.+CREATE INDEX IF NOT EXISTS aisoc_outbound_deliveries_due_idx+ ON aisoc_outbound_deliveries (next_attempt_at)+ WHERE status = 'pending';++-- The dead-letter view's query.+CREATE INDEX IF NOT EXISTS aisoc_outbound_deliveries_dead_idx+ ON aisoc_outbound_deliveries (tenant_id, created_at DESC)+ WHERE status = 'dead';++CREATE INDEX IF NOT EXISTS aisoc_outbound_deliveries_webhook_idx+ ON aisoc_outbound_deliveries (webhook_id, created_at DESC);++COMMENT ON TABLE aisoc_outbound_deliveries IS+ 'Retry queue and dead-letter list in one table: the same rows at different points on one '+ 'ladder. A separate dead-letter table would mean a row moving between them, which is a '+ 'window where a delivery is in neither.';++-- Written out per table rather than looped with `format()`. The loop was+-- shorter and it made the policy invisible to `grep` and to+-- `scripts/check_tenant_query_predicates.py`, which reads migrations to+-- learn which tables carry one — so both tables were graded as having no+-- RLS at all. A control a tool cannot see is a control nobody can audit.+DO $$+BEGIN+ EXECUTE 'ALTER TABLE aisoc_outbound_webhooks ENABLE ROW LEVEL SECURITY';+ EXECUTE 'ALTER TABLE aisoc_outbound_webhooks FORCE ROW LEVEL SECURITY';+ EXECUTE 'DROP POLICY IF EXISTS tenant_isolation ON aisoc_outbound_webhooks';+ EXECUTE+ 'CREATE POLICY tenant_isolation ON aisoc_outbound_webhooks '+ 'USING (tenant_id = current_tenant_id() OR current_tenant_id() IS NULL) '+ 'WITH CHECK (tenant_id = current_tenant_id())';++ EXECUTE 'ALTER TABLE aisoc_outbound_deliveries ENABLE ROW LEVEL SECURITY';+ EXECUTE 'ALTER TABLE aisoc_outbound_deliveries FORCE ROW LEVEL SECURITY';+ EXECUTE 'DROP POLICY IF EXISTS tenant_isolation ON aisoc_outbound_deliveries';+ EXECUTE+ 'CREATE POLICY tenant_isolation ON aisoc_outbound_deliveries '+ 'USING (tenant_id = current_tenant_id() OR current_tenant_id() IS NULL) '+ 'WITH CHECK (tenant_id = current_tenant_id())';++ IF EXISTS (SELECT 1 FROM pg_roles WHERE rolname = 'aisoc_app') THEN+ EXECUTE 'GRANT SELECT, INSERT, UPDATE, DELETE ON aisoc_outbound_webhooks TO aisoc_app';+ EXECUTE 'GRANT SELECT, INSERT, UPDATE, DELETE ON aisoc_outbound_deliveries TO aisoc_app';+ END IF;+END+$$;diff --git a/services/api/migrations/098_report_delivery.sql b/services/api/migrations/098_report_delivery.sqlnew file mode 100644index 000000000..5eb931498--- /dev/null+++ b/services/api/migrations/098_report_delivery.sql@@ -0,0 +1,33 @@+-- 098_report_delivery.sql — record whether a generated report reached anybody.+--+-- Depth plan 5.2. `report_artefacts.delivered_at` existed and nothing set+-- it, because nothing delivered. Now that the scheduler does, a timestamp+-- alone is not enough to read the row honestly: "no SMTP relay is+-- configured", "the template names no recipients" and "the relay rejected+-- the address" are three different facts, and all three leave+-- `delivered_at` NULL exactly as a successful delivery that has not+-- finished would.+--+-- So the outcome is a column with a closed vocabulary and the reason is+-- beside it. A console rendering "generated, not delivered: no SMTP relay+-- is configured" sends an operator to the right place; one rendering a+-- blank timestamp sends them nowhere.+--+-- `skipped` is deliberately distinct from `failed`: the first is a choice+-- the deployment made (or did not make) and the second is something going+-- wrong. Collapsing them would make an unconfigured relay look like an+-- outage every five minutes.+--+-- Reversible: DROP COLUMN IF EXISTS on both.++ALTER TABLE report_artefacts+ ADD COLUMN IF NOT EXISTS delivery_status TEXT NOT NULL DEFAULT 'not_attempted'+ CHECK (delivery_status IN ('not_attempted', 'sent', 'skipped', 'failed'));++ALTER TABLE report_artefacts+ ADD COLUMN IF NOT EXISTS delivery_detail TEXT NOT NULL DEFAULT '';++COMMENT ON COLUMN report_artefacts.delivery_status IS+ 'Whether the generated report reached anybody. `skipped` is a choice the deployment made '+ '(no relay, no recipients); `failed` is something going wrong. Collapsing the two would '+ 'make an unconfigured relay look like an outage on every pass.';diff --git a/services/api/tests/test_delivery_5_2.py b/services/api/tests/test_delivery_5_2.pynew file mode 100644index 000000000..c3917a9e6--- /dev/null+++ b/services/api/tests/test_delivery_5_2.py@@ -0,0 +1,437 @@+"""Depth 5.2 — the five delivery claims, each of which was false.++Reproduce, measured on the tree this started from:++1. `report_templates.cron_schedule` is a storable, editable field and+ **nothing in `services/api` reads it**, so a tenant could set a schedule+ in the console, see it saved, and never receive a report. Separately,+ `POST /reports/generate` wrote `status="pending"` and nothing ever moved+ it off pending.+2. **No SMTP anywhere in `services/api`** — `grep -rl smtplib` returned+ nothing — while `destinations.dispatch("email", ...)` answered+ `detail="formatted"` and sent nothing.+3. `send_approval_email` had **one reference in the whole tree, its own+ definition**. The verifier, the renderer, the signed URLs and the+ `/approvals/email/decide` route had all shipped, waiting for a link+ nothing ever sent.+4. `services/teams-bot` appeared in **no compose file**, had no Dockerfile,+ and had no route that could post a card.+5. `grep -rln outbound_webhook services/ apps/` returned nothing: AiSOC+ could be told things and could tell nobody anything.++What is asserted below is mostly the honest no-op. Every one of these+channels can be unconfigured, and the thing that matters is that an+unconfigured channel says so instead of reporting a delivery.+"""++from __future__ import annotations++import json+import uuid+from datetime import UTC, datetime, timedelta+from types import SimpleNamespace+from typing import Any++import httpx+import pytest+from app.services import approval_delivery, outbound_webhooks, smtp_delivery+from app.workers import report_scheduler++pytestmark = pytest.mark.asyncio+++class _Recorder:+ """A stand-in transport that records requests and replays answers.++ ``httpx.MockTransport`` rather than ``respx``: the actions and+ connectors services lock respx and this one does not, and adding a test+ dependency to the service with the largest dependency set to avoid+ twenty lines is the wrong trade. The transport sits *under* the real+ ``AsyncClient``, so request building, header assembly and the body the+ signature covers all run exactly as they do in production.+ """++ def __init__(self, *responses: httpx.Response | Exception) -> None:+ self._responses = list(responses)+ self.requests: list[httpx.Request] = []++ def __call__(self, request: httpx.Request) -> httpx.Response:+ self.requests.append(request)+ answer = self._responses[min(len(self.requests) - 1, len(self._responses) - 1)]+ if isinstance(answer, Exception):+ raise answer+ return httpx.Response(answer.status_code, content=answer.content, headers=answer.headers)++ @property+ def called(self) -> bool:+ return bool(self.requests)++ def install(self, monkeypatch: pytest.MonkeyPatch, module: Any) -> _Recorder:+ """Make every ``AsyncClient`` built by ``module`` use this transport."""+ real = httpx.AsyncClient++ def _factory(*args: Any, **kwargs: Any) -> httpx.AsyncClient:+ kwargs["transport"] = httpx.MockTransport(self)+ return real(*args, **kwargs)++ monkeypatch.setattr(module.httpx, "AsyncClient", _factory)+ return self+++def _ok(status_code: int = 200, payload: dict[str, Any] | None = None, text: str = "") -> httpx.Response:+ if payload is not None:+ return httpx.Response(status_code, json=payload)+ return httpx.Response(status_code, text=text)+++# ---------------------------------------------------------------------------+# 2. SMTP+# ---------------------------------------------------------------------------+++class TestSmtpDelivery:+ async def test_no_relay_is_skipped_and_says_which_setting(self) -> None:+ result = await smtp_delivery.send_mail(+ recipients=["[email protected]"],+ subject="x",+ text_body="y",+ settings=smtp_delivery.SmtpSettings(),+ )+ assert result.delivered is False+ assert result.status == "skipped"+ assert "SMTP_HOST" in result.detail++ async def test_no_recipients_is_skipped_not_sent(self) -> None:+ result = await smtp_delivery.send_mail(+ recipients=[],+ subject="x",+ text_body="y",+ settings=smtp_delivery.SmtpSettings(host="relay.example.com", sender="[email protected]"),+ )+ assert (result.delivered, result.status) == (False, "skipped")++ async def test_a_relay_failure_is_reported_not_swallowed(self, monkeypatch: pytest.MonkeyPatch) -> None:+ def _boom(*args: Any, **kwargs: Any) -> None:+ raise OSError("connection refused")++ monkeypatch.setattr(smtp_delivery, "_send_blocking", _boom)+ result = await smtp_delivery.send_mail(+ recipients=["[email protected]"],+ subject="x",+ text_body="y",+ settings=smtp_delivery.SmtpSettings(host="relay.example.com", sender="[email protected]"),+ )+ assert (result.delivered, result.status) == (False, "failed")+ assert "connection refused" in result.detail++ async def test_a_send_builds_a_message_with_both_parts_and_the_attachment(self, monkeypatch: pytest.MonkeyPatch) -> None:+ captured: list[Any] = []+ monkeypatch.setattr(smtp_delivery, "_send_blocking", lambda settings, message: captured.append(message))++ result = await smtp_delivery.send_mail(+ recipients="[email protected], [email protected]",+ subject="Weekly report",+ text_body="plain",+ html_body="<p>rich</p>",+ attachment=("report.html", b"<html></html>", "text/html"),+ settings=smtp_delivery.SmtpSettings(host="relay.example.com", sender="[email protected]"),+ )+ assert result.delivered is True+ message = captured[0]+ assert message["To"] == "[email protected], [email protected]"+ # A plain-text alternative is always present: an analyst on a+ # locked-down gateway still needs the decision link.+ assert {part.get_content_type() for part in message.walk()} >= {"text/plain", "text/html"}++ def test_the_smtp_mailer_matches_the_mail_delivery_protocol(self) -> None:+ """`email_approval.send_approval_email` calls it by that Protocol.++ A signature mismatch would be invisible until the first real+ approval, because nothing else constructs this class.+ """+ import inspect++ from app.services.email_approval import MailDeliveryClient++ expected = inspect.signature(MailDeliveryClient.send).parameters+ actual = inspect.signature(smtp_delivery.SmtpApprovalMailer.send).parameters+ assert set(expected) == set(actual)+++# ---------------------------------------------------------------------------+# 3 and 4. Approval delivery: email, Slack, Teams+# ---------------------------------------------------------------------------+++def _approval(**overrides: Any) -> SimpleNamespace:+ return SimpleNamespace(+ id=uuid.uuid4(),+ case_id=uuid.uuid4(),+ title="Isolate web-prod-04",+ summary="Credential access observed",+ risk_level="high",+ action={"action_type": "isolate_host", "target": "web-prod-04", **overrides.pop("action", {})},+ **overrides,+ )+++class TestApprovalDelivery:+ async def test_every_channel_is_skipped_with_its_setting_when_unconfigured(self, monkeypatch: pytest.MonkeyPatch) -> None:+ for name in (+ "AISOC_APPROVAL_TOKEN_SECRET",+ "SMTP_HOST",+ "SMTP_SENDER",+ "AISOC_SLACK_BOT_URL",+ "AISOC_TEAMS_BOT_URL",+ "AISOC_INTERNAL_TOKEN",+ ):+ monkeypatch.delenv(name, raising=False)++ report = await approval_delivery.deliver_approval(_approval())+ assert set(report.channels) == {"email", "slack", "teams"}+ assert report.delivered_to == []+ assert all(result["status"] == "skipped" for result in report.channels.values())+ assert "AISOC_SLACK_BOT_URL" in report.channels["slack"]["detail"]+ assert "AISOC_TEAMS_BOT_URL" in report.channels["teams"]["detail"]++ async def test_email_refuses_to_send_an_unsigned_decision_link(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """A one-click approve anybody could forge is worse than no email."""+ monkeypatch.delenv("AISOC_APPROVAL_TOKEN_SECRET", raising=False)+ monkeypatch.setenv("SMTP_HOST", "relay.example.com")+ monkeypatch.setenv("SMTP_SENDER", "[email protected]")++ report = await approval_delivery.deliver_approval(_approval(action={"approver_emails": ["[email protected]"]}))+ assert report.channels["email"]["delivered"] is False+ assert "AISOC_APPROVAL_TOKEN_SECRET" in report.channels["email"]["detail"]++ async def test_a_card_the_bot_did_not_post_is_not_a_delivery(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """The bots answer 202 with `posted: false` when no channel is set.++ Reading the status code alone — which is the obvious+ implementation — would record that as delivered.+ """+ monkeypatch.setenv("AISOC_SLACK_BOT_URL", "http://slack-bot:8089")+ monkeypatch.setenv("AISOC_INTERNAL_TOKEN", "tok")+ monkeypatch.delenv("AISOC_TEAMS_BOT_URL", raising=False)+ _Recorder(_ok(202, {"posted": False, "reason": "no approvals channel configured"})).install(monkeypatch, approval_delivery)++ report = await approval_delivery.deliver_approval(_approval())+ assert report.channels["slack"]["delivered"] is False+ assert "no approvals channel" in report.channels["slack"]["detail"]++ async def test_a_posted_card_is_a_delivery_and_carries_the_internal_token(self, monkeypatch: pytest.MonkeyPatch) -> None:+ monkeypatch.setenv("AISOC_TEAMS_BOT_URL", "http://teams-bot:8090")+ monkeypatch.setenv("AISOC_INTERNAL_TOKEN", "tok")+ monkeypatch.delenv("AISOC_SLACK_BOT_URL", raising=False)+ recorder = _Recorder(_ok(202, {"posted": True})).install(monkeypatch, approval_delivery)++ report = await approval_delivery.deliver_approval(_approval())+ assert report.channels["teams"]["delivered"] is True+ assert report.delivered_to == ["teams"]+ assert recorder.requests[0].headers["X-AiSOC-Internal-Token"] == "tok"++ async def test_a_bot_that_is_down_does_not_take_the_approval_with_it(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """The row is already durable; a channel outage must leave a record."""+ monkeypatch.setenv("AISOC_SLACK_BOT_URL", "http://slack-bot:8089")+ monkeypatch.setenv("AISOC_INTERNAL_TOKEN", "tok")+ _Recorder(httpx.ConnectError("refused")).install(monkeypatch, approval_delivery)++ report = await approval_delivery.deliver_approval(_approval())+ assert report.channels["slack"]["status"] == "failed"++ async def test_a_missing_internal_token_is_named_rather_than_round_tripped(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """Both bots refuse an unset token, so posting is a guaranteed 401."""+ monkeypatch.setenv("AISOC_SLACK_BOT_URL", "http://slack-bot:8089")+ monkeypatch.delenv("AISOC_INTERNAL_TOKEN", raising=False)+ report = await approval_delivery.deliver_approval(_approval())+ assert "AISOC_INTERNAL_TOKEN" in report.channels["slack"]["detail"]+++# ---------------------------------------------------------------------------+# 5. Outbound event webhooks+# ---------------------------------------------------------------------------+++class TestWebhookSignature:+ def test_a_signature_round_trips(self) -> None:+ body = b'{"a":1}'+ header = outbound_webhooks.sign(body, secret="s3cret", timestamp=1_700_000_000)+ assert outbound_webhooks.verify(body, header, secret="s3cret", now=1_700_000_000)++ def test_a_modified_body_fails(self) -> None:+ header = outbound_webhooks.sign(b'{"a":1}', secret="s3cret", timestamp=1_700_000_000)+ assert not outbound_webhooks.verify(b'{"a":2}', header, secret="s3cret", now=1_700_000_000)++ def test_a_different_secret_fails(self) -> None:+ header = outbound_webhooks.sign(b"x", secret="s3cret", timestamp=1_700_000_000)+ assert not outbound_webhooks.verify(b"x", header, secret="other", now=1_700_000_000)++ def test_a_replayed_body_outside_the_window_fails(self) -> None:+ """The timestamp is inside the signed material, so an attacker+ cannot re-stamp a captured body."""+ header = outbound_webhooks.sign(b"x", secret="s3cret", timestamp=1_700_000_000)+ assert not outbound_webhooks.verify(b"x", header, secret="s3cret", now=1_700_000_000 + 3600)+ forged = header.replace("t=1700000000", "t=1700003600")+ assert not outbound_webhooks.verify(b"x", forged, secret="s3cret", now=1_700_000_000 + 3600)++ @pytest.mark.parametrize("header", ["", "garbage", "t=abc,v1=xx", "v1=xx", "t=1700000000"])+ def test_a_malformed_header_fails_rather_than_raising(self, header: str) -> None:+ assert not outbound_webhooks.verify(b"x", header, secret="s", now=1_700_000_000)+++def _delivery(**overrides: Any) -> Any:+ from app.models.outbound_webhook import OutboundDelivery++ defaults = {+ "id": uuid.uuid4(),+ "tenant_id": uuid.uuid4(),+ "webhook_id": uuid.uuid4(),+ "event_type": "alert.created",+ "payload": {"alert_id": "a-1"},+ "status": "pending",+ "attempts": 0,+ "created_at": datetime.now(UTC),+ "last_error": "",+ }+ return OutboundDelivery(**{**defaults, **overrides})+++def _webhook(url: str = "https://receiver.example.com/hook") -> Any:+ from app.models.outbound_webhook import OutboundWebhook++ return OutboundWebhook(id=uuid.uuid4(), tenant_id=uuid.uuid4(), name="dest", url=url, event_types=[], secret="", enabled=True)+++class TestWebhookDelivery:+ async def test_a_2xx_is_delivered_and_stops_retrying(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _Recorder(_ok(200)).install(monkeypatch, outbound_webhooks)+ delivery = await outbound_webhooks.attempt_delivery(_delivery(), _webhook())+ assert delivery.status == "delivered"+ assert delivery.next_attempt_at is None+ assert delivery.delivered_at is not None++ async def test_a_5xx_is_retried_with_a_backoff(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _Recorder(_ok(503)).install(monkeypatch, outbound_webhooks)+ delivery = await outbound_webhooks.attempt_delivery(_delivery(), _webhook())+ assert delivery.status == "pending"+ assert delivery.next_attempt_at is not None+ assert delivery.next_attempt_at > datetime.now(UTC)++ async def test_a_4xx_is_dead_immediately(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """The receiver understood and refused. Six identical refusals+ would be six identical answers, and the operator needs the first."""+ _Recorder(_ok(410, text="gone")).install(monkeypatch, outbound_webhooks)+ delivery = await outbound_webhooks.attempt_delivery(_delivery(), _webhook())+ assert delivery.status == "dead"+ assert delivery.attempts == 1+ assert "410" in delivery.last_error++ @pytest.mark.parametrize("code", [408, 425, 429])+ async def test_the_three_retryable_client_statuses_are_retried(self, code: int, monkeypatch: pytest.MonkeyPatch) -> None:+ _Recorder(_ok(code)).install(monkeypatch, outbound_webhooks)+ delivery = await outbound_webhooks.attempt_delivery(_delivery(), _webhook())+ assert delivery.status == "pending"++ async def test_the_ladder_ends_in_the_dead_letter_view(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _Recorder(_ok(503)).install(monkeypatch, outbound_webhooks)+ delivery = _delivery(attempts=outbound_webhooks.MAX_ATTEMPTS - 1)+ await outbound_webhooks.attempt_delivery(delivery, _webhook())+ assert delivery.attempts == outbound_webhooks.MAX_ATTEMPTS+ assert delivery.status == "dead"+ assert delivery.next_attempt_at is None++ async def test_a_connection_failure_is_retried_not_dead(self, monkeypatch: pytest.MonkeyPatch) -> None:+ _Recorder(httpx.ConnectError("refused")).install(monkeypatch, outbound_webhooks)+ delivery = await outbound_webhooks.attempt_delivery(_delivery(), _webhook())+ assert delivery.status == "pending"+ assert "could not be reached" in delivery.last_error++ async def test_the_body_is_byte_identical_across_attempts(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """The signature covers the body. A dict that serialised in a+ different order would be a differently-signed message claiming to+ be the same event."""+ recorder = _Recorder(_ok(503)).install(monkeypatch, outbound_webhooks)+ delivery = _delivery(payload={"b": 2, "a": 1, "c": {"z": 1, "y": 2}})+ webhook = _webhook()+ await outbound_webhooks.attempt_delivery(delivery, webhook)+ await outbound_webhooks.attempt_delivery(delivery, webhook)+ assert recorder.requests[0].read() == recorder.requests[1].read()+ envelope = json.loads(recorder.requests[0].read())+ assert envelope["id"] == str(delivery.id)+ assert envelope["type"] == "alert.created"++ async def test_a_signed_delivery_verifies_at_the_receiver(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """End to end over the wire: sign here, verify as a receiver would."""+ monkeypatch.setattr(outbound_webhooks, "_secret_for", lambda webhook: "shared")+ recorder = _Recorder(_ok(200)).install(monkeypatch, outbound_webhooks)+ await outbound_webhooks.attempt_delivery(_delivery(), _webhook())+ request = recorder.requests[0]+ assert outbound_webhooks.verify(request.read(), request.headers["X-AiSOC-Signature"], secret="shared")++ async def test_an_undecryptable_secret_is_dead_not_sent_unsigned(self, monkeypatch: pytest.MonkeyPatch) -> None:+ """Sending unsigned because a key would not decrypt is how a+ receiver that checks signatures starts rejecting everything."""++ def _boom(webhook: Any) -> str:+ raise ValueError("key rotated out")++ monkeypatch.setattr(outbound_webhooks, "_secret_for", _boom)+ recorder = _Recorder(_ok(200)).install(monkeypatch, outbound_webhooks)+ delivery = await outbound_webhooks.attempt_delivery(_delivery(), _webhook())+ assert delivery.status == "dead"+ assert not recorder.called+++# ---------------------------------------------------------------------------+# 1. The report scheduler+# ---------------------------------------------------------------------------+++class TestCronMatching:+ @pytest.mark.parametrize(+ ("expression", "moment", "expected"),+ [+ ("* * * * *", datetime(2026, 10, 7, 9, 0, tzinfo=UTC), True),+ ("0 9 * * *", datetime(2026, 10, 7, 9, 0, tzinfo=UTC), True),+ ("0 9 * * *", datetime(2026, 10, 7, 9, 1, tzinfo=UTC), False),+ ("*/15 * * * *", datetime(2026, 10, 7, 9, 30, tzinfo=UTC), True),+ ("*/15 * * * *", datetime(2026, 10, 7, 9, 31, tzinfo=UTC), False),+ # 2026-10-07 is a Wednesday; cron Sunday=0, so Wednesday is 3.+ ("0 9 * * 3", datetime(2026, 10, 7, 9, 0, tzinfo=UTC), True),+ ("0 9 * * 1", datetime(2026, 10, 7, 9, 0, tzinfo=UTC), False),+ ("0 9 * * 1-5", datetime(2026, 10, 7, 9, 0, tzinfo=UTC), True),+ ("0 9,17 * * *", datetime(2026, 10, 7, 17, 0, tzinfo=UTC), True),+ ],+ )+ def test_matches(self, expression: str, moment: datetime, expected: bool) -> None:+ assert report_scheduler.cron_is_due(expression, moment) is expected++ @pytest.mark.parametrize("expression", ["", "* * * *", "@weekly", "x * * * *", "0 99 * * *", "*/0 * * * *"])+ def test_an_unparseable_expression_raises_rather_than_matching_everything(self, expression: str) -> None:+ """Treating what it cannot parse as "every minute" is how one bad+ field becomes 168 emails."""+ with pytest.raises(ValueError):+ report_scheduler.cron_is_due(expression, datetime(2026, 10, 7, 9, 0, tzinfo=UTC))++ def test_a_schedule_between_polls_is_not_missed(self) -> None:+ """The poll is five minutes and cron granularity is one.++ Asking "does the current minute match" — the obvious+ implementation — would check 08:57 and 09:02 and never 09:00, so a+ weekly report would simply never go out and nothing would say so.+ """+ now = datetime(2026, 10, 7, 9, 2, tzinfo=UTC)+ assert report_scheduler.cron_is_due("0 9 * * *", now) is False+ assert report_scheduler.due_since("0 9 * * *", since=now - timedelta(minutes=5), now=now) is True++ def test_an_already_run_schedule_does_not_run_twice(self) -> None:+ now = datetime(2026, 10, 7, 9, 2, tzinfo=UTC)+ assert report_scheduler.due_since("0 9 * * *", since=datetime(2026, 10, 7, 9, 1, tzinfo=UTC), now=now) is False++ def test_a_long_outage_collapses_into_one_report(self) -> None:+ """Seven daily emails on restart is its own incident."""+ now = datetime(2026, 10, 7, 9, 2, tzinfo=UTC)+ assert report_scheduler.due_since("0 9 * * *", since=now - timedelta(days=7), now=now) is True+ # The catch-up window is bounded, so the pass that follows this one+ # cannot find six more occurrences to fire.+ assert report_scheduler._MAX_CATCHUP == timedelta(hours=24)diff --git a/services/teams-bot/Dockerfile b/services/teams-bot/Dockerfilenew file mode 100644index 000000000..adce190e7--- /dev/null+++ b/services/teams-bot/Dockerfile@@ -0,0 +1,29 @@+FROM python:3.11-slim++WORKDIR /app++RUN pip install --no-cache-dir poetry==2.4.1 && \+ poetry config virtualenvs.create false++COPY pyproject.toml poetry.lock ./++# Dependencies come from poetry.lock and nowhere else, so two builds of this+# commit install byte-identical versions. `poetry install` refuses a lock that+# no longer matches pyproject.toml ("changed significantly since poetry.lock+# was last generated"), so editing the manifest without re-locking is a build+# error rather than a silent re-resolution nobody reviewed. The matching+# `poetry check --lock` runs in CI, where the README the check wants is on+# disk; this layer has only the two files it installs from.+#+# There is deliberately no pip fallback here. The one this replaced carried its+# own copy of the dependency list, drifted from pyproject.toml, and pinned+# versions no test ever ran. When a transient poetry failure triggered it on+# 2026-09-24 the api image shipped FastAPI 0.111.1 and died at import. A build+# that fails is recoverable; an image that boots on untested versions is not.+RUN poetry install --no-interaction --no-ansi --only main --no-root++COPY app/ ./app/++EXPOSE 8090++CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8090"]diff --git a/services/teams-bot/app/main.py b/services/teams-bot/app/main.pyindex 6b01af0a6..e5da9981c 100644--- a/services/teams-bot/app/main.py+++ b/services/teams-bot/app/main.py@@ -29,6 +29,7 @@ from fastapi import FastAPI, Requestfrom app._health import install_health_routesfrom app.callbacks import callback_max_age_seconds, handle_card_action+from app.notify import router as notify_routerapp = FastAPI(title="AiSOC Teams Bot",@@ -37,6 +38,8 @@ app = FastAPI()# Phase 2.6 — k8s liveness + readiness probes (see app/_health.py).+app.include_router(notify_router)+_mark_ready, _mark_not_ready = install_health_routes(app, service_name="aisoc-teams-bot")app.state.mark_ready = _mark_readyapp.state.mark_not_ready = _mark_not_readydiff --git a/services/teams-bot/app/notify.py b/services/teams-bot/app/notify.pynew file mode 100644index 000000000..9837ff868--- /dev/null+++ b/services/teams-bot/app/notify.py@@ -0,0 +1,138 @@+"""Let the Teams bot start a conversation.++Exactly the gap ``services/slack-bot/app/notify.py`` closed, one service+over, and for the same reason: every route here was inbound. Teams posts an+``invoke`` activity, the bot verifies the signed card payload and answers —+so the bot could *answer* a question and could not *ask* one. The card+builder (``cards.approval_card``) and the signed callback handler were both+written and both reachable only from a conversation a human had already+started.++So the documented "an agent stops and asks Teams for approval" flow could+only ever begin with a person typing first. This is one route and one+outbound call to the Bot Framework.++Why an incoming webhook and not the Bot Framework proactive API+----------------------------------------------------------------+Posting proactively through the Bot Framework needs a stored conversation+reference per channel, a Microsoft App ID and a client secret this service+has never held — the module docstring in ``main.py`` is explicit that the+outer Bot Framework auth is terminated by the fronting proxy and that the+bot "never minted or holds the Microsoft secret". A Workflows incoming+webhook needs none of that: the URL is the credential, an operator creates+it on the channel they want, and the card payload is the same Adaptive Card+the callback handler already signs.++Authentication is the same shared internal token the Slack bot uses, for+the same reason: the caller is an AiSOC service, not Teams, so there is no+Teams signature to verify and a bearer token the operator generates is what+``scripts/ensure_env.py`` already provisions.+"""++from __future__ import annotations++import hmac+import os+from typing import Any++import httpx+import structlog+from fastapi import APIRouter, Header, HTTPException, status+from pydantic import BaseModel, Field++from app.cards import approval_card++logger = structlog.get_logger(__name__)++router = APIRouter(prefix="/internal", tags=["internal"])++_TIMEOUT_SECONDS = 10.0+++class ApprovalCardRequest(BaseModel):+ """An approval an agent wants a human in Teams to decide."""++ action: dict[str, Any] = Field(..., description="The action record: id, action_type, target, risk_level, rationale.")+ case: dict[str, Any] = Field(default_factory=dict, description="Case context rendered into the card.")+ webhook_url: str | None = Field(+ default=None,+ description="Override the configured channel webhook. Useful for per-tenant routing.",+ )+ requested_by: str = Field(+ default="",+ description=(+ "Who asked, when somebody did. An agent-raised approval has nobody, and the card says so rather than "+ "attributing it to whoever happens to be on call."+ ),+ )+ timeout_seconds: int | None = None+++def _authorized(supplied: str | None) -> bool:+ """Constant-time compare, failing closed on an unset token.++ No dev-mode exemption, matching the Slack bot: that exemption used to+ exist there and it was the state a stock install ran in, so a route+ that posts into a workspace channel was open to anything that could+ reach the pod.+ """+ expected = os.environ.get("AISOC_INTERNAL_TOKEN", "").strip()+ if not expected:+ return False+ return bool(supplied) and hmac.compare_digest(supplied.strip(), expected)++[email protected]("/approval-card", status_code=status.HTTP_202_ACCEPTED)+async def post_approval_card(+ body: ApprovalCardRequest,+ x_internal_token: str | None = Header(default=None, alias="X-AiSOC-Internal-Token"),+) -> dict[str, Any]:+ """Post an approval card into the configured Teams channel.++ Returns 202 with ``posted: false`` rather than failing when no webhook+ is configured. The approval is already durable in Postgres by the time+ this is called and the console and responder app can both act on it;+ Teams is one delivery route, not the record. The caller reads ``posted``+ rather than the status code, so a skipped post is not recorded as a+ delivery.+ """+ if not _authorized(x_internal_token):+ raise HTTPException(status_code=status.HTTP_401_UNAUTHORIZED, detail="internal token required")++ webhook_url = (body.webhook_url or os.environ.get("TEAMS_APPROVALS_WEBHOOK_URL", "")).strip()+ if not webhook_url:+ logger.info("teams_bot.approval_card_skipped", reason="TEAMS_APPROVALS_WEBHOOK_URL is unset", action_id=body.action.get("id"))+ return {"posted": False, "reason": "no approvals webhook configured"}++ signing_secret = os.environ.get("AISOC_TEAMS_CALLBACK_SECRET", "").strip()+ if not signing_secret:+ # Refused rather than posted unsigned. The buttons carry an+ # HMAC-signed payload that `handle_card_action` verifies, so a card+ # signed with an empty secret is one whose Approve button the+ # callback handler will reject — a card that looks actionable and+ # is not is worse than no card.+ logger.warning("teams_bot.approval_card_unsigned", action_id=body.action.get("id"))+ return {"posted": False, "reason": "AISOC_TEAMS_CALLBACK_SECRET is unset, so the card's buttons could not be signed"}++ card = approval_card(+ action=body.action,+ case=body.case,+ requested_by=body.requested_by,+ web_base=os.environ.get("AISOC_WEB_BASE_URL", ""),+ signing_secret=signing_secret,+ timeout_seconds=body.timeout_seconds,+ )++ try:+ async with httpx.AsyncClient(timeout=_TIMEOUT_SECONDS) as client:+ response = await client.post(webhook_url, json=card)+ except Exception as exc: # noqa: BLE001 — Teams being down must not fail the agent+ logger.warning("teams_bot.approval_card_failed", action_id=body.action.get("id"), error=str(exc))+ return {"posted": False, "reason": f"Teams rejected the message ({type(exc).__name__})"}++ if not 200 <= response.status_code < 300:+ logger.warning("teams_bot.approval_card_rejected", action_id=body.action.get("id"), status_code=response.status_code)+ return {"posted": False, "reason": f"Teams returned HTTP {response.status_code}"}++ logger.info("teams_bot.approval_card_posted", action_id=body.action.get("id"))+ return {"posted": True}diff --git a/services/teams-bot/tests/test_notify.py b/services/teams-bot/tests/test_notify.pynew file mode 100644index 000000000..9119b534b--- /dev/null+++ b/services/teams-bot/tests/test_notify.py@@ -0,0 +1,111 @@+"""The route that lets this bot start a conversation.++Reproduce: before this, every route here was inbound. Teams posted an+``invoke`` activity, the bot verified the signed card payload and answered —+so it could answer a question and could not ask one, and the documented+"an agent stops and asks Teams for approval" flow could only begin with a+person typing first. The card builder and the signed callback handler were+both written and both reachable only from a conversation a human started.+"""++from __future__ import annotations++import httpx+import pytest+from app import notify+from app.main import app+from fastapi.testclient import TestClient+++def client(monkeypatch: pytest.MonkeyPatch) -> TestClient:+ monkeypatch.setenv("AISOC_INTERNAL_TOKEN", "tok")+ monkeypatch.setenv("AISOC_TEAMS_CALLBACK_SECRET", "signing-secret")+ monkeypatch.setenv("AISOC_WEB_BASE_URL", "https://console.example.com")+ return TestClient(app)+++_ACTION = {"id": "act-1", "action_type": "isolate_host", "target": "web-prod-04", "rationale": "credential access"}+++def test_the_route_exists_and_refuses_an_unauthenticated_caller(client: TestClient) -> None:+ assert client.post("/internal/approval-card", json={"action": _ACTION}).status_code == 401+++def test_an_unset_internal_token_refuses_everything(monkeypatch: pytest.MonkeyPatch) -> None:+ """Failing closed, with no dev-mode exemption.++ That exemption existed on the Slack bot and it was the state a stock+ install ran in, so a route that posts into a workspace channel was open+ to anything that could reach the pod.+ """+ monkeypatch.delenv("AISOC_INTERNAL_TOKEN", raising=False)+ response = TestClient(app).post("/internal/approval-card", json={"action": _ACTION}, headers={"X-AiSOC-Internal-Token": "anything"})+ assert response.status_code == 401+++def test_no_configured_webhook_is_reported_not_failed(client: TestClient, monkeypatch: pytest.MonkeyPatch) -> None:+ """202 with ``posted: false``.++ The approval is already durable in Postgres and the console and the+ responder app can both act on it; Teams is one delivery route, not the+ record. The caller reads ``posted`` rather than the status code, which+ is what stops a skipped post being recorded as a delivery.+ """+ monkeypatch.delenv("TEAMS_APPROVALS_WEBHOOK_URL", raising=False)+ response = client.post("/internal/approval-card", json={"action": _ACTION}, headers={"X-AiSOC-Internal-Token": "tok"})+ assert response.status_code == 202+ assert response.json() == {"posted": False, "reason": "no approvals webhook configured"}+++def test_an_unsigned_card_is_refused(client: TestClient, monkeypatch: pytest.MonkeyPatch) -> None:+ """A card whose Approve button the callback handler will reject looks+ actionable and is not, which is worse than no card."""+ monkeypatch.setenv("AISOC_TEAMS_CALLBACK_SECRET", "")+ response = client.post(+ "/internal/approval-card",+ json={"action": _ACTION, "webhook_url": "https://teams.example/hook"},+ headers={"X-AiSOC-Internal-Token": "tok"},+ )+ assert response.json()["posted"] is False+ assert "AISOC_TEAMS_CALLBACK_SECRET" in response.json()["reason"]+++def test_a_posted_card_is_an_adaptive_card_envelope(client: TestClient, monkeypatch: pytest.MonkeyPatch) -> None:+ sent: list[httpx.Request] = []++ def _transport(request: httpx.Request) -> httpx.Response:+ sent.append(request)+ return httpx.Response(202)++ real = httpx.AsyncClient+ monkeypatch.setattr(+ notify.httpx,+ "AsyncClient",+ lambda *args, **kwargs: real(*args, **{**kwargs, "transport": httpx.MockTransport(_transport)}),+ )+ response = client.post(+ "/internal/approval-card",+ json={"action": _ACTION, "case": {"id": "c-1", "severity": "high"}, "webhook_url": "https://teams.example/hook"},+ headers={"X-AiSOC-Internal-Token": "tok"},+ )+ assert response.json() == {"posted": True}+ body = sent[0].read().decode()+ assert "AdaptiveCard" in body+ assert "web-prod-04" in body+++def test_a_rejected_post_is_not_reported_as_delivered(client: TestClient, monkeypatch: pytest.MonkeyPatch) -> None:+ real = httpx.AsyncClient+ monkeypatch.setattr(+ notify.httpx,+ "AsyncClient",+ lambda *args, **kwargs: real(*args, **{**kwargs, "transport": httpx.MockTransport(lambda r: httpx.Response(400, text="bad"))}),+ )+ response = client.post(+ "/internal/approval-card",+ json={"action": _ACTION, "webhook_url": "https://teams.example/hook"},+ headers={"X-AiSOC-Internal-Token": "tok"},+ )+ assert response.json()["posted"] is False+ assert "400" in response.json()["reason"]