Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
Make a dead consumer say so, and make the four alembic chains run (#831)
Beenu Arora1a9c0851b89a
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.ymlindex a210c2679..aa35c6e91 100644--- a/.github/workflows/ci.yml+++ b/.github/workflows/ci.yml@@ -556,6 +556,32 @@ jobs:- name: Verify FastAPI services install /livez + /readyz probes (Phase 2.6)run: python scripts/audit_health_probes.py --check+ # A column a model declares and no migration creates is not a latent+ # problem: SQLAlchemy names every mapped column in its SELECT, so the+ # first query against that table fails. `ueba_entity_baselines`+ # declared `peer_group_id`, migration 0001 never created it, and UEBA+ # could not write a baseline or an anomaly on any deployment while its+ # container reported healthy throughout. Gated for the four services+ # that own their schema through alembic and therefore have no+ # `create_all` fallback; `--all` reports the rest as advisory.+ #+ # The self-test runs first, and deliberately: it injects one of each+ # kind of drift into a scratch tree, so a parser that has quietly+ # stopped matching cannot report the repository clean.+ - name: Verify ORM columns exist in migrations (both directions)+ run: |+ python3 scripts/check_orm_migration_parity.py --self-test+ python3 scripts/check_orm_migration_parity.py --check++ # The four alembic chains have to be invoked by something. They were+ # not: every container command was a plain `uvicorn`, so the+ # documented quickstart booted two services with no tables and no RLS+ # policies while reporting success at every step.+ - name: Verify the alembic services run their chain on the way up+ run: |+ python3 scripts/check_service_migration_bootstrap.py --self-test+ python3 scripts/check_service_migration_bootstrap.py --check+# Phase 4.2 — every managed-mode tenant manifest under# `infra/fly/managed/tenants/*.yaml` must parse, carry the# required fields (slug, display_name, base_domain, contact_email,diff --git a/CHANGELOG.md b/CHANGELOG.mdindex f64fd2ed7..68e58a685 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -530,6 +530,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0the view holds no graph at **first paint**, so a future `fallbackData` (whichdisables revalidation, making a mock permanent rather than provisional)fails the build.+- **`scripts/check_orm_migration_parity.py` — a column a model declares and+ no migration creates now fails the build.** Structural on both sides+ (Python `ast` over the models and the alembic revisions, plus the raw SQL+ those revisions and `services/api/migrations/*.sql` execute) and in both+ directions: `missing-in-migration`, `missing-in-model`,+ `type-narrower-in-migration`, `type-family-mismatch`, and+ `no-migration-source` for a mapped table no migration mentions at all —+ which is the finding that exists because pairing nothing produces zero+ findings, and zero findings over zero comparisons prints the same word as+ a clean result. `--credits` enumerates what each verdict was *based on*+ for the same reason.++ Scoped structurally rather than by a list: a service that calls+ `metadata.create_all` builds its tables from the models, so a column no+ migration creates is created anyway. `services/api` is the one such+ service and is reported as advisory with its count printed, not hidden.+ The four alembic-managed services have no such fallback, which is exactly+ why UEBA's missing column was fatal.++ Verified non-vacuous against the tree as it was: it reports all three of+ the real UEBA defects. The `type-family-mismatch` kind exists *because*+ an earlier version reported only two of them — a length comparison could+ not see `UUID` against `String(64)`, since neither side declared a length+ the other contradicted.++- **`scripts/check_service_migration_bootstrap.py` — a migration chain+ nothing invokes is not a migration chain.** Asserts every service with an+ `alembic.ini` ships `app/_migrate.py`, keeps it byte-identical to the+ others, runs it from its `CMD`/`ENTRYPOINT` (matched on the command lines,+ so a mention in a comment does not count), and carries both an owner DSN+ and a compose healthcheck. The other direction too: a service shipping+ the module with no chain to apply would exit 1 on every start.++- **`scripts/audit_health_probes.py` now checks the thirteen copies of+ `app/_health.py` are identical.** The module's own docstring claimed they+ were kept in sync and nothing verified it — they happened to be, so the+ claim was true by luck. It also pointed at `scripts/sync_health_module.py`,+ which does not exist in this tree.- **Every test file in the tree is now executed by a workflow, and a gateholds it that way in both directions.** 63 were executed by nothing at all:@@ -2040,6 +2078,115 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0before the gate in CI and injects nine defects — a dropped connector, arenamed one, a class that never registers, a deleted artefact, an unhookedconsumer, an empty corpus — requiring each to be caught.+- **UEBA could never write a baseline or an anomaly, and said it was+ healthy the whole time.** Three separate defects on one code path, each+ reached by the first scoreable event rather than by some rare branch:++ - `EntityBaseline.peer_group_id` was declared by the model and created by+ no migration. SQLAlchemy names every mapped column in its `SELECT`, so+ the first database call `score_event` makes raised+ `UndefinedColumnError`.+ - `ueba_peer_groups.id` was created `UUID DEFAULT gen_random_uuid()` and+ declared `String(64)`, while `PeerGroupService` writes group names like+ `dept:engineering` into it — `InvalidTextRepresentation` on insert. The+ column follows the model, because the model matches the data.+ - `ueba_anomalies.event_type` was 64 characters wide against a model+ declaring 128, so an event type between the two lengths passed every+ test and failed at insert time.++ Fixed in `services/ueba/alembic/versions/0004_baseline_peer_group_and_column_parity.py`.++- **And with the schema right, the baseline still could not accumulate.**+ `_welford_update` edited the per-feature dictionary in place. Its caller+ passes a *shallow* copy of `EntityBaseline.feature_stats`, so the object+ being edited was the one SQLAlchemy had loaded from the column; by the+ time the caller assigned the result back, old and new compared equal and+ SQLAlchemy left the column out of the `UPDATE` — a `JSON`/`JSONB` column+ has no change tracking unless wrapped in `MutableDict`. Measured on a+ live stack: 36 events for one entity, every one processed without error,+ and the stored baseline frozen at `count: 1` while `window_end` advanced+ on every one. No entity could reach `min_baseline_samples` (30), so+ `compute_z_score` returned `None` forever and no anomaly was ever scored.+ The service logged `features_unscoreable` 36 times, which reads as "too+ quiet to score" rather than as a bug. The updater now returns new+ dictionaries and mutates nothing.++- **A UEBA handler exception killed the subscription silently.** The+ consume loop had no `except` at all: the first event that raised left the+ `async for`, the `finally` stopped the consumer, and the exception went+ into a task held in a module global — so it was never+ garbage-collected, asyncio never emitted "Task exception was never+ retrieved", and nothing was logged. The container stayed `running` with+ restarts 0 and `/health` answered a hardcoded 200, permanently.++ The policy is now the one `services/fusion` and `services/agents` already+ follow: **log loudly, dead-letter, and keep the subscription.** A single+ unprocessable event must not stop the stream. Refused events go to+ `aisoc_dead_letters` — the table fusion already writes and+ `GET /api/v1/health/dead-letters` already reads, with its per-reason+ breakdown — rather than to a second dead-letter path nobody would know to+ check.++- **`/readyz` reported that a consumer task had been *created*, not that it+ was running.** Three services created theirs as a fire-and-forget+ `asyncio.create_task` and called `mark_ready()` on the next line+ (`ueba`, `fusion`, `agents`). `app/_health.py` gains+ `register_subscription(app, name, probe)`; `/readyz` evaluates every+ registered probe per request and answers 503 naming the detached ones.+ `/livez` deliberately still does not consult them — wiring a detached+ consumer into liveness makes an orchestrator restart a pod over a fault a+ restart cannot fix. Each of the three also gained a done-callback that+ retrieves and logs its task's outcome, and UEBA's `/health` now carries+ `attached`, the processed / failed / dead-lettered counters and the last+ error instead of a hardcoded `{"status": "ok"}`.++- **The same audit across every other consumer.** `services/fusion`'s+ `_consume_loop` guarded `_process_message` but left+ `lake.flush_if_stale()` outside the guard, so a ClickHouse outage raising+ there would have ended the loop with the archive fault reported as+ nothing; it is now guarded and the alert path survives it.+ `services/agents`' `_process_with_retry` was already total, and the loop+ now guards it anyway rather than depending on a callee staying that way.+ `services/realtime` logged "Kafka consumer failed to start (will retry)"+ and never retried — kafkajs reports a dead consumer through its `CRASH`+ event, not by rejecting the promise, so a consumer that died after a+ successful start stayed recorded as attached for the life of the process.+ Both of its consumers now run under a supervisor with bounded backoff,+ and `/health` answers 503 listing any detached topic instead of a+ hardcoded `status: 'healthy'`.++- **Nothing invoked the four alembic chains.** `honeytokens`,+ `osquery-tls`, `purple-team` and `ueba` own their schemas through+ alembic; every container command was a plain `uvicorn`, so the documented+ quickstart booted them with empty schemas and none of their+ row-level-security policies. Each image now runs `python -m app._migrate`+ before its server: it applies the chain as the owner credential, takes a+ shared transaction-scoped advisory lock so four chains starting at once+ queue instead of deadlocking, reads the applied revision back, and+ refuses to start the server unless it matches head. An exit code is not+ evidence — `alembic upgrade head` exits 0 when it applies nothing, which+ is exactly what happened when the chains shared one version table. All+ four also gained a compose healthcheck; they had none, which is why a+ container that had done nothing still read as `running`.++- **`docker compose down && docker compose up` could not bring Kafka back.**+ `kafka_data` is a named volume and ZooKeeper had none, so the second `up`+ on kept data generated a fresh cluster id and the broker refused it with+ `InconsistentClusterIdException`. `make down && make up` — documented as+ "stop the stack, keep the data" — therefore left every Kafka-dependent+ service in `created`, and only `make clean`, which deletes the data,+ recovered. ZooKeeper now persists `/var/lib/zookeeper/data` and `/log`.++- **`osquery-tls` published a container port nothing listened on.** Compose+ mapped `127.0.0.1:8091:8007` while the service runs uvicorn on 9001. With+ no healthcheck either, the container read as `running` against a closed+ socket. Found by adding the healthcheck and watching it never leave+ `starting`.++- **`docker-compose.override.yml` was committed.** Its own header reads+ "untracked, never committed": it is a QA port remap that moved PostgreSQL+ to 55432 and ClickHouse to 58123/59000, contradicting every port the+ documentation quotes. Removed.- **Two more gates were exempt from the empty-corpus rule only by accident.**`check_route_auth.py` and `check_tenant_query_predicates.py` did not exit 0diff --git a/apps/docs/docs/deployment/env-vars.md b/apps/docs/docs/deployment/env-vars.mdindex 2083c54d7..1c1825526 100644--- a/apps/docs/docs/deployment/env-vars.md+++ b/apps/docs/docs/deployment/env-vars.md@@ -161,6 +161,41 @@ Two things changed alongside, both of which an operator can observe:deployment is adopted automatically on the next `alembic upgrade`: therecorded version is copied into the per-chain table when, and only when,that chain's own tables are already present.++##### Who runs the chain++Nothing did. All four container commands were a plain `uvicorn`, so on the+documented quickstart these services booted against empty schemas with none+of their policies installed, and every command in that sequence reported+success. The only documented invocation was a manual step, in a page whose+"start the stack" step did not start three of the four services.++Each image now runs `python -m app._migrate` before its server. That module:++- resolves the owner DSN by the precedence in the table above — it runs+ `alembic` as a subprocess so `env.py` stays the single answer to *which+ credential*, rather than a second copy of the rule here;+- takes a shared transaction-scoped advisory lock inside the upgrade+ (`MIGRATION_LOCK_KEY` in each `env.py`), so four chains starting at once+ against one database queue instead of deadlocking on catalog locks;+- reads the applied revision back and compares it to head. `alembic upgrade+ head` exits 0 when it applies **nothing**, which is exactly what happened+ when the chains shared a version table — so an exit code is not evidence,+ and the module refuses to start the server unless the version table+ actually holds head;+- provides no way to skip. A service that cannot reach head does not serve.++`AISOC_APP_DB_PASSWORD` is passed to all four for the same reason the API+gets it: the postgres init hook fires only on an empty data directory, so an+existing volume never reaches it and the runtime role would have no+credential to authenticate with.++Two CI gates keep this true:+`scripts/check_service_migration_bootstrap.py` asserts every service with an+`alembic.ini` ships the module, keeps it identical to the others, invokes it+from its `CMD`, and carries both an owner DSN and a healthcheck;+`scripts/check_orm_migration_parity.py` asserts no model declares a column+its migrations do not create.| `AISOC_APP_DB_PASSWORD` | `aisoc_app_dev_secret` in compose; unset elsewhere | Password applied to the runtime role, by the postgres init hook on a fresh volume and by the migration runner on every run. Unset leaves the role's credential alone. || `DATABASE_POOL_SIZE` | `20` | SQLAlchemy pool size || `DATABASE_MAX_OVERFLOW` | `10` | SQLAlchemy max overflow |diff --git a/apps/docs/docs/quickstart.md b/apps/docs/docs/quickstart.mdindex 9d6f6e4a8..803ad0048 100644--- a/apps/docs/docs/quickstart.md+++ b/apps/docs/docs/quickstart.md@@ -124,17 +124,34 @@ SECRET_KEY=change-me-in-production-at-least-32-chars### 2. Start the full stack+Most of the services below are behind a compose profile, so a bare+`docker compose up -d` starts neither UEBA nor Honeytokens nor Purple Team.+Name the profiles you want:+```bash-docker compose up -d-docker compose ps+docker compose --profile full --profile extras up -d+docker compose --profile full --profile extras ps```-**What this gives you about AI.** `docker compose up -d` starts the CORE+That starts:++| Profile | Services |+| --- | --- |+| *(default)* | **PostgreSQL** (5432) · **Redis** (6379) · **Kafka** (9092) · **litellm** (4000, LLM gateway) · **api** (8000, FastAPI core) · **agents** (8001, LangGraph) · **realtime** (8086, Node.js + VAPID Web Push) · **web** (3000) · **fusion** (8003) · **ingest** (8081, Go) |+| `full` | **ClickHouse** · **OpenSearch** · **Neo4j** · **Qdrant** · **kafka-ui** (8080) · **actions** (8002) · **threatintel** (8005) · **ueba** (8007) · **enrichment** (8080, Go) · **connectors** (8003) |+| `extras` | **honeytokens** (8008) · **purple-team** (8006) |+| `osquery` | **osquery-tls** (8091) |++`make up` and `make up-full` are the supported spellings for the first two+rows and additionally wait for every container to report healthy, which a+bare `docker compose up -d` does not.++**What this gives you about AI.** Every command above starts the COREprofile, which includes the `litellm` LLM gateway. AiSOC asks for logical taskaliases (`aisoc-triage`, `aisoc-investigation`, …) and the gateway is the onlything that resolves them, so it has to be running for AI triage to happen at-all — it was a `full`-profile service until 2026-09, which meant this exact-command could not do AI triage even with the key you just set+all — it was a `full`-profile service until 2026-09, which meant the default+install could not do AI triage even with the key you just set([ADR-0006](https://github.com/beenuar/AiSOC/blob/main/docs/decisions/0006-llm-gateway-in-core.md)).- **With a provider key** in `.env`: alerts are triaged by the AI, and the@@ -148,32 +165,48 @@ Path A above needs no gateway: the demo compose pins every role to a concreteprovider model, and a concrete pin goes straight to the provider rather thanthrough the gateway.-This starts the full set of services:--- **PostgreSQL** (5432) · **Redis** (6379) · **Kafka** (9092)-- **ClickHouse** · **OpenSearch** · **Neo4j** · **Qdrant**-- **api** (8000, FastAPI core) · **agents** (8001, LangGraph) ·- **realtime** (8086, Node.js + VAPID Web Push) · **web** (3000)-- **fusion** (8003) · **actions** (8002) · **threatintel** (8005) ·- **ueba** (8007) · **honeytokens** (8008) · **purple-team** (8006) ·- **ingest** (8081, Go) · **enrichment** (8080, Go) · **mcp** (TypeScript)+### 3. Database migrations — nothing to run++Every schema is applied by the stack itself; there is no manual migration+step, and there is nothing to remember to run in the right order.++- **api** applies its raw-SQL chain+ (`services/api/migrations/*.sql`) from+ [`app.scripts.run_migrations`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/scripts/run_migrations.py)+ during startup. It has no `alembic.ini` and never did, so+ `docker compose exec api alembic upgrade head` — which earlier revisions of+ this page told you to run — fails with `No 'script_location' key found`.+- **ueba**, **honeytokens**, **purple-team** and **osquery-tls** own their+ schemas through alembic. Each container runs+ [`app/_migrate.py`](https://github.com/beenuar/AiSOC/blob/main/services/ueba/app/_migrate.py)+ before its server: it applies the chain, reads the applied revision back,+ and refuses to start the service unless it matches head. Re-running is+ free — a database already at head applies nothing.++Those four apply their chain as the **owner**, not as the role they serve+requests with: `docker-compose.yml` sets `DATABASE_MIGRATION_URL` alongside+`DATABASE_URL` for each, and their `env.py` prefers it. The runtime role+holds no `CREATE` on schema public, by design — that is what stops it+turning row-level security off — so a chain applied as the runtime+credential fails on the first `CREATE TABLE`. See+[Env vars → the four services that manage their own schema](deployment/env-vars#the-four-services-that-manage-their-own-schema).-### 3. Run database migrations+To confirm the schemas landed, ask the database rather than the logs. Each+chain records its revision in its **own** version table, because four chains+numbering their revisions `0001`, `0002`, … against one database previously+shared `alembic_version` and the second chain to run believed it was already+done:```bash-docker compose exec api alembic upgrade head-docker compose exec ueba alembic upgrade head-docker compose exec honeytokens alembic upgrade head-docker compose exec purple-team alembic upgrade head+docker compose exec postgres psql -U aisoc -d aisoc -c \+ "SELECT table_name FROM information_schema.tables WHERE table_name LIKE 'alembic_version%' ORDER BY 1"```-Each of those four runs as the **owner**, not as the role the service serves-requests with: `docker-compose.yml` sets `DATABASE_MIGRATION_URL` alongside-`DATABASE_URL` for all four, and their `alembic/env.py` prefers it. The-runtime role holds no `CREATE` on schema public, by design — that is what-stops it turning row-level security off — so a chain applied as the runtime-credential fails on the first `CREATE TABLE`. See-[Env vars → the four services that manage their own schema](deployment/env-vars#the-four-services-that-manage-their-own-schema).+Four rows — `alembic_version_honeytokens`, `alembic_version_osquery_tls`,+`alembic_version_purple_team`, `alembic_version_ueba` — means all four ran.+A service whose chain did not reach head does not serve at all: its+container exits and `docker compose ps` shows it unhealthy rather than+running.The `api` migrations include[`008_investigation_ledger.sql`](https://github.com/beenuar/AiSOC/blob/main/services/api/migrations/008_investigation_ledger.sql)diff --git a/docker-compose.override.yml b/docker-compose.override.ymldeleted file mode 100644index 15c88f10e..000000000--- a/docker-compose.override.yml+++ /dev/null@@ -1,12 +0,0 @@-# QA-only host port remap (untracked, never committed). Another stack on this-# machine already publishes 5432, 8123 and 9000. Container ports are unchanged,-# so service-to-service traffic over the compose network is identical to the-# documented deployment; only the host-side bindings move.-services:- postgres:- ports: !override- - "127.0.0.1:55432:5432"- clickhouse:- ports: !override- - "127.0.0.1:58123:8123"- - "127.0.0.1:59000:9000"diff --git a/docker-compose.yml b/docker-compose.ymlindex a3c580149..ae41360d7 100644--- a/docker-compose.yml+++ b/docker-compose.yml@@ -55,6 +55,8 @@ volumes:postgres_data:redis_data:kafka_data:+ zookeeper_data:+ zookeeper_log:clickhouse_data:opensearch_data:qdrant_data:@@ -123,6 +125,23 @@ services:environment:ZOOKEEPER_CLIENT_PORT: 2181ZOOKEEPER_TICK_TIME: 2000+ # Kafka keeps its cluster id in `kafka_data`, which IS persisted, and+ # takes the authoritative one from ZooKeeper, which was not. So the+ # second `up` on a kept volume generated a fresh cluster id and the+ # broker refused to join it:+ #+ # InconsistentClusterIdException: The Cluster ID <new> doesn't match+ # stored clusterId Some(<old>) in meta.properties+ #+ # `make down && make up` — documented as "stop the stack, keep the data"+ # — therefore could not bring Kafka back, and every service that depends+ # on it stayed in `created`. Only `make clean`, which deletes the data,+ # recovered. Persisting both halves of the pair is the fix; persisting+ # neither would also be consistent, but it would discard the event log+ # on every restart.+ volumes:+ - zookeeper_data:/var/lib/zookeeper/data+ - zookeeper_log:/var/lib/zookeeper/lognetworks:- aisochealthcheck:@@ -616,8 +635,15 @@ services:condition: service_healthy# Host port 8007 was previously colliding with the `ueba` service's# 127.0.0.1:8007 binding; remapped to 8091 here. UEBA still owns 8007.+ #+ # The container side was 8007 too, and nothing has ever listened there:+ # this service's Dockerfile runs uvicorn on 9001 and EXPOSEs 9001. The+ # published port therefore resolved to a closed socket, and because the+ # service had no healthcheck either, the container still read as+ # `running`. Found by adding the healthcheck below and watching it sit+ # at `starting` against a port with no server on it.ports:- - "127.0.0.1:8091:8007"+ - "127.0.0.1:8091:9001"profiles:- osqueryenvironment:@@ -628,10 +654,25 @@ services:DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisoc# aisoc-db-role: owner — alembic applies this service's chain; the runtime role holds no DDLDATABASE_MIGRATION_URL: postgresql+asyncpg://aisoc:${POSTGRES_PASSWORD:-aisoc_dev_secret}@postgres:5432/aisoc+ # Read by the postgres init hook on a fresh volume and by the api+ # migration runner otherwise. Passed here too so this service's own+ # DATABASE_URL and the role's actual credential come from one value.+ AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}AISOC_OSQUERY_TLS_ENROLL_SECRET: ${AISOC_OSQUERY_TLS_ENROLL_SECRET:-change-me-in-production}# The ingest service is registered as `ingest-worker` in this compose# file; the previous `http://ingest:8080` would not resolve via DNS.AISOC_INGEST_BASE_URL: http://ingest-worker:8080+ # A detached consumer used to read as `running`. /readyz answers 503 when+ # the subscription is gone or the schema is not at head, so `docker compose+ # ps` and `make _wait` now see the difference.+ healthcheck:+ test:+ - CMD-SHELL+ - "python -c \"import urllib.request,sys; r=urllib.request.urlopen('http://127.0.0.1:9001/readyz',timeout=2); sys.exit(0 if r.status==200 else 1)\" || exit 1"+ interval: 10s+ timeout: 5s+ retries: 20+ start_period: 90snetworks:- aisocrestart: unless-stopped@@ -794,12 +835,27 @@ services:DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisoc# aisoc-db-role: owner — alembic applies this service's chain; the runtime role holds no DDLDATABASE_MIGRATION_URL: postgresql+asyncpg://aisoc:${POSTGRES_PASSWORD:-aisoc_dev_secret}@postgres:5432/aisoc+ # Read by the postgres init hook on a fresh volume and by the api+ # migration runner otherwise. Passed here too so this service's own+ # DATABASE_URL and the role's actual credential come from one value.+ AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}REDIS_URL: redis://:redis_dev_secret@redis:6379/8KAFKA_BOOTSTRAP_SERVERS: kafka:29092CLICKHOUSE_URL: http://aisoc:clickhouse_dev_secret@clickhouse:8123/aisocCORE_API_URL: http://api:8000AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:-}AISOC_DEV_MODE: ${AISOC_DEV_MODE:-1}+ # A detached consumer used to read as `running`. /readyz answers 503 when+ # the subscription is gone or the schema is not at head, so `docker compose+ # ps` and `make _wait` now see the difference.+ healthcheck:+ test:+ - CMD-SHELL+ - "python -c \"import urllib.request,sys; r=urllib.request.urlopen('http://127.0.0.1:8004/readyz',timeout=2); sys.exit(0 if r.status==200 else 1)\" || exit 1"+ interval: 10s+ timeout: 5s+ retries: 20+ start_period: 90snetworks:- aisocrestart: unless-stopped@@ -832,11 +888,26 @@ services:DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisoc# aisoc-db-role: owner — alembic applies this service's chain; the runtime role holds no DDLDATABASE_MIGRATION_URL: postgresql+asyncpg://aisoc:${POSTGRES_PASSWORD:-aisoc_dev_secret}@postgres:5432/aisoc+ # Read by the postgres init hook on a fresh volume and by the api+ # migration runner otherwise. Passed here too so this service's own+ # DATABASE_URL and the role's actual credential come from one value.+ AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}REDIS_URL: redis://:redis_dev_secret@redis:6379/9KAFKA_BOOTSTRAP_SERVERS: kafka:29092CORE_API_URL: http://api:8000AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:-}AISOC_DEV_MODE: ${AISOC_DEV_MODE:-1}+ # A detached consumer used to read as `running`. /readyz answers 503 when+ # the subscription is gone or the schema is not at head, so `docker compose+ # ps` and `make _wait` now see the difference.+ healthcheck:+ test:+ - CMD-SHELL+ - "python -c \"import urllib.request,sys; r=urllib.request.urlopen('http://127.0.0.1:8005/readyz',timeout=2); sys.exit(0 if r.status==200 else 1)\" || exit 1"+ interval: 10s+ timeout: 5s+ retries: 20+ start_period: 90snetworks:- aisocrestart: unless-stopped@@ -867,11 +938,26 @@ services:DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisoc# aisoc-db-role: owner — alembic applies this service's chain; the runtime role holds no DDLDATABASE_MIGRATION_URL: postgresql+asyncpg://aisoc:${POSTGRES_PASSWORD:-aisoc_dev_secret}@postgres:5432/aisoc+ # Read by the postgres init hook on a fresh volume and by the api+ # migration runner otherwise. Passed here too so this service's own+ # DATABASE_URL and the role's actual credential come from one value.+ AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}REDIS_URL: redis://:redis_dev_secret@redis:6379/10CORE_API_URL: http://api:8000ATTCK_DATA_PATH: /app/data/enterprise-attack.jsonAISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:-}AISOC_DEV_MODE: ${AISOC_DEV_MODE:-1}+ # A detached consumer used to read as `running`. /readyz answers 503 when+ # the subscription is gone or the schema is not at head, so `docker compose+ # ps` and `make _wait` now see the difference.+ healthcheck:+ test:+ - CMD-SHELL+ - "python -c \"import urllib.request,sys; r=urllib.request.urlopen('http://127.0.0.1:8006/readyz',timeout=2); sys.exit(0 if r.status==200 else 1)\" || exit 1"+ interval: 10s+ timeout: 5s+ retries: 20+ start_period: 90snetworks:- aisocrestart: unless-stoppeddiff --git a/scripts/audit_health_probes.py b/scripts/audit_health_probes.pyindex b6c21e109..bfaeaa2df 100644--- a/scripts/audit_health_probes.py+++ b/scripts/audit_health_probes.py@@ -64,19 +64,39 @@ def discover_fastapi_services() -> list[str]:return out-def audit_service(svc: str) -> tuple[bool, bool, bool]:- """Return ``(has_module, imports_helper, wires_routes)``.+#: The copy every other one is compared against. Arbitrary but fixed: what+#: matters is that there is one answer to "which is right" rather than+#: thirteen files that each look plausible on their own.+REFERENCE_SERVICE = "api"+++def audit_service(svc: str, reference: bytes | None) -> tuple[bool, bool, bool, bool]:+ """Return ``(has_module, imports_helper, wires_routes, in_sync)``.* ``has_module`` — ``services/<svc>/app/_health.py`` exists.* ``imports_helper``— ``app/main.py`` imports ``install_health_routes``.* ``wires_routes`` — ``app/main.py`` calls ``install_health_routes``.+ * ``in_sync`` — that copy is byte-identical to the reference one.++ ``in_sync`` is checked because the module's own docstring claims the+ thirteen copies are kept in sync and nothing verified it. They happened+ to be identical, so the claim was true by luck: any change landing in one+ tree would have left the other twelve serving the older contract, and a+ probe that answers differently per service is worse than one that is+ wrong everywhere — it makes a fleet-wide readiness answer unreadable.+ The docstring also pointed at ``scripts/sync_health_module.py``, which+ does not exist in this tree."""base = SERVICES_DIR / svc / "app"- has_module = (base / "_health.py").is_file()+ module = base / "_health.py"+ has_module = module.is_file()main_text = (base / "main.py").read_text(encoding="utf-8")imports_helper = "from app._health import install_health_routes" in main_textwires_routes = "install_health_routes(app" in main_text- return has_module, imports_helper, wires_routes+ # Unknown rather than false when there is nothing to compare against, so a+ # missing reference is reported as the missing module it is.+ in_sync = True if reference is None or not has_module else module.read_bytes() == reference+ return has_module, imports_helper, wires_routes, in_syncdef main() -> int:@@ -101,19 +121,35 @@ def main() -> int:)return 1- print(f"{'service':<18} {'_health.py':<11} {'import':<7} {'wire':<5}")- print(f"{'-' * 18} {'-' * 11} {'-' * 7} {'-' * 5}")+ reference_path = SERVICES_DIR / REFERENCE_SERVICE / "app" / "_health.py"+ reference = reference_path.read_bytes() if reference_path.is_file() else None+ if reference is None:+ print(+ f"health-probes: reference module {reference_path} is missing, so the " f"copies have nothing to be in sync with.",+ file=sys.stderr,+ )+ return 1++ print(f"{'service':<18} {'_health.py':<11} {'import':<7} {'wire':<5} {'in-sync':<7}")+ print(f"{'-' * 18} {'-' * 11} {'-' * 7} {'-' * 5} {'-' * 7}")drift: list[str] = []+ out_of_sync: list[str] = []for svc in services:- has_mod, has_import, has_wire = audit_service(svc)+ has_mod, has_import, has_wire, in_sync = audit_service(svc, reference)marker_mod = "yes" if has_mod else "NO"marker_import = "yes" if has_import else "NO"marker_wire = "yes" if has_wire else "NO"- print(f"{svc:<18} {marker_mod:<11} {marker_import:<7} {marker_wire:<5}")+ marker_sync = "yes" if in_sync else "NO"+ print(f"{svc:<18} {marker_mod:<11} {marker_import:<7} {marker_wire:<5} {marker_sync:<7}")if not (has_mod and has_import and has_wire):drift.append(svc)+ if not in_sync:+ out_of_sync.append(svc)++ if not args.check:+ return 0- if args.check and drift:+ if drift:print("\nFAIL: the following FastAPI services do not install the ""Phase 2.6 /livez + /readyz probes from app._health:\n - " + "\n - ".join(drift),@@ -125,8 +161,20 @@ def main() -> int:"app, service_name='aisoc-<svc>') in app/main.py.",file=sys.stderr,)- return 1- return 0+ if out_of_sync:+ print(+ f"\nFAIL: these copies of app/_health.py differ from "+ f"services/{REFERENCE_SERVICE}/app/_health.py:\n - " + "\n - ".join(out_of_sync),+ file=sys.stderr,+ )+ print(+ f"\nFix: cp services/{REFERENCE_SERVICE}/app/_health.py "+ f"services/<svc>/app/_health.py for each one above. The probes are a "+ f"contract the whole fleet answers; one tree answering an older "+ f"version of it cannot be read off a dashboard.",+ file=sys.stderr,+ )+ return 1 if (drift or out_of_sync) else 0if __name__ == "__main__":diff --git a/scripts/check_orm_migration_parity.py b/scripts/check_orm_migration_parity.pynew file mode 100644index 000000000..744cd4357--- /dev/null+++ b/scripts/check_orm_migration_parity.py@@ -0,0 +1,884 @@+#!/usr/bin/env python3+"""Every column a model declares must be a column some migration creates.++Why this exists+---------------++``services/ueba/app/models/ueba.py`` declared ``EntityBaseline.peer_group_id``+and migration ``0001`` never created it. SQLAlchemy names every mapped column+in its ``SELECT``, so the *first* database call UEBA's scoring path makes+raised ``UndefinedColumnError``. UEBA could therefore never write a baseline+or an anomaly, on any deployment, from the day the column was added — and+because the consumer had no ``except``, the container stayed ``running`` with+``/health`` answering 200 the whole time.++Two more of the same kind were in the same three tables:+``ueba_peer_groups.id`` was created ``UUID`` and declared ``String(64)`` while+the code wrote ``dept:engineering`` into it, and ``ueba_anomalies.event_type``+was created 64 characters wide and declared 128.++None of this needed a database to find. The model and the migration are both+in the tree, and the disagreement is structural. That is what this reads.++What it checks, in both directions+----------------------------------++``missing-in-migration``+ A model declares a column no migration creates. This is the direction+ that breaks at runtime, and it breaks on the first query rather than on+ some rare path, because the column list is in the ``SELECT``.++``missing-in-model``+ A migration creates a column no model declares. Not a crash, so it is+ reported separately and carries a recorded reason — but it is checked,+ because a gate that only looks one way passes while drift accumulates in+ the other. Three of the four cases here turned out to be+ ``server_default`` columns nothing maps, and one was a table whose model+ had been renamed.++``type-narrower-in-migration``+ Both sides have the column and the database's is too small to hold what+ the model will send. ``String(64)`` in a table against ``String(128)`` in+ the model passes every test that inserts a short value.++``no-migration-source``+ A model declares a table **no migration anywhere mentions**. This is the+ finding that exists because of how this gate could have been useless: a+ table the parser fails to pair contributes zero column comparisons, and+ zero findings over zero comparisons prints the same word as a clean+ result. ``--credits`` exists for the same reason — it enumerates what+ each verdict was *based on*, so a parser that silently credits nothing+ can be caught by reading rather than by trusting the exit code.++What it reads+-------------++Structurally, both sides. Models: Python ``ast`` over+``services/*/app/**`` for classes carrying ``__tablename__``, collecting+``mapped_column`` / ``Column`` assignments. Migrations: Python ``ast`` over+each alembic chain for ``op.create_table`` / ``op.add_column`` /+``op.drop_column``, plus the raw-SQL statements those revisions and+``services/api/migrations/*.sql`` execute, because half the column additions+in this repository are ``ALTER TABLE … ADD COLUMN IF NOT EXISTS`` inside a+string.++Usage+-----++::++ python3 scripts/check_orm_migration_parity.py # table + verdict+ python3 scripts/check_orm_migration_parity.py --check # gate+ python3 scripts/check_orm_migration_parity.py --credits # what it credited+ python3 scripts/check_orm_migration_parity.py --json+ python3 scripts/check_orm_migration_parity.py --self-test++Exit codes: 0 clean, 1 findings, 2 the scan itself could not run.+"""++from __future__ import annotations++import argparse+import ast+import json+import pathlib+import re+import sys++# `scripts/` is on sys.path when this file is run as a program, but not when a+# test loads it by path with importlib. gate_toolkit sits beside it either way.+sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))++from gate_toolkit import repo_root, self_test_main++REPO_ROOT = repo_root()+SERVICES_DIR = REPO_ROOT / "services"++#: Where a service's raw-SQL chain lives, if it has one. Alembic chains are+#: discovered from ``alembic.ini`` instead, since two of the four do not put+#: them under a directory named ``alembic``.+SQL_MIGRATION_DIRS = ("migrations",)++#: Directories under ``services/<svc>/`` that never hold models worth pairing.+_SKIP_PARTS = frozenset({"tests", "test", "_vendor", "node_modules", "__pycache__", "alembic", "versions"})+++# ---------------------------------------------------------------------------+# Recorded exceptions+# ---------------------------------------------------------------------------+#+# Shrink-only, and verified in both directions: an entry that stops being+# needed fails the build rather than sitting here. Each one says why, because+# a bare list of table names is a second place for drift to hide.++#: ``(table, column)`` a migration creates and no model maps. Harmless by+#: construction — every one carries a ``server_default``, so an insert that+#: does not name it still succeeds — but recorded rather than ignored.+UNMAPPED_COLUMNS: dict[tuple[str, str], str] = {+ ("ueba_entity_baselines", "observation_count"): "server_default '0'; the rolling count lives in feature_stats instead",+ ("ueba_entity_baselines", "created_at"): "server_default now(); the model exposes window_start/updated_at",+ ("ueba_anomalies", "z_scores"): "server_default '{}'; superseded by the per-feature z_score inside features",+ ("ueba_peer_groups", "created_at"): "server_default now(); the model exposes updated_at",+}++#: Tables whose model is mapped but whose creation this gate cannot see, with+#: the reason. Empty on purpose: it exists so that adding one is a deliberate,+#: reviewed act rather than the default outcome of a parser that missed+#: something.+UNSOURCED_TABLES: dict[str, str] = {}+++# ---------------------------------------------------------------------------+# Model side+# ---------------------------------------------------------------------------+++def _string(node: ast.AST) -> str | None:+ return node.value if isinstance(node, ast.Constant) and isinstance(node.value, str) else None+++#: SQL type name → family. Two spellings of the same family are not drift+#: (``Text`` against ``String``, ``BigInteger`` against ``Integer``); two+#: families are, and one such mismatch is why this mapping exists.+#:+#: ``ueba_peer_groups.id`` was created ``UUID DEFAULT gen_random_uuid()`` and+#: declared ``String(64)``, and the code wrote ``dept:engineering`` into it —+#: an ``InvalidTextRepresentation`` on the first insert. A length comparison+#: could not see it: neither side declares a length that the other contradicts,+#: so the column read as agreed. Found by re-running this gate against the+#: tree as it was and noticing that it reported two of the three known+#: defects, which is the question `--credits` is for: what does it credit as+#: clean, not what does it flag.+_TYPE_FAMILIES: dict[str, str] = {+ # SQLAlchemy constructors and SQL type names share this table on purpose:+ # the two sides have to be classified identically or the comparison is+ # between two different vocabularies.+ "string": "string",+ "str": "string",+ "text": "string",+ "unicode": "string",+ "unicodetext": "string",+ "varchar": "string",+ "char": "string",+ "citext": "string",+ "uuid": "uuid",+ "guid": "uuid",+ "integer": "integer",+ "int": "integer",+ "int4": "integer",+ "int8": "integer",+ "smallint": "integer",+ "bigint": "integer",+ "smallinteger": "integer",+ "biginteger": "integer",+ "serial": "integer",+ "bigserial": "integer",+ "float": "numeric",+ "numeric": "numeric",+ "decimal": "numeric",+ "real": "numeric",+ "double": "numeric",+ "boolean": "boolean",+ "bool": "boolean",+ "json": "json",+ "jsonb": "json",+ "datetime": "datetime",+ "timestamp": "datetime",+ "timestamptz": "datetime",+ "date": "datetime",+ "time": "datetime",+ "interval": "interval",+ "largebinary": "binary",+ "bytea": "binary",+ "inet": "inet",+ "array": "array",+ "enum": "enum",+}+++def _family(name: str | None) -> str | None:+ """The family of a type name, or None when it is one this gate cannot rank.++ Unknown is not a mismatch. A vendor type or a custom ``TypeDecorator``+ must not be reported as drift just because this table has not heard of+ it — a gate that invents findings gets turned off, and then it is not+ checking the ones it was right about either.+ """+ return _TYPE_FAMILIES.get(name.strip().lower()) if name else None+++def _declared_type(call: ast.Call) -> tuple[str | None, int | None]:+ """``(family, string_length)`` of the type in a column definition.++ The length is only read for ``String``: the other families either carry+ no length or carry one whose mismatch is not silently lossy the way a+ truncating varchar is.+ """+ for node in ast.walk(call):+ if not isinstance(node, ast.Call):+ continue+ name = node.func.attr if isinstance(node.func, ast.Attribute) else getattr(node.func, "id", None)+ family = _family(name)+ if family is None:+ continue+ if family != "string":+ return family, None+ for arg in node.args:+ if isinstance(arg, ast.Constant) and isinstance(arg.value, int):+ return family, arg.value+ for kw in node.keywords:+ if kw.arg == "length" and isinstance(kw.value, ast.Constant) and isinstance(kw.value.value, int):+ return family, kw.value.value+ # ``String`` with no length is unbounded text — never too narrow.+ return family, None+ return None, None+++def _column_from_assignment(node: ast.AST) -> tuple[str, tuple[str | None, int | None]] | None:+ """``(column_name, string_length)`` for one class-body assignment.++ Handles the two spellings in this tree: SQLAlchemy 2.0+ ``name: Mapped[T] = mapped_column(...)`` and the older+ ``name = Column(...)``. The column's name is the attribute name unless+ the call passes an explicit string first — ``mapped_column("other", ...)``+ — which is the case a regex over attribute names would get wrong.+ """+ target: ast.expr+ value: ast.expr | None+ if isinstance(node, ast.AnnAssign):+ target, value = node.target, node.value+ elif isinstance(node, ast.Assign) and len(node.targets) == 1:+ target, value = node.targets[0], node.value+ else:+ return None+ if not isinstance(target, ast.Name) or not isinstance(value, ast.Call):+ return None++ func = value.func+ callee = func.attr if isinstance(func, ast.Attribute) else getattr(func, "id", None)+ if callee not in {"mapped_column", "Column"}:+ return None++ name = target.id+ if value.args and (explicit := _string(value.args[0])):+ name = explicit+ return name, _declared_type(value)+++def model_tables(paths: list[pathlib.Path]) -> dict[str, dict]:+ """Mapped tables, by table name, from the model modules given."""+ tables: dict[str, dict] = {}+ for path in paths:+ try:+ tree = ast.parse(path.read_text(encoding="utf-8", errors="replace"))+ except SyntaxError:+ continue+ for node in ast.walk(tree):+ if not isinstance(node, ast.ClassDef):+ continue+ tablename: str | None = None+ columns: dict[str, tuple[str | None, int | None]] = {}+ for stmt in node.body:+ if isinstance(stmt, ast.Assign) and len(stmt.targets) == 1:+ only = stmt.targets[0]+ if isinstance(only, ast.Name) and only.id == "__tablename__":+ tablename = _string(stmt.value)+ found = _column_from_assignment(stmt)+ if found is not None:+ columns[found[0]] = found[1]+ if tablename and columns:+ tables[tablename] = {+ "columns": columns,+ "source": str(path.relative_to(REPO_ROOT)),+ "model": node.name,+ }+ return tables+++# ---------------------------------------------------------------------------+# Migration side+# ---------------------------------------------------------------------------++#: ``CREATE TABLE [IF NOT EXISTS] name ( ... )`` — the body is scanned for+#: leading identifiers, one per column definition.+_CREATE_TABLE = re.compile(r"CREATE\s+TABLE\s+(?:IF\s+NOT\s+EXISTS\s+)?[\"']?(\w+)[\"']?\s*\((.*?)\);", re.I | re.S)+_ADD_COLUMN = re.compile(+ r"ALTER\s+TABLE\s+(?:ONLY\s+)?[\"']?(\w+)[\"']?\s+ADD\s+COLUMN\s+"+ r"(?:IF\s+NOT\s+EXISTS\s+)?[\"']?(\w+)[\"']?\s*([A-Za-z]*)\s*(?:\((\d+)\))?",+ re.I,+)+_DROP_COLUMN = re.compile(r"ALTER\s+TABLE\s+(?:ONLY\s+)?[\"']?(\w+)[\"']?\s+DROP\s+COLUMN\s+(?:IF\s+EXISTS\s+)?[\"']?(\w+)[\"']?", re.I)+_ALTER_TYPE = re.compile(+ r"ALTER\s+TABLE\s+(?:ONLY\s+)?[\"']?(\w+)[\"']?\s+ALTER\s+COLUMN\s+[\"']?(\w+)[\"']?\s+"+ r"(?:SET\s+DATA\s+)?TYPE\s+([A-Za-z]+)\s*(?:\((\d+)\))?",+ re.I,+)++#: Words that begin a table-level constraint rather than a column.+_CONSTRAINT_WORDS = frozenset({"primary", "unique", "foreign", "constraint", "check", "exclude", "like"})+++def _split_columns(body: str) -> list[tuple[str, str | None, int | None]]:+ """Column ``(name, length)`` pairs from a ``CREATE TABLE`` body."""+ depth = 0+ current: list[str] = []+ parts: list[str] = []+ for char in body:+ if char == "(":+ depth += 1+ elif char == ")":+ depth -= 1+ if char == "," and depth == 0:+ parts.append("".join(current))+ current = []+ continue+ current.append(char)+ parts.append("".join(current))++ out: list[tuple[str, str | None, int | None]] = []+ for part in parts:+ tokens = part.strip().strip('"').split()+ if not tokens:+ continue+ name = tokens[0].strip('"').strip(",")+ if not name or not name.replace("_", "").isalnum() or name.lower() in _CONSTRAINT_WORDS:+ continue+ family: str | None = None+ length: int | None = None+ if len(tokens) > 1:+ # ``VARCHAR(64)`` arrives as one token; the family is the part+ # before the parenthesis.+ declared = tokens[1].split("(")[0]+ family = _family(declared)+ if match := re.search(r"\((\d+)\)", tokens[1]):+ length = int(match.group(1))+ out.append((name, family, length))+ return out+++def _record(+ store: dict[str, dict[str, tuple[str | None, int | None]]],+ table: str,+ column: str,+ family: str | None,+ length: int | None,+) -> None:+ store.setdefault(table, {})[column] = (family, length)+++def _scan_sql(text: str, store: dict[str, dict[str, tuple[str | None, int | None]]], dropped: set[tuple[str, str]]) -> None:+ for table, body in _CREATE_TABLE.findall(text):+ for column, family, length in _split_columns(body):+ _record(store, table, column, family, length)+ for table, column, type_name, length in _ADD_COLUMN.findall(text):+ _record(store, table, column, _family(type_name), int(length) if length else None)+ for table, column, type_name, length in _ALTER_TYPE.findall(text):+ # Only widens what is already known; an ALTER TYPE on a column this+ # scan never saw created is still a column it never saw created.+ if column in store.get(table, {}):+ _record(store, table, column, _family(type_name), int(length) if length else None)+ for table, column in _DROP_COLUMN.findall(text):+ dropped.add((table, column))+++def _scan_alembic_module(tree: ast.AST, store: dict[str, dict[str, tuple[str | None, int | None]]], dropped: set[tuple[str, str]]) -> None:+ """``op.*`` table and column operations, plus the SQL they execute."""+ for node in ast.walk(tree):+ if not isinstance(node, ast.Call) or not isinstance(node.func, ast.Attribute):+ continue+ op_name = node.func.attr++ if op_name == "execute":+ for arg in node.args:+ for literal in ast.walk(arg):+ if (sql := _string(literal)) and "TABLE" in sql.upper():+ _scan_sql(sql, store, dropped)+ continue++ if op_name == "create_table" and node.args:+ table = _string(node.args[0])+ if not table:+ continue+ for arg in node.args[1:]:+ if isinstance(arg, ast.Call) and (column := _string(arg.args[0]) if arg.args else None):+ _record(store, table, column, *_declared_type(arg))+ continue++ if op_name == "add_column" and len(node.args) >= 2:+ table = _string(node.args[0])+ spec = node.args[1]+ if table and isinstance(spec, ast.Call) and spec.args and (column := _string(spec.args[0])):+ _record(store, table, column, *_declared_type(spec))+ continue++ if op_name == "alter_column" and len(node.args) >= 2:+ table, column = _string(node.args[0]), _string(node.args[1])+ if not table or not column or column not in store.get(table, {}):+ continue+ for kw in node.keywords:+ if kw.arg == "type_" and isinstance(kw.value, ast.Call):+ _record(store, table, column, *_declared_type(kw.value))+ continue++ if op_name == "drop_column" and len(node.args) >= 2:+ table, column = _string(node.args[0]), _string(node.args[1])+ if table and column:+ dropped.add((table, column))+++def _upgrade_only(path: pathlib.Path) -> ast.Module | None:+ """The revision's ``upgrade()`` body, with ``downgrade()`` removed.++ ``downgrade`` drops what ``upgrade`` creates. Scanning the whole module+ made every revision cancel itself out — the parser credited a column and+ then recorded it as dropped, so a correct chain reported every column+ missing. Caught by reading ``--credits``, not by the exit code.+ """+ try:+ tree = ast.parse(path.read_text(encoding="utf-8", errors="replace"))+ except (SyntaxError, OSError):+ return None+ tree.body = [node for node in tree.body if not (isinstance(node, ast.FunctionDef) and node.name.startswith("downgrade"))]+ return tree+++def migration_columns(service: pathlib.Path) -> tuple[dict[str, dict[str, tuple[str | None, int | None]]], set[tuple[str, str]], list[str]]:+ """Columns this service's migrations create, columns they drop, files read."""+ store: dict[str, dict[str, tuple[str | None, int | None]]] = {}+ dropped: set[tuple[str, str]] = set()+ read: list[str] = []++ # Alembic chain, located from alembic.ini rather than assumed: osquery-tls+ # keeps its chain under app/db, and a gate that globbed for a directory+ # named `alembic` would have reported that service as having no migrations+ # at all — which is a clean result, wrongly.+ ini = service / "alembic.ini"+ if ini.is_file():+ match = re.search(r"^\s*script_location\s*=\s*(.+?)\s*$", ini.read_text(encoding="utf-8"), re.M)+ if match:+ versions = service / match.group(1).strip() / "versions"+ for path in sorted(versions.glob("*.py")):+ tree = _upgrade_only(path)+ if tree is not None:+ _scan_alembic_module(tree, store, dropped)+ read.append(str(path.relative_to(REPO_ROOT)))++ for name in SQL_MIGRATION_DIRS:+ for path in sorted((service / name).glob("*.sql")):+ _scan_sql(path.read_text(encoding="utf-8", errors="replace"), store, dropped)+ read.append(str(path.relative_to(REPO_ROOT)))++ return store, dropped, read+++# ---------------------------------------------------------------------------+# Comparison+# ---------------------------------------------------------------------------+++def model_paths(service: pathlib.Path) -> list[pathlib.Path]:+ return [path for path in sorted((service / "app").rglob("*.py")) if not _SKIP_PARTS & set(path.relative_to(service).parts)]+++def creates_tables_from_metadata(paths: list[pathlib.Path]) -> str | None:+ """The file where this service calls ``metadata.create_all``, if it does.++ This is the line that decides whether the check applies, and why it is+ read structurally rather than assumed per service.++ A service that calls ``create_all`` on the way up builds its tables from+ the models, so a column the models declare and no migration creates is+ *created anyway* — the migrations are supplemental there, adding columns+ to tables the ORM already owns. ``services/api`` is the one such service+ in this tree, and it accounts for 173 of the 173 disagreements found+ before this distinction existed. Holding it to the same rule would have+ meant either 173 allowlist entries or a gate nobody could turn on.++ A migration-managed service has no such fallback, which is exactly why+ UEBA's missing column was fatal rather than cosmetic.++ Those services are still scanned, counted, and printed — ``--all`` shows+ their findings — because "excluded" has to be visible. An exclusion that+ prints nothing is the same defect one level up.+ """+ for path in paths:+ try:+ tree = ast.parse(path.read_text(encoding="utf-8", errors="replace"))+ except (SyntaxError, OSError):+ continue+ for node in ast.walk(tree):+ if isinstance(node, ast.Attribute) and node.attr == "create_all":+ return str(path.relative_to(REPO_ROOT))+ return None+++def scan() -> dict:+ if not SERVICES_DIR.is_dir():+ return {"error": f"no services directory at {SERVICES_DIR}"}++ services: list[dict] = []+ for service in sorted(p for p in SERVICES_DIR.iterdir() if p.is_dir()):+ if not (service / "app").is_dir():+ continue+ paths = model_paths(service)+ models = model_tables(paths)+ if not models:+ continue+ create_all = creates_tables_from_metadata(paths)+ created, dropped, files = migration_columns(service)+ if not files:+ # A service with models and no migration chain manages its schema+ # elsewhere (create_all, or another service's chain). Not a+ # finding, but counted, so `--credits` shows it was considered.+ services.append(+ {+ "service": service.name,+ "schema_source": "orm" if create_all else "none",+ "tables": len(models),+ "columns": sum(len(t["columns"]) for t in models.values()),+ "migration_files": 0,+ "findings": [],+ "credits": {},+ }+ )+ continue++ findings: list[dict] = []+ credits: dict[str, dict] = {}+ for table, spec in sorted(models.items()):+ table_columns = {c: spec_t for c, spec_t in created.get(table, {}).items() if (table, c) not in dropped}+ credits[table] = {+ "model": spec["model"],+ "model_source": spec["source"],+ "migration_columns": sorted(table_columns),+ }+ if not table_columns:+ if table in UNSOURCED_TABLES:+ credits[table]["recorded"] = UNSOURCED_TABLES[table]+ continue+ findings.append(+ {+ "kind": "no-migration-source",+ "table": table,+ "column": "*",+ "detail": f"{spec['source']} maps {spec['model']} to {table}; no migration in this service creates it",+ }+ )+ continue++ for column, (model_family, model_length) in sorted(spec["columns"].items()):+ if column not in table_columns:+ findings.append(+ {+ "kind": "missing-in-migration",+ "table": table,+ "column": column,+ "detail": f"{spec['model']} declares it; no migration creates it. Every query naming this table will fail.",+ }+ )+ continue+ table_family, table_length = table_columns[column]++ if model_family and table_family and model_family != table_family:+ findings.append(+ {+ "kind": "type-family-mismatch",+ "table": table,+ "column": column,+ "detail": f"{spec['model']} declares {model_family}; the table is {table_family}",+ }+ )+ # The width question is meaningless across families.+ continue++ if model_length is not None and table_length is not None and table_length < model_length:+ findings.append(+ {+ "kind": "type-narrower-in-migration",+ "table": table,+ "column": column,+ "detail": f"{spec['model']} declares String({model_length}); the table is {table_length} wide",+ }+ )++ for column in sorted(table_columns):+ if column in spec["columns"] or (table, column) in UNMAPPED_COLUMNS:+ continue+ findings.append(+ {+ "kind": "missing-in-model",+ "table": table,+ "column": column,+ "detail": f"a migration creates it; {spec['model']} does not map it",+ }+ )++ services.append(+ {+ "service": service.name,+ "schema_source": "orm" if create_all else "migrations",+ "create_all_at": create_all,+ "tables": len(models),+ "columns": sum(len(t["columns"]) for t in models.values()),+ "migration_files": len(files),+ "findings": findings,+ "credits": credits,+ }+ )++ gated = [s for s in services if s["schema_source"] == "migrations"]+ return {+ "root": str(REPO_ROOT),+ "services": services,+ "gated_services": [s["service"] for s in gated],+ "advisory_services": [s["service"] for s in services if s["schema_source"] != "migrations"],+ "tables": sum(s["tables"] for s in services),+ "columns": sum(s["columns"] for s in services),+ # Only the gated services: quoting the total next to the verdict would+ # credit this check with 832 columns in `api` that it deliberately+ # does not gate.+ "gated_columns": sum(s["columns"] for s in gated),+ "compared": sum(len(s["credits"]) for s in gated),+ "findings": [dict(f, service=s["service"]) for s in gated for f in s["findings"]],+ "advisory_findings": [dict(f, service=s["service"]) for s in services if s["schema_source"] != "migrations" for f in s["findings"]],+ }+++def unused_exceptions(result: dict) -> list[str]:+ """Recorded exceptions nothing needs any more.++ The other direction of the same check: an allowlist verified only when it+ forgives something keeps growing, and every stale entry is a column the+ gate has stopped looking at for a reason that no longer exists.+ """+ seen_columns: set[tuple[str, str]] = set()+ seen_tables: set[str] = set()+ for service in result["services"]:+ if service["schema_source"] != "migrations":+ continue+ for table, credit in service["credits"].items():+ seen_tables.add(table)+ for column in credit["migration_columns"]:+ seen_columns.add((table, column))++ stale = [+ f"UNMAPPED_COLUMNS[{key!r}] — {key[1]} is no longer created by any migration" for key in UNMAPPED_COLUMNS if key not in seen_columns+ ]+ stale += [f"UNSOURCED_TABLES[{name!r}] — {name} is no longer a mapped table" for name in UNSOURCED_TABLES if name not in seen_tables]+ return stale+++# ---------------------------------------------------------------------------+# Self-test+# ---------------------------------------------------------------------------++_INJECTED_MODEL = """+from sqlalchemy.orm import Mapped, mapped_column+from sqlalchemy import String++class Widget:+ __tablename__ = "widgets"+ id: Mapped[str] = mapped_column(String(16), primary_key=True)+ ghost: Mapped[str] = mapped_column(String(32))+ squeezed: Mapped[str] = mapped_column(String(128))+ mistyped: Mapped[str] = mapped_column(String(64))+"""++_INJECTED_MIGRATION = """+from alembic import op+import sqlalchemy as sa+from sqlalchemy.dialects import postgresql+revision = "0001"+down_revision = None++def upgrade() -> None:+ op.create_table(+ "widgets",+ sa.Column("id", sa.String(16), primary_key=True),+ sa.Column("squeezed", sa.String(64)),+ sa.Column("mistyped", postgresql.UUID(as_uuid=True)),+ sa.Column("orphan", sa.String(8), server_default=""),+ )++def downgrade() -> None:+ op.drop_table("widgets")+"""+++def _detects_injected_drift() -> list[tuple[str, bool]]:+ """Prove the gate still catches each kind, on a tree built for the purpose.++ Three synthetic disagreements in one service, one per kind. Run before the+ gate renders any verdict about the real tree, which is the point of a+ self-test: a gate whose parser has quietly stopped matching anything+ reports a clean repository in exactly the same words as a clean one.+ """+ import tempfile++ global REPO_ROOT, SERVICES_DIR # noqa: PLW0603 — restored below+ saved_root, saved_services = REPO_ROOT, SERVICES_DIR+ try:+ with tempfile.TemporaryDirectory() as raw:+ root = pathlib.Path(raw)+ service = root / "services" / "widgetry"+ (service / "app" / "models").mkdir(parents=True)+ (service / "app" / "models" / "widget.py").write_text(_INJECTED_MODEL)+ (service / "alembic" / "versions").mkdir(parents=True)+ (service / "alembic.ini").write_text("[alembic]\nscript_location = alembic\n")+ (service / "alembic" / "versions" / "0001_init.py").write_text(_INJECTED_MIGRATION)++ REPO_ROOT, SERVICES_DIR = root, root / "services"+ kinds = {f["kind"] for f in scan()["findings"]}+ finally:+ REPO_ROOT, SERVICES_DIR = saved_root, saved_services++ return [+ ("detects a model column no migration creates", "missing-in-migration" in kinds),+ ("detects a migration column no model maps", "missing-in-model" in kinds),+ ("detects a column the table is too narrow for", "type-narrower-in-migration" in kinds),+ ("detects a column whose type family disagrees", "type-family-mismatch" in kinds),+ ]+++def _detects_unsourced_table() -> list[tuple[str, bool]]:+ """And a mapped table whose creation the parser cannot find at all.++ Separate from the three above because it is the failure mode of this gate+ rather than of the tree: pairing nothing has to be a finding, or a parser+ that matches no migration statement at all reports every service clean.+ """+ import tempfile++ global REPO_ROOT, SERVICES_DIR # noqa: PLW0603 — restored below+ saved_root, saved_services = REPO_ROOT, SERVICES_DIR+ try:+ with tempfile.TemporaryDirectory() as raw:+ root = pathlib.Path(raw)+ service = root / "services" / "widgetry"+ (service / "app" / "models").mkdir(parents=True)+ (service / "app" / "models" / "widget.py").write_text(_INJECTED_MODEL)+ (service / "alembic" / "versions").mkdir(parents=True)+ (service / "alembic.ini").write_text("[alembic]\nscript_location = alembic\n")+ # A revision that creates a different table, so the chain is real+ # and the mapped table is still unaccounted for.+ (service / "alembic" / "versions" / "0001_init.py").write_text(_INJECTED_MIGRATION.replace('"widgets"', '"sprockets"'))+ REPO_ROOT, SERVICES_DIR = root, root / "services"+ kinds = {f["kind"] for f in scan()["findings"]}+ finally:+ REPO_ROOT, SERVICES_DIR = saved_root, saved_services++ return [("detects a mapped table no migration mentions", "no-migration-source" in kinds)]+++# ---------------------------------------------------------------------------+# Entry point+# ---------------------------------------------------------------------------+++def main() -> int:+ parser = argparse.ArgumentParser(description="ORM/migration column parity")+ parser.add_argument("--check", action="store_true", help="exit non-zero on findings")+ parser.add_argument("--all", action="store_true", help="also list findings for services that build tables from the models")+ parser.add_argument("--credits", action="store_true", help="print what each verdict was based on")+ parser.add_argument("--json", action="store_true", help="machine-readable output")+ parser.add_argument("--self-test", action="store_true", help="prove this gate still detects drift")+ args = parser.parse_args()++ if args.self_test:+ return self_test_main(+ pathlib.Path(__file__).name,+ ["--check"],+ extra=[*_detects_injected_drift(), *_detects_unsourced_table()],+ )++ result = scan()+ if "error" in result:+ print(f"orm-parity: {result['error']}", file=sys.stderr)+ return 2++ if args.json:+ print(json.dumps(result, indent=2, sort_keys=True))++ # Naming the root and the corpus size is not decoration. A gate that+ # printed "OK" without them could be reporting on a different checkout,+ # or on none: `found nothing` and `scanned nothing` print the same word.+ if not args.json:+ print(f"orm-parity: root {result['root']}")+ print(f"{'service':<14} {'schema from':<11} {'tables':>6} {'columns':>8} {'migrations':>11} {'findings':>9}")+ print(f"{'-' * 14} {'-' * 11} {'-' * 6} {'-' * 8} {'-' * 11} {'-' * 9}")+ for service in result["services"]:+ print(+ f"{service['service']:<14} {service['schema_source']:<11} {service['tables']:>6} "+ f"{service['columns']:>8} {service['migration_files']:>11} {len(service['findings']):>9}"+ )+ for service in result["services"]:+ if service["schema_source"] == "orm":+ print(+ f"\n{service['service']}: advisory only — builds its tables from the models at "+ f"{service['create_all_at']}, so a column no migration creates is created anyway. "+ f"{len(service['findings'])} disagreement(s); see --all."+ )++ if args.all and result["advisory_findings"]:+ print(f"\nAdvisory ({len(result['advisory_findings'])}), in services whose tables come from the models:")+ for finding in result["advisory_findings"]:+ print(f" [{finding['kind']}] {finding['service']}: {finding['table']}.{finding['column']}")++ if args.credits:+ print("\nWhat each table's verdict was based on:")+ for service in result["services"]:+ for table, credit in sorted(service["credits"].items()):+ columns = ", ".join(credit["migration_columns"]) or "(nothing — this is a finding)"+ print(f" {service['service']}/{table} ← {credit['model_source']}::{credit['model']}")+ print(f" migrations credit: {columns}")++ if not result["gated_services"] or not result["compared"]:+ # The corpus refusal. A tree with no migration-managed table produces+ # no findings, and that is not a clean repository — it is a scan that+ # read nothing, which this must never report as a pass. It also+ # catches the scoping above going wrong: if every service were+ # classified as building its tables from the models, this gate would+ # have nothing left to judge and has to say so rather than pass.+ print(+ f"\norm-parity: paired 0 migration-managed tables under {SERVICES_DIR}. Zero tables "+ f"compared is not zero drift; check the root above is the tree you meant.",+ file=sys.stderr,+ )+ return 1++ stale = unused_exceptions(result)+ findings = result["findings"]++ if not args.json:+ print(+ f"\ngated: {result['compared']} migration-managed table(s), "+ f"{result['gated_columns']} declared column(s) across {len(result['gated_services'])} service(s)"+ )++ if findings:+ print(f"\nFAIL: {len(findings)} ORM/migration disagreement(s):", file=sys.stderr)+ for finding in findings:+ print(+ f" [{finding['kind']}] {finding['service']}: {finding['table']}.{finding['column']}\n" f" {finding['detail']}",+ file=sys.stderr,+ )+ print(+ "\nFix: add the column in a new migration (the direction that breaks at runtime), "+ "or record it in UNMAPPED_COLUMNS in this file with the reason it is harmless.",+ file=sys.stderr,+ )+ if stale:+ print("\nFAIL: recorded exceptions that nothing needs any more:", file=sys.stderr)+ for entry in stale:+ print(f" {entry}", file=sys.stderr)+ print("\nFix: delete them. This list only shrinks.", file=sys.stderr)++ if args.check:+ return 1 if (findings or stale) else 0+ if findings or stale:+ print("\n(advisory run — use --check to make this a gate)")+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/scripts/check_service_migration_bootstrap.py b/scripts/check_service_migration_bootstrap.pynew file mode 100644index 000000000..2aed085d8--- /dev/null+++ b/scripts/check_service_migration_bootstrap.py@@ -0,0 +1,304 @@+#!/usr/bin/env python3+"""A migration chain nothing invokes is not a migration chain.++Why this exists+---------------++Four services own their schema through alembic — ``ueba``, ``honeytokens``,+``purple-team`` and ``osquery-tls``. Their chains were correct: run by hand+they applied cleanly, nine tables each with a row-level-security policy, each+with its own version table. **No compose path invoked any of them.** Every+container command was a plain ``uvicorn``, so on the documented quickstart+those services booted healthy with empty schemas and no policies.++That is how two services shipped with no tables while every command in the+sequence reported success. The chain existing, the chain being correct, and+the chain having run are three different facts, and only the third one+matters to a running deployment.++What it checks+--------------++For every service with an ``alembic.ini`` — discovered from the file, not+from a list kept here, so a fifth service cannot be added without being+held to this:++``no-bootstrap-module``+ ``app/_migrate.py`` is missing. That is the module that applies the+ chain and verifies it reached head.++``bootstrap-not-invoked``+ The Dockerfile never runs it. This is the defect as it actually was: the+ module could have been present and the ``CMD`` still a bare ``uvicorn``.+ Checked against the ``CMD``/``ENTRYPOINT`` lines rather than the whole+ file, so a mention in a comment does not satisfy it.++``bootstrap-module-drift``+ The copy differs from the reference one. The four are vendored because+ services share no Python path, and a wrapper that verifies "reached+ head" in one tree and not in another is worse than none: the fleet-wide+ claim becomes unreadable.++``no-owner-credential``+ ``docker-compose.yml`` does not give the service a ``*DATABASE_MIGRATION_URL``.+ Alembic issues DDL and the runtime role deliberately holds no ``CREATE``+ on schema public — that is what stops it turning RLS off — so a chain+ applied as the runtime credential dies on the first ``CREATE TABLE``.++``no-healthcheck``+ The service has no compose healthcheck, so a container that refused to+ start still reads as ``running``. Invisibility is the whole reason both+ of these defects survived a full acceptance pass.++And the other direction: a service shipping ``app/_migrate.py`` with no+``alembic.ini`` is reported too, because a bootstrap for a chain that is not+there will exit 1 on every start of a service that has nothing to migrate.++Usage+-----++::++ python3 scripts/check_service_migration_bootstrap.py # table+ python3 scripts/check_service_migration_bootstrap.py --check # gate+ python3 scripts/check_service_migration_bootstrap.py --json+ python3 scripts/check_service_migration_bootstrap.py --self-test++Exit codes: 0 clean, 1 findings, 2 the scan itself could not run.+"""++from __future__ import annotations++import argparse+import json+import pathlib+import re+import sys++sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))++from gate_toolkit import repo_root, self_test_main++REPO_ROOT = repo_root()+SERVICES_DIR = REPO_ROOT / "services"+COMPOSE = REPO_ROOT / "docker-compose.yml"++#: The module each service runs before its server, and the reference copy the+#: others are compared against.+BOOTSTRAP = pathlib.Path("app") / "_migrate.py"+REFERENCE_SERVICE = "ueba"++#: How the Dockerfile must invoke it. Matched on the module path so the shell+#: spelling is free to change.+INVOCATION = "app._migrate"+++def _compose_block(text: str, service: str) -> str:+ """The ``docker-compose.yml`` stanza for one service.++ Sliced on indentation rather than parsed with a YAML library: this gate+ runs on a bare interpreter in CI before any ``pip install``, and pulling+ in PyYAML to read two keys would take that away from it.+ """+ match = re.search(rf"^ {re.escape(service)}:$", text, re.M)+ if not match:+ return ""+ rest = text[match.end() :]+ end = re.search(r"^ \S", rest, re.M)+ return rest[: end.start()] if end else rest+++def _command_lines(dockerfile: str) -> str:+ """Just the CMD / ENTRYPOINT lines, so a comment cannot satisfy the check."""+ return "\n".join(line for line in dockerfile.splitlines() if line.strip().upper().startswith(("CMD", "ENTRYPOINT")))+++def scan() -> dict:+ if not SERVICES_DIR.is_dir():+ return {"error": f"no services directory at {SERVICES_DIR}"}++ compose_text = COMPOSE.read_text(encoding="utf-8") if COMPOSE.is_file() else ""+ reference_path = SERVICES_DIR / REFERENCE_SERVICE / BOOTSTRAP+ reference = reference_path.read_bytes() if reference_path.is_file() else None++ services: list[dict] = []+ findings: list[dict] = []++ candidates = sorted(p for p in SERVICES_DIR.iterdir() if p.is_dir())++ for service in candidates:+ has_chain = (service / "alembic.ini").is_file()+ module = service / BOOTSTRAP++ # The other direction: a bootstrap with no chain to apply.+ if module.is_file() and not has_chain:+ findings.append(+ {+ "service": service.name,+ "kind": "bootstrap-without-chain",+ "detail": f"ships {BOOTSTRAP} but has no alembic.ini; it would exit 1 on every start",+ }+ )+ if not has_chain:+ continue++ dockerfile = service / "Dockerfile"+ commands = _command_lines(dockerfile.read_text(encoding="utf-8")) if dockerfile.is_file() else ""+ block = _compose_block(compose_text, service.name)++ row = {+ "service": service.name,+ "module": module.is_file(),+ "in_sync": reference is not None and module.is_file() and module.read_bytes() == reference,+ "invoked": INVOCATION in commands,+ "owner_credential": bool(re.search(r"^\s+\w*DATABASE_MIGRATION_URL:", block, re.M)),+ "healthcheck": "healthcheck:" in block,+ "in_compose": bool(block),+ }+ services.append(row)++ if not row["module"]:+ findings.append(+ {+ "service": service.name,+ "kind": "no-bootstrap-module",+ "detail": f"{BOOTSTRAP} is missing; nothing applies this service's chain",+ }+ )+ elif not row["in_sync"]:+ findings.append(+ {+ "service": service.name,+ "kind": "bootstrap-module-drift",+ "detail": f"{BOOTSTRAP} differs from services/{REFERENCE_SERVICE}/{BOOTSTRAP}",+ }+ )+ if not row["invoked"]:+ findings.append(+ {+ "service": service.name,+ "kind": "bootstrap-not-invoked",+ "detail": f"no CMD/ENTRYPOINT runs {INVOCATION}; this service would boot on whatever schema is there",+ }+ )+ if row["in_compose"] and not row["owner_credential"]:+ findings.append(+ {+ "service": service.name,+ "kind": "no-owner-credential",+ "detail": (+ "no *DATABASE_MIGRATION_URL in its compose environment; the chain would be "+ "applied as the runtime role and fail on the first CREATE TABLE"+ ),+ }+ )+ if row["in_compose"] and not row["healthcheck"]:+ findings.append(+ {+ "service": service.name,+ "kind": "no-healthcheck",+ "detail": "no compose healthcheck, so a container that refused to start still reads as running",+ }+ )++ return {"root": str(REPO_ROOT), "services": services, "findings": findings}+++# ---------------------------------------------------------------------------+# Self-test+# ---------------------------------------------------------------------------+++def _detects_uninvoked_chain() -> list[tuple[str, bool]]:+ """Prove each kind is still detected, on a tree built for the purpose."""+ import tempfile++ global REPO_ROOT, SERVICES_DIR, COMPOSE # noqa: PLW0603 — restored below+ saved = (REPO_ROOT, SERVICES_DIR, COMPOSE)+ try:+ with tempfile.TemporaryDirectory() as raw:+ root = pathlib.Path(raw)+ service = root / "services" / "widgetry"+ (service / "app").mkdir(parents=True)+ (service / "alembic.ini").write_text("[alembic]\nscript_location = alembic\n")+ # The defect exactly as it was: a chain, and a bare uvicorn CMD.+ (service / "Dockerfile").write_text('FROM python:3.11-slim\nCMD ["uvicorn", "app.main:app", "--port", "8000"]\n')+ (root / "docker-compose.yml").write_text("services:\n widgetry:\n image: x\n")++ # A second service with the bootstrap and no chain — the other+ # direction.+ stray = root / "services" / "strayville"+ (stray / "app").mkdir(parents=True)+ (stray / "app" / "_migrate.py").write_text("# stub\n")++ REPO_ROOT, SERVICES_DIR, COMPOSE = root, root / "services", root / "docker-compose.yml"+ kinds = {f["kind"] for f in scan()["findings"]}+ finally:+ REPO_ROOT, SERVICES_DIR, COMPOSE = saved++ return [+ ("detects a chain with no bootstrap module", "no-bootstrap-module" in kinds),+ ("detects a bootstrap module the Dockerfile never runs", "bootstrap-not-invoked" in kinds),+ ("detects a service with no owner credential", "no-owner-credential" in kinds),+ ("detects a service with no compose healthcheck", "no-healthcheck" in kinds),+ ("detects a bootstrap module with no chain to apply", "bootstrap-without-chain" in kinds),+ ]+++def main() -> int:+ parser = argparse.ArgumentParser(description="alembic chains must be invoked on the way up")+ parser.add_argument("--check", action="store_true", help="exit non-zero on findings")+ parser.add_argument("--json", action="store_true", help="machine-readable output")+ parser.add_argument("--self-test", action="store_true", help="prove this gate still detects drift")+ args = parser.parse_args()++ if args.self_test:+ return self_test_main(pathlib.Path(__file__).name, ["--check"], extra=_detects_uninvoked_chain())++ result = scan()+ if "error" in result:+ print(f"migration-bootstrap: {result['error']}", file=sys.stderr)+ return 2++ if args.json:+ print(json.dumps(result, indent=2, sort_keys=True))+ else:+ print(f"migration-bootstrap: root {result['root']}")+ print(f"{'service':<14} {'_migrate':<9} {'in-sync':<8} {'invoked':<8} {'owner':<6} {'health':<7}")+ print(f"{'-' * 14} {'-' * 9} {'-' * 8} {'-' * 8} {'-' * 6} {'-' * 7}")+ for row in result["services"]:+ mark = lambda flag: "yes" if flag else "NO" # noqa: E731 — table formatting only+ print(+ f"{row['service']:<14} {mark(row['module']):<9} {mark(row['in_sync']):<8} "+ f"{mark(row['invoked']):<8} {mark(row['owner_credential']):<6} {mark(row['healthcheck']):<7}"+ )++ if not result["services"]:+ # The corpus refusal. No service with an alembic.ini means this gate+ # inspected nothing, and "nothing to check" must not print the same+ # verdict as "everything checked out".+ print(+ f"\nmigration-bootstrap: found no service with an alembic.ini under {SERVICES_DIR}. "+ f"Four services own their schema that way; zero found means the root above is not "+ f"the tree you meant.",+ file=sys.stderr,+ )+ return 1++ if not result["findings"]:+ print(f"\nOK: {len(result['services'])} alembic-managed service(s) apply their chain before serving")+ return 0++ print(f"\nFAIL: {len(result['findings'])} finding(s):", file=sys.stderr)+ for finding in result["findings"]:+ print(f" [{finding['kind']}] {finding['service']}: {finding['detail']}", file=sys.stderr)+ print(+ f"\nFix: copy services/{REFERENCE_SERVICE}/{BOOTSTRAP} into the service, and make its "+ f"Dockerfile CMD run `python -m {INVOCATION} -- <server command>`.",+ file=sys.stderr,+ )+ return 1 if args.check else 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/scripts/mypy_baseline.json b/scripts/mypy_baseline.jsonindex 06122f20a..a34a48c4e 100644--- a/scripts/mypy_baseline.json+++ b/scripts/mypy_baseline.json@@ -510,10 +510,6 @@"tests/test_event_warehouse_registry.py": {"arg-type": 3},- "tests/test_health_probes.py": {- "misc": 6,- "valid-type": 1- },"tests/test_hunts_tenant_isolation.py": {"misc": 1},diff --git a/services/actions/app/_health.py b/services/actions/app/_health.pyindex 974269731..0acb9e4da 100644--- a/services/actions/app/_health.py+++ b/services/actions/app/_health.py@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /``mark_not_ready()`` from those handlers respectively.+Readiness is a question, not a latch+------------------------------------++``mark_ready()`` answers "did startup finish", and for a service+whose only job is to answer HTTP that is the whole story. For a+service whose job is to *consume a topic* it is not: three+services in this repository created the consumer as a+fire-and-forget ``asyncio.create_task`` and called+``mark_ready()`` on the next line, so readiness recorded that the+task had been **created**, never that it was still running.++UEBA is what that cost. Its handler raised+``UndefinedColumnError`` on the first scoreable event, the+exception left the ``async for``, the ``finally`` stopped the+consumer, and the task ended — holding the exception, which+nothing retrieved because the task was parked in a module global+and so never garbage-collected. The container stayed ``running``+with restarts 0, ``/health`` returned 200 and ``/readyz``+returned 200, permanently. A consumer that has silently detached+is indistinguishable from an idle one, and the whole point of a+readiness probe is to tell those two apart.++So a service that owns a subscription registers a probe for it+with :func:`register_subscription`, and ``/readyz`` evaluates+every probe on each request. A probe that returns false — or+raises — is reported as detached and ``/readyz`` answers 503 with+the names, which is a readiness failure rather than a liveness+one on purpose: the orchestrator should stop routing to a+half-working replica, and killing the process would only hide a+schema or broker fault behind a restart loop.++``/livez`` deliberately does **not** consult the probes. It+answers "is the interpreter responsive", and wiring a detached+consumer into it would make every orchestrator restart the pod on+a fault a restart cannot fix.+This module is intentionally dependency-free (only FastAPI) so itcan be vendored into every service tree without import-graphside-effects. Each service ships its own copy under``services/<svc>/app/_health.py`` because services don't share a-Python path — keep the file in sync via-``scripts/sync_health_module.py`` (CI gate).+Python path; ``scripts/audit_health_probes.py --check`` is the CI+gate that every service has one and wires it in."""from __future__ import annotations+import jsonfrom collections.abc import Callablefrom typing import Anyfrom fastapi import FastAPI, Response+#: Where the probes registered against an app are kept. Read by ``/readyz``,+#: written by :func:`register_subscription`. On ``app.state`` rather than in a+#: module global because a module global is shared by every app in the+#: process, which makes two test clients interfere.+_STATE_ATTR = "health_subscriptions"+++def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:+ """Names of the subscriptions currently reporting not-attached.++ A probe that raises counts as detached. It is asking a live object+ whether it is still working; if that question itself fails, the honest+ answer is not "yes".+ """+ detached: list[str] = []+ for name, probe in sorted(probes.items()):+ try:+ attached = bool(probe())+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one+ attached = False+ if not attached:+ detached.append(name)+ return detached+++def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:+ """Make ``/readyz`` depend on *probe* returning true.++ Call this once per subscription the service owns, after+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —+ it runs on every ``/readyz`` request — so it should read a flag the+ consumer maintains or check ``task.done()``, not round-trip to a broker.+ """+ probes = getattr(app.state, _STATE_ATTR, None)+ if probes is None: # pragma: no cover — install_health_routes was not called+ raise RuntimeError("register_subscription requires install_health_routes(app) first")+ probes[name] = probe+def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:"""Install ``/livez`` and ``/readyz`` on ``app``.@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[``mark_ready()`` is called — exactly the right behaviour for arolling deploy where the orchestrator should hold trafficuntil the pod's dependencies are connected.++ Startup having finished is necessary and not sufficient: any+ subscription registered with :func:`register_subscription`+ must also still be attached. See the module docstring."""state: dict[str, bool] = {"ready": False}+ probes: dict[str, Callable[[], bool]] = {}+ setattr(app.state, _STATE_ATTR, probes)@app.get("/livez",@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[include_in_schema=False,)async def _readyz() -> Response:- if state["ready"]:- return Response(- content='{"status":"ready","service":"' + service_name + '"}',- media_type="application/json",- status_code=200,- )+ body: dict[str, Any] = {"status": "ready", "service": service_name}+ status_code = 200++ if not state["ready"]:+ body["status"] = "starting"+ status_code = 503+ else:+ detached = _probe_detached(probes)+ if detached:+ body["status"] = "degraded"+ body["detached"] = detached+ status_code = 503++ # Named subscriptions are reported either way, so a 200 says which+ # ones were checked rather than only that nothing was wrong. A probe+ # list that is unexpectedly empty is then visible in the same place+ # the verdict is.+ if probes:+ body["subscriptions"] = sorted(probes)+return Response(- content='{"status":"starting","service":"' + service_name + '"}',+ content=json.dumps(body),media_type="application/json",- status_code=503,+ status_code=status_code,)def mark_ready() -> None:diff --git a/services/agents/app/_health.py b/services/agents/app/_health.pyindex 974269731..0acb9e4da 100644--- a/services/agents/app/_health.py+++ b/services/agents/app/_health.py@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /``mark_not_ready()`` from those handlers respectively.+Readiness is a question, not a latch+------------------------------------++``mark_ready()`` answers "did startup finish", and for a service+whose only job is to answer HTTP that is the whole story. For a+service whose job is to *consume a topic* it is not: three+services in this repository created the consumer as a+fire-and-forget ``asyncio.create_task`` and called+``mark_ready()`` on the next line, so readiness recorded that the+task had been **created**, never that it was still running.++UEBA is what that cost. Its handler raised+``UndefinedColumnError`` on the first scoreable event, the+exception left the ``async for``, the ``finally`` stopped the+consumer, and the task ended — holding the exception, which+nothing retrieved because the task was parked in a module global+and so never garbage-collected. The container stayed ``running``+with restarts 0, ``/health`` returned 200 and ``/readyz``+returned 200, permanently. A consumer that has silently detached+is indistinguishable from an idle one, and the whole point of a+readiness probe is to tell those two apart.++So a service that owns a subscription registers a probe for it+with :func:`register_subscription`, and ``/readyz`` evaluates+every probe on each request. A probe that returns false — or+raises — is reported as detached and ``/readyz`` answers 503 with+the names, which is a readiness failure rather than a liveness+one on purpose: the orchestrator should stop routing to a+half-working replica, and killing the process would only hide a+schema or broker fault behind a restart loop.++``/livez`` deliberately does **not** consult the probes. It+answers "is the interpreter responsive", and wiring a detached+consumer into it would make every orchestrator restart the pod on+a fault a restart cannot fix.+This module is intentionally dependency-free (only FastAPI) so itcan be vendored into every service tree without import-graphside-effects. Each service ships its own copy under``services/<svc>/app/_health.py`` because services don't share a-Python path — keep the file in sync via-``scripts/sync_health_module.py`` (CI gate).+Python path; ``scripts/audit_health_probes.py --check`` is the CI+gate that every service has one and wires it in."""from __future__ import annotations+import jsonfrom collections.abc import Callablefrom typing import Anyfrom fastapi import FastAPI, Response+#: Where the probes registered against an app are kept. Read by ``/readyz``,+#: written by :func:`register_subscription`. On ``app.state`` rather than in a+#: module global because a module global is shared by every app in the+#: process, which makes two test clients interfere.+_STATE_ATTR = "health_subscriptions"+++def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:+ """Names of the subscriptions currently reporting not-attached.++ A probe that raises counts as detached. It is asking a live object+ whether it is still working; if that question itself fails, the honest+ answer is not "yes".+ """+ detached: list[str] = []+ for name, probe in sorted(probes.items()):+ try:+ attached = bool(probe())+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one+ attached = False+ if not attached:+ detached.append(name)+ return detached+++def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:+ """Make ``/readyz`` depend on *probe* returning true.++ Call this once per subscription the service owns, after+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —+ it runs on every ``/readyz`` request — so it should read a flag the+ consumer maintains or check ``task.done()``, not round-trip to a broker.+ """+ probes = getattr(app.state, _STATE_ATTR, None)+ if probes is None: # pragma: no cover — install_health_routes was not called+ raise RuntimeError("register_subscription requires install_health_routes(app) first")+ probes[name] = probe+def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:"""Install ``/livez`` and ``/readyz`` on ``app``.@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[``mark_ready()`` is called — exactly the right behaviour for arolling deploy where the orchestrator should hold trafficuntil the pod's dependencies are connected.++ Startup having finished is necessary and not sufficient: any+ subscription registered with :func:`register_subscription`+ must also still be attached. See the module docstring."""state: dict[str, bool] = {"ready": False}+ probes: dict[str, Callable[[], bool]] = {}+ setattr(app.state, _STATE_ATTR, probes)@app.get("/livez",@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[include_in_schema=False,)async def _readyz() -> Response:- if state["ready"]:- return Response(- content='{"status":"ready","service":"' + service_name + '"}',- media_type="application/json",- status_code=200,- )+ body: dict[str, Any] = {"status": "ready", "service": service_name}+ status_code = 200++ if not state["ready"]:+ body["status"] = "starting"+ status_code = 503+ else:+ detached = _probe_detached(probes)+ if detached:+ body["status"] = "degraded"+ body["detached"] = detached+ status_code = 503++ # Named subscriptions are reported either way, so a 200 says which+ # ones were checked rather than only that nothing was wrong. A probe+ # list that is unexpectedly empty is then visible in the same place+ # the verdict is.+ if probes:+ body["subscriptions"] = sorted(probes)+return Response(- content='{"status":"starting","service":"' + service_name + '"}',+ content=json.dumps(body),media_type="application/json",- status_code=503,+ status_code=status_code,)def mark_ready() -> None:diff --git a/services/agents/app/main.py b/services/agents/app/main.pyindex f434c64ec..5de9c6269 100644--- a/services/agents/app/main.py+++ b/services/agents/app/main.py@@ -7,7 +7,7 @@ import structlogfrom fastapi import FastAPIfrom fastapi.middleware.cors import CORSMiddleware-from app._health import install_health_routes+from app._health import install_health_routes, register_subscriptionfrom app.api.contextual import router as contextual_routerfrom app.api.copilot import router as copilot_routerfrom app.api.explain import router as explain_router@@ -32,6 +32,18 @@ from app.workers.fused_alert_consumer import FusedAlertTriageWorker, worker_enablogger = structlog.get_logger()+def _log_triage_worker_exit(task: asyncio.Task) -> None:+ """Report the auto-triage worker's exit instead of losing it in the task."""+ if task.cancelled():+ logger.info("auto_triage_worker.cancelled")+ return+ exc = task.exception()+ if exc is None:+ logger.error("auto_triage_worker.exited", detail="consume loop returned; fused alerts are no longer triaged")+ return+ logger.error("auto_triage_worker.died", error=str(exc), error_type=type(exc).__name__, exc_info=exc)++@asynccontextmanagerasync def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:"""Load resources on startup."""@@ -100,7 +112,15 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:business_context=applier,)app.state.triage_worker = triage_worker- app.state.triage_worker_task = asyncio.create_task(triage_worker.start())+ task = asyncio.create_task(triage_worker.start())+ # Same pair as fusion and ueba: retrieve the task's outcome so a+ # dead worker is in the log, and make readiness depend on the+ # subscription rather than on this block having run. Without the+ # probe, /readyz answered 200 for a worker whose loop had exited —+ # and nothing auto-triages a fused alert after that.+ task.add_done_callback(_log_triage_worker_exit)+ app.state.triage_worker_task = task+ register_subscription(app, triage_worker.topic, lambda: triage_worker.attached)logger.info("auto_triage_worker.enabled")except Exception as exc: # noqa: BLE001 — never block API startuplogger.warning("auto_triage_worker.start_failed", error=str(exc))diff --git a/services/agents/app/workers/fused_alert_consumer.py b/services/agents/app/workers/fused_alert_consumer.pyindex 620d95e0f..6a3df7abf 100644--- a/services/agents/app/workers/fused_alert_consumer.py+++ b/services/agents/app/workers/fused_alert_consumer.py@@ -241,6 +241,7 @@ class FusedAlertTriageWorker:self._consumer: Any | None = Noneself._producer: Any | None = None # lazily created for the DLQself._running = False+ self._attached = False# Phase B4 — environment-specific noise reduction applied post-fusion →# pre-triage. None = disabled (no rules file / flag off).self._business_context = business_context@@ -260,6 +261,7 @@ class FusedAlertTriageWorker:)await self._consumer.start()self._running = True+ self._attached = Truelogger.info("auto_triage_worker.started", topic=self._topic, group=self._group_id)try:async for msg in self._consumer:@@ -269,12 +271,33 @@ class FusedAlertTriageWorker:# or after the poison alert is dead-lettered — so a crash between# inference and persistence replays the alert instead of losing# it, and offsets advance only on durable completion (issue #571).- await self._process_with_retry(msg.value)+ #+ # ``_process_with_retry`` catches and dead-letters rather than+ # raising, which is the reason this loop survives a poison+ # alert. The guard is here as well because that is a property+ # of another method: the loop must not depend on a callee+ # staying total, which is exactly how the ueba consumer came+ # to have no ``except`` on its only handler.+ try:+ await self._process_with_retry(msg.value)+ except Exception as exc: # noqa: BLE001 — one alert must not end the subscription+ _METRICS["errors"] += 1+ logger.error(+ "auto_triage_worker.handler_escaped",+ error=str(exc),+ error_type=type(exc).__name__,+ partition=getattr(msg, "partition", None),+ offset=getattr(msg, "offset", None),+ exc_info=True,+ )try:await self._consumer.commit()except Exception as commit_exc: # noqa: BLE001 — reprocess on restartlogger.warning("auto_triage_worker.commit_failed", error=str(commit_exc))finally:+ # Cleared before the teardown await, which can itself raise on a+ # broker that has gone away.+ self._attached = Falseawait self._consumer.stop()async def _process_with_retry(self, message: Any) -> bool:@@ -349,8 +372,19 @@ class FusedAlertTriageWorker:except Exception as exc: # noqa: BLE001 — never wedge the loop on a DLQ failurelogger.error("auto_triage_worker.dlq_send_failed", alert_id=alert_id, error=str(exc))+ @property+ def topic(self) -> str:+ """The topic this worker subscribes to (names its readiness probe)."""+ return self._topic++ @property+ def attached(self) -> bool:+ """Whether the consume loop is currently iterating the subscription."""+ return self._attached+async def stop(self) -> None:self._running = False+ self._attached = Falseif self._consumer is not None:await self._consumer.stop()self._consumer = Nonediff --git a/services/api/app/_health.py b/services/api/app/_health.pyindex 974269731..0acb9e4da 100644--- a/services/api/app/_health.py+++ b/services/api/app/_health.py@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /``mark_not_ready()`` from those handlers respectively.+Readiness is a question, not a latch+------------------------------------++``mark_ready()`` answers "did startup finish", and for a service+whose only job is to answer HTTP that is the whole story. For a+service whose job is to *consume a topic* it is not: three+services in this repository created the consumer as a+fire-and-forget ``asyncio.create_task`` and called+``mark_ready()`` on the next line, so readiness recorded that the+task had been **created**, never that it was still running.++UEBA is what that cost. Its handler raised+``UndefinedColumnError`` on the first scoreable event, the+exception left the ``async for``, the ``finally`` stopped the+consumer, and the task ended — holding the exception, which+nothing retrieved because the task was parked in a module global+and so never garbage-collected. The container stayed ``running``+with restarts 0, ``/health`` returned 200 and ``/readyz``+returned 200, permanently. A consumer that has silently detached+is indistinguishable from an idle one, and the whole point of a+readiness probe is to tell those two apart.++So a service that owns a subscription registers a probe for it+with :func:`register_subscription`, and ``/readyz`` evaluates+every probe on each request. A probe that returns false — or+raises — is reported as detached and ``/readyz`` answers 503 with+the names, which is a readiness failure rather than a liveness+one on purpose: the orchestrator should stop routing to a+half-working replica, and killing the process would only hide a+schema or broker fault behind a restart loop.++``/livez`` deliberately does **not** consult the probes. It+answers "is the interpreter responsive", and wiring a detached+consumer into it would make every orchestrator restart the pod on+a fault a restart cannot fix.+This module is intentionally dependency-free (only FastAPI) so itcan be vendored into every service tree without import-graphside-effects. Each service ships its own copy under``services/<svc>/app/_health.py`` because services don't share a-Python path — keep the file in sync via-``scripts/sync_health_module.py`` (CI gate).+Python path; ``scripts/audit_health_probes.py --check`` is the CI+gate that every service has one and wires it in."""from __future__ import annotations+import jsonfrom collections.abc import Callablefrom typing import Anyfrom fastapi import FastAPI, Response+#: Where the probes registered against an app are kept. Read by ``/readyz``,+#: written by :func:`register_subscription`. On ``app.state`` rather than in a+#: module global because a module global is shared by every app in the+#: process, which makes two test clients interfere.+_STATE_ATTR = "health_subscriptions"+++def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:+ """Names of the subscriptions currently reporting not-attached.++ A probe that raises counts as detached. It is asking a live object+ whether it is still working; if that question itself fails, the honest+ answer is not "yes".+ """+ detached: list[str] = []+ for name, probe in sorted(probes.items()):+ try:+ attached = bool(probe())+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one+ attached = False+ if not attached:+ detached.append(name)+ return detached+++def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:+ """Make ``/readyz`` depend on *probe* returning true.++ Call this once per subscription the service owns, after+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —+ it runs on every ``/readyz`` request — so it should read a flag the+ consumer maintains or check ``task.done()``, not round-trip to a broker.+ """+ probes = getattr(app.state, _STATE_ATTR, None)+ if probes is None: # pragma: no cover — install_health_routes was not called+ raise RuntimeError("register_subscription requires install_health_routes(app) first")+ probes[name] = probe+def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:"""Install ``/livez`` and ``/readyz`` on ``app``.@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[``mark_ready()`` is called — exactly the right behaviour for arolling deploy where the orchestrator should hold trafficuntil the pod's dependencies are connected.++ Startup having finished is necessary and not sufficient: any+ subscription registered with :func:`register_subscription`+ must also still be attached. See the module docstring."""state: dict[str, bool] = {"ready": False}+ probes: dict[str, Callable[[], bool]] = {}+ setattr(app.state, _STATE_ATTR, probes)@app.get("/livez",@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[include_in_schema=False,)async def _readyz() -> Response:- if state["ready"]:- return Response(- content='{"status":"ready","service":"' + service_name + '"}',- media_type="application/json",- status_code=200,- )+ body: dict[str, Any] = {"status": "ready", "service": service_name}+ status_code = 200++ if not state["ready"]:+ body["status"] = "starting"+ status_code = 503+ else:+ detached = _probe_detached(probes)+ if detached:+ body["status"] = "degraded"+ body["detached"] = detached+ status_code = 503++ # Named subscriptions are reported either way, so a 200 says which+ # ones were checked rather than only that nothing was wrong. A probe+ # list that is unexpectedly empty is then visible in the same place+ # the verdict is.+ if probes:+ body["subscriptions"] = sorted(probes)+return Response(- content='{"status":"starting","service":"' + service_name + '"}',+ content=json.dumps(body),media_type="application/json",- status_code=503,+ status_code=status_code,)def mark_ready() -> None:diff --git a/services/api/tests/test_health_probes.py b/services/api/tests/test_health_probes.pyindex 37f51e968..d0f83b69a 100644--- a/services/api/tests/test_health_probes.py+++ b/services/api/tests/test_health_probes.py@@ -19,12 +19,20 @@ canonical copy.from __future__ import annotations-from app._health import install_health_routes+from collections.abc import Callable++import pytest+from app._health import install_health_routes, register_subscriptionfrom fastapi import FastAPIfrom fastapi.testclient import TestClient-def _build_app() -> tuple[FastAPI, callable, callable]:+def _build_app() -> tuple[FastAPI, Callable[[], None], Callable[[], None]]:+ # `callable` (the builtin) was used here as a type, which is not one:+ # mypy read every `mark_ready()` in this file as calling a value of type+ # `callable?` and recorded six `misc` findings plus a `valid-type` for+ # the annotation. Adding tests below made the count grow rather than the+ # cause get fixed.app = FastAPI()mark_ready, mark_not_ready = install_health_routes(app, service_name="aisoc-test")return app, mark_ready, mark_not_ready@@ -89,6 +97,97 @@ def test_readyz_can_recover_after_not_ready() -> None:assert client.get("/readyz").status_code == 200+class TestSubscriptionProbes:+ """Readiness has to answer "is it running", not "was it started".++ UEBA is what made this necessary. Its consumer was created as a+ fire-and-forget ``asyncio.create_task`` and ``mark_ready()`` was called+ on the next line, so readiness recorded that the task had been created.+ The first scoreable event raised ``UndefinedColumnError``, the loop+ exited, and ``/readyz`` answered 200 for as long as the container was+ left up. Two other services had the same pairing.+ """++ def test_ready_stays_200_while_the_subscription_is_attached(self) -> None:+ app, mark_ready, _ = _build_app()+ register_subscription(app, "aisoc.raw_events", lambda: True)+ mark_ready()++ response = TestClient(app).get("/readyz")++ assert response.status_code == 200+ # Named on the way past, so a 200 says what it checked rather than+ # only that nothing was wrong.+ assert response.json()["subscriptions"] == ["aisoc.raw_events"]++ def test_detached_subscription_makes_readiness_503(self) -> None:+ app, mark_ready, _ = _build_app()+ attached = {"value": True}+ register_subscription(app, "aisoc.raw_events", lambda: attached["value"])+ mark_ready()+ client = TestClient(app)+ assert client.get("/readyz").status_code == 200++ attached["value"] = False++ response = client.get("/readyz")+ assert response.status_code == 503+ assert response.json()["status"] == "degraded"+ assert response.json()["detached"] == ["aisoc.raw_events"]++ def test_a_probe_that_raises_counts_as_detached(self) -> None:+ """Fail closed. The probe asks a live object whether it still+ works; if that question itself raises, the answer is not "yes"."""++ def explode() -> bool:+ raise RuntimeError("consumer object is gone")++ app, mark_ready, _ = _build_app()+ register_subscription(app, "aisoc.alerts.fused", explode)+ mark_ready()++ response = TestClient(app).get("/readyz")++ assert response.status_code == 503+ assert response.json()["detached"] == ["aisoc.alerts.fused"]++ def test_livez_ignores_a_detached_subscription(self) -> None:+ """Deliberate. Liveness failing tells an orchestrator to restart the+ pod, and a restart does not fix a missing column or a revoked grant+ — it just hides it behind a crash loop."""+ app, mark_ready, _ = _build_app()+ register_subscription(app, "aisoc.raw_events", lambda: False)+ mark_ready()++ assert TestClient(app).get("/livez").status_code == 200++ def test_probes_are_not_consulted_before_startup_finishes(self) -> None:+ """An attached consumer does not make a half-started service ready."""+ app, _, _ = _build_app()+ register_subscription(app, "aisoc.raw_events", lambda: True)++ response = TestClient(app).get("/readyz")++ assert response.status_code == 503+ assert response.json()["status"] == "starting"++ def test_registration_requires_the_routes_to_be_installed(self) -> None:+ with pytest.raises(RuntimeError):+ register_subscription(FastAPI(), "aisoc.raw_events", lambda: True)++ def test_two_apps_do_not_share_a_registry(self) -> None:+ """The registry hangs off ``app.state``, not a module global: two+ test clients in one process must not see each other's probes."""+ first, first_ready, _ = _build_app()+ second, second_ready, _ = _build_app()+ register_subscription(first, "first.topic", lambda: False)+ first_ready()+ second_ready()++ assert TestClient(first).get("/readyz").status_code == 503+ assert TestClient(second).get("/readyz").status_code == 200++def test_probes_excluded_from_openapi() -> None:"""``/livez`` and ``/readyz`` are operator-facing; we don't wantthem spamming the SDK / OpenAPI surface that the rest of thediff --git a/services/connectors/app/_health.py b/services/connectors/app/_health.pyindex 974269731..0acb9e4da 100644--- a/services/connectors/app/_health.py+++ b/services/connectors/app/_health.py@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /``mark_not_ready()`` from those handlers respectively.+Readiness is a question, not a latch+------------------------------------++``mark_ready()`` answers "did startup finish", and for a service+whose only job is to answer HTTP that is the whole story. For a+service whose job is to *consume a topic* it is not: three+services in this repository created the consumer as a+fire-and-forget ``asyncio.create_task`` and called+``mark_ready()`` on the next line, so readiness recorded that the+task had been **created**, never that it was still running.++UEBA is what that cost. Its handler raised+``UndefinedColumnError`` on the first scoreable event, the+exception left the ``async for``, the ``finally`` stopped the+consumer, and the task ended — holding the exception, which+nothing retrieved because the task was parked in a module global+and so never garbage-collected. The container stayed ``running``+with restarts 0, ``/health`` returned 200 and ``/readyz``+returned 200, permanently. A consumer that has silently detached+is indistinguishable from an idle one, and the whole point of a+readiness probe is to tell those two apart.++So a service that owns a subscription registers a probe for it+with :func:`register_subscription`, and ``/readyz`` evaluates+every probe on each request. A probe that returns false — or+raises — is reported as detached and ``/readyz`` answers 503 with+the names, which is a readiness failure rather than a liveness+one on purpose: the orchestrator should stop routing to a+half-working replica, and killing the process would only hide a+schema or broker fault behind a restart loop.++``/livez`` deliberately does **not** consult the probes. It+answers "is the interpreter responsive", and wiring a detached+consumer into it would make every orchestrator restart the pod on+a fault a restart cannot fix.+This module is intentionally dependency-free (only FastAPI) so itcan be vendored into every service tree without import-graphside-effects. Each service ships its own copy under``services/<svc>/app/_health.py`` because services don't share a-Python path — keep the file in sync via-``scripts/sync_health_module.py`` (CI gate).+Python path; ``scripts/audit_health_probes.py --check`` is the CI+gate that every service has one and wires it in."""from __future__ import annotations+import jsonfrom collections.abc import Callablefrom typing import Anyfrom fastapi import FastAPI, Response+#: Where the probes registered against an app are kept. Read by ``/readyz``,+#: written by :func:`register_subscription`. On ``app.state`` rather than in a+#: module global because a module global is shared by every app in the+#: process, which makes two test clients interfere.+_STATE_ATTR = "health_subscriptions"+++def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:+ """Names of the subscriptions currently reporting not-attached.++ A probe that raises counts as detached. It is asking a live object+ whether it is still working; if that question itself fails, the honest+ answer is not "yes".+ """+ detached: list[str] = []+ for name, probe in sorted(probes.items()):+ try:+ attached = bool(probe())+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one+ attached = False+ if not attached:+ detached.append(name)+ return detached+++def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:+ """Make ``/readyz`` depend on *probe* returning true.++ Call this once per subscription the service owns, after+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —+ it runs on every ``/readyz`` request — so it should read a flag the+ consumer maintains or check ``task.done()``, not round-trip to a broker.+ """+ probes = getattr(app.state, _STATE_ATTR, None)+ if probes is None: # pragma: no cover — install_health_routes was not called+ raise RuntimeError("register_subscription requires install_health_routes(app) first")+ probes[name] = probe+def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:"""Install ``/livez`` and ``/readyz`` on ``app``.@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[``mark_ready()`` is called — exactly the right behaviour for arolling deploy where the orchestrator should hold trafficuntil the pod's dependencies are connected.++ Startup having finished is necessary and not sufficient: any+ subscription registered with :func:`register_subscription`+ must also still be attached. See the module docstring."""state: dict[str, bool] = {"ready": False}+ probes: dict[str, Callable[[], bool]] = {}+ setattr(app.state, _STATE_ATTR, probes)@app.get("/livez",@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[include_in_schema=False,)async def _readyz() -> Response:- if state["ready"]:- return Response(- content='{"status":"ready","service":"' + service_name + '"}',- media_type="application/json",- status_code=200,- )+ body: dict[str, Any] = {"status": "ready", "service": service_name}+ status_code = 200++ if not state["ready"]:+ body["status"] = "starting"+ status_code = 503+ else:+ detached = _probe_detached(probes)+ if detached:+ body["status"] = "degraded"+ body["detached"] = detached+ status_code = 503++ # Named subscriptions are reported either way, so a 200 says which+ # ones were checked rather than only that nothing was wrong. A probe+ # list that is unexpectedly empty is then visible in the same place+ # the verdict is.+ if probes:+ body["subscriptions"] = sorted(probes)+return Response(- content='{"status":"starting","service":"' + service_name + '"}',+ content=json.dumps(body),media_type="application/json",- status_code=503,+ status_code=status_code,)def mark_ready() -> None:diff --git a/services/fusion/app/_health.py b/services/fusion/app/_health.pyindex 974269731..0acb9e4da 100644--- a/services/fusion/app/_health.py+++ b/services/fusion/app/_health.py@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /``mark_not_ready()`` from those handlers respectively.+Readiness is a question, not a latch+------------------------------------++``mark_ready()`` answers "did startup finish", and for a service+whose only job is to answer HTTP that is the whole story. For a+service whose job is to *consume a topic* it is not: three+services in this repository created the consumer as a+fire-and-forget ``asyncio.create_task`` and called+``mark_ready()`` on the next line, so readiness recorded that the+task had been **created**, never that it was still running.++UEBA is what that cost. Its handler raised+``UndefinedColumnError`` on the first scoreable event, the+exception left the ``async for``, the ``finally`` stopped the+consumer, and the task ended — holding the exception, which+nothing retrieved because the task was parked in a module global+and so never garbage-collected. The container stayed ``running``+with restarts 0, ``/health`` returned 200 and ``/readyz``+returned 200, permanently. A consumer that has silently detached+is indistinguishable from an idle one, and the whole point of a+readiness probe is to tell those two apart.++So a service that owns a subscription registers a probe for it+with :func:`register_subscription`, and ``/readyz`` evaluates+every probe on each request. A probe that returns false — or+raises — is reported as detached and ``/readyz`` answers 503 with+the names, which is a readiness failure rather than a liveness+one on purpose: the orchestrator should stop routing to a+half-working replica, and killing the process would only hide a+schema or broker fault behind a restart loop.++``/livez`` deliberately does **not** consult the probes. It+answers "is the interpreter responsive", and wiring a detached+consumer into it would make every orchestrator restart the pod on+a fault a restart cannot fix.+This module is intentionally dependency-free (only FastAPI) so itcan be vendored into every service tree without import-graphside-effects. Each service ships its own copy under``services/<svc>/app/_health.py`` because services don't share a-Python path — keep the file in sync via-``scripts/sync_health_module.py`` (CI gate).+Python path; ``scripts/audit_health_probes.py --check`` is the CI+gate that every service has one and wires it in."""from __future__ import annotations+import jsonfrom collections.abc import Callablefrom typing import Anyfrom fastapi import FastAPI, Response+#: Where the probes registered against an app are kept. Read by ``/readyz``,+#: written by :func:`register_subscription`. On ``app.state`` rather than in a+#: module global because a module global is shared by every app in the+#: process, which makes two test clients interfere.+_STATE_ATTR = "health_subscriptions"+++def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:+ """Names of the subscriptions currently reporting not-attached.++ A probe that raises counts as detached. It is asking a live object+ whether it is still working; if that question itself fails, the honest+ answer is not "yes".+ """+ detached: list[str] = []+ for name, probe in sorted(probes.items()):+ try:+ attached = bool(probe())+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one+ attached = False+ if not attached:+ detached.append(name)+ return detached+++def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:+ """Make ``/readyz`` depend on *probe* returning true.++ Call this once per subscription the service owns, after+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —+ it runs on every ``/readyz`` request — so it should read a flag the+ consumer maintains or check ``task.done()``, not round-trip to a broker.+ """+ probes = getattr(app.state, _STATE_ATTR, None)+ if probes is None: # pragma: no cover — install_health_routes was not called+ raise RuntimeError("register_subscription requires install_health_routes(app) first")+ probes[name] = probe+def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:"""Install ``/livez`` and ``/readyz`` on ``app``.@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[``mark_ready()`` is called — exactly the right behaviour for arolling deploy where the orchestrator should hold trafficuntil the pod's dependencies are connected.++ Startup having finished is necessary and not sufficient: any+ subscription registered with :func:`register_subscription`+ must also still be attached. See the module docstring."""state: dict[str, bool] = {"ready": False}+ probes: dict[str, Callable[[], bool]] = {}+ setattr(app.state, _STATE_ATTR, probes)@app.get("/livez",@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[include_in_schema=False,)async def _readyz() -> Response:- if state["ready"]:- return Response(- content='{"status":"ready","service":"' + service_name + '"}',- media_type="application/json",- status_code=200,- )+ body: dict[str, Any] = {"status": "ready", "service": service_name}+ status_code = 200++ if not state["ready"]:+ body["status"] = "starting"+ status_code = 503+ else:+ detached = _probe_detached(probes)+ if detached:+ body["status"] = "degraded"+ body["detached"] = detached+ status_code = 503++ # Named subscriptions are reported either way, so a 200 says which+ # ones were checked rather than only that nothing was wrong. A probe+ # list that is unexpectedly empty is then visible in the same place+ # the verdict is.+ if probes:+ body["subscriptions"] = sorted(probes)+return Response(- content='{"status":"starting","service":"' + service_name + '"}',+ content=json.dumps(body),media_type="application/json",- status_code=503,+ status_code=status_code,)def mark_ready() -> None:diff --git a/services/fusion/app/main.py b/services/fusion/app/main.pyindex fd7e4ba3c..9a9164168 100644--- a/services/fusion/app/main.py+++ b/services/fusion/app/main.py@@ -5,7 +5,7 @@ from contextlib import asynccontextmanagerimport redis.asyncio as aioredisfrom fastapi import FastAPI-from app._health import install_health_routes+from app._health import install_health_routes, register_subscriptionfrom app.api.router import router, set_workerfrom app.core.config import settingsfrom app.core.logging import configure_logging, logger@@ -26,6 +26,23 @@ from app.services.windowed_detection import WindowedDetectionEnginefrom app.workers.consumer import FusionWorker+def _log_worker_exit(task: asyncio.Task) -> None:+ """Say why the consume loop stopped, whatever the reason.++ A worker that ends cleanly is as much of a problem as one that raises:+ either way nothing is consuming ``raw_events`` any more, and both used to+ be silent.+ """+ if task.cancelled():+ logger.info("fusion.worker_cancelled")+ return+ exc = task.exception()+ if exc is None:+ logger.error("fusion.worker_exited", detail="consume loop returned; the subscription is gone")+ return+ logger.error("fusion.worker_died", error=str(exc), error_type=type(exc).__name__, exc_info=exc)++@asynccontextmanagerasync def lifespan(app: FastAPI):configure_logging()@@ -114,8 +131,16 @@ async def lifespan(app: FastAPI):)set_worker(worker)- # Start Kafka worker as a background task+ # Start Kafka worker as a background task.+ #+ # The done-callback is what makes a dead worker findable. The task is held+ # on app.state, so it is never garbage-collected and asyncio never emits+ # its "Task exception was never retrieved" warning — without this, a+ # consume loop that died took the reason with it. The matching readiness+ # probe below is what stops /readyz answering 200 afterwards.worker_task = asyncio.create_task(worker.start())+ worker_task.add_done_callback(_log_worker_exit)+ register_subscription(app, "alerts+raw_events", lambda: worker.attached)app.state.worker_task = worker_taskapp.state.redis = redis_clientdiff --git a/services/fusion/app/workers/consumer.py b/services/fusion/app/workers/consumer.pyindex 9c22423c1..dcc76a78d 100644--- a/services/fusion/app/workers/consumer.py+++ b/services/fusion/app/workers/consumer.py@@ -67,12 +67,24 @@ class FusionWorker:self._consumer: AIOKafkaConsumer | None = Noneself._producer: AIOKafkaProducer | None = Noneself._running = False+ self._attached = Falseself._flush_task: asyncio.Task | None = None@propertydef engine(self) -> FusionEngine:return self._engine+ @property+ def attached(self) -> bool:+ """Whether the consume loop is currently iterating the subscription.++ ``_running`` is a request ("keep going"); this is an observation ("it+ is going"). Readiness needs the second one: the worker was started as+ a fire-and-forget task and ``/readyz`` latched to 200 next to it, so a+ loop that exited left the service reporting ready forever.+ """+ return self._attached+async def start(self) -> None:topics = [settings.kafka_topic_alerts_raw]# Subscribe to raw_events when EITHER promotion or lake archival needs@@ -155,27 +167,44 @@ class FusionWorker:consumer = self._consumerif consumer is None: # pragma: no cover — unreachable via start()raise RuntimeError("fusion consume loop started before the Kafka consumer")- async for msg in consumer:- if not self._running:- break- try:- await self._process_message(msg.value, topic=msg.topic)- except Exception as exc:- _METRICS["errors"] += 1- logger.error("Failed to process message", error=str(exc), exc_info=True)- else:- # At-least-once: commit only after the message is handled- # (validated + laked + promoted, or dead-lettered). A crash- # before this line re-delivers the in-flight message on restart- # instead of losing it. Commit failure => reprocess on restart.+ # Attached for as long as this loop is iterating; read by the+ # readiness probe app/main.py registers. Cleared in the ``finally`` so+ # every exit path reports the same way, including one that raises.+ self._attached = True+ try:+ async for msg in consumer:+ if not self._running:+ breaktry:- await consumer.commit()- except Exception as commit_exc: # noqa: BLE001- logger.warning("fusion.commit_failed", error=str(commit_exc))- # Flush any stale lake batch so archival isn't stranded during a- # low-traffic window (batch fills by size OR age).- if self._lake is not None:- await self._lake.flush_if_stale()+ await self._process_message(msg.value, topic=msg.topic)+ except Exception as exc:+ _METRICS["errors"] += 1+ logger.error("Failed to process message", error=str(exc), exc_info=True)+ else:+ # At-least-once: commit only after the message is handled+ # (validated + laked + promoted, or dead-lettered). A crash+ # before this line re-delivers the in-flight message on restart+ # instead of losing it. Commit failure => reprocess on restart.+ try:+ await consumer.commit()+ except Exception as commit_exc: # noqa: BLE001+ logger.warning("fusion.commit_failed", error=str(commit_exc))+ # Flush any stale lake batch so archival isn't stranded during a+ # low-traffic window (batch fills by size OR age).+ #+ # Guarded, because this await sat outside the per-message+ # try/except: a ClickHouse outage raising here left the+ # ``async for`` entirely and stopped fusion consuming, with the+ # archive fault reported as nothing at all. Archival falling+ # behind must not cost the alert path.+ if self._lake is not None:+ try:+ await self._lake.flush_if_stale()+ except Exception as flush_exc: # noqa: BLE001 — the lake is not the alert path+ _METRICS["errors"] += 1+ logger.error("fusion.lake_flush_failed", error=str(flush_exc), exc_info=True)+ finally:+ self._attached = Falseasync def _dead_letter(self,diff --git a/services/honeytokens/Dockerfile b/services/honeytokens/Dockerfileindex fe6f67cda..8280f7cb2 100644--- a/services/honeytokens/Dockerfile+++ b/services/honeytokens/Dockerfile@@ -25,4 +25,17 @@ COPY app/ app/EXPOSE 8005-CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8005"]+# The chain runs on the way up, then this process is replaced by the server.+#+# Nothing invoked these four alembic chains. This was a bare `uvicorn` line, so+# the documented quickstart booted the service against an empty schema with+# none of its row-level-security policies installed, and every command in that+# sequence reported success. `app/_migrate.py` applies the chain as the owner+# credential, verifies it reached head, and refuses to start the server if it+# did not.+#+# `sh -c` because an exec-form CMD cannot run two things. Neither shell nor+# wrapper stays in the process tree: `sh -c` with a single command execs it,+# and `_migrate` hands over with `os.execvp`, so uvicorn ends up as PID 1 and+# receives SIGTERM directly rather than through a parent that would swallow it.+CMD ["sh", "-c", "python -m app._migrate -- uvicorn app.main:app --host 0.0.0.0 --port 8005"]diff --git a/services/honeytokens/alembic/env.py b/services/honeytokens/alembic/env.pyindex b8d41012e..f27d751cf 100644--- a/services/honeytokens/alembic/env.py+++ b/services/honeytokens/alembic/env.py@@ -157,9 +157,27 @@ def run_migrations_offline() -> None:context.run_migrations()+#: Serialises the four chains that share one database against each other.+#:+#: They are started simultaneously by ``docker compose up``, and while each+#: now owns its own version table, they do not own their own *database*: the+#: legacy-version probe below reads a table a sibling may be creating, and+#: ``000N_runtime_role_grants`` issues ``GRANT`` statements that take+#: catalog locks on ``pg_class``. Two chains arriving together deadlocked+#: rather than queued.+#:+#: ``pg_advisory_xact_lock`` and not the session form: it is released by the+#: commit or rollback that ends the upgrade, so a chain that dies mid-apply+#: cannot leave the other three waiting on a lock nobody holds a connection+#: for. The key is an arbitrary constant, shared by every chain on purpose —+#: the point is mutual exclusion, not per-chain exclusion.+MIGRATION_LOCK_KEY = 0x4149_5354++def do_run_migrations(connection): # type: ignore[no-untyped-def]context.configure(connection=connection, target_metadata=target_metadata, version_table=VERSION_TABLE)with context.begin_transaction():+ connection.execute(sa.text("SELECT pg_advisory_xact_lock(:key)"), {"key": MIGRATION_LOCK_KEY})# Inside ``begin_transaction()``, not before it. Alembic's# ``begin_transaction`` is a no-op when the connection is *already* in a# transaction, and in SQLAlchemy 2.0 any execute on a plain ``connect()``diff --git a/services/honeytokens/app/_health.py b/services/honeytokens/app/_health.pyindex 974269731..0acb9e4da 100644--- a/services/honeytokens/app/_health.py+++ b/services/honeytokens/app/_health.py@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /``mark_not_ready()`` from those handlers respectively.+Readiness is a question, not a latch+------------------------------------++``mark_ready()`` answers "did startup finish", and for a service+whose only job is to answer HTTP that is the whole story. For a+service whose job is to *consume a topic* it is not: three+services in this repository created the consumer as a+fire-and-forget ``asyncio.create_task`` and called+``mark_ready()`` on the next line, so readiness recorded that the+task had been **created**, never that it was still running.++UEBA is what that cost. Its handler raised+``UndefinedColumnError`` on the first scoreable event, the+exception left the ``async for``, the ``finally`` stopped the+consumer, and the task ended — holding the exception, which+nothing retrieved because the task was parked in a module global+and so never garbage-collected. The container stayed ``running``+with restarts 0, ``/health`` returned 200 and ``/readyz``+returned 200, permanently. A consumer that has silently detached+is indistinguishable from an idle one, and the whole point of a+readiness probe is to tell those two apart.++So a service that owns a subscription registers a probe for it+with :func:`register_subscription`, and ``/readyz`` evaluates+every probe on each request. A probe that returns false — or+raises — is reported as detached and ``/readyz`` answers 503 with+the names, which is a readiness failure rather than a liveness+one on purpose: the orchestrator should stop routing to a+half-working replica, and killing the process would only hide a+schema or broker fault behind a restart loop.++``/livez`` deliberately does **not** consult the probes. It+answers "is the interpreter responsive", and wiring a detached+consumer into it would make every orchestrator restart the pod on+a fault a restart cannot fix.+This module is intentionally dependency-free (only FastAPI) so itcan be vendored into every service tree without import-graphside-effects. Each service ships its own copy under``services/<svc>/app/_health.py`` because services don't share a-Python path — keep the file in sync via-``scripts/sync_health_module.py`` (CI gate).+Python path; ``scripts/audit_health_probes.py --check`` is the CI+gate that every service has one and wires it in."""from __future__ import annotations+import jsonfrom collections.abc import Callablefrom typing import Anyfrom fastapi import FastAPI, Response+#: Where the probes registered against an app are kept. Read by ``/readyz``,+#: written by :func:`register_subscription`. On ``app.state`` rather than in a+#: module global because a module global is shared by every app in the+#: process, which makes two test clients interfere.+_STATE_ATTR = "health_subscriptions"+++def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:+ """Names of the subscriptions currently reporting not-attached.++ A probe that raises counts as detached. It is asking a live object+ whether it is still working; if that question itself fails, the honest+ answer is not "yes".+ """+ detached: list[str] = []+ for name, probe in sorted(probes.items()):+ try:+ attached = bool(probe())+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one+ attached = False+ if not attached:+ detached.append(name)+ return detached+++def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:+ """Make ``/readyz`` depend on *probe* returning true.++ Call this once per subscription the service owns, after+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —+ it runs on every ``/readyz`` request — so it should read a flag the+ consumer maintains or check ``task.done()``, not round-trip to a broker.+ """+ probes = getattr(app.state, _STATE_ATTR, None)+ if probes is None: # pragma: no cover — install_health_routes was not called+ raise RuntimeError("register_subscription requires install_health_routes(app) first")+ probes[name] = probe+def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:"""Install ``/livez`` and ``/readyz`` on ``app``.@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[``mark_ready()`` is called — exactly the right behaviour for arolling deploy where the orchestrator should hold trafficuntil the pod's dependencies are connected.++ Startup having finished is necessary and not sufficient: any+ subscription registered with :func:`register_subscription`+ must also still be attached. See the module docstring."""state: dict[str, bool] = {"ready": False}+ probes: dict[str, Callable[[], bool]] = {}+ setattr(app.state, _STATE_ATTR, probes)@app.get("/livez",@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[include_in_schema=False,)async def _readyz() -> Response:- if state["ready"]:- return Response(- content='{"status":"ready","service":"' + service_name + '"}',- media_type="application/json",- status_code=200,- )+ body: dict[str, Any] = {"status": "ready", "service": service_name}+ status_code = 200++ if not state["ready"]:+ body["status"] = "starting"+ status_code = 503+ else:+ detached = _probe_detached(probes)+ if detached:+ body["status"] = "degraded"+ body["detached"] = detached+ status_code = 503++ # Named subscriptions are reported either way, so a 200 says which+ # ones were checked rather than only that nothing was wrong. A probe+ # list that is unexpectedly empty is then visible in the same place+ # the verdict is.+ if probes:+ body["subscriptions"] = sorted(probes)+return Response(- content='{"status":"starting","service":"' + service_name + '"}',+ content=json.dumps(body),media_type="application/json",- status_code=503,+ status_code=status_code,)def mark_ready() -> None:diff --git a/services/honeytokens/app/_migrate.py b/services/honeytokens/app/_migrate.pynew file mode 100644index 000000000..1297f113b--- /dev/null+++ b/services/honeytokens/app/_migrate.py@@ -0,0 +1,228 @@+"""Apply this service's alembic chain, then exec what comes after it.++Why this exists+---------------++Four services in this repository own their schema through alembic —+``ueba``, ``honeytokens``, ``purple-team`` and ``osquery-tls`` — and **no+compose path invoked any of them**. Their container command was a plain+``uvicorn``, so on the documented quickstart they booted healthy against+empty schemas with none of their row-level-security policies installed. The+only documented invocation was a manual step in a docs-portal quickstart+whose own "start the stack" step does not start three of the four services,+and whose first line failed outright.++That is how two services shipped with no tables while every command+reported success. The API does not have this problem because its chain runs+from its own process on the way up; this is the same idea for the four that+use alembic.++What it guarantees+------------------++**Idempotent.** ``alembic upgrade head`` on a database already at head+applies nothing. Running this on every container start is the point: there+is no separate "have you migrated yet" state for an operator to get wrong.++**Race-free.** Four chains start at once against one database. The chains+take a shared transaction-scoped advisory lock (``MIGRATION_LOCK_KEY`` in+each ``env.py``), so they queue instead of deadlocking on catalog locks.+This wrapper deliberately holds no lock of its own — a lock held here would+be a second mechanism to reason about, and it would not cover the window+that actually matters.++**Loud.** The service does not start unless the chain reached head. Not "the+command exited 0": alembic exits 0 when it applies *nothing*, which is+exactly what happened when four chains shared one ``alembic_version`` table+and the second one to run believed it was already done. So ``current`` is+read back and compared to ``heads``, and a mismatch — or an empty+``current``, meaning the chain never ran — stops the container with the+reason on stderr. A service booting on an empty schema is the failure this+replaces, and it must not be reachable by a wrapper that only checks an exit+code.++Why the owner credential, and why a subprocess+----------------------------------------------++Alembic issues DDL. The runtime role every service serves requests as holds+no ``CREATE`` on schema public, by design — that is what stops it turning+row-level security off — so the chain must be applied as the owner. Each+``env.py`` already resolves that (its own ``*_DATABASE_MIGRATION_URL``+first, then ``DATABASE_MIGRATION_URL``), and this runs alembic as a+subprocess precisely so ``env.py`` stays the single answer to "which+credential". A second resolution here would be a second thing to keep in+step, and the two would disagree the first time either changed.++This module is vendored per service because services do not share a Python+path. ``scripts/check_service_migration_bootstrap.py`` is the gate that each+of the four ships a copy, keeps them identical, and runs one on the way up.++Usage+-----++::++ python -m app._migrate # apply the chain, exit+ python -m app._migrate -- uvicorn app.main:app --port 8004+"""++from __future__ import annotations++import os+import subprocess+import sys+from pathlib import Path++#: Where ``alembic.ini`` sits relative to the working directory in every one+#: of the four images (``WORKDIR /app``, and the ini is copied to ``/app``).+ALEMBIC_INI = "alembic.ini"++#: Suffix that marks a DSN as the migration (owner) credential. Matches the+#: convention each ``env.py`` documents and ``scripts/check_runtime_db_role.py``+#: enforces on the deployment surfaces: a service-prefixed+#: ``<SVC>_DATABASE_MIGRATION_URL`` wins over the bare form.+MIGRATION_URL_SUFFIX = "DATABASE_MIGRATION_URL"+++def migration_credential_present(env: dict[str, str] | None = None) -> bool:+ """Whether an owner DSN is configured.++ Used only for the warning below. The chain is applied either way — a+ deployment that has not split the roles yet behaves as it did before —+ but it is announced, because a chain silently applied as the runtime+ credential fails on the first ``CREATE TABLE`` and the message names a+ permission rather than a configuration.+ """+ source = os.environ if env is None else env+ return any(name.endswith(MIGRATION_URL_SUFFIX) and value.strip() for name, value in source.items())+++def _alembic(args: list[str], *, capture: bool) -> subprocess.CompletedProcess[str]:+ return subprocess.run( # noqa: S603 — fixed argv, no shell+ [sys.executable, "-m", "alembic", "-c", ALEMBIC_INI, *args],+ capture_output=capture,+ text=True,+ check=False,+ )+++def _revisions(output: str) -> set[str]:+ """Revision ids in ``alembic current`` / ``alembic heads`` output.++ Both print ``<rev> (head)`` or ``<rev>``, one per line, with informational+ lines alembic sends to stderr. An empty set means the version table holds+ nothing — the chain has not run — which is the case this exists to catch+ and is therefore never treated as agreement.+ """+ found: set[str] = set()+ for line in output.splitlines():+ token = line.strip().split(" ")[0].strip()+ if token and not token.startswith(("INFO", "WARNING", "ERROR", "alembic:")):+ found.add(token)+ return found+++def apply_chain() -> int:+ """Apply the chain and verify it reached head. Returns an exit code."""+ if not Path(ALEMBIC_INI).is_file():+ print(+ f"migrate: {ALEMBIC_INI} not found in {Path.cwd()}. This service's image must "+ f"copy its alembic config and chain; see its Dockerfile.",+ file=sys.stderr,+ )+ return 1++ if not migration_credential_present():+ print(+ f"migrate: no *{MIGRATION_URL_SUFFIX} is set, so the chain will be applied as "+ f"the runtime credential. That works only while the two are the same role; "+ f"under the DML-only runtime role the first CREATE TABLE fails with "+ f"'permission denied for schema public'.",+ file=sys.stderr,+ )++ upgrade = _alembic(["upgrade", "head"], capture=False)+ if upgrade.returncode != 0:+ print(+ f"migrate: 'alembic upgrade head' failed (exit {upgrade.returncode}). Refusing to "+ f"start: this service would serve requests against whatever schema is there now.",+ file=sys.stderr,+ )+ return upgrade.returncode++ # Read back rather than trusting the exit code. `alembic upgrade head`+ # exits 0 having applied nothing when it believes the database is already+ # at head, and believing that wrongly is the documented way these four+ # services ended up with no tables.+ current = _alembic(["current"], capture=True)+ heads = _alembic(["heads"], capture=True)+ if current.returncode != 0 or heads.returncode != 0:+ print(+ "migrate: could not read back the applied revision "+ f"(current exit {current.returncode}, heads exit {heads.returncode}). "+ "Refusing to start: an unverified migration is the state this check exists "+ "to reject.\n"+ f"{(current.stderr or '') + (heads.stderr or '')}",+ file=sys.stderr,+ )+ return 1++ applied = _revisions(current.stdout)+ expected = _revisions(heads.stdout)++ if not expected:+ print(+ "migrate: this chain declares no head revision, so there is nothing to verify "+ "against. Refusing to start rather than reporting an empty chain as applied.",+ file=sys.stderr,+ )+ return 1++ if not applied:+ print(+ f"migrate: 'alembic upgrade head' reported success but the version table is "+ f"empty — the chain never ran. Expected {sorted(expected)}. Refusing to start "+ f"on an unmigrated schema.",+ file=sys.stderr,+ )+ return 1++ if not expected.issubset(applied):+ print(+ f"migrate: chain did not reach head. At {sorted(applied)}, head is " f"{sorted(expected)}. Refusing to start.",+ file=sys.stderr,+ )+ return 1++ print(f"migrate: schema at head {sorted(expected)}", file=sys.stderr)+ return 0+++def main(argv: list[str] | None = None) -> int:+ args = list(sys.argv[1:] if argv is None else argv)++ # Everything after `--` is the real service command. Exec'd rather than+ # spawned so the server keeps PID 1 and receives SIGTERM directly — a+ # Python parent waiting on a child swallows the orchestrator's stop+ # signal and turns every shutdown into a ten-second kill.+ command: list[str] = []+ if "--" in args:+ split = args.index("--")+ command = args[split + 1 :]+ args = args[:split]++ if args:+ print(f"migrate: unexpected arguments {args}; usage: python -m app._migrate [-- CMD ...]", file=sys.stderr)+ return 2++ code = apply_chain()+ if code != 0:+ return code++ if command:+ os.execvp(command[0], command) # noqa: S606 — replaces this process by design+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/services/mesh/app/_health.py b/services/mesh/app/_health.pyindex 974269731..0acb9e4da 100644--- a/services/mesh/app/_health.py+++ b/services/mesh/app/_health.py@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /``mark_not_ready()`` from those handlers respectively.+Readiness is a question, not a latch+------------------------------------++``mark_ready()`` answers "did startup finish", and for a service+whose only job is to answer HTTP that is the whole story. For a+service whose job is to *consume a topic* it is not: three+services in this repository created the consumer as a+fire-and-forget ``asyncio.create_task`` and called+``mark_ready()`` on the next line, so readiness recorded that the+task had been **created**, never that it was still running.++UEBA is what that cost. Its handler raised+``UndefinedColumnError`` on the first scoreable event, the+exception left the ``async for``, the ``finally`` stopped the+consumer, and the task ended — holding the exception, which+nothing retrieved because the task was parked in a module global+and so never garbage-collected. The container stayed ``running``+with restarts 0, ``/health`` returned 200 and ``/readyz``+returned 200, permanently. A consumer that has silently detached+is indistinguishable from an idle one, and the whole point of a+readiness probe is to tell those two apart.++So a service that owns a subscription registers a probe for it+with :func:`register_subscription`, and ``/readyz`` evaluates+every probe on each request. A probe that returns false — or+raises — is reported as detached and ``/readyz`` answers 503 with+the names, which is a readiness failure rather than a liveness+one on purpose: the orchestrator should stop routing to a+half-working replica, and killing the process would only hide a+schema or broker fault behind a restart loop.++``/livez`` deliberately does **not** consult the probes. It+answers "is the interpreter responsive", and wiring a detached+consumer into it would make every orchestrator restart the pod on+a fault a restart cannot fix.+This module is intentionally dependency-free (only FastAPI) so itcan be vendored into every service tree without import-graphside-effects. Each service ships its own copy under``services/<svc>/app/_health.py`` because services don't share a-Python path — keep the file in sync via-``scripts/sync_health_module.py`` (CI gate).+Python path; ``scripts/audit_health_probes.py --check`` is the CI+gate that every service has one and wires it in."""from __future__ import annotations+import jsonfrom collections.abc import Callablefrom typing import Anyfrom fastapi import FastAPI, Response+#: Where the probes registered against an app are kept. Read by ``/readyz``,+#: written by :func:`register_subscription`. On ``app.state`` rather than in a+#: module global because a module global is shared by every app in the+#: process, which makes two test clients interfere.+_STATE_ATTR = "health_subscriptions"+++def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:+ """Names of the subscriptions currently reporting not-attached.++ A probe that raises counts as detached. It is asking a live object+ whether it is still working; if that question itself fails, the honest+ answer is not "yes".+ """+ detached: list[str] = []+ for name, probe in sorted(probes.items()):+ try:+ attached = bool(probe())+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one+ attached = False+ if not attached:+ detached.append(name)+ return detached+++def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:+ """Make ``/readyz`` depend on *probe* returning true.++ Call this once per subscription the service owns, after+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —+ it runs on every ``/readyz`` request — so it should read a flag the+ consumer maintains or check ``task.done()``, not round-trip to a broker.+ """+ probes = getattr(app.state, _STATE_ATTR, None)+ if probes is None: # pragma: no cover — install_health_routes was not called+ raise RuntimeError("register_subscription requires install_health_routes(app) first")+ probes[name] = probe+def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:"""Install ``/livez`` and ``/readyz`` on ``app``.@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[``mark_ready()`` is called — exactly the right behaviour for arolling deploy where the orchestrator should hold trafficuntil the pod's dependencies are connected.++ Startup having finished is necessary and not sufficient: any+ subscription registered with :func:`register_subscription`+ must also still be attached. See the module docstring."""state: dict[str, bool] = {"ready": False}+ probes: dict[str, Callable[[], bool]] = {}+ setattr(app.state, _STATE_ATTR, probes)@app.get("/livez",@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[include_in_schema=False,)async def _readyz() -> Response:- if state["ready"]:- return Response(- content='{"status":"ready","service":"' + service_name + '"}',- media_type="application/json",- status_code=200,- )+ body: dict[str, Any] = {"status": "ready", "service": service_name}+ status_code = 200++ if not state["ready"]:+ body["status"] = "starting"+ status_code = 503+ else:+ detached = _probe_detached(probes)+ if detached:+ body["status"] = "degraded"+ body["detached"] = detached+ status_code = 503++ # Named subscriptions are reported either way, so a 200 says which+ # ones were checked rather than only that nothing was wrong. A probe+ # list that is unexpectedly empty is then visible in the same place+ # the verdict is.+ if probes:+ body["subscriptions"] = sorted(probes)+return Response(- content='{"status":"starting","service":"' + service_name + '"}',+ content=json.dumps(body),media_type="application/json",- status_code=503,+ status_code=status_code,)def mark_ready() -> None:diff --git a/services/osquery-tls/Dockerfile b/services/osquery-tls/Dockerfileindex 24191e267..bedcf7846 100644--- a/services/osquery-tls/Dockerfile+++ b/services/osquery-tls/Dockerfile@@ -35,4 +35,17 @@ COPY . .EXPOSE 9001-CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "9001"]+# The chain runs on the way up, then this process is replaced by the server.+#+# Nothing invoked these four alembic chains. This was a bare `uvicorn` line, so+# the documented quickstart booted the service against an empty schema with+# none of its row-level-security policies installed, and every command in that+# sequence reported success. `app/_migrate.py` applies the chain as the owner+# credential, verifies it reached head, and refuses to start the server if it+# did not.+#+# `sh -c` because an exec-form CMD cannot run two things. Neither shell nor+# wrapper stays in the process tree: `sh -c` with a single command execs it,+# and `_migrate` hands over with `os.execvp`, so uvicorn ends up as PID 1 and+# receives SIGTERM directly rather than through a parent that would swallow it.+CMD ["sh", "-c", "python -m app._migrate -- uvicorn app.main:app --host 0.0.0.0 --port 9001"]diff --git a/services/osquery-tls/app/_health.py b/services/osquery-tls/app/_health.pyindex 974269731..0acb9e4da 100644--- a/services/osquery-tls/app/_health.py+++ b/services/osquery-tls/app/_health.py@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /``mark_not_ready()`` from those handlers respectively.+Readiness is a question, not a latch+------------------------------------++``mark_ready()`` answers "did startup finish", and for a service+whose only job is to answer HTTP that is the whole story. For a+service whose job is to *consume a topic* it is not: three+services in this repository created the consumer as a+fire-and-forget ``asyncio.create_task`` and called+``mark_ready()`` on the next line, so readiness recorded that the+task had been **created**, never that it was still running.++UEBA is what that cost. Its handler raised+``UndefinedColumnError`` on the first scoreable event, the+exception left the ``async for``, the ``finally`` stopped the+consumer, and the task ended — holding the exception, which+nothing retrieved because the task was parked in a module global+and so never garbage-collected. The container stayed ``running``+with restarts 0, ``/health`` returned 200 and ``/readyz``+returned 200, permanently. A consumer that has silently detached+is indistinguishable from an idle one, and the whole point of a+readiness probe is to tell those two apart.++So a service that owns a subscription registers a probe for it+with :func:`register_subscription`, and ``/readyz`` evaluates+every probe on each request. A probe that returns false — or+raises — is reported as detached and ``/readyz`` answers 503 with+the names, which is a readiness failure rather than a liveness+one on purpose: the orchestrator should stop routing to a+half-working replica, and killing the process would only hide a+schema or broker fault behind a restart loop.++``/livez`` deliberately does **not** consult the probes. It+answers "is the interpreter responsive", and wiring a detached+consumer into it would make every orchestrator restart the pod on+a fault a restart cannot fix.+This module is intentionally dependency-free (only FastAPI) so itcan be vendored into every service tree without import-graphside-effects. Each service ships its own copy under``services/<svc>/app/_health.py`` because services don't share a-Python path — keep the file in sync via-``scripts/sync_health_module.py`` (CI gate).+Python path; ``scripts/audit_health_probes.py --check`` is the CI+gate that every service has one and wires it in."""from __future__ import annotations+import jsonfrom collections.abc import Callablefrom typing import Anyfrom fastapi import FastAPI, Response+#: Where the probes registered against an app are kept. Read by ``/readyz``,+#: written by :func:`register_subscription`. On ``app.state`` rather than in a+#: module global because a module global is shared by every app in the+#: process, which makes two test clients interfere.+_STATE_ATTR = "health_subscriptions"+++def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:+ """Names of the subscriptions currently reporting not-attached.++ A probe that raises counts as detached. It is asking a live object+ whether it is still working; if that question itself fails, the honest+ answer is not "yes".+ """+ detached: list[str] = []+ for name, probe in sorted(probes.items()):+ try:+ attached = bool(probe())+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one+ attached = False+ if not attached:+ detached.append(name)+ return detached+++def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:+ """Make ``/readyz`` depend on *probe* returning true.++ Call this once per subscription the service owns, after+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —+ it runs on every ``/readyz`` request — so it should read a flag the+ consumer maintains or check ``task.done()``, not round-trip to a broker.+ """+ probes = getattr(app.state, _STATE_ATTR, None)+ if probes is None: # pragma: no cover — install_health_routes was not called+ raise RuntimeError("register_subscription requires install_health_routes(app) first")+ probes[name] = probe+def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:"""Install ``/livez`` and ``/readyz`` on ``app``.@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[``mark_ready()`` is called — exactly the right behaviour for arolling deploy where the orchestrator should hold trafficuntil the pod's dependencies are connected.++ Startup having finished is necessary and not sufficient: any+ subscription registered with :func:`register_subscription`+ must also still be attached. See the module docstring."""state: dict[str, bool] = {"ready": False}+ probes: dict[str, Callable[[], bool]] = {}+ setattr(app.state, _STATE_ATTR, probes)@app.get("/livez",@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[include_in_schema=False,)async def _readyz() -> Response:- if state["ready"]:- return Response(- content='{"status":"ready","service":"' + service_name + '"}',- media_type="application/json",- status_code=200,- )+ body: dict[str, Any] = {"status": "ready", "service": service_name}+ status_code = 200++ if not state["ready"]:+ body["status"] = "starting"+ status_code = 503+ else:+ detached = _probe_detached(probes)+ if detached:+ body["status"] = "degraded"+ body["detached"] = detached+ status_code = 503++ # Named subscriptions are reported either way, so a 200 says which+ # ones were checked rather than only that nothing was wrong. A probe+ # list that is unexpectedly empty is then visible in the same place+ # the verdict is.+ if probes:+ body["subscriptions"] = sorted(probes)+return Response(- content='{"status":"starting","service":"' + service_name + '"}',+ content=json.dumps(body),media_type="application/json",- status_code=503,+ status_code=status_code,)def mark_ready() -> None:diff --git a/services/osquery-tls/app/_migrate.py b/services/osquery-tls/app/_migrate.pynew file mode 100644index 000000000..1297f113b--- /dev/null+++ b/services/osquery-tls/app/_migrate.py@@ -0,0 +1,228 @@+"""Apply this service's alembic chain, then exec what comes after it.++Why this exists+---------------++Four services in this repository own their schema through alembic —+``ueba``, ``honeytokens``, ``purple-team`` and ``osquery-tls`` — and **no+compose path invoked any of them**. Their container command was a plain+``uvicorn``, so on the documented quickstart they booted healthy against+empty schemas with none of their row-level-security policies installed. The+only documented invocation was a manual step in a docs-portal quickstart+whose own "start the stack" step does not start three of the four services,+and whose first line failed outright.++That is how two services shipped with no tables while every command+reported success. The API does not have this problem because its chain runs+from its own process on the way up; this is the same idea for the four that+use alembic.++What it guarantees+------------------++**Idempotent.** ``alembic upgrade head`` on a database already at head+applies nothing. Running this on every container start is the point: there+is no separate "have you migrated yet" state for an operator to get wrong.++**Race-free.** Four chains start at once against one database. The chains+take a shared transaction-scoped advisory lock (``MIGRATION_LOCK_KEY`` in+each ``env.py``), so they queue instead of deadlocking on catalog locks.+This wrapper deliberately holds no lock of its own — a lock held here would+be a second mechanism to reason about, and it would not cover the window+that actually matters.++**Loud.** The service does not start unless the chain reached head. Not "the+command exited 0": alembic exits 0 when it applies *nothing*, which is+exactly what happened when four chains shared one ``alembic_version`` table+and the second one to run believed it was already done. So ``current`` is+read back and compared to ``heads``, and a mismatch — or an empty+``current``, meaning the chain never ran — stops the container with the+reason on stderr. A service booting on an empty schema is the failure this+replaces, and it must not be reachable by a wrapper that only checks an exit+code.++Why the owner credential, and why a subprocess+----------------------------------------------++Alembic issues DDL. The runtime role every service serves requests as holds+no ``CREATE`` on schema public, by design — that is what stops it turning+row-level security off — so the chain must be applied as the owner. Each+``env.py`` already resolves that (its own ``*_DATABASE_MIGRATION_URL``+first, then ``DATABASE_MIGRATION_URL``), and this runs alembic as a+subprocess precisely so ``env.py`` stays the single answer to "which+credential". A second resolution here would be a second thing to keep in+step, and the two would disagree the first time either changed.++This module is vendored per service because services do not share a Python+path. ``scripts/check_service_migration_bootstrap.py`` is the gate that each+of the four ships a copy, keeps them identical, and runs one on the way up.++Usage+-----++::++ python -m app._migrate # apply the chain, exit+ python -m app._migrate -- uvicorn app.main:app --port 8004+"""++from __future__ import annotations++import os+import subprocess+import sys+from pathlib import Path++#: Where ``alembic.ini`` sits relative to the working directory in every one+#: of the four images (``WORKDIR /app``, and the ini is copied to ``/app``).+ALEMBIC_INI = "alembic.ini"++#: Suffix that marks a DSN as the migration (owner) credential. Matches the+#: convention each ``env.py`` documents and ``scripts/check_runtime_db_role.py``+#: enforces on the deployment surfaces: a service-prefixed+#: ``<SVC>_DATABASE_MIGRATION_URL`` wins over the bare form.+MIGRATION_URL_SUFFIX = "DATABASE_MIGRATION_URL"+++def migration_credential_present(env: dict[str, str] | None = None) -> bool:+ """Whether an owner DSN is configured.++ Used only for the warning below. The chain is applied either way — a+ deployment that has not split the roles yet behaves as it did before —+ but it is announced, because a chain silently applied as the runtime+ credential fails on the first ``CREATE TABLE`` and the message names a+ permission rather than a configuration.+ """+ source = os.environ if env is None else env+ return any(name.endswith(MIGRATION_URL_SUFFIX) and value.strip() for name, value in source.items())+++def _alembic(args: list[str], *, capture: bool) -> subprocess.CompletedProcess[str]:+ return subprocess.run( # noqa: S603 — fixed argv, no shell+ [sys.executable, "-m", "alembic", "-c", ALEMBIC_INI, *args],+ capture_output=capture,+ text=True,+ check=False,+ )+++def _revisions(output: str) -> set[str]:+ """Revision ids in ``alembic current`` / ``alembic heads`` output.++ Both print ``<rev> (head)`` or ``<rev>``, one per line, with informational+ lines alembic sends to stderr. An empty set means the version table holds+ nothing — the chain has not run — which is the case this exists to catch+ and is therefore never treated as agreement.+ """+ found: set[str] = set()+ for line in output.splitlines():+ token = line.strip().split(" ")[0].strip()+ if token and not token.startswith(("INFO", "WARNING", "ERROR", "alembic:")):+ found.add(token)+ return found+++def apply_chain() -> int:+ """Apply the chain and verify it reached head. Returns an exit code."""+ if not Path(ALEMBIC_INI).is_file():+ print(+ f"migrate: {ALEMBIC_INI} not found in {Path.cwd()}. This service's image must "+ f"copy its alembic config and chain; see its Dockerfile.",+ file=sys.stderr,+ )+ return 1++ if not migration_credential_present():+ print(+ f"migrate: no *{MIGRATION_URL_SUFFIX} is set, so the chain will be applied as "+ f"the runtime credential. That works only while the two are the same role; "+ f"under the DML-only runtime role the first CREATE TABLE fails with "+ f"'permission denied for schema public'.",+ file=sys.stderr,+ )++ upgrade = _alembic(["upgrade", "head"], capture=False)+ if upgrade.returncode != 0:+ print(+ f"migrate: 'alembic upgrade head' failed (exit {upgrade.returncode}). Refusing to "+ f"start: this service would serve requests against whatever schema is there now.",+ file=sys.stderr,+ )+ return upgrade.returncode++ # Read back rather than trusting the exit code. `alembic upgrade head`+ # exits 0 having applied nothing when it believes the database is already+ # at head, and believing that wrongly is the documented way these four+ # services ended up with no tables.+ current = _alembic(["current"], capture=True)+ heads = _alembic(["heads"], capture=True)+ if current.returncode != 0 or heads.returncode != 0:+ print(+ "migrate: could not read back the applied revision "+ f"(current exit {current.returncode}, heads exit {heads.returncode}). "+ "Refusing to start: an unverified migration is the state this check exists "+ "to reject.\n"+ f"{(current.stderr or '') + (heads.stderr or '')}",+ file=sys.stderr,+ )+ return 1++ applied = _revisions(current.stdout)+ expected = _revisions(heads.stdout)++ if not expected:+ print(+ "migrate: this chain declares no head revision, so there is nothing to verify "+ "against. Refusing to start rather than reporting an empty chain as applied.",+ file=sys.stderr,+ )+ return 1++ if not applied:+ print(+ f"migrate: 'alembic upgrade head' reported success but the version table is "+ f"empty — the chain never ran. Expected {sorted(expected)}. Refusing to start "+ f"on an unmigrated schema.",+ file=sys.stderr,+ )+ return 1++ if not expected.issubset(applied):+ print(+ f"migrate: chain did not reach head. At {sorted(applied)}, head is " f"{sorted(expected)}. Refusing to start.",+ file=sys.stderr,+ )+ return 1++ print(f"migrate: schema at head {sorted(expected)}", file=sys.stderr)+ return 0+++def main(argv: list[str] | None = None) -> int:+ args = list(sys.argv[1:] if argv is None else argv)++ # Everything after `--` is the real service command. Exec'd rather than+ # spawned so the server keeps PID 1 and receives SIGTERM directly — a+ # Python parent waiting on a child swallows the orchestrator's stop+ # signal and turns every shutdown into a ten-second kill.+ command: list[str] = []+ if "--" in args:+ split = args.index("--")+ command = args[split + 1 :]+ args = args[:split]++ if args:+ print(f"migrate: unexpected arguments {args}; usage: python -m app._migrate [-- CMD ...]", file=sys.stderr)+ return 2++ code = apply_chain()+ if code != 0:+ return code++ if command:+ os.execvp(command[0], command) # noqa: S606 — replaces this process by design+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/services/osquery-tls/app/db/env.py b/services/osquery-tls/app/db/env.pyindex 32b2a400c..fbaab4e57 100644--- a/services/osquery-tls/app/db/env.py+++ b/services/osquery-tls/app/db/env.py@@ -154,9 +154,27 @@ def run_migrations_offline() -> None:context.run_migrations()+#: Serialises the four chains that share one database against each other.+#:+#: They are started simultaneously by ``docker compose up``, and while each+#: now owns its own version table, they do not own their own *database*: the+#: legacy-version probe below reads a table a sibling may be creating, and+#: ``000N_runtime_role_grants`` issues ``GRANT`` statements that take+#: catalog locks on ``pg_class``. Two chains arriving together deadlocked+#: rather than queued.+#:+#: ``pg_advisory_xact_lock`` and not the session form: it is released by the+#: commit or rollback that ends the upgrade, so a chain that dies mid-apply+#: cannot leave the other three waiting on a lock nobody holds a connection+#: for. The key is an arbitrary constant, shared by every chain on purpose —+#: the point is mutual exclusion, not per-chain exclusion.+MIGRATION_LOCK_KEY = 0x4149_5354++def do_run_migrations(connection):context.configure(connection=connection, target_metadata=target_metadata, version_table=VERSION_TABLE)with context.begin_transaction():+ connection.execute(sa.text("SELECT pg_advisory_xact_lock(:key)"), {"key": MIGRATION_LOCK_KEY})# Inside ``begin_transaction()``, not before it. Alembic's# ``begin_transaction`` is a no-op when the connection is *already* in a# transaction, and in SQLAlchemy 2.0 any execute on a plain ``connect()``diff --git a/services/purple-team/Dockerfile b/services/purple-team/Dockerfileindex 023b60fae..1adb44ab0 100644--- a/services/purple-team/Dockerfile+++ b/services/purple-team/Dockerfile@@ -25,4 +25,17 @@ COPY app/ app/EXPOSE 8006-CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8006"]+# The chain runs on the way up, then this process is replaced by the server.+#+# Nothing invoked these four alembic chains. This was a bare `uvicorn` line, so+# the documented quickstart booted the service against an empty schema with+# none of its row-level-security policies installed, and every command in that+# sequence reported success. `app/_migrate.py` applies the chain as the owner+# credential, verifies it reached head, and refuses to start the server if it+# did not.+#+# `sh -c` because an exec-form CMD cannot run two things. Neither shell nor+# wrapper stays in the process tree: `sh -c` with a single command execs it,+# and `_migrate` hands over with `os.execvp`, so uvicorn ends up as PID 1 and+# receives SIGTERM directly rather than through a parent that would swallow it.+CMD ["sh", "-c", "python -m app._migrate -- uvicorn app.main:app --host 0.0.0.0 --port 8006"]diff --git a/services/purple-team/alembic/env.py b/services/purple-team/alembic/env.pyindex be505bc8e..0e1a5452b 100644--- a/services/purple-team/alembic/env.py+++ b/services/purple-team/alembic/env.py@@ -157,9 +157,27 @@ def run_migrations_offline() -> None:context.run_migrations()+#: Serialises the four chains that share one database against each other.+#:+#: They are started simultaneously by ``docker compose up``, and while each+#: now owns its own version table, they do not own their own *database*: the+#: legacy-version probe below reads a table a sibling may be creating, and+#: ``000N_runtime_role_grants`` issues ``GRANT`` statements that take+#: catalog locks on ``pg_class``. Two chains arriving together deadlocked+#: rather than queued.+#:+#: ``pg_advisory_xact_lock`` and not the session form: it is released by the+#: commit or rollback that ends the upgrade, so a chain that dies mid-apply+#: cannot leave the other three waiting on a lock nobody holds a connection+#: for. The key is an arbitrary constant, shared by every chain on purpose —+#: the point is mutual exclusion, not per-chain exclusion.+MIGRATION_LOCK_KEY = 0x4149_5354++def do_run_migrations(connection): # type: ignore[no-untyped-def]context.configure(connection=connection, target_metadata=target_metadata, version_table=VERSION_TABLE)with context.begin_transaction():+ connection.execute(sa.text("SELECT pg_advisory_xact_lock(:key)"), {"key": MIGRATION_LOCK_KEY})# Inside ``begin_transaction()``, not before it. Alembic's# ``begin_transaction`` is a no-op when the connection is *already* in a# transaction, and in SQLAlchemy 2.0 any execute on a plain ``connect()``diff --git a/services/purple-team/app/_health.py b/services/purple-team/app/_health.pyindex 974269731..0acb9e4da 100644--- a/services/purple-team/app/_health.py+++ b/services/purple-team/app/_health.py@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /``mark_not_ready()`` from those handlers respectively.+Readiness is a question, not a latch+------------------------------------++``mark_ready()`` answers "did startup finish", and for a service+whose only job is to answer HTTP that is the whole story. For a+service whose job is to *consume a topic* it is not: three+services in this repository created the consumer as a+fire-and-forget ``asyncio.create_task`` and called+``mark_ready()`` on the next line, so readiness recorded that the+task had been **created**, never that it was still running.++UEBA is what that cost. Its handler raised+``UndefinedColumnError`` on the first scoreable event, the+exception left the ``async for``, the ``finally`` stopped the+consumer, and the task ended — holding the exception, which+nothing retrieved because the task was parked in a module global+and so never garbage-collected. The container stayed ``running``+with restarts 0, ``/health`` returned 200 and ``/readyz``+returned 200, permanently. A consumer that has silently detached+is indistinguishable from an idle one, and the whole point of a+readiness probe is to tell those two apart.++So a service that owns a subscription registers a probe for it+with :func:`register_subscription`, and ``/readyz`` evaluates+every probe on each request. A probe that returns false — or+raises — is reported as detached and ``/readyz`` answers 503 with+the names, which is a readiness failure rather than a liveness+one on purpose: the orchestrator should stop routing to a+half-working replica, and killing the process would only hide a+schema or broker fault behind a restart loop.++``/livez`` deliberately does **not** consult the probes. It+answers "is the interpreter responsive", and wiring a detached+consumer into it would make every orchestrator restart the pod on+a fault a restart cannot fix.+This module is intentionally dependency-free (only FastAPI) so itcan be vendored into every service tree without import-graphside-effects. Each service ships its own copy under``services/<svc>/app/_health.py`` because services don't share a-Python path — keep the file in sync via-``scripts/sync_health_module.py`` (CI gate).+Python path; ``scripts/audit_health_probes.py --check`` is the CI+gate that every service has one and wires it in."""from __future__ import annotations+import jsonfrom collections.abc import Callablefrom typing import Anyfrom fastapi import FastAPI, Response+#: Where the probes registered against an app are kept. Read by ``/readyz``,+#: written by :func:`register_subscription`. On ``app.state`` rather than in a+#: module global because a module global is shared by every app in the+#: process, which makes two test clients interfere.+_STATE_ATTR = "health_subscriptions"+++def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:+ """Names of the subscriptions currently reporting not-attached.++ A probe that raises counts as detached. It is asking a live object+ whether it is still working; if that question itself fails, the honest+ answer is not "yes".+ """+ detached: list[str] = []+ for name, probe in sorted(probes.items()):+ try:+ attached = bool(probe())+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one+ attached = False+ if not attached:+ detached.append(name)+ return detached+++def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:+ """Make ``/readyz`` depend on *probe* returning true.++ Call this once per subscription the service owns, after+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —+ it runs on every ``/readyz`` request — so it should read a flag the+ consumer maintains or check ``task.done()``, not round-trip to a broker.+ """+ probes = getattr(app.state, _STATE_ATTR, None)+ if probes is None: # pragma: no cover — install_health_routes was not called+ raise RuntimeError("register_subscription requires install_health_routes(app) first")+ probes[name] = probe+def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:"""Install ``/livez`` and ``/readyz`` on ``app``.@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[``mark_ready()`` is called — exactly the right behaviour for arolling deploy where the orchestrator should hold trafficuntil the pod's dependencies are connected.++ Startup having finished is necessary and not sufficient: any+ subscription registered with :func:`register_subscription`+ must also still be attached. See the module docstring."""state: dict[str, bool] = {"ready": False}+ probes: dict[str, Callable[[], bool]] = {}+ setattr(app.state, _STATE_ATTR, probes)@app.get("/livez",@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[include_in_schema=False,)async def _readyz() -> Response:- if state["ready"]:- return Response(- content='{"status":"ready","service":"' + service_name + '"}',- media_type="application/json",- status_code=200,- )+ body: dict[str, Any] = {"status": "ready", "service": service_name}+ status_code = 200++ if not state["ready"]:+ body["status"] = "starting"+ status_code = 503+ else:+ detached = _probe_detached(probes)+ if detached:+ body["status"] = "degraded"+ body["detached"] = detached+ status_code = 503++ # Named subscriptions are reported either way, so a 200 says which+ # ones were checked rather than only that nothing was wrong. A probe+ # list that is unexpectedly empty is then visible in the same place+ # the verdict is.+ if probes:+ body["subscriptions"] = sorted(probes)+return Response(- content='{"status":"starting","service":"' + service_name + '"}',+ content=json.dumps(body),media_type="application/json",- status_code=503,+ status_code=status_code,)def mark_ready() -> None:diff --git a/services/purple-team/app/_migrate.py b/services/purple-team/app/_migrate.pynew file mode 100644index 000000000..1297f113b--- /dev/null+++ b/services/purple-team/app/_migrate.py@@ -0,0 +1,228 @@+"""Apply this service's alembic chain, then exec what comes after it.++Why this exists+---------------++Four services in this repository own their schema through alembic —+``ueba``, ``honeytokens``, ``purple-team`` and ``osquery-tls`` — and **no+compose path invoked any of them**. Their container command was a plain+``uvicorn``, so on the documented quickstart they booted healthy against+empty schemas with none of their row-level-security policies installed. The+only documented invocation was a manual step in a docs-portal quickstart+whose own "start the stack" step does not start three of the four services,+and whose first line failed outright.++That is how two services shipped with no tables while every command+reported success. The API does not have this problem because its chain runs+from its own process on the way up; this is the same idea for the four that+use alembic.++What it guarantees+------------------++**Idempotent.** ``alembic upgrade head`` on a database already at head+applies nothing. Running this on every container start is the point: there+is no separate "have you migrated yet" state for an operator to get wrong.++**Race-free.** Four chains start at once against one database. The chains+take a shared transaction-scoped advisory lock (``MIGRATION_LOCK_KEY`` in+each ``env.py``), so they queue instead of deadlocking on catalog locks.+This wrapper deliberately holds no lock of its own — a lock held here would+be a second mechanism to reason about, and it would not cover the window+that actually matters.++**Loud.** The service does not start unless the chain reached head. Not "the+command exited 0": alembic exits 0 when it applies *nothing*, which is+exactly what happened when four chains shared one ``alembic_version`` table+and the second one to run believed it was already done. So ``current`` is+read back and compared to ``heads``, and a mismatch — or an empty+``current``, meaning the chain never ran — stops the container with the+reason on stderr. A service booting on an empty schema is the failure this+replaces, and it must not be reachable by a wrapper that only checks an exit+code.++Why the owner credential, and why a subprocess+----------------------------------------------++Alembic issues DDL. The runtime role every service serves requests as holds+no ``CREATE`` on schema public, by design — that is what stops it turning+row-level security off — so the chain must be applied as the owner. Each+``env.py`` already resolves that (its own ``*_DATABASE_MIGRATION_URL``+first, then ``DATABASE_MIGRATION_URL``), and this runs alembic as a+subprocess precisely so ``env.py`` stays the single answer to "which+credential". A second resolution here would be a second thing to keep in+step, and the two would disagree the first time either changed.++This module is vendored per service because services do not share a Python+path. ``scripts/check_service_migration_bootstrap.py`` is the gate that each+of the four ships a copy, keeps them identical, and runs one on the way up.++Usage+-----++::++ python -m app._migrate # apply the chain, exit+ python -m app._migrate -- uvicorn app.main:app --port 8004+"""++from __future__ import annotations++import os+import subprocess+import sys+from pathlib import Path++#: Where ``alembic.ini`` sits relative to the working directory in every one+#: of the four images (``WORKDIR /app``, and the ini is copied to ``/app``).+ALEMBIC_INI = "alembic.ini"++#: Suffix that marks a DSN as the migration (owner) credential. Matches the+#: convention each ``env.py`` documents and ``scripts/check_runtime_db_role.py``+#: enforces on the deployment surfaces: a service-prefixed+#: ``<SVC>_DATABASE_MIGRATION_URL`` wins over the bare form.+MIGRATION_URL_SUFFIX = "DATABASE_MIGRATION_URL"+++def migration_credential_present(env: dict[str, str] | None = None) -> bool:+ """Whether an owner DSN is configured.++ Used only for the warning below. The chain is applied either way — a+ deployment that has not split the roles yet behaves as it did before —+ but it is announced, because a chain silently applied as the runtime+ credential fails on the first ``CREATE TABLE`` and the message names a+ permission rather than a configuration.+ """+ source = os.environ if env is None else env+ return any(name.endswith(MIGRATION_URL_SUFFIX) and value.strip() for name, value in source.items())+++def _alembic(args: list[str], *, capture: bool) -> subprocess.CompletedProcess[str]:+ return subprocess.run( # noqa: S603 — fixed argv, no shell+ [sys.executable, "-m", "alembic", "-c", ALEMBIC_INI, *args],+ capture_output=capture,+ text=True,+ check=False,+ )+++def _revisions(output: str) -> set[str]:+ """Revision ids in ``alembic current`` / ``alembic heads`` output.++ Both print ``<rev> (head)`` or ``<rev>``, one per line, with informational+ lines alembic sends to stderr. An empty set means the version table holds+ nothing — the chain has not run — which is the case this exists to catch+ and is therefore never treated as agreement.+ """+ found: set[str] = set()+ for line in output.splitlines():+ token = line.strip().split(" ")[0].strip()+ if token and not token.startswith(("INFO", "WARNING", "ERROR", "alembic:")):+ found.add(token)+ return found+++def apply_chain() -> int:+ """Apply the chain and verify it reached head. Returns an exit code."""+ if not Path(ALEMBIC_INI).is_file():+ print(+ f"migrate: {ALEMBIC_INI} not found in {Path.cwd()}. This service's image must "+ f"copy its alembic config and chain; see its Dockerfile.",+ file=sys.stderr,+ )+ return 1++ if not migration_credential_present():+ print(+ f"migrate: no *{MIGRATION_URL_SUFFIX} is set, so the chain will be applied as "+ f"the runtime credential. That works only while the two are the same role; "+ f"under the DML-only runtime role the first CREATE TABLE fails with "+ f"'permission denied for schema public'.",+ file=sys.stderr,+ )++ upgrade = _alembic(["upgrade", "head"], capture=False)+ if upgrade.returncode != 0:+ print(+ f"migrate: 'alembic upgrade head' failed (exit {upgrade.returncode}). Refusing to "+ f"start: this service would serve requests against whatever schema is there now.",+ file=sys.stderr,+ )+ return upgrade.returncode++ # Read back rather than trusting the exit code. `alembic upgrade head`+ # exits 0 having applied nothing when it believes the database is already+ # at head, and believing that wrongly is the documented way these four+ # services ended up with no tables.+ current = _alembic(["current"], capture=True)+ heads = _alembic(["heads"], capture=True)+ if current.returncode != 0 or heads.returncode != 0:+ print(+ "migrate: could not read back the applied revision "+ f"(current exit {current.returncode}, heads exit {heads.returncode}). "+ "Refusing to start: an unverified migration is the state this check exists "+ "to reject.\n"+ f"{(current.stderr or '') + (heads.stderr or '')}",+ file=sys.stderr,+ )+ return 1++ applied = _revisions(current.stdout)+ expected = _revisions(heads.stdout)++ if not expected:+ print(+ "migrate: this chain declares no head revision, so there is nothing to verify "+ "against. Refusing to start rather than reporting an empty chain as applied.",+ file=sys.stderr,+ )+ return 1++ if not applied:+ print(+ f"migrate: 'alembic upgrade head' reported success but the version table is "+ f"empty — the chain never ran. Expected {sorted(expected)}. Refusing to start "+ f"on an unmigrated schema.",+ file=sys.stderr,+ )+ return 1++ if not expected.issubset(applied):+ print(+ f"migrate: chain did not reach head. At {sorted(applied)}, head is " f"{sorted(expected)}. Refusing to start.",+ file=sys.stderr,+ )+ return 1++ print(f"migrate: schema at head {sorted(expected)}", file=sys.stderr)+ return 0+++def main(argv: list[str] | None = None) -> int:+ args = list(sys.argv[1:] if argv is None else argv)++ # Everything after `--` is the real service command. Exec'd rather than+ # spawned so the server keeps PID 1 and receives SIGTERM directly — a+ # Python parent waiting on a child swallows the orchestrator's stop+ # signal and turns every shutdown into a ten-second kill.+ command: list[str] = []+ if "--" in args:+ split = args.index("--")+ command = args[split + 1 :]+ args = args[:split]++ if args:+ print(f"migrate: unexpected arguments {args}; usage: python -m app._migrate [-- CMD ...]", file=sys.stderr)+ return 2++ code = apply_chain()+ if code != 0:+ return code++ if command:+ os.execvp(command[0], command) # noqa: S606 — replaces this process by design+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/services/realtime/src/index.ts b/services/realtime/src/index.tsindex 82cff5417..daf6c1ec0 100644--- a/services/realtime/src/index.ts+++ b/services/realtime/src/index.ts@@ -414,9 +414,89 @@ const kafka = new Kafka({retry: { retries: 5 },});+// --- Subscription state, reported on /health ---+//+// `/health` returned a hardcoded `status: 'healthy'` with a client count and+// nothing about Kafka, so a realtime service whose consumers had never+// connected was indistinguishable from one fanning out every alert. The two+// consumers below register here and `reportHealth` answers 503 while either+// is detached — the same contract the Python services answer on /readyz.+type SubscriptionState = {+ attached: boolean;+ lastError?: string;+ attempts: number;+};+const subscriptions = new Map<string, SubscriptionState>();++// Bounded backoff, capped. A detached consumer is retried because the usual+// cause is a broker or topic that is not there *yet*, which a retry fixes;+// the cap stops a permanently-broken cluster becoming a busy loop, and the+// state above stops a permanent failure being silent while it retries.+const RETRY_BASE_MS = 2_000;+const RETRY_MAX_MS = 60_000;++/**+ * Keep `start` running, and record whether it is.+ *+ * The previous code called each starter inside a try/catch that logged+ * "Kafka consumer failed to start (will retry)" and then returned. Nothing+ * retried. The message described behaviour the code did not have, which is+ * worse than no message: it tells a reader the gap is already handled.+ */+function superviseConsumer(+ name: string,+ start: (onCrash: (err: unknown) => void) => Promise<void>,+): void {+ subscriptions.set(name, { attached: false, attempts: 0 });++ const run = async (): Promise<void> => {+ const state = subscriptions.get(name)!;+ for (;;) {+ state.attempts += 1;+ try {+ let crashed = false;+ await start((err: unknown) => {+ // kafkajs surfaces a dead consumer loop through its CRASH event,+ // not by rejecting the promise `run()` returned — so without this+ // hook a consumer that died after a successful start stayed+ // recorded as attached for the life of the process.+ if (crashed) return;+ crashed = true;+ state.attached = false;+ state.lastError = err instanceof Error ? err.message : String(err);+ log.error({ err, name }, 'Kafka consumer crashed; reconnecting');+ });+ state.attached = true;+ state.lastError = undefined;+ log.info({ name, attempts: state.attempts }, 'Kafka consumer attached');+ // Wait for a crash rather than returning: this function owns the+ // retry, so it has to still be here when the consumer stops.+ while (!crashed) {+ await new Promise((resolve) => setTimeout(resolve, 5_000));+ }+ } catch (err) {+ state.attached = false;+ state.lastError = err instanceof Error ? err.message : String(err);+ const delay = Math.min(+ RETRY_BASE_MS * 2 ** Math.min(state.attempts - 1, 5),+ RETRY_MAX_MS,+ );+ log.warn(+ { err, name, attempts: state.attempts, retryInMs: delay },+ 'Kafka consumer not attached; retrying',+ );+ await new Promise((resolve) => setTimeout(resolve, delay));+ }+ }+ };++ void run();+}+// --- Kafka consumer: bridge fused alerts to WebSocket clients ----async function startKafkaConsumer() {+async function startKafkaConsumer(onCrash: (err: unknown) => void) {const consumer = kafka.consumer({ groupId: 'aisoc-realtime-ws' });+ consumer.on(consumer.events.CRASH, (event) => onCrash(event.payload.error));await consumer.connect();await consumer.subscribe({ topic: KAFKA_TOPIC_FUSED, fromBeginning: false });@@ -486,13 +566,14 @@ async function startKafkaConsumer() {// versa. A failure to start (Kafka unreachable, topic missing, etc.) is// logged and surfaced via the outer retry wrapper; it does NOT crash the// process, because the alert fan-out path is the higher-priority surface.-async function startGraphUpdateConsumer() {+async function startGraphUpdateConsumer(onCrash: (err: unknown) => void) {if (!KAFKA_TOPIC_GRAPH_UPDATES) {log.info('Graph update consumer disabled (empty KAFKA_TOPIC_GRAPH_UPDATES)');return;}const consumer = kafka.consumer({ groupId: 'aisoc-realtime-graph' });+ consumer.on(consumer.events.CRASH, (event) => onCrash(event.payload.error));await consumer.connect();await consumer.subscribe({@@ -708,10 +789,22 @@ app.post('/internal/agent-event', internalEventRateLimit, (req, res) => {// Expose both `/health` (canonical) and `/healthz` (k8s + frontend default) so// callers don't have to guess.const reportHealth = (_req: express.Request, res: express.Response) => {- res.json({- status: 'healthy',+ const detached = [...subscriptions.entries()]+ .filter(([, state]) => !state.attached)+ .map(([name]) => name);++ res.status(detached.length > 0 ? 503 : 200).json({+ status: detached.length > 0 ? 'degraded' : 'healthy',service: 'aisoc-realtime',clients: wss.clients.size,+ // Reported whether or not anything is wrong, so a healthy answer says+ // which subscriptions were checked rather than only that none failed.+ subscriptions: [...subscriptions.entries()].map(([name, state]) => ({+ topic: name,+ attached: state.attached,+ attempts: state.attempts,+ ...(state.lastError ? { last_error: state.lastError } : {}),+ })),});};app.get('/health', reportHealth);@@ -743,30 +836,19 @@ server.listen(PORT, '::', () => {}})();- // Start the two Kafka consumers concurrently. Each is wrapped in its own- // try/catch so a failure on the graph topic (e.g. it doesn't exist yet on- // a brand-new cluster) does NOT block the higher-priority fused-alerts- // fan-out. We deliberately fire them in parallel rather than awaiting- // sequentially so the HTTP/WS listener is fully up before either Kafka- // round-trip completes.- void (async () => {- try {- await startKafkaConsumer();- } catch (err) {- log.warn({ err }, 'Kafka consumer failed to start (will retry)');- }- })();-- void (async () => {- try {- await startGraphUpdateConsumer();- } catch (err) {- // T1.4 fan-out is best-effort: if the graph topic isn't available the- // alerts/cases/agents/insights channels still work, the graph panel- // just won't light up in real time.- log.warn({ err }, 'Graph update consumer failed to start (will retry)');- }- })();+ // Start the two Kafka consumers under their own supervisors, so a failure+ // on the graph topic (e.g. it doesn't exist yet on a brand-new cluster)+ // does NOT block the higher-priority fused-alerts fan-out, and neither one+ // stays silently detached. Fired rather than awaited so the HTTP/WS+ // listener is fully up before either Kafka round-trip completes.+ superviseConsumer(KAFKA_TOPIC_FUSED, startKafkaConsumer);++ // Only supervised when it is configured. An empty KAFKA_TOPIC_GRAPH_UPDATES+ // turns this fan-out off deliberately, and reporting a switched-off consumer+ // as detached would make /health red for a supported configuration.+ if (KAFKA_TOPIC_GRAPH_UPDATES) {+ superviseConsumer(KAFKA_TOPIC_GRAPH_UPDATES, startGraphUpdateConsumer);+ }});// Flush spans on the way out. Without this the exporter drops the spansdiff --git a/services/slack-bot/app/_health.py b/services/slack-bot/app/_health.pyindex 974269731..0acb9e4da 100644--- a/services/slack-bot/app/_health.py+++ b/services/slack-bot/app/_health.py@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /``mark_not_ready()`` from those handlers respectively.+Readiness is a question, not a latch+------------------------------------++``mark_ready()`` answers "did startup finish", and for a service+whose only job is to answer HTTP that is the whole story. For a+service whose job is to *consume a topic* it is not: three+services in this repository created the consumer as a+fire-and-forget ``asyncio.create_task`` and called+``mark_ready()`` on the next line, so readiness recorded that the+task had been **created**, never that it was still running.++UEBA is what that cost. Its handler raised+``UndefinedColumnError`` on the first scoreable event, the+exception left the ``async for``, the ``finally`` stopped the+consumer, and the task ended — holding the exception, which+nothing retrieved because the task was parked in a module global+and so never garbage-collected. The container stayed ``running``+with restarts 0, ``/health`` returned 200 and ``/readyz``+returned 200, permanently. A consumer that has silently detached+is indistinguishable from an idle one, and the whole point of a+readiness probe is to tell those two apart.++So a service that owns a subscription registers a probe for it+with :func:`register_subscription`, and ``/readyz`` evaluates+every probe on each request. A probe that returns false — or+raises — is reported as detached and ``/readyz`` answers 503 with+the names, which is a readiness failure rather than a liveness+one on purpose: the orchestrator should stop routing to a+half-working replica, and killing the process would only hide a+schema or broker fault behind a restart loop.++``/livez`` deliberately does **not** consult the probes. It+answers "is the interpreter responsive", and wiring a detached+consumer into it would make every orchestrator restart the pod on+a fault a restart cannot fix.+This module is intentionally dependency-free (only FastAPI) so itcan be vendored into every service tree without import-graphside-effects. Each service ships its own copy under``services/<svc>/app/_health.py`` because services don't share a-Python path — keep the file in sync via-``scripts/sync_health_module.py`` (CI gate).+Python path; ``scripts/audit_health_probes.py --check`` is the CI+gate that every service has one and wires it in."""from __future__ import annotations+import jsonfrom collections.abc import Callablefrom typing import Anyfrom fastapi import FastAPI, Response+#: Where the probes registered against an app are kept. Read by ``/readyz``,+#: written by :func:`register_subscription`. On ``app.state`` rather than in a+#: module global because a module global is shared by every app in the+#: process, which makes two test clients interfere.+_STATE_ATTR = "health_subscriptions"+++def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:+ """Names of the subscriptions currently reporting not-attached.++ A probe that raises counts as detached. It is asking a live object+ whether it is still working; if that question itself fails, the honest+ answer is not "yes".+ """+ detached: list[str] = []+ for name, probe in sorted(probes.items()):+ try:+ attached = bool(probe())+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one+ attached = False+ if not attached:+ detached.append(name)+ return detached+++def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:+ """Make ``/readyz`` depend on *probe* returning true.++ Call this once per subscription the service owns, after+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —+ it runs on every ``/readyz`` request — so it should read a flag the+ consumer maintains or check ``task.done()``, not round-trip to a broker.+ """+ probes = getattr(app.state, _STATE_ATTR, None)+ if probes is None: # pragma: no cover — install_health_routes was not called+ raise RuntimeError("register_subscription requires install_health_routes(app) first")+ probes[name] = probe+def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:"""Install ``/livez`` and ``/readyz`` on ``app``.@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[``mark_ready()`` is called — exactly the right behaviour for arolling deploy where the orchestrator should hold trafficuntil the pod's dependencies are connected.++ Startup having finished is necessary and not sufficient: any+ subscription registered with :func:`register_subscription`+ must also still be attached. See the module docstring."""state: dict[str, bool] = {"ready": False}+ probes: dict[str, Callable[[], bool]] = {}+ setattr(app.state, _STATE_ATTR, probes)@app.get("/livez",@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[include_in_schema=False,)async def _readyz() -> Response:- if state["ready"]:- return Response(- content='{"status":"ready","service":"' + service_name + '"}',- media_type="application/json",- status_code=200,- )+ body: dict[str, Any] = {"status": "ready", "service": service_name}+ status_code = 200++ if not state["ready"]:+ body["status"] = "starting"+ status_code = 503+ else:+ detached = _probe_detached(probes)+ if detached:+ body["status"] = "degraded"+ body["detached"] = detached+ status_code = 503++ # Named subscriptions are reported either way, so a 200 says which+ # ones were checked rather than only that nothing was wrong. A probe+ # list that is unexpectedly empty is then visible in the same place+ # the verdict is.+ if probes:+ body["subscriptions"] = sorted(probes)+return Response(- content='{"status":"starting","service":"' + service_name + '"}',+ content=json.dumps(body),media_type="application/json",- status_code=503,+ status_code=status_code,)def mark_ready() -> None:diff --git a/services/slack-bot/tests/test_health_probes.py b/services/slack-bot/tests/test_health_probes.pynew file mode 100644index 000000000..00bd7914e--- /dev/null+++ b/services/slack-bot/tests/test_health_probes.py@@ -0,0 +1,98 @@+"""The shared probe helper, exercised in this tree too.++``app/_health.py`` is vendored byte-identical into thirteen services, and its+whole point is that every copy answers the same contract — which is only+worth something if more than one copy is ever run. ``services/api`` owns the+exhaustive suite; this asserts the parts an operator reads off a dashboard,+against *this* service's copy.++Worth keeping rather than deduplicating: when the helper grew+``register_subscription``, the api tree's tests covered the new branches and+none of the other twelve did. Here that showed up as this service's coverage+falling below its floor, which is a coarse signal and, that time, the only+one.+"""++from __future__ import annotations++from collections.abc import Callable++import pytest+from app._health import install_health_routes, register_subscription+from fastapi import FastAPI+from fastapi.testclient import TestClient+++def _build_app() -> tuple[FastAPI, Callable[[], None], Callable[[], None]]:+ app = FastAPI()+ mark_ready, mark_not_ready = install_health_routes(app, service_name="aisoc-slack-bot")+ return app, mark_ready, mark_not_ready+++def test_livez_is_200_before_startup_finishes() -> None:+ app, _, _ = _build_app()++ response = TestClient(app).get("/livez")++ assert response.status_code == 200+ assert response.json()["service"] == "aisoc-slack-bot"+++def test_readyz_is_503_until_marked_ready() -> None:+ app, mark_ready, _ = _build_app()+ client = TestClient(app)++ assert client.get("/readyz").status_code == 503+ mark_ready()+ assert client.get("/readyz").status_code == 200+++def test_readyz_is_503_again_after_draining() -> None:+ app, mark_ready, mark_not_ready = _build_app()+ client = TestClient(app)++ mark_ready()+ mark_not_ready()++ assert client.get("/readyz").status_code == 503+++def test_a_detached_subscription_is_named_in_the_503() -> None:+ app, mark_ready, _ = _build_app()+ attached = {"value": True}+ register_subscription(app, "aisoc.alerts.fused", lambda: attached["value"])+ mark_ready()+ client = TestClient(app)+ assert client.get("/readyz").json()["subscriptions"] == ["aisoc.alerts.fused"]++ attached["value"] = False++ response = client.get("/readyz")+ assert response.status_code == 503+ assert response.json()["detached"] == ["aisoc.alerts.fused"]+++def test_a_probe_that_raises_is_not_a_healthy_answer() -> None:+ def explode() -> bool:+ raise RuntimeError("gone")++ app, mark_ready, _ = _build_app()+ register_subscription(app, "aisoc.alerts.fused", explode)+ mark_ready()++ assert TestClient(app).get("/readyz").status_code == 503+++def test_livez_ignores_a_detached_subscription() -> None:+ """Liveness failing means "restart me", and a restart does not fix a+ missing column or a revoked grant."""+ app, mark_ready, _ = _build_app()+ register_subscription(app, "aisoc.alerts.fused", lambda: False)+ mark_ready()++ assert TestClient(app).get("/livez").status_code == 200+++def test_registration_requires_the_routes_first() -> None:+ with pytest.raises(RuntimeError):+ register_subscription(FastAPI(), "aisoc.alerts.fused", lambda: True)diff --git a/services/teams-bot/app/_health.py b/services/teams-bot/app/_health.pyindex 974269731..0acb9e4da 100644--- a/services/teams-bot/app/_health.py+++ b/services/teams-bot/app/_health.py@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /``mark_not_ready()`` from those handlers respectively.+Readiness is a question, not a latch+------------------------------------++``mark_ready()`` answers "did startup finish", and for a service+whose only job is to answer HTTP that is the whole story. For a+service whose job is to *consume a topic* it is not: three+services in this repository created the consumer as a+fire-and-forget ``asyncio.create_task`` and called+``mark_ready()`` on the next line, so readiness recorded that the+task had been **created**, never that it was still running.++UEBA is what that cost. Its handler raised+``UndefinedColumnError`` on the first scoreable event, the+exception left the ``async for``, the ``finally`` stopped the+consumer, and the task ended — holding the exception, which+nothing retrieved because the task was parked in a module global+and so never garbage-collected. The container stayed ``running``+with restarts 0, ``/health`` returned 200 and ``/readyz``+returned 200, permanently. A consumer that has silently detached+is indistinguishable from an idle one, and the whole point of a+readiness probe is to tell those two apart.++So a service that owns a subscription registers a probe for it+with :func:`register_subscription`, and ``/readyz`` evaluates+every probe on each request. A probe that returns false — or+raises — is reported as detached and ``/readyz`` answers 503 with+the names, which is a readiness failure rather than a liveness+one on purpose: the orchestrator should stop routing to a+half-working replica, and killing the process would only hide a+schema or broker fault behind a restart loop.++``/livez`` deliberately does **not** consult the probes. It+answers "is the interpreter responsive", and wiring a detached+consumer into it would make every orchestrator restart the pod on+a fault a restart cannot fix.+This module is intentionally dependency-free (only FastAPI) so itcan be vendored into every service tree without import-graphside-effects. Each service ships its own copy under``services/<svc>/app/_health.py`` because services don't share a-Python path — keep the file in sync via-``scripts/sync_health_module.py`` (CI gate).+Python path; ``scripts/audit_health_probes.py --check`` is the CI+gate that every service has one and wires it in."""from __future__ import annotations+import jsonfrom collections.abc import Callablefrom typing import Anyfrom fastapi import FastAPI, Response+#: Where the probes registered against an app are kept. Read by ``/readyz``,+#: written by :func:`register_subscription`. On ``app.state`` rather than in a+#: module global because a module global is shared by every app in the+#: process, which makes two test clients interfere.+_STATE_ATTR = "health_subscriptions"+++def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:+ """Names of the subscriptions currently reporting not-attached.++ A probe that raises counts as detached. It is asking a live object+ whether it is still working; if that question itself fails, the honest+ answer is not "yes".+ """+ detached: list[str] = []+ for name, probe in sorted(probes.items()):+ try:+ attached = bool(probe())+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one+ attached = False+ if not attached:+ detached.append(name)+ return detached+++def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:+ """Make ``/readyz`` depend on *probe* returning true.++ Call this once per subscription the service owns, after+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —+ it runs on every ``/readyz`` request — so it should read a flag the+ consumer maintains or check ``task.done()``, not round-trip to a broker.+ """+ probes = getattr(app.state, _STATE_ATTR, None)+ if probes is None: # pragma: no cover — install_health_routes was not called+ raise RuntimeError("register_subscription requires install_health_routes(app) first")+ probes[name] = probe+def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:"""Install ``/livez`` and ``/readyz`` on ``app``.@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[``mark_ready()`` is called — exactly the right behaviour for arolling deploy where the orchestrator should hold trafficuntil the pod's dependencies are connected.++ Startup having finished is necessary and not sufficient: any+ subscription registered with :func:`register_subscription`+ must also still be attached. See the module docstring."""state: dict[str, bool] = {"ready": False}+ probes: dict[str, Callable[[], bool]] = {}+ setattr(app.state, _STATE_ATTR, probes)@app.get("/livez",@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[include_in_schema=False,)async def _readyz() -> Response:- if state["ready"]:- return Response(- content='{"status":"ready","service":"' + service_name + '"}',- media_type="application/json",- status_code=200,- )+ body: dict[str, Any] = {"status": "ready", "service": service_name}+ status_code = 200++ if not state["ready"]:+ body["status"] = "starting"+ status_code = 503+ else:+ detached = _probe_detached(probes)+ if detached:+ body["status"] = "degraded"+ body["detached"] = detached+ status_code = 503++ # Named subscriptions are reported either way, so a 200 says which+ # ones were checked rather than only that nothing was wrong. A probe+ # list that is unexpectedly empty is then visible in the same place+ # the verdict is.+ if probes:+ body["subscriptions"] = sorted(probes)+return Response(- content='{"status":"starting","service":"' + service_name + '"}',+ content=json.dumps(body),media_type="application/json",- status_code=503,+ status_code=status_code,)def mark_ready() -> None:diff --git a/services/threatintel/app/_health.py b/services/threatintel/app/_health.pyindex 974269731..0acb9e4da 100644--- a/services/threatintel/app/_health.py+++ b/services/threatintel/app/_health.py@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /``mark_not_ready()`` from those handlers respectively.+Readiness is a question, not a latch+------------------------------------++``mark_ready()`` answers "did startup finish", and for a service+whose only job is to answer HTTP that is the whole story. For a+service whose job is to *consume a topic* it is not: three+services in this repository created the consumer as a+fire-and-forget ``asyncio.create_task`` and called+``mark_ready()`` on the next line, so readiness recorded that the+task had been **created**, never that it was still running.++UEBA is what that cost. Its handler raised+``UndefinedColumnError`` on the first scoreable event, the+exception left the ``async for``, the ``finally`` stopped the+consumer, and the task ended — holding the exception, which+nothing retrieved because the task was parked in a module global+and so never garbage-collected. The container stayed ``running``+with restarts 0, ``/health`` returned 200 and ``/readyz``+returned 200, permanently. A consumer that has silently detached+is indistinguishable from an idle one, and the whole point of a+readiness probe is to tell those two apart.++So a service that owns a subscription registers a probe for it+with :func:`register_subscription`, and ``/readyz`` evaluates+every probe on each request. A probe that returns false — or+raises — is reported as detached and ``/readyz`` answers 503 with+the names, which is a readiness failure rather than a liveness+one on purpose: the orchestrator should stop routing to a+half-working replica, and killing the process would only hide a+schema or broker fault behind a restart loop.++``/livez`` deliberately does **not** consult the probes. It+answers "is the interpreter responsive", and wiring a detached+consumer into it would make every orchestrator restart the pod on+a fault a restart cannot fix.+This module is intentionally dependency-free (only FastAPI) so itcan be vendored into every service tree without import-graphside-effects. Each service ships its own copy under``services/<svc>/app/_health.py`` because services don't share a-Python path — keep the file in sync via-``scripts/sync_health_module.py`` (CI gate).+Python path; ``scripts/audit_health_probes.py --check`` is the CI+gate that every service has one and wires it in."""from __future__ import annotations+import jsonfrom collections.abc import Callablefrom typing import Anyfrom fastapi import FastAPI, Response+#: Where the probes registered against an app are kept. Read by ``/readyz``,+#: written by :func:`register_subscription`. On ``app.state`` rather than in a+#: module global because a module global is shared by every app in the+#: process, which makes two test clients interfere.+_STATE_ATTR = "health_subscriptions"+++def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:+ """Names of the subscriptions currently reporting not-attached.++ A probe that raises counts as detached. It is asking a live object+ whether it is still working; if that question itself fails, the honest+ answer is not "yes".+ """+ detached: list[str] = []+ for name, probe in sorted(probes.items()):+ try:+ attached = bool(probe())+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one+ attached = False+ if not attached:+ detached.append(name)+ return detached+++def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:+ """Make ``/readyz`` depend on *probe* returning true.++ Call this once per subscription the service owns, after+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —+ it runs on every ``/readyz`` request — so it should read a flag the+ consumer maintains or check ``task.done()``, not round-trip to a broker.+ """+ probes = getattr(app.state, _STATE_ATTR, None)+ if probes is None: # pragma: no cover — install_health_routes was not called+ raise RuntimeError("register_subscription requires install_health_routes(app) first")+ probes[name] = probe+def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:"""Install ``/livez`` and ``/readyz`` on ``app``.@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[``mark_ready()`` is called — exactly the right behaviour for arolling deploy where the orchestrator should hold trafficuntil the pod's dependencies are connected.++ Startup having finished is necessary and not sufficient: any+ subscription registered with :func:`register_subscription`+ must also still be attached. See the module docstring."""state: dict[str, bool] = {"ready": False}+ probes: dict[str, Callable[[], bool]] = {}+ setattr(app.state, _STATE_ATTR, probes)@app.get("/livez",@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[include_in_schema=False,)async def _readyz() -> Response:- if state["ready"]:- return Response(- content='{"status":"ready","service":"' + service_name + '"}',- media_type="application/json",- status_code=200,- )+ body: dict[str, Any] = {"status": "ready", "service": service_name}+ status_code = 200++ if not state["ready"]:+ body["status"] = "starting"+ status_code = 503+ else:+ detached = _probe_detached(probes)+ if detached:+ body["status"] = "degraded"+ body["detached"] = detached+ status_code = 503++ # Named subscriptions are reported either way, so a 200 says which+ # ones were checked rather than only that nothing was wrong. A probe+ # list that is unexpectedly empty is then visible in the same place+ # the verdict is.+ if probes:+ body["subscriptions"] = sorted(probes)+return Response(- content='{"status":"starting","service":"' + service_name + '"}',+ content=json.dumps(body),media_type="application/json",- status_code=503,+ status_code=status_code,)def mark_ready() -> None:diff --git a/services/ueba/Dockerfile b/services/ueba/Dockerfileindex 810b570f0..7ce8f612f 100644--- a/services/ueba/Dockerfile+++ b/services/ueba/Dockerfile@@ -25,4 +25,17 @@ COPY app/ app/EXPOSE 8004-CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8004"]+# The chain runs on the way up, then this process is replaced by the server.+#+# Nothing invoked these four alembic chains. This was a bare `uvicorn` line, so+# the documented quickstart booted the service against an empty schema with+# none of its row-level-security policies installed, and every command in that+# sequence reported success. `app/_migrate.py` applies the chain as the owner+# credential, verifies it reached head, and refuses to start the server if it+# did not.+#+# `sh -c` because an exec-form CMD cannot run two things. Neither shell nor+# wrapper stays in the process tree: `sh -c` with a single command execs it,+# and `_migrate` hands over with `os.execvp`, so uvicorn ends up as PID 1 and+# receives SIGTERM directly rather than through a parent that would swallow it.+CMD ["sh", "-c", "python -m app._migrate -- uvicorn app.main:app --host 0.0.0.0 --port 8004"]diff --git a/services/ueba/alembic/env.py b/services/ueba/alembic/env.pyindex b7318aeda..12e7567c2 100644--- a/services/ueba/alembic/env.py+++ b/services/ueba/alembic/env.py@@ -157,9 +157,27 @@ def run_migrations_offline() -> None:context.run_migrations()+#: Serialises the four chains that share one database against each other.+#:+#: They are started simultaneously by ``docker compose up``, and while each+#: now owns its own version table, they do not own their own *database*: the+#: legacy-version probe below reads a table a sibling may be creating, and+#: ``000N_runtime_role_grants`` issues ``GRANT`` statements that take+#: catalog locks on ``pg_class``. Two chains arriving together deadlocked+#: rather than queued.+#:+#: ``pg_advisory_xact_lock`` and not the session form: it is released by the+#: commit or rollback that ends the upgrade, so a chain that dies mid-apply+#: cannot leave the other three waiting on a lock nobody holds a connection+#: for. The key is an arbitrary constant, shared by every chain on purpose —+#: the point is mutual exclusion, not per-chain exclusion.+MIGRATION_LOCK_KEY = 0x4149_5354++def do_run_migrations(connection): # type: ignore[no-untyped-def]context.configure(connection=connection, target_metadata=target_metadata, version_table=VERSION_TABLE)with context.begin_transaction():+ connection.execute(sa.text("SELECT pg_advisory_xact_lock(:key)"), {"key": MIGRATION_LOCK_KEY})# Inside ``begin_transaction()``, not before it. Alembic's# ``begin_transaction`` is a no-op when the connection is *already* in a# transaction, and in SQLAlchemy 2.0 any execute on a plain ``connect()``diff --git a/services/ueba/alembic/versions/0004_baseline_peer_group_and_column_parity.py b/services/ueba/alembic/versions/0004_baseline_peer_group_and_column_parity.pynew file mode 100644index 000000000..d4293a7f5--- /dev/null+++ b/services/ueba/alembic/versions/0004_baseline_peer_group_and_column_parity.py@@ -0,0 +1,90 @@+"""Bring the UEBA tables back in line with the models that read them.++Revision ID: 0004+Revises: 0003+Create Date: 2026-09-24++``0001`` and ``app/models/ueba.py`` disagreed in three places, and every one of+them was reached on the first scoreable event rather than on some rare path:++``ueba_entity_baselines.peer_group_id``+ Declared by :class:`~app.models.ueba.EntityBaseline`, created by no+ migration. SQLAlchemy names every mapped column in its ``SELECT``, so+ ``BaselineService.get_or_create`` — the first database call+ ``score_event`` makes — raised ``UndefinedColumnError``. UEBA could+ therefore never write a baseline or an anomaly, on any deployment, and+ the container stayed healthy throughout.++``ueba_peer_groups.id``+ Created as ``UUID DEFAULT gen_random_uuid()``, declared as+ ``String(64)``, and written with values like ``dept:engineering`` — the+ ids are externally-supplied group names, not surrogate keys, which is+ why ``PeerGroupService.get_or_create`` passes one explicitly. An insert+ raised ``InvalidTextRepresentation``. The column follows the model+ because the model matches the data; a UUID here would mean the caller+ could not look a group up by the name it knows it by.++``ueba_anomalies.event_type``+ ``String(64)`` in the table against ``String(128)`` in the model, so an+ event type between the two lengths passed every test and failed at+ insert time in production. Widened to the declared length rather than+ narrowing the model, which would only move the failure.++The remaining differences are the harmless direction — ``observation_count``,+``created_at`` and ``z_scores`` exist in the tables and are not mapped — and+are recorded as such in ``scripts/check_orm_migration_parity.py`` rather than+dropped. A column the models never name costs a default and nothing else;+removing it would be an irreversible change made for tidiness.+"""++from __future__ import annotations++import sqlalchemy as sa+from alembic import op # type: ignore[attr-defined] # re-exported at runtime, absent from the stub++revision = "0004"+down_revision = "0003"+branch_labels = None+depends_on = None+++def upgrade() -> None:+ # Guarded so the revision is safe on a database whose tables were created+ # from the models instead of from this chain (``create_all`` in a test+ # fixture, and the sandbox), where the column is already present.+ op.execute("ALTER TABLE ueba_entity_baselines ADD COLUMN IF NOT EXISTS peer_group_id VARCHAR(64)")++ # ``USING id::text`` rather than a drop-and-add: an existing row holds a+ # group whose members' baselines point at it, and a generated UUID is not+ # a name anything can look up again.+ op.execute("ALTER TABLE ueba_peer_groups ALTER COLUMN id DROP DEFAULT")+ op.execute("ALTER TABLE ueba_peer_groups ALTER COLUMN id TYPE VARCHAR(64) USING id::text")++ op.alter_column(+ "ueba_anomalies",+ "event_type",+ existing_type=sa.String(64),+ type_=sa.String(128),+ existing_nullable=False,+ )+++def downgrade() -> None:+ op.alter_column(+ "ueba_anomalies",+ "event_type",+ existing_type=sa.String(128),+ type_=sa.String(64),+ existing_nullable=False,+ )++ # A group id that is not a UUID cannot be cast back to one, so the+ # downgrade discards the names rather than failing halfway: anything this+ # revision made storable is not representable in the old column. Stated+ # here because a downgrade that silently drops rows is worse than one that+ # says it will.+ op.execute("DELETE FROM ueba_peer_groups WHERE id !~* '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$'")+ op.execute("ALTER TABLE ueba_peer_groups ALTER COLUMN id TYPE UUID USING id::uuid")+ op.execute("ALTER TABLE ueba_peer_groups ALTER COLUMN id SET DEFAULT gen_random_uuid()")++ op.execute("ALTER TABLE ueba_entity_baselines DROP COLUMN IF EXISTS peer_group_id")diff --git a/services/ueba/app/_health.py b/services/ueba/app/_health.pyindex 974269731..0acb9e4da 100644--- a/services/ueba/app/_health.py+++ b/services/ueba/app/_health.py@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /``mark_not_ready()`` from those handlers respectively.+Readiness is a question, not a latch+------------------------------------++``mark_ready()`` answers "did startup finish", and for a service+whose only job is to answer HTTP that is the whole story. For a+service whose job is to *consume a topic* it is not: three+services in this repository created the consumer as a+fire-and-forget ``asyncio.create_task`` and called+``mark_ready()`` on the next line, so readiness recorded that the+task had been **created**, never that it was still running.++UEBA is what that cost. Its handler raised+``UndefinedColumnError`` on the first scoreable event, the+exception left the ``async for``, the ``finally`` stopped the+consumer, and the task ended — holding the exception, which+nothing retrieved because the task was parked in a module global+and so never garbage-collected. The container stayed ``running``+with restarts 0, ``/health`` returned 200 and ``/readyz``+returned 200, permanently. A consumer that has silently detached+is indistinguishable from an idle one, and the whole point of a+readiness probe is to tell those two apart.++So a service that owns a subscription registers a probe for it+with :func:`register_subscription`, and ``/readyz`` evaluates+every probe on each request. A probe that returns false — or+raises — is reported as detached and ``/readyz`` answers 503 with+the names, which is a readiness failure rather than a liveness+one on purpose: the orchestrator should stop routing to a+half-working replica, and killing the process would only hide a+schema or broker fault behind a restart loop.++``/livez`` deliberately does **not** consult the probes. It+answers "is the interpreter responsive", and wiring a detached+consumer into it would make every orchestrator restart the pod on+a fault a restart cannot fix.+This module is intentionally dependency-free (only FastAPI) so itcan be vendored into every service tree without import-graphside-effects. Each service ships its own copy under``services/<svc>/app/_health.py`` because services don't share a-Python path — keep the file in sync via-``scripts/sync_health_module.py`` (CI gate).+Python path; ``scripts/audit_health_probes.py --check`` is the CI+gate that every service has one and wires it in."""from __future__ import annotations+import jsonfrom collections.abc import Callablefrom typing import Anyfrom fastapi import FastAPI, Response+#: Where the probes registered against an app are kept. Read by ``/readyz``,+#: written by :func:`register_subscription`. On ``app.state`` rather than in a+#: module global because a module global is shared by every app in the+#: process, which makes two test clients interfere.+_STATE_ATTR = "health_subscriptions"+++def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:+ """Names of the subscriptions currently reporting not-attached.++ A probe that raises counts as detached. It is asking a live object+ whether it is still working; if that question itself fails, the honest+ answer is not "yes".+ """+ detached: list[str] = []+ for name, probe in sorted(probes.items()):+ try:+ attached = bool(probe())+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one+ attached = False+ if not attached:+ detached.append(name)+ return detached+++def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:+ """Make ``/readyz`` depend on *probe* returning true.++ Call this once per subscription the service owns, after+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —+ it runs on every ``/readyz`` request — so it should read a flag the+ consumer maintains or check ``task.done()``, not round-trip to a broker.+ """+ probes = getattr(app.state, _STATE_ATTR, None)+ if probes is None: # pragma: no cover — install_health_routes was not called+ raise RuntimeError("register_subscription requires install_health_routes(app) first")+ probes[name] = probe+def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:"""Install ``/livez`` and ``/readyz`` on ``app``.@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[``mark_ready()`` is called — exactly the right behaviour for arolling deploy where the orchestrator should hold trafficuntil the pod's dependencies are connected.++ Startup having finished is necessary and not sufficient: any+ subscription registered with :func:`register_subscription`+ must also still be attached. See the module docstring."""state: dict[str, bool] = {"ready": False}+ probes: dict[str, Callable[[], bool]] = {}+ setattr(app.state, _STATE_ATTR, probes)@app.get("/livez",@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[include_in_schema=False,)async def _readyz() -> Response:- if state["ready"]:- return Response(- content='{"status":"ready","service":"' + service_name + '"}',- media_type="application/json",- status_code=200,- )+ body: dict[str, Any] = {"status": "ready", "service": service_name}+ status_code = 200++ if not state["ready"]:+ body["status"] = "starting"+ status_code = 503+ else:+ detached = _probe_detached(probes)+ if detached:+ body["status"] = "degraded"+ body["detached"] = detached+ status_code = 503++ # Named subscriptions are reported either way, so a 200 says which+ # ones were checked rather than only that nothing was wrong. A probe+ # list that is unexpectedly empty is then visible in the same place+ # the verdict is.+ if probes:+ body["subscriptions"] = sorted(probes)+return Response(- content='{"status":"starting","service":"' + service_name + '"}',+ content=json.dumps(body),media_type="application/json",- status_code=503,+ status_code=status_code,)def mark_ready() -> None:diff --git a/services/ueba/app/_migrate.py b/services/ueba/app/_migrate.pynew file mode 100644index 000000000..1297f113b--- /dev/null+++ b/services/ueba/app/_migrate.py@@ -0,0 +1,228 @@+"""Apply this service's alembic chain, then exec what comes after it.++Why this exists+---------------++Four services in this repository own their schema through alembic —+``ueba``, ``honeytokens``, ``purple-team`` and ``osquery-tls`` — and **no+compose path invoked any of them**. Their container command was a plain+``uvicorn``, so on the documented quickstart they booted healthy against+empty schemas with none of their row-level-security policies installed. The+only documented invocation was a manual step in a docs-portal quickstart+whose own "start the stack" step does not start three of the four services,+and whose first line failed outright.++That is how two services shipped with no tables while every command+reported success. The API does not have this problem because its chain runs+from its own process on the way up; this is the same idea for the four that+use alembic.++What it guarantees+------------------++**Idempotent.** ``alembic upgrade head`` on a database already at head+applies nothing. Running this on every container start is the point: there+is no separate "have you migrated yet" state for an operator to get wrong.++**Race-free.** Four chains start at once against one database. The chains+take a shared transaction-scoped advisory lock (``MIGRATION_LOCK_KEY`` in+each ``env.py``), so they queue instead of deadlocking on catalog locks.+This wrapper deliberately holds no lock of its own — a lock held here would+be a second mechanism to reason about, and it would not cover the window+that actually matters.++**Loud.** The service does not start unless the chain reached head. Not "the+command exited 0": alembic exits 0 when it applies *nothing*, which is+exactly what happened when four chains shared one ``alembic_version`` table+and the second one to run believed it was already done. So ``current`` is+read back and compared to ``heads``, and a mismatch — or an empty+``current``, meaning the chain never ran — stops the container with the+reason on stderr. A service booting on an empty schema is the failure this+replaces, and it must not be reachable by a wrapper that only checks an exit+code.++Why the owner credential, and why a subprocess+----------------------------------------------++Alembic issues DDL. The runtime role every service serves requests as holds+no ``CREATE`` on schema public, by design — that is what stops it turning+row-level security off — so the chain must be applied as the owner. Each+``env.py`` already resolves that (its own ``*_DATABASE_MIGRATION_URL``+first, then ``DATABASE_MIGRATION_URL``), and this runs alembic as a+subprocess precisely so ``env.py`` stays the single answer to "which+credential". A second resolution here would be a second thing to keep in+step, and the two would disagree the first time either changed.++This module is vendored per service because services do not share a Python+path. ``scripts/check_service_migration_bootstrap.py`` is the gate that each+of the four ships a copy, keeps them identical, and runs one on the way up.++Usage+-----++::++ python -m app._migrate # apply the chain, exit+ python -m app._migrate -- uvicorn app.main:app --port 8004+"""++from __future__ import annotations++import os+import subprocess+import sys+from pathlib import Path++#: Where ``alembic.ini`` sits relative to the working directory in every one+#: of the four images (``WORKDIR /app``, and the ini is copied to ``/app``).+ALEMBIC_INI = "alembic.ini"++#: Suffix that marks a DSN as the migration (owner) credential. Matches the+#: convention each ``env.py`` documents and ``scripts/check_runtime_db_role.py``+#: enforces on the deployment surfaces: a service-prefixed+#: ``<SVC>_DATABASE_MIGRATION_URL`` wins over the bare form.+MIGRATION_URL_SUFFIX = "DATABASE_MIGRATION_URL"+++def migration_credential_present(env: dict[str, str] | None = None) -> bool:+ """Whether an owner DSN is configured.++ Used only for the warning below. The chain is applied either way — a+ deployment that has not split the roles yet behaves as it did before —+ but it is announced, because a chain silently applied as the runtime+ credential fails on the first ``CREATE TABLE`` and the message names a+ permission rather than a configuration.+ """+ source = os.environ if env is None else env+ return any(name.endswith(MIGRATION_URL_SUFFIX) and value.strip() for name, value in source.items())+++def _alembic(args: list[str], *, capture: bool) -> subprocess.CompletedProcess[str]:+ return subprocess.run( # noqa: S603 — fixed argv, no shell+ [sys.executable, "-m", "alembic", "-c", ALEMBIC_INI, *args],+ capture_output=capture,+ text=True,+ check=False,+ )+++def _revisions(output: str) -> set[str]:+ """Revision ids in ``alembic current`` / ``alembic heads`` output.++ Both print ``<rev> (head)`` or ``<rev>``, one per line, with informational+ lines alembic sends to stderr. An empty set means the version table holds+ nothing — the chain has not run — which is the case this exists to catch+ and is therefore never treated as agreement.+ """+ found: set[str] = set()+ for line in output.splitlines():+ token = line.strip().split(" ")[0].strip()+ if token and not token.startswith(("INFO", "WARNING", "ERROR", "alembic:")):+ found.add(token)+ return found+++def apply_chain() -> int:+ """Apply the chain and verify it reached head. Returns an exit code."""+ if not Path(ALEMBIC_INI).is_file():+ print(+ f"migrate: {ALEMBIC_INI} not found in {Path.cwd()}. This service's image must "+ f"copy its alembic config and chain; see its Dockerfile.",+ file=sys.stderr,+ )+ return 1++ if not migration_credential_present():+ print(+ f"migrate: no *{MIGRATION_URL_SUFFIX} is set, so the chain will be applied as "+ f"the runtime credential. That works only while the two are the same role; "+ f"under the DML-only runtime role the first CREATE TABLE fails with "+ f"'permission denied for schema public'.",+ file=sys.stderr,+ )++ upgrade = _alembic(["upgrade", "head"], capture=False)+ if upgrade.returncode != 0:+ print(+ f"migrate: 'alembic upgrade head' failed (exit {upgrade.returncode}). Refusing to "+ f"start: this service would serve requests against whatever schema is there now.",+ file=sys.stderr,+ )+ return upgrade.returncode++ # Read back rather than trusting the exit code. `alembic upgrade head`+ # exits 0 having applied nothing when it believes the database is already+ # at head, and believing that wrongly is the documented way these four+ # services ended up with no tables.+ current = _alembic(["current"], capture=True)+ heads = _alembic(["heads"], capture=True)+ if current.returncode != 0 or heads.returncode != 0:+ print(+ "migrate: could not read back the applied revision "+ f"(current exit {current.returncode}, heads exit {heads.returncode}). "+ "Refusing to start: an unverified migration is the state this check exists "+ "to reject.\n"+ f"{(current.stderr or '') + (heads.stderr or '')}",+ file=sys.stderr,+ )+ return 1++ applied = _revisions(current.stdout)+ expected = _revisions(heads.stdout)++ if not expected:+ print(+ "migrate: this chain declares no head revision, so there is nothing to verify "+ "against. Refusing to start rather than reporting an empty chain as applied.",+ file=sys.stderr,+ )+ return 1++ if not applied:+ print(+ f"migrate: 'alembic upgrade head' reported success but the version table is "+ f"empty — the chain never ran. Expected {sorted(expected)}. Refusing to start "+ f"on an unmigrated schema.",+ file=sys.stderr,+ )+ return 1++ if not expected.issubset(applied):+ print(+ f"migrate: chain did not reach head. At {sorted(applied)}, head is " f"{sorted(expected)}. Refusing to start.",+ file=sys.stderr,+ )+ return 1++ print(f"migrate: schema at head {sorted(expected)}", file=sys.stderr)+ return 0+++def main(argv: list[str] | None = None) -> int:+ args = list(sys.argv[1:] if argv is None else argv)++ # Everything after `--` is the real service command. Exec'd rather than+ # spawned so the server keeps PID 1 and receives SIGTERM directly — a+ # Python parent waiting on a child swallows the orchestrator's stop+ # signal and turns every shutdown into a ten-second kill.+ command: list[str] = []+ if "--" in args:+ split = args.index("--")+ command = args[split + 1 :]+ args = args[:split]++ if args:+ print(f"migrate: unexpected arguments {args}; usage: python -m app._migrate [-- CMD ...]", file=sys.stderr)+ return 2++ code = apply_chain()+ if code != 0:+ return code++ if command:+ os.execvp(command[0], command) # noqa: S606 — replaces this process by design+ return 0+++if __name__ == "__main__":+ sys.exit(main())diff --git a/services/ueba/app/main.py b/services/ueba/app/main.pyindex a9166d041..bec289484 100644--- a/services/ueba/app/main.py+++ b/services/ueba/app/main.py@@ -3,12 +3,14 @@from __future__ import annotationsimport asyncio+import jsonimport logging+from typing import Any-from fastapi import FastAPI+from fastapi import FastAPI, Responsefrom fastapi.middleware.cors import CORSMiddleware-from app._health import install_health_routes+from app._health import install_health_routes, register_subscriptionfrom app.core.config import settingsfrom app.core.cors import build_cors_kwargs@@ -53,8 +55,12 @@ app = FastAPI()# Phase 2.6 — k8s liveness + readiness probes (see app/_health.py).-# /readyz flips on once the Kafka consumer task is created in the-# startup hook.+#+# /readyz used to flip on the moment the Kafka consumer task was *created*,+# which is not the same thing as it running: the first scoreable event raised+# UndefinedColumnError, the task ended, and this endpoint kept answering 200.+# The startup hook now registers a probe over the consumer itself, so+# readiness is re-evaluated per request instead of latched once._mark_ready, _mark_not_ready = install_health_routes(app, service_name="aisoc-ueba")app.state.mark_ready = _mark_readyapp.state.mark_not_ready = _mark_not_ready@@ -77,15 +83,49 @@ if _otel_enabled:# Kafka consumer lifecycle# ---------------------------------------------------------------------------_consumer_task: asyncio.Task | None = None # type: ignore[type-arg]+_consumer: Any = None+++def _log_consumer_exit(task: asyncio.Task) -> None: # type: ignore[type-arg]+ """Retrieve the consumer task's outcome and say what it was.++ This callback is the reason a dead consumer is now findable. The task was+ parked in a module global, so it was never garbage-collected, so asyncio+ never emitted its "Task exception was never retrieved" warning — the+ exception that stopped UEBA from ever writing an anomaly existed only+ inside an object nobody read.+ """+ if task.cancelled():+ LOG.info("UEBA Kafka consumer task cancelled (shutdown).")+ return+ exc = task.exception()+ if exc is None:+ LOG.error(+ "UEBA Kafka consumer task ended without an error. The subscription is gone "+ "and no further events will be scored; /readyz now reports 503."+ )+ return+ LOG.error(+ "UEBA Kafka consumer task died: %s: %s. The subscription is gone and no further " "events will be scored; /readyz now reports 503.",+ type(exc).__name__,+ exc,+ exc_info=exc,+ )@app.on_event("startup")async def _start_kafka() -> None:- global _consumer_task+ global _consumer_task, _consumerfrom app.services.kafka_consumer import UEBAKafkaConsumer- consumer = UEBAKafkaConsumer()- _consumer_task = asyncio.create_task(consumer.run(), name="ueba-kafka-consumer")+ _consumer = UEBAKafkaConsumer()+ _consumer_task = asyncio.create_task(_consumer.run(), name="ueba-kafka-consumer")+ _consumer_task.add_done_callback(_log_consumer_exit)++ # Readiness depends on the consumer still being attached, not on this+ # function having reached its last line.+ register_subscription(app, settings.kafka_input_topic, lambda: bool(_consumer and _consumer.attached))+LOG.info("UEBA service started (OTel=%s)", _otel_enabled)# Phase 2.6 — Kafka consumer is up; HTTP surface is serving.app.state.mark_ready()@@ -105,5 +145,33 @@ async def _stop_kafka() -> None:@app.get("/health")-async def health() -> dict:- return {"status": "ok", "service": settings.service_name}+async def health() -> Response:+ """Service status, including whether the consumer is actually consuming.++ This endpoint returned a hardcoded ``{"status": "ok"}`` and so reported+ exactly the same thing whether UEBA was scoring every event or had been+ detached from its topic since the first one. It now answers 503 when the+ consumer is not attached, and carries the counters — including the last+ handler error — that say which of the two it is.+ """+ attached = bool(_consumer and _consumer.attached)+ body: dict[str, Any] = {+ "status": "ok" if attached else "degraded",+ "service": settings.service_name,+ "consumer": {+ "topic": settings.kafka_input_topic,+ "attached": attached,+ **(_consumer.stats() if _consumer else {}),+ },+ }+ if not attached:+ body["consumer"]["hint"] = (+ "the subscription is not attached — see this service's logs for the "+ "consumer task's exception, and GET /api/v1/health/dead-letters for "+ "events it refused"+ )+ return Response(+ content=json.dumps(body),+ media_type="application/json",+ status_code=200 if attached else 503,+ )diff --git a/services/ueba/app/services/baseline.py b/services/ueba/app/services/baseline.pyindex a209e4e9f..f0ed35e9b 100644--- a/services/ueba/app/services/baseline.py+++ b/services/ueba/app/services/baseline.py@@ -27,7 +27,7 @@ def _welford_update(feature: str,value: float,) -> dict[str, dict[str, float]]:- """Update incremental mean/variance for *feature* with new *value*.+ """Return *stats* with *feature* advanced by one observation of *value*.Uses Welford's online algorithm:n ← n + 1@@ -36,11 +36,32 @@ def _welford_update(δ2 ← x - meanM2 ← M2 + δ·δ2variance = M2 / (n−1) for n > 1++ Returns new dictionaries and mutates neither *stats* nor the per-feature+ dictionaries inside it. That is not a style preference — it is the+ difference between a baseline that accumulates and one that does not.++ This used to edit ``stats[feature]`` in place. Its caller passed a+ *shallow* copy of ``EntityBaseline.feature_stats``, so the per-feature+ dictionary being edited was the very object SQLAlchemy had loaded from+ the column. By the time the caller assigned the result back, the loaded+ value and the new value were equal, and SQLAlchemy decides whether to+ include a column in an ``UPDATE`` by comparing exactly those two — a+ ``JSON``/``JSONB`` column has no change tracking of its own unless it is+ wrapped in ``MutableDict``. So the statement went out without the column.++ Measured on a live stack: 36 events for one entity, every one processed+ without error, and the stored baseline stayed at ``count: 1`` with the+ first event's mean while ``window_end`` — an ordinary timestamp column,+ with a value that genuinely differed — advanced on every one. No entity+ could ever reach ``min_baseline_samples``, so ``compute_z_score``+ returned ``None`` for every feature of every event, so no anomaly was+ ever scored. The service logged ``features_unscoreable`` 36 times, which+ reads as "this entity is too quiet to score" rather than as a bug."""- if feature not in stats:- stats[feature] = {"mean": 0.0, "M2": 0.0, "count": 0}+ current = stats.get(feature)+ s: dict[str, float] = dict(current) if current else {"mean": 0.0, "M2": 0.0, "count": 0}- s = stats[feature]s["count"] += 1n = s["count"]delta = value - s["mean"]@@ -52,7 +73,7 @@ def _welford_update(variance = s["M2"] / (n - 1) if n > 1 else 0.0s["std"] = math.sqrt(variance)- return stats+ return {**stats, feature: s}def compute_z_score(@@ -145,7 +166,10 @@ class BaselineService:) -> EntityBaseline:"""Incrementally update the baseline with new feature observations."""baseline = await self.get_or_create(tenant_id, entity_type, entity_id)- stats = dict(baseline.feature_stats) # copy so SQLAlchemy detects mutation+ # A shallow copy is enough only because ``_welford_update`` replaces+ # the per-feature dictionary rather than editing it; see its docstring+ # for what happens when it does not.+ stats = dict(baseline.feature_stats)for feature, value in features.items():stats = _welford_update(stats, feature, value)diff --git a/services/ueba/app/services/dead_letters.py b/services/ueba/app/services/dead_letters.pynew file mode 100644index 000000000..52076669e--- /dev/null+++ b/services/ueba/app/services/dead_letters.py@@ -0,0 +1,125 @@+"""Record events this service refused, where an operator already looks.++The UEBA consumer had no ``except`` around its handler at all, so the first+event that raised ended the ``async for``, ran the ``finally`` that stopped+the consumer, and left the container ``running`` with ``/health`` answering+200. Nothing was logged, because the task holding the exception lived in a+module global and was never garbage-collected, so asyncio's+"exception was never retrieved" warning never fired either.++Guarding the handler is half the fix; the other half is that a guarded+failure has to go somewhere. It goes to ``aisoc_dead_letters`` — the table+``services/fusion`` already writes and ``GET /api/v1/health/dead-letters``+already reads, with a per-reason breakdown that turns fifty identical rows+into one finding. A second dead-letter path would mean an operator had to+know to check two.++Two properties, both inherited deliberately from+``services/fusion/app/services/dlq_sink.py``:++**A failure to record must not become a second failure.** The sink exists to+let the consumer keep going; if writing the record can raise, one bad event+stops the stream anyway. Every error here is logged and swallowed.++**The payload is truncated, never stored whole.** It is the thing that was+refused, so it is untrusted by definition and may be large.+"""++from __future__ import annotations++import json+from typing import Any++import structlog+from sqlalchemy import text+from sqlalchemy.ext.asyncio import async_sessionmaker++logger = structlog.get_logger(__name__)++#: Characters of payload kept. Matches the fusion sink so the same column+#: holds comparable rows: enough to recognise the shape and find the+#: producer, far short of enough to replay it.+MAX_EXCERPT = 2000++#: This service's schema version marker on the shared table. The column is+#: ``NOT NULL DEFAULT 'unknown'`` and "unknown" would be a worse answer than+#: naming the envelope the consumer was reading.+SCHEMA_VERSION = "ueba.raw_events.v1"++_INSERT_SQL = text(+ """+ INSERT INTO aisoc_dead_letters+ (tenant_id, topic, reason, schema_version, payload_excerpt, source_event_id)+ VALUES (:tenant_id, :topic, :reason, :schema_version, :payload_excerpt, :source_event_id)+ """+)+++def _excerpt(payload: Any) -> str:+ try:+ return json.dumps(payload, default=str)[:MAX_EXCERPT]+ except (TypeError, ValueError):+ return str(payload)[:MAX_EXCERPT]+++class DeadLetterSink:+ """Persist a refused event to the shared dead-letter table."""++ def __init__(self, session_factory: async_sessionmaker) -> None:+ self._session_factory = session_factory++ async def record(+ self,+ *,+ topic: str,+ reason: str,+ payload: Any,+ tenant_id: str | None = None,+ source_event_id: str | None = None,+ ) -> bool:+ """Write one dead letter. Returns whether it landed.++ The return value is for the caller's counters, not for control flow —+ there is no branch here a consumer should take differently.+ """+ params = {+ "tenant_id": tenant_id,+ "topic": topic,+ "reason": reason[:1000],+ "schema_version": SCHEMA_VERSION,+ "payload_excerpt": _excerpt(payload),+ "source_event_id": source_event_id,+ }++ # Two attempts, and the second is not a retry of the same statement.+ # ``tenant_id`` carries a foreign key to ``tenants``, and UEBA's+ # tenants come from the event envelope rather than from a join — an+ # event naming a tenant this database has never seen would otherwise+ # lose the dead letter to a constraint violation, and that event is+ # precisely the evidence that something upstream is producing them.+ # The reason string already names the tenant, so nothing is lost by+ # detaching the column.+ last_error: Exception | None = None+ for attempt_params in (params, {**params, "tenant_id": None}):+ try:+ async with self._session_factory() as session:+ async with session.begin():+ await session.execute(_INSERT_SQL, attempt_params)+ return True+ except Exception as exc: # noqa: BLE001 — see the module docstring+ last_error = exc+ if attempt_params["tenant_id"] is None:+ break++ logger.warning(+ "ueba.dead_letter_persist_failed",+ topic=topic,+ reason=reason[:200],+ error=str(last_error)[:300],+ hint=(+ "the refused event is in the log line above this one; "+ "aisoc_dead_letters is created by services/api migration "+ "053_dead_letters.sql"+ ),+ )+ return Falsediff --git a/services/ueba/app/services/kafka_consumer.py b/services/ueba/app/services/kafka_consumer.pyindex 609b6c9cd..4acf5e08f 100644--- a/services/ueba/app/services/kafka_consumer.py+++ b/services/ueba/app/services/kafka_consumer.py@@ -34,6 +34,31 @@ Anomalies are published to ``ueba.anomalies``:"features": {...},"detected_at": "2026-05-03T12:00:01Z"}++What happens when the handler raises+------------------------------------++It logs at ``error``, dead-letters the event, and carries on. A single+unprocessable event must not stop the stream, which is already the policy in+``services/fusion`` and ``services/agents`` — one answer across the three+consumers is worth more than a cleverer answer here.++What must never happen again is the previous behaviour. There was no+``except`` at all: the first event that raised left the ``async for``, the+``finally`` stopped the consumer, and the exception went into the task+object. Because the task was held in a module global it was never+garbage-collected, so asyncio never emitted "Task exception was never+retrieved". The container stayed ``running`` with restarts 0, ``/health``+answered 200, and ``/readyz`` answered 200, for as long as anyone left it up.++Continuing is only safe if a subscription that *does* end says so, so this+class tracks whether it is attached and ``app/main.py`` registers that as a+readiness probe. Detaching is not treated as recoverable in place: there is+no reconnect loop here, because a loop would turn a permanent fault — a+missing column, a revoked grant — into an indefinite retry that reads as+ordinary churn. The honest report is a 503 on ``/readyz``, the exception in+the log, and the restart decision left to whatever is supervising the+container."""from __future__ import annotations@@ -46,6 +71,7 @@ from typing import Anyfrom sqlalchemy.ext.asyncio import async_sessionmaker, create_async_enginefrom app.core.config import settings+from app.services.dead_letters import DeadLetterSinkfrom app.services.feature_extraction import extractfrom app.services.peer_group import PeerGroupServicefrom app.services.scoring import ScoringService@@ -58,6 +84,27 @@ class UEBAKafkaConsumer:self._engine = create_async_engine(settings.database_url, pool_size=5)self._session_factory = async_sessionmaker(self._engine, expire_on_commit=False)self._running = False+ # True only while the ``async for`` is iterating a started consumer.+ # Read through ``attached`` by the readiness probe; set False in the+ # loop's ``finally`` so every exit path — clean stop, broker loss,+ # unhandled error — is reported the same way.+ self._attached = False+ self._dlq = DeadLetterSink(self._session_factory)+ self._stats: dict[str, Any] = {+ "processed": 0,+ "handler_failures": 0,+ "dead_lettered": 0,+ "last_error": None,+ }++ @property+ def attached(self) -> bool:+ """Whether this consumer is currently subscribed and iterating."""+ return self._attached++ def stats(self) -> dict[str, Any]:+ """Counters for ``/health``. A copy, so a caller cannot edit them."""+ return dict(self._stats)async def _process_message(self, raw: bytes, producer: Any) -> None:try:@@ -151,17 +198,74 @@ class UEBAKafkaConsumer:await consumer.start()await producer.start()self._running = True+ self._attached = TrueLOG.info("UEBA Kafka consumer started (topic=%s)", settings.kafka_input_topic)try:async for msg in consumer:if not self._running:break- await self._process_message(msg.value, producer)+ try:+ await self._process_message(msg.value, producer)+ except Exception as exc: # noqa: BLE001 — one event must not end the subscription+ await self._handler_failed(msg, exc)+ else:+ self._stats["processed"] += 1finally:+ # Before the awaits below, not after: ``consumer.stop()`` can+ # itself raise on a broker that has gone away, and readiness must+ # already be reporting detached by then rather than staying 200+ # through a failed teardown.+ self._attached = Falseawait consumer.stop()await producer.stop()LOG.info("UEBA Kafka consumer stopped.")+ async def _handler_failed(self, msg: Any, exc: Exception) -> None:+ """Report one unprocessable event and keep the subscription.++ Loud on purpose. This is the exact point the old code exited from+ without writing anything anywhere, so the log line carries the+ partition and offset needed to find the event again, and+ ``exc_info`` carries the traceback that says which column or grant+ was missing.+ """+ self._stats["handler_failures"] += 1+ self._stats["last_error"] = f"{type(exc).__name__}: {str(exc)[:300]}"++ topic = getattr(msg, "topic", settings.kafka_input_topic)+ partition = getattr(msg, "partition", None)+ offset = getattr(msg, "offset", None)+ raw = getattr(msg, "value", msg)++ LOG.error(+ "UEBA handler failed; event dead-lettered and the subscription kept. " "topic=%s partition=%s offset=%s error=%s: %s",+ topic,+ partition,+ offset,+ type(exc).__name__,+ exc,+ exc_info=True,+ )++ payload: Any+ try:+ payload = json.loads(raw) if isinstance(raw, bytes | bytearray) else raw+ except (json.JSONDecodeError, UnicodeDecodeError):+ payload = {"undecodable": str(raw)[:500]}++ tenant_id = payload.get("tenant_id") if isinstance(payload, dict) else None+ source_event_id = payload.get("event_id") if isinstance(payload, dict) else None++ recorded = await self._dlq.record(+ topic=str(topic),+ reason=f"{type(exc).__name__}: {str(exc)[:400]}",+ payload=payload,+ tenant_id=str(tenant_id) if tenant_id else None,+ source_event_id=str(source_event_id) if source_event_id else None,+ )+ if recorded:+ self._stats["dead_lettered"] += 1+def stop(self) -> None:self._running = Falsediff --git a/services/ueba/tests/test_baseline_scoring.py b/services/ueba/tests/test_baseline_scoring.pyindex e83d1b478..e3116b8a5 100644--- a/services/ueba/tests/test_baseline_scoring.py+++ b/services/ueba/tests/test_baseline_scoring.py@@ -15,6 +15,7 @@ raise an anomaly.from __future__ import annotations+import jsonimport mathimport pytest@@ -192,6 +193,72 @@ class TestUnscoreableFeatures:assert _unscoreable_features({"a": {"z_score": 1.5}}) == []+class TestWelfordDoesNotMutateItsInput:+ """The property that decides whether a baseline accumulates at all.++ ``_welford_update`` is correct arithmetic either way, which is why the+ existing tests above passed while no entity in any deployment ever got+ past one observation. The caller hands it a shallow copy of the JSONB+ column SQLAlchemy loaded; editing the per-feature dictionary in place+ therefore edited the loaded value too, so the new value compared equal to+ the old one and SQLAlchemy left the column out of the ``UPDATE``+ entirely — a ``JSON`` column has no change tracking unless it is wrapped+ in ``MutableDict``.++ These assert the shape of the return value rather than the numbers, so+ they fail if anyone reintroduces in-place editing even if the statistics+ stay right.+ """++ def test_leaves_the_dict_it_was_given_untouched(self):+ original = {"f": {"mean": 10.0, "M2": 0.0, "count": 1, "std": 0.0}}+ before = json.dumps(original, sort_keys=True)++ _welford_update(original, "f", 20.0)++ assert json.dumps(original, sort_keys=True) == before++ def test_returns_a_distinct_per_feature_dict(self):+ # Identity, not equality: a returned dict that *is* the loaded one is+ # exactly what made the change invisible to the flush.+ original = {"f": {"mean": 10.0, "M2": 0.0, "count": 1, "std": 0.0}}++ updated = _welford_update(original, "f", 20.0)++ assert updated is not original+ assert updated["f"] is not original["f"]++ def test_a_shallow_copy_of_the_previous_value_still_differs(self):+ """The caller's exact pattern, asserted end to end.++ ``BaselineService.update`` does ``dict(baseline.feature_stats)`` and+ assigns the result back. If the two compare equal, the column is not+ written and the baseline is frozen at its first observation.+ """+ loaded = {"f": {"mean": 10.0, "M2": 0.0, "count": 1, "std": 0.0}}++ new_value = _welford_update(dict(loaded), "f", 20.0)++ assert new_value != loaded+ assert new_value["f"]["count"] == 2++ def test_accumulates_across_repeated_round_trips(self):+ """Thirty observations must produce a baseline of thirty.++ ``min_baseline_samples`` defaults to 30, so an updater that does not+ accumulate makes ``compute_z_score`` return ``None`` forever and no+ anomaly is ever scored. Measured on a live stack before the fix: 36+ events, ``count: 1``.+ """+ stored: dict[str, dict[str, float]] = {}+ for value in range(30):+ # Round-tripped through JSON each time, as a JSONB column does.+ stored = json.loads(json.dumps(_welford_update(dict(stored), "f", float(value))))++ assert stored["f"]["count"] == 30+ assert stored["f"]["std"] > 0++class TestRiskLevel:"""`critical` and `high` are hard-coded; only `medium` follows settings."""