Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

Make a dead consumer say so, and make the four alembic chains run (#831)

Beenu Arora1a9c0851b89a
diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml
index a210c2679..aa35c6e91 100644
--- a/.github/workflows/ci.yml
+++ b/.github/workflows/ci.yml
@@ -556,6 +556,32 @@ jobs:
- name: Verify FastAPI services install /livez + /readyz probes (Phase 2.6)
run: python scripts/audit_health_probes.py --check
+ # A column a model declares and no migration creates is not a latent
+ # problem: SQLAlchemy names every mapped column in its SELECT, so the
+ # first query against that table fails. `ueba_entity_baselines`
+ # declared `peer_group_id`, migration 0001 never created it, and UEBA
+ # could not write a baseline or an anomaly on any deployment while its
+ # container reported healthy throughout. Gated for the four services
+ # that own their schema through alembic and therefore have no
+ # `create_all` fallback; `--all` reports the rest as advisory.
+ #
+ # The self-test runs first, and deliberately: it injects one of each
+ # kind of drift into a scratch tree, so a parser that has quietly
+ # stopped matching cannot report the repository clean.
+ - name: Verify ORM columns exist in migrations (both directions)
+ run: |
+ python3 scripts/check_orm_migration_parity.py --self-test
+ python3 scripts/check_orm_migration_parity.py --check
+
+ # The four alembic chains have to be invoked by something. They were
+ # not: every container command was a plain `uvicorn`, so the
+ # documented quickstart booted two services with no tables and no RLS
+ # policies while reporting success at every step.
+ - name: Verify the alembic services run their chain on the way up
+ run: |
+ python3 scripts/check_service_migration_bootstrap.py --self-test
+ python3 scripts/check_service_migration_bootstrap.py --check
+
# Phase 4.2 — every managed-mode tenant manifest under
# `infra/fly/managed/tenants/*.yaml` must parse, carry the
# required fields (slug, display_name, base_domain, contact_email,
diff --git a/CHANGELOG.md b/CHANGELOG.md
index f64fd2ed7..68e58a685 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -530,6 +530,44 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
the view holds no graph at **first paint**, so a future `fallbackData` (which
disables revalidation, making a mock permanent rather than provisional)
fails the build.
+- **`scripts/check_orm_migration_parity.py` — a column a model declares and
+ no migration creates now fails the build.** Structural on both sides
+ (Python `ast` over the models and the alembic revisions, plus the raw SQL
+ those revisions and `services/api/migrations/*.sql` execute) and in both
+ directions: `missing-in-migration`, `missing-in-model`,
+ `type-narrower-in-migration`, `type-family-mismatch`, and
+ `no-migration-source` for a mapped table no migration mentions at all —
+ which is the finding that exists because pairing nothing produces zero
+ findings, and zero findings over zero comparisons prints the same word as
+ a clean result. `--credits` enumerates what each verdict was *based on*
+ for the same reason.
+
+ Scoped structurally rather than by a list: a service that calls
+ `metadata.create_all` builds its tables from the models, so a column no
+ migration creates is created anyway. `services/api` is the one such
+ service and is reported as advisory with its count printed, not hidden.
+ The four alembic-managed services have no such fallback, which is exactly
+ why UEBA's missing column was fatal.
+
+ Verified non-vacuous against the tree as it was: it reports all three of
+ the real UEBA defects. The `type-family-mismatch` kind exists *because*
+ an earlier version reported only two of them — a length comparison could
+ not see `UUID` against `String(64)`, since neither side declared a length
+ the other contradicted.
+
+- **`scripts/check_service_migration_bootstrap.py` — a migration chain
+ nothing invokes is not a migration chain.** Asserts every service with an
+ `alembic.ini` ships `app/_migrate.py`, keeps it byte-identical to the
+ others, runs it from its `CMD`/`ENTRYPOINT` (matched on the command lines,
+ so a mention in a comment does not count), and carries both an owner DSN
+ and a compose healthcheck. The other direction too: a service shipping
+ the module with no chain to apply would exit 1 on every start.
+
+- **`scripts/audit_health_probes.py` now checks the thirteen copies of
+ `app/_health.py` are identical.** The module's own docstring claimed they
+ were kept in sync and nothing verified it — they happened to be, so the
+ claim was true by luck. It also pointed at `scripts/sync_health_module.py`,
+ which does not exist in this tree.
- **Every test file in the tree is now executed by a workflow, and a gate
holds it that way in both directions.** 63 were executed by nothing at all:
@@ -2040,6 +2078,115 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
before the gate in CI and injects nine defects — a dropped connector, a
renamed one, a class that never registers, a deleted artefact, an unhooked
consumer, an empty corpus — requiring each to be caught.
+- **UEBA could never write a baseline or an anomaly, and said it was
+ healthy the whole time.** Three separate defects on one code path, each
+ reached by the first scoreable event rather than by some rare branch:
+
+ - `EntityBaseline.peer_group_id` was declared by the model and created by
+ no migration. SQLAlchemy names every mapped column in its `SELECT`, so
+ the first database call `score_event` makes raised
+ `UndefinedColumnError`.
+ - `ueba_peer_groups.id` was created `UUID DEFAULT gen_random_uuid()` and
+ declared `String(64)`, while `PeerGroupService` writes group names like
+ `dept:engineering` into it — `InvalidTextRepresentation` on insert. The
+ column follows the model, because the model matches the data.
+ - `ueba_anomalies.event_type` was 64 characters wide against a model
+ declaring 128, so an event type between the two lengths passed every
+ test and failed at insert time.
+
+ Fixed in `services/ueba/alembic/versions/0004_baseline_peer_group_and_column_parity.py`.
+
+- **And with the schema right, the baseline still could not accumulate.**
+ `_welford_update` edited the per-feature dictionary in place. Its caller
+ passes a *shallow* copy of `EntityBaseline.feature_stats`, so the object
+ being edited was the one SQLAlchemy had loaded from the column; by the
+ time the caller assigned the result back, old and new compared equal and
+ SQLAlchemy left the column out of the `UPDATE` — a `JSON`/`JSONB` column
+ has no change tracking unless wrapped in `MutableDict`. Measured on a
+ live stack: 36 events for one entity, every one processed without error,
+ and the stored baseline frozen at `count: 1` while `window_end` advanced
+ on every one. No entity could reach `min_baseline_samples` (30), so
+ `compute_z_score` returned `None` forever and no anomaly was ever scored.
+ The service logged `features_unscoreable` 36 times, which reads as "too
+ quiet to score" rather than as a bug. The updater now returns new
+ dictionaries and mutates nothing.
+
+- **A UEBA handler exception killed the subscription silently.** The
+ consume loop had no `except` at all: the first event that raised left the
+ `async for`, the `finally` stopped the consumer, and the exception went
+ into a task held in a module global — so it was never
+ garbage-collected, asyncio never emitted "Task exception was never
+ retrieved", and nothing was logged. The container stayed `running` with
+ restarts 0 and `/health` answered a hardcoded 200, permanently.
+
+ The policy is now the one `services/fusion` and `services/agents` already
+ follow: **log loudly, dead-letter, and keep the subscription.** A single
+ unprocessable event must not stop the stream. Refused events go to
+ `aisoc_dead_letters` — the table fusion already writes and
+ `GET /api/v1/health/dead-letters` already reads, with its per-reason
+ breakdown — rather than to a second dead-letter path nobody would know to
+ check.
+
+- **`/readyz` reported that a consumer task had been *created*, not that it
+ was running.** Three services created theirs as a fire-and-forget
+ `asyncio.create_task` and called `mark_ready()` on the next line
+ (`ueba`, `fusion`, `agents`). `app/_health.py` gains
+ `register_subscription(app, name, probe)`; `/readyz` evaluates every
+ registered probe per request and answers 503 naming the detached ones.
+ `/livez` deliberately still does not consult them — wiring a detached
+ consumer into liveness makes an orchestrator restart a pod over a fault a
+ restart cannot fix. Each of the three also gained a done-callback that
+ retrieves and logs its task's outcome, and UEBA's `/health` now carries
+ `attached`, the processed / failed / dead-lettered counters and the last
+ error instead of a hardcoded `{"status": "ok"}`.
+
+- **The same audit across every other consumer.** `services/fusion`'s
+ `_consume_loop` guarded `_process_message` but left
+ `lake.flush_if_stale()` outside the guard, so a ClickHouse outage raising
+ there would have ended the loop with the archive fault reported as
+ nothing; it is now guarded and the alert path survives it.
+ `services/agents`' `_process_with_retry` was already total, and the loop
+ now guards it anyway rather than depending on a callee staying that way.
+ `services/realtime` logged "Kafka consumer failed to start (will retry)"
+ and never retried — kafkajs reports a dead consumer through its `CRASH`
+ event, not by rejecting the promise, so a consumer that died after a
+ successful start stayed recorded as attached for the life of the process.
+ Both of its consumers now run under a supervisor with bounded backoff,
+ and `/health` answers 503 listing any detached topic instead of a
+ hardcoded `status: 'healthy'`.
+
+- **Nothing invoked the four alembic chains.** `honeytokens`,
+ `osquery-tls`, `purple-team` and `ueba` own their schemas through
+ alembic; every container command was a plain `uvicorn`, so the documented
+ quickstart booted them with empty schemas and none of their
+ row-level-security policies. Each image now runs `python -m app._migrate`
+ before its server: it applies the chain as the owner credential, takes a
+ shared transaction-scoped advisory lock so four chains starting at once
+ queue instead of deadlocking, reads the applied revision back, and
+ refuses to start the server unless it matches head. An exit code is not
+ evidence — `alembic upgrade head` exits 0 when it applies nothing, which
+ is exactly what happened when the chains shared one version table. All
+ four also gained a compose healthcheck; they had none, which is why a
+ container that had done nothing still read as `running`.
+
+- **`docker compose down && docker compose up` could not bring Kafka back.**
+ `kafka_data` is a named volume and ZooKeeper had none, so the second `up`
+ on kept data generated a fresh cluster id and the broker refused it with
+ `InconsistentClusterIdException`. `make down && make up` — documented as
+ "stop the stack, keep the data" — therefore left every Kafka-dependent
+ service in `created`, and only `make clean`, which deletes the data,
+ recovered. ZooKeeper now persists `/var/lib/zookeeper/data` and `/log`.
+
+- **`osquery-tls` published a container port nothing listened on.** Compose
+ mapped `127.0.0.1:8091:8007` while the service runs uvicorn on 9001. With
+ no healthcheck either, the container read as `running` against a closed
+ socket. Found by adding the healthcheck and watching it never leave
+ `starting`.
+
+- **`docker-compose.override.yml` was committed.** Its own header reads
+ "untracked, never committed": it is a QA port remap that moved PostgreSQL
+ to 55432 and ClickHouse to 58123/59000, contradicting every port the
+ documentation quotes. Removed.
- **Two more gates were exempt from the empty-corpus rule only by accident.**
`check_route_auth.py` and `check_tenant_query_predicates.py` did not exit 0
diff --git a/apps/docs/docs/deployment/env-vars.md b/apps/docs/docs/deployment/env-vars.md
index 2083c54d7..1c1825526 100644
--- a/apps/docs/docs/deployment/env-vars.md
+++ b/apps/docs/docs/deployment/env-vars.md
@@ -161,6 +161,41 @@ Two things changed alongside, both of which an operator can observe:
deployment is adopted automatically on the next `alembic upgrade`: the
recorded version is copied into the per-chain table when, and only when,
that chain's own tables are already present.
+
+##### Who runs the chain
+
+Nothing did. All four container commands were a plain `uvicorn`, so on the
+documented quickstart these services booted against empty schemas with none
+of their policies installed, and every command in that sequence reported
+success. The only documented invocation was a manual step, in a page whose
+"start the stack" step did not start three of the four services.
+
+Each image now runs `python -m app._migrate` before its server. That module:
+
+- resolves the owner DSN by the precedence in the table above — it runs
+ `alembic` as a subprocess so `env.py` stays the single answer to *which
+ credential*, rather than a second copy of the rule here;
+- takes a shared transaction-scoped advisory lock inside the upgrade
+ (`MIGRATION_LOCK_KEY` in each `env.py`), so four chains starting at once
+ against one database queue instead of deadlocking on catalog locks;
+- reads the applied revision back and compares it to head. `alembic upgrade
+ head` exits 0 when it applies **nothing**, which is exactly what happened
+ when the chains shared a version table — so an exit code is not evidence,
+ and the module refuses to start the server unless the version table
+ actually holds head;
+- provides no way to skip. A service that cannot reach head does not serve.
+
+`AISOC_APP_DB_PASSWORD` is passed to all four for the same reason the API
+gets it: the postgres init hook fires only on an empty data directory, so an
+existing volume never reaches it and the runtime role would have no
+credential to authenticate with.
+
+Two CI gates keep this true:
+`scripts/check_service_migration_bootstrap.py` asserts every service with an
+`alembic.ini` ships the module, keeps it identical to the others, invokes it
+from its `CMD`, and carries both an owner DSN and a healthcheck;
+`scripts/check_orm_migration_parity.py` asserts no model declares a column
+its migrations do not create.
| `AISOC_APP_DB_PASSWORD` | `aisoc_app_dev_secret` in compose; unset elsewhere | Password applied to the runtime role, by the postgres init hook on a fresh volume and by the migration runner on every run. Unset leaves the role's credential alone. |
| `DATABASE_POOL_SIZE` | `20` | SQLAlchemy pool size |
| `DATABASE_MAX_OVERFLOW` | `10` | SQLAlchemy max overflow |
diff --git a/apps/docs/docs/quickstart.md b/apps/docs/docs/quickstart.md
index 9d6f6e4a8..803ad0048 100644
--- a/apps/docs/docs/quickstart.md
+++ b/apps/docs/docs/quickstart.md
@@ -124,17 +124,34 @@ SECRET_KEY=change-me-in-production-at-least-32-chars
### 2. Start the full stack
+Most of the services below are behind a compose profile, so a bare
+`docker compose up -d` starts neither UEBA nor Honeytokens nor Purple Team.
+Name the profiles you want:
+
```bash
-docker compose up -d
-docker compose ps
+docker compose --profile full --profile extras up -d
+docker compose --profile full --profile extras ps
```
-**What this gives you about AI.** `docker compose up -d` starts the CORE
+That starts:
+
+| Profile | Services |
+| --- | --- |
+| *(default)* | **PostgreSQL** (5432) · **Redis** (6379) · **Kafka** (9092) · **litellm** (4000, LLM gateway) · **api** (8000, FastAPI core) · **agents** (8001, LangGraph) · **realtime** (8086, Node.js + VAPID Web Push) · **web** (3000) · **fusion** (8003) · **ingest** (8081, Go) |
+| `full` | **ClickHouse** · **OpenSearch** · **Neo4j** · **Qdrant** · **kafka-ui** (8080) · **actions** (8002) · **threatintel** (8005) · **ueba** (8007) · **enrichment** (8080, Go) · **connectors** (8003) |
+| `extras` | **honeytokens** (8008) · **purple-team** (8006) |
+| `osquery` | **osquery-tls** (8091) |
+
+`make up` and `make up-full` are the supported spellings for the first two
+rows and additionally wait for every container to report healthy, which a
+bare `docker compose up -d` does not.
+
+**What this gives you about AI.** Every command above starts the CORE
profile, which includes the `litellm` LLM gateway. AiSOC asks for logical task
aliases (`aisoc-triage`, `aisoc-investigation`, …) and the gateway is the only
thing that resolves them, so it has to be running for AI triage to happen at
-all — it was a `full`-profile service until 2026-09, which meant this exact
-command could not do AI triage even with the key you just set
+all — it was a `full`-profile service until 2026-09, which meant the default
+install could not do AI triage even with the key you just set
([ADR-0006](https://github.com/beenuar/AiSOC/blob/main/docs/decisions/0006-llm-gateway-in-core.md)).
- **With a provider key** in `.env`: alerts are triaged by the AI, and the
@@ -148,32 +165,48 @@ Path A above needs no gateway: the demo compose pins every role to a concrete
provider model, and a concrete pin goes straight to the provider rather than
through the gateway.
-This starts the full set of services:
-
-- **PostgreSQL** (5432) · **Redis** (6379) · **Kafka** (9092)
-- **ClickHouse** · **OpenSearch** · **Neo4j** · **Qdrant**
-- **api** (8000, FastAPI core) · **agents** (8001, LangGraph) ·
- **realtime** (8086, Node.js + VAPID Web Push) · **web** (3000)
-- **fusion** (8003) · **actions** (8002) · **threatintel** (8005) ·
- **ueba** (8007) · **honeytokens** (8008) · **purple-team** (8006) ·
- **ingest** (8081, Go) · **enrichment** (8080, Go) · **mcp** (TypeScript)
+### 3. Database migrations — nothing to run
+
+Every schema is applied by the stack itself; there is no manual migration
+step, and there is nothing to remember to run in the right order.
+
+- **api** applies its raw-SQL chain
+ (`services/api/migrations/*.sql`) from
+ [`app.scripts.run_migrations`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/scripts/run_migrations.py)
+ during startup. It has no `alembic.ini` and never did, so
+ `docker compose exec api alembic upgrade head` — which earlier revisions of
+ this page told you to run — fails with `No 'script_location' key found`.
+- **ueba**, **honeytokens**, **purple-team** and **osquery-tls** own their
+ schemas through alembic. Each container runs
+ [`app/_migrate.py`](https://github.com/beenuar/AiSOC/blob/main/services/ueba/app/_migrate.py)
+ before its server: it applies the chain, reads the applied revision back,
+ and refuses to start the service unless it matches head. Re-running is
+ free — a database already at head applies nothing.
+
+Those four apply their chain as the **owner**, not as the role they serve
+requests with: `docker-compose.yml` sets `DATABASE_MIGRATION_URL` alongside
+`DATABASE_URL` for each, and their `env.py` prefers it. The runtime role
+holds no `CREATE` on schema public, by design — that is what stops it
+turning row-level security off — so a chain applied as the runtime
+credential fails on the first `CREATE TABLE`. See
+[Env vars → the four services that manage their own schema](deployment/env-vars#the-four-services-that-manage-their-own-schema).
-### 3. Run database migrations
+To confirm the schemas landed, ask the database rather than the logs. Each
+chain records its revision in its **own** version table, because four chains
+numbering their revisions `0001`, `0002`, … against one database previously
+shared `alembic_version` and the second chain to run believed it was already
+done:
```bash
-docker compose exec api alembic upgrade head
-docker compose exec ueba alembic upgrade head
-docker compose exec honeytokens alembic upgrade head
-docker compose exec purple-team alembic upgrade head
+docker compose exec postgres psql -U aisoc -d aisoc -c \
+ "SELECT table_name FROM information_schema.tables WHERE table_name LIKE 'alembic_version%' ORDER BY 1"
```
-Each of those four runs as the **owner**, not as the role the service serves
-requests with: `docker-compose.yml` sets `DATABASE_MIGRATION_URL` alongside
-`DATABASE_URL` for all four, and their `alembic/env.py` prefers it. The
-runtime role holds no `CREATE` on schema public, by design — that is what
-stops it turning row-level security off — so a chain applied as the runtime
-credential fails on the first `CREATE TABLE`. See
-[Env vars → the four services that manage their own schema](deployment/env-vars#the-four-services-that-manage-their-own-schema).
+Four rows — `alembic_version_honeytokens`, `alembic_version_osquery_tls`,
+`alembic_version_purple_team`, `alembic_version_ueba` — means all four ran.
+A service whose chain did not reach head does not serve at all: its
+container exits and `docker compose ps` shows it unhealthy rather than
+running.
The `api` migrations include
[`008_investigation_ledger.sql`](https://github.com/beenuar/AiSOC/blob/main/services/api/migrations/008_investigation_ledger.sql)
diff --git a/docker-compose.override.yml b/docker-compose.override.yml
deleted file mode 100644
index 15c88f10e..000000000
--- a/docker-compose.override.yml
+++ /dev/null
@@ -1,12 +0,0 @@
-# QA-only host port remap (untracked, never committed). Another stack on this
-# machine already publishes 5432, 8123 and 9000. Container ports are unchanged,
-# so service-to-service traffic over the compose network is identical to the
-# documented deployment; only the host-side bindings move.
-services:
- postgres:
- ports: !override
- - "127.0.0.1:55432:5432"
- clickhouse:
- ports: !override
- - "127.0.0.1:58123:8123"
- - "127.0.0.1:59000:9000"
diff --git a/docker-compose.yml b/docker-compose.yml
index a3c580149..ae41360d7 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -55,6 +55,8 @@ volumes:
postgres_data:
redis_data:
kafka_data:
+ zookeeper_data:
+ zookeeper_log:
clickhouse_data:
opensearch_data:
qdrant_data:
@@ -123,6 +125,23 @@ services:
environment:
ZOOKEEPER_CLIENT_PORT: 2181
ZOOKEEPER_TICK_TIME: 2000
+ # Kafka keeps its cluster id in `kafka_data`, which IS persisted, and
+ # takes the authoritative one from ZooKeeper, which was not. So the
+ # second `up` on a kept volume generated a fresh cluster id and the
+ # broker refused to join it:
+ #
+ # InconsistentClusterIdException: The Cluster ID <new> doesn't match
+ # stored clusterId Some(<old>) in meta.properties
+ #
+ # `make down && make up` — documented as "stop the stack, keep the data"
+ # — therefore could not bring Kafka back, and every service that depends
+ # on it stayed in `created`. Only `make clean`, which deletes the data,
+ # recovered. Persisting both halves of the pair is the fix; persisting
+ # neither would also be consistent, but it would discard the event log
+ # on every restart.
+ volumes:
+ - zookeeper_data:/var/lib/zookeeper/data
+ - zookeeper_log:/var/lib/zookeeper/log
networks:
- aisoc
healthcheck:
@@ -616,8 +635,15 @@ services:
condition: service_healthy
# Host port 8007 was previously colliding with the `ueba` service's
# 127.0.0.1:8007 binding; remapped to 8091 here. UEBA still owns 8007.
+ #
+ # The container side was 8007 too, and nothing has ever listened there:
+ # this service's Dockerfile runs uvicorn on 9001 and EXPOSEs 9001. The
+ # published port therefore resolved to a closed socket, and because the
+ # service had no healthcheck either, the container still read as
+ # `running`. Found by adding the healthcheck below and watching it sit
+ # at `starting` against a port with no server on it.
ports:
- - "127.0.0.1:8091:8007"
+ - "127.0.0.1:8091:9001"
profiles:
- osquery
environment:
@@ -628,10 +654,25 @@ services:
DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisoc
# aisoc-db-role: owner — alembic applies this service's chain; the runtime role holds no DDL
DATABASE_MIGRATION_URL: postgresql+asyncpg://aisoc:${POSTGRES_PASSWORD:-aisoc_dev_secret}@postgres:5432/aisoc
+ # Read by the postgres init hook on a fresh volume and by the api
+ # migration runner otherwise. Passed here too so this service's own
+ # DATABASE_URL and the role's actual credential come from one value.
+ AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}
AISOC_OSQUERY_TLS_ENROLL_SECRET: ${AISOC_OSQUERY_TLS_ENROLL_SECRET:-change-me-in-production}
# The ingest service is registered as `ingest-worker` in this compose
# file; the previous `http://ingest:8080` would not resolve via DNS.
AISOC_INGEST_BASE_URL: http://ingest-worker:8080
+ # A detached consumer used to read as `running`. /readyz answers 503 when
+ # the subscription is gone or the schema is not at head, so `docker compose
+ # ps` and `make _wait` now see the difference.
+ healthcheck:
+ test:
+ - CMD-SHELL
+ - "python -c \"import urllib.request,sys; r=urllib.request.urlopen('http://127.0.0.1:9001/readyz',timeout=2); sys.exit(0 if r.status==200 else 1)\" || exit 1"
+ interval: 10s
+ timeout: 5s
+ retries: 20
+ start_period: 90s
networks:
- aisoc
restart: unless-stopped
@@ -794,12 +835,27 @@ services:
DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisoc
# aisoc-db-role: owner — alembic applies this service's chain; the runtime role holds no DDL
DATABASE_MIGRATION_URL: postgresql+asyncpg://aisoc:${POSTGRES_PASSWORD:-aisoc_dev_secret}@postgres:5432/aisoc
+ # Read by the postgres init hook on a fresh volume and by the api
+ # migration runner otherwise. Passed here too so this service's own
+ # DATABASE_URL and the role's actual credential come from one value.
+ AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}
REDIS_URL: redis://:redis_dev_secret@redis:6379/8
KAFKA_BOOTSTRAP_SERVERS: kafka:29092
CLICKHOUSE_URL: http://aisoc:clickhouse_dev_secret@clickhouse:8123/aisoc
CORE_API_URL: http://api:8000
AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:-}
AISOC_DEV_MODE: ${AISOC_DEV_MODE:-1}
+ # A detached consumer used to read as `running`. /readyz answers 503 when
+ # the subscription is gone or the schema is not at head, so `docker compose
+ # ps` and `make _wait` now see the difference.
+ healthcheck:
+ test:
+ - CMD-SHELL
+ - "python -c \"import urllib.request,sys; r=urllib.request.urlopen('http://127.0.0.1:8004/readyz',timeout=2); sys.exit(0 if r.status==200 else 1)\" || exit 1"
+ interval: 10s
+ timeout: 5s
+ retries: 20
+ start_period: 90s
networks:
- aisoc
restart: unless-stopped
@@ -832,11 +888,26 @@ services:
DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisoc
# aisoc-db-role: owner — alembic applies this service's chain; the runtime role holds no DDL
DATABASE_MIGRATION_URL: postgresql+asyncpg://aisoc:${POSTGRES_PASSWORD:-aisoc_dev_secret}@postgres:5432/aisoc
+ # Read by the postgres init hook on a fresh volume and by the api
+ # migration runner otherwise. Passed here too so this service's own
+ # DATABASE_URL and the role's actual credential come from one value.
+ AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}
REDIS_URL: redis://:redis_dev_secret@redis:6379/9
KAFKA_BOOTSTRAP_SERVERS: kafka:29092
CORE_API_URL: http://api:8000
AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:-}
AISOC_DEV_MODE: ${AISOC_DEV_MODE:-1}
+ # A detached consumer used to read as `running`. /readyz answers 503 when
+ # the subscription is gone or the schema is not at head, so `docker compose
+ # ps` and `make _wait` now see the difference.
+ healthcheck:
+ test:
+ - CMD-SHELL
+ - "python -c \"import urllib.request,sys; r=urllib.request.urlopen('http://127.0.0.1:8005/readyz',timeout=2); sys.exit(0 if r.status==200 else 1)\" || exit 1"
+ interval: 10s
+ timeout: 5s
+ retries: 20
+ start_period: 90s
networks:
- aisoc
restart: unless-stopped
@@ -867,11 +938,26 @@ services:
DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisoc
# aisoc-db-role: owner — alembic applies this service's chain; the runtime role holds no DDL
DATABASE_MIGRATION_URL: postgresql+asyncpg://aisoc:${POSTGRES_PASSWORD:-aisoc_dev_secret}@postgres:5432/aisoc
+ # Read by the postgres init hook on a fresh volume and by the api
+ # migration runner otherwise. Passed here too so this service's own
+ # DATABASE_URL and the role's actual credential come from one value.
+ AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}
REDIS_URL: redis://:redis_dev_secret@redis:6379/10
CORE_API_URL: http://api:8000
ATTCK_DATA_PATH: /app/data/enterprise-attack.json
AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:-}
AISOC_DEV_MODE: ${AISOC_DEV_MODE:-1}
+ # A detached consumer used to read as `running`. /readyz answers 503 when
+ # the subscription is gone or the schema is not at head, so `docker compose
+ # ps` and `make _wait` now see the difference.
+ healthcheck:
+ test:
+ - CMD-SHELL
+ - "python -c \"import urllib.request,sys; r=urllib.request.urlopen('http://127.0.0.1:8006/readyz',timeout=2); sys.exit(0 if r.status==200 else 1)\" || exit 1"
+ interval: 10s
+ timeout: 5s
+ retries: 20
+ start_period: 90s
networks:
- aisoc
restart: unless-stopped
diff --git a/scripts/audit_health_probes.py b/scripts/audit_health_probes.py
index b6c21e109..bfaeaa2df 100644
--- a/scripts/audit_health_probes.py
+++ b/scripts/audit_health_probes.py
@@ -64,19 +64,39 @@ def discover_fastapi_services() -> list[str]:
return out
-def audit_service(svc: str) -> tuple[bool, bool, bool]:
- """Return ``(has_module, imports_helper, wires_routes)``.
+#: The copy every other one is compared against. Arbitrary but fixed: what
+#: matters is that there is one answer to "which is right" rather than
+#: thirteen files that each look plausible on their own.
+REFERENCE_SERVICE = "api"
+
+
+def audit_service(svc: str, reference: bytes | None) -> tuple[bool, bool, bool, bool]:
+ """Return ``(has_module, imports_helper, wires_routes, in_sync)``.
* ``has_module`` — ``services/<svc>/app/_health.py`` exists.
* ``imports_helper``— ``app/main.py`` imports ``install_health_routes``.
* ``wires_routes`` — ``app/main.py`` calls ``install_health_routes``.
+ * ``in_sync`` — that copy is byte-identical to the reference one.
+
+ ``in_sync`` is checked because the module's own docstring claims the
+ thirteen copies are kept in sync and nothing verified it. They happened
+ to be identical, so the claim was true by luck: any change landing in one
+ tree would have left the other twelve serving the older contract, and a
+ probe that answers differently per service is worse than one that is
+ wrong everywhere — it makes a fleet-wide readiness answer unreadable.
+ The docstring also pointed at ``scripts/sync_health_module.py``, which
+ does not exist in this tree.
"""
base = SERVICES_DIR / svc / "app"
- has_module = (base / "_health.py").is_file()
+ module = base / "_health.py"
+ has_module = module.is_file()
main_text = (base / "main.py").read_text(encoding="utf-8")
imports_helper = "from app._health import install_health_routes" in main_text
wires_routes = "install_health_routes(app" in main_text
- return has_module, imports_helper, wires_routes
+ # Unknown rather than false when there is nothing to compare against, so a
+ # missing reference is reported as the missing module it is.
+ in_sync = True if reference is None or not has_module else module.read_bytes() == reference
+ return has_module, imports_helper, wires_routes, in_sync
def main() -> int:
@@ -101,19 +121,35 @@ def main() -> int:
)
return 1
- print(f"{'service':<18} {'_health.py':<11} {'import':<7} {'wire':<5}")
- print(f"{'-' * 18} {'-' * 11} {'-' * 7} {'-' * 5}")
+ reference_path = SERVICES_DIR / REFERENCE_SERVICE / "app" / "_health.py"
+ reference = reference_path.read_bytes() if reference_path.is_file() else None
+ if reference is None:
+ print(
+ f"health-probes: reference module {reference_path} is missing, so the " f"copies have nothing to be in sync with.",
+ file=sys.stderr,
+ )
+ return 1
+
+ print(f"{'service':<18} {'_health.py':<11} {'import':<7} {'wire':<5} {'in-sync':<7}")
+ print(f"{'-' * 18} {'-' * 11} {'-' * 7} {'-' * 5} {'-' * 7}")
drift: list[str] = []
+ out_of_sync: list[str] = []
for svc in services:
- has_mod, has_import, has_wire = audit_service(svc)
+ has_mod, has_import, has_wire, in_sync = audit_service(svc, reference)
marker_mod = "yes" if has_mod else "NO"
marker_import = "yes" if has_import else "NO"
marker_wire = "yes" if has_wire else "NO"
- print(f"{svc:<18} {marker_mod:<11} {marker_import:<7} {marker_wire:<5}")
+ marker_sync = "yes" if in_sync else "NO"
+ print(f"{svc:<18} {marker_mod:<11} {marker_import:<7} {marker_wire:<5} {marker_sync:<7}")
if not (has_mod and has_import and has_wire):
drift.append(svc)
+ if not in_sync:
+ out_of_sync.append(svc)
+
+ if not args.check:
+ return 0
- if args.check and drift:
+ if drift:
print(
"\nFAIL: the following FastAPI services do not install the "
"Phase 2.6 /livez + /readyz probes from app._health:\n - " + "\n - ".join(drift),
@@ -125,8 +161,20 @@ def main() -> int:
"app, service_name='aisoc-<svc>') in app/main.py.",
file=sys.stderr,
)
- return 1
- return 0
+ if out_of_sync:
+ print(
+ f"\nFAIL: these copies of app/_health.py differ from "
+ f"services/{REFERENCE_SERVICE}/app/_health.py:\n - " + "\n - ".join(out_of_sync),
+ file=sys.stderr,
+ )
+ print(
+ f"\nFix: cp services/{REFERENCE_SERVICE}/app/_health.py "
+ f"services/<svc>/app/_health.py for each one above. The probes are a "
+ f"contract the whole fleet answers; one tree answering an older "
+ f"version of it cannot be read off a dashboard.",
+ file=sys.stderr,
+ )
+ return 1 if (drift or out_of_sync) else 0
if __name__ == "__main__":
diff --git a/scripts/check_orm_migration_parity.py b/scripts/check_orm_migration_parity.py
new file mode 100644
index 000000000..744cd4357
--- /dev/null
+++ b/scripts/check_orm_migration_parity.py
@@ -0,0 +1,884 @@
+#!/usr/bin/env python3
+"""Every column a model declares must be a column some migration creates.
+
+Why this exists
+---------------
+
+``services/ueba/app/models/ueba.py`` declared ``EntityBaseline.peer_group_id``
+and migration ``0001`` never created it. SQLAlchemy names every mapped column
+in its ``SELECT``, so the *first* database call UEBA's scoring path makes
+raised ``UndefinedColumnError``. UEBA could therefore never write a baseline
+or an anomaly, on any deployment, from the day the column was added — and
+because the consumer had no ``except``, the container stayed ``running`` with
+``/health`` answering 200 the whole time.
+
+Two more of the same kind were in the same three tables:
+``ueba_peer_groups.id`` was created ``UUID`` and declared ``String(64)`` while
+the code wrote ``dept:engineering`` into it, and ``ueba_anomalies.event_type``
+was created 64 characters wide and declared 128.
+
+None of this needed a database to find. The model and the migration are both
+in the tree, and the disagreement is structural. That is what this reads.
+
+What it checks, in both directions
+----------------------------------
+
+``missing-in-migration``
+ A model declares a column no migration creates. This is the direction
+ that breaks at runtime, and it breaks on the first query rather than on
+ some rare path, because the column list is in the ``SELECT``.
+
+``missing-in-model``
+ A migration creates a column no model declares. Not a crash, so it is
+ reported separately and carries a recorded reason — but it is checked,
+ because a gate that only looks one way passes while drift accumulates in
+ the other. Three of the four cases here turned out to be
+ ``server_default`` columns nothing maps, and one was a table whose model
+ had been renamed.
+
+``type-narrower-in-migration``
+ Both sides have the column and the database's is too small to hold what
+ the model will send. ``String(64)`` in a table against ``String(128)`` in
+ the model passes every test that inserts a short value.
+
+``no-migration-source``
+ A model declares a table **no migration anywhere mentions**. This is the
+ finding that exists because of how this gate could have been useless: a
+ table the parser fails to pair contributes zero column comparisons, and
+ zero findings over zero comparisons prints the same word as a clean
+ result. ``--credits`` exists for the same reason — it enumerates what
+ each verdict was *based on*, so a parser that silently credits nothing
+ can be caught by reading rather than by trusting the exit code.
+
+What it reads
+-------------
+
+Structurally, both sides. Models: Python ``ast`` over
+``services/*/app/**`` for classes carrying ``__tablename__``, collecting
+``mapped_column`` / ``Column`` assignments. Migrations: Python ``ast`` over
+each alembic chain for ``op.create_table`` / ``op.add_column`` /
+``op.drop_column``, plus the raw-SQL statements those revisions and
+``services/api/migrations/*.sql`` execute, because half the column additions
+in this repository are ``ALTER TABLE … ADD COLUMN IF NOT EXISTS`` inside a
+string.
+
+Usage
+-----
+
+::
+
+ python3 scripts/check_orm_migration_parity.py # table + verdict
+ python3 scripts/check_orm_migration_parity.py --check # gate
+ python3 scripts/check_orm_migration_parity.py --credits # what it credited
+ python3 scripts/check_orm_migration_parity.py --json
+ python3 scripts/check_orm_migration_parity.py --self-test
+
+Exit codes: 0 clean, 1 findings, 2 the scan itself could not run.
+"""
+
+from __future__ import annotations
+
+import argparse
+import ast
+import json
+import pathlib
+import re
+import sys
+
+# `scripts/` is on sys.path when this file is run as a program, but not when a
+# test loads it by path with importlib. gate_toolkit sits beside it either way.
+sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
+
+from gate_toolkit import repo_root, self_test_main
+
+REPO_ROOT = repo_root()
+SERVICES_DIR = REPO_ROOT / "services"
+
+#: Where a service's raw-SQL chain lives, if it has one. Alembic chains are
+#: discovered from ``alembic.ini`` instead, since two of the four do not put
+#: them under a directory named ``alembic``.
+SQL_MIGRATION_DIRS = ("migrations",)
+
+#: Directories under ``services/<svc>/`` that never hold models worth pairing.
+_SKIP_PARTS = frozenset({"tests", "test", "_vendor", "node_modules", "__pycache__", "alembic", "versions"})
+
+
+# ---------------------------------------------------------------------------
+# Recorded exceptions
+# ---------------------------------------------------------------------------
+#
+# Shrink-only, and verified in both directions: an entry that stops being
+# needed fails the build rather than sitting here. Each one says why, because
+# a bare list of table names is a second place for drift to hide.
+
+#: ``(table, column)`` a migration creates and no model maps. Harmless by
+#: construction — every one carries a ``server_default``, so an insert that
+#: does not name it still succeeds — but recorded rather than ignored.
+UNMAPPED_COLUMNS: dict[tuple[str, str], str] = {
+ ("ueba_entity_baselines", "observation_count"): "server_default '0'; the rolling count lives in feature_stats instead",
+ ("ueba_entity_baselines", "created_at"): "server_default now(); the model exposes window_start/updated_at",
+ ("ueba_anomalies", "z_scores"): "server_default '{}'; superseded by the per-feature z_score inside features",
+ ("ueba_peer_groups", "created_at"): "server_default now(); the model exposes updated_at",
+}
+
+#: Tables whose model is mapped but whose creation this gate cannot see, with
+#: the reason. Empty on purpose: it exists so that adding one is a deliberate,
+#: reviewed act rather than the default outcome of a parser that missed
+#: something.
+UNSOURCED_TABLES: dict[str, str] = {}
+
+
+# ---------------------------------------------------------------------------
+# Model side
+# ---------------------------------------------------------------------------
+
+
+def _string(node: ast.AST) -> str | None:
+ return node.value if isinstance(node, ast.Constant) and isinstance(node.value, str) else None
+
+
+#: SQL type name → family. Two spellings of the same family are not drift
+#: (``Text`` against ``String``, ``BigInteger`` against ``Integer``); two
+#: families are, and one such mismatch is why this mapping exists.
+#:
+#: ``ueba_peer_groups.id`` was created ``UUID DEFAULT gen_random_uuid()`` and
+#: declared ``String(64)``, and the code wrote ``dept:engineering`` into it —
+#: an ``InvalidTextRepresentation`` on the first insert. A length comparison
+#: could not see it: neither side declares a length that the other contradicts,
+#: so the column read as agreed. Found by re-running this gate against the
+#: tree as it was and noticing that it reported two of the three known
+#: defects, which is the question `--credits` is for: what does it credit as
+#: clean, not what does it flag.
+_TYPE_FAMILIES: dict[str, str] = {
+ # SQLAlchemy constructors and SQL type names share this table on purpose:
+ # the two sides have to be classified identically or the comparison is
+ # between two different vocabularies.
+ "string": "string",
+ "str": "string",
+ "text": "string",
+ "unicode": "string",
+ "unicodetext": "string",
+ "varchar": "string",
+ "char": "string",
+ "citext": "string",
+ "uuid": "uuid",
+ "guid": "uuid",
+ "integer": "integer",
+ "int": "integer",
+ "int4": "integer",
+ "int8": "integer",
+ "smallint": "integer",
+ "bigint": "integer",
+ "smallinteger": "integer",
+ "biginteger": "integer",
+ "serial": "integer",
+ "bigserial": "integer",
+ "float": "numeric",
+ "numeric": "numeric",
+ "decimal": "numeric",
+ "real": "numeric",
+ "double": "numeric",
+ "boolean": "boolean",
+ "bool": "boolean",
+ "json": "json",
+ "jsonb": "json",
+ "datetime": "datetime",
+ "timestamp": "datetime",
+ "timestamptz": "datetime",
+ "date": "datetime",
+ "time": "datetime",
+ "interval": "interval",
+ "largebinary": "binary",
+ "bytea": "binary",
+ "inet": "inet",
+ "array": "array",
+ "enum": "enum",
+}
+
+
+def _family(name: str | None) -> str | None:
+ """The family of a type name, or None when it is one this gate cannot rank.
+
+ Unknown is not a mismatch. A vendor type or a custom ``TypeDecorator``
+ must not be reported as drift just because this table has not heard of
+ it — a gate that invents findings gets turned off, and then it is not
+ checking the ones it was right about either.
+ """
+ return _TYPE_FAMILIES.get(name.strip().lower()) if name else None
+
+
+def _declared_type(call: ast.Call) -> tuple[str | None, int | None]:
+ """``(family, string_length)`` of the type in a column definition.
+
+ The length is only read for ``String``: the other families either carry
+ no length or carry one whose mismatch is not silently lossy the way a
+ truncating varchar is.
+ """
+ for node in ast.walk(call):
+ if not isinstance(node, ast.Call):
+ continue
+ name = node.func.attr if isinstance(node.func, ast.Attribute) else getattr(node.func, "id", None)
+ family = _family(name)
+ if family is None:
+ continue
+ if family != "string":
+ return family, None
+ for arg in node.args:
+ if isinstance(arg, ast.Constant) and isinstance(arg.value, int):
+ return family, arg.value
+ for kw in node.keywords:
+ if kw.arg == "length" and isinstance(kw.value, ast.Constant) and isinstance(kw.value.value, int):
+ return family, kw.value.value
+ # ``String`` with no length is unbounded text — never too narrow.
+ return family, None
+ return None, None
+
+
+def _column_from_assignment(node: ast.AST) -> tuple[str, tuple[str | None, int | None]] | None:
+ """``(column_name, string_length)`` for one class-body assignment.
+
+ Handles the two spellings in this tree: SQLAlchemy 2.0
+ ``name: Mapped[T] = mapped_column(...)`` and the older
+ ``name = Column(...)``. The column's name is the attribute name unless
+ the call passes an explicit string first — ``mapped_column("other", ...)``
+ — which is the case a regex over attribute names would get wrong.
+ """
+ target: ast.expr
+ value: ast.expr | None
+ if isinstance(node, ast.AnnAssign):
+ target, value = node.target, node.value
+ elif isinstance(node, ast.Assign) and len(node.targets) == 1:
+ target, value = node.targets[0], node.value
+ else:
+ return None
+ if not isinstance(target, ast.Name) or not isinstance(value, ast.Call):
+ return None
+
+ func = value.func
+ callee = func.attr if isinstance(func, ast.Attribute) else getattr(func, "id", None)
+ if callee not in {"mapped_column", "Column"}:
+ return None
+
+ name = target.id
+ if value.args and (explicit := _string(value.args[0])):
+ name = explicit
+ return name, _declared_type(value)
+
+
+def model_tables(paths: list[pathlib.Path]) -> dict[str, dict]:
+ """Mapped tables, by table name, from the model modules given."""
+ tables: dict[str, dict] = {}
+ for path in paths:
+ try:
+ tree = ast.parse(path.read_text(encoding="utf-8", errors="replace"))
+ except SyntaxError:
+ continue
+ for node in ast.walk(tree):
+ if not isinstance(node, ast.ClassDef):
+ continue
+ tablename: str | None = None
+ columns: dict[str, tuple[str | None, int | None]] = {}
+ for stmt in node.body:
+ if isinstance(stmt, ast.Assign) and len(stmt.targets) == 1:
+ only = stmt.targets[0]
+ if isinstance(only, ast.Name) and only.id == "__tablename__":
+ tablename = _string(stmt.value)
+ found = _column_from_assignment(stmt)
+ if found is not None:
+ columns[found[0]] = found[1]
+ if tablename and columns:
+ tables[tablename] = {
+ "columns": columns,
+ "source": str(path.relative_to(REPO_ROOT)),
+ "model": node.name,
+ }
+ return tables
+
+
+# ---------------------------------------------------------------------------
+# Migration side
+# ---------------------------------------------------------------------------
+
+#: ``CREATE TABLE [IF NOT EXISTS] name ( ... )`` — the body is scanned for
+#: leading identifiers, one per column definition.
+_CREATE_TABLE = re.compile(r"CREATE\s+TABLE\s+(?:IF\s+NOT\s+EXISTS\s+)?[\"']?(\w+)[\"']?\s*\((.*?)\);", re.I | re.S)
+_ADD_COLUMN = re.compile(
+ r"ALTER\s+TABLE\s+(?:ONLY\s+)?[\"']?(\w+)[\"']?\s+ADD\s+COLUMN\s+"
+ r"(?:IF\s+NOT\s+EXISTS\s+)?[\"']?(\w+)[\"']?\s*([A-Za-z]*)\s*(?:\((\d+)\))?",
+ re.I,
+)
+_DROP_COLUMN = re.compile(r"ALTER\s+TABLE\s+(?:ONLY\s+)?[\"']?(\w+)[\"']?\s+DROP\s+COLUMN\s+(?:IF\s+EXISTS\s+)?[\"']?(\w+)[\"']?", re.I)
+_ALTER_TYPE = re.compile(
+ r"ALTER\s+TABLE\s+(?:ONLY\s+)?[\"']?(\w+)[\"']?\s+ALTER\s+COLUMN\s+[\"']?(\w+)[\"']?\s+"
+ r"(?:SET\s+DATA\s+)?TYPE\s+([A-Za-z]+)\s*(?:\((\d+)\))?",
+ re.I,
+)
+
+#: Words that begin a table-level constraint rather than a column.
+_CONSTRAINT_WORDS = frozenset({"primary", "unique", "foreign", "constraint", "check", "exclude", "like"})
+
+
+def _split_columns(body: str) -> list[tuple[str, str | None, int | None]]:
+ """Column ``(name, length)`` pairs from a ``CREATE TABLE`` body."""
+ depth = 0
+ current: list[str] = []
+ parts: list[str] = []
+ for char in body:
+ if char == "(":
+ depth += 1
+ elif char == ")":
+ depth -= 1
+ if char == "," and depth == 0:
+ parts.append("".join(current))
+ current = []
+ continue
+ current.append(char)
+ parts.append("".join(current))
+
+ out: list[tuple[str, str | None, int | None]] = []
+ for part in parts:
+ tokens = part.strip().strip('"').split()
+ if not tokens:
+ continue
+ name = tokens[0].strip('"').strip(",")
+ if not name or not name.replace("_", "").isalnum() or name.lower() in _CONSTRAINT_WORDS:
+ continue
+ family: str | None = None
+ length: int | None = None
+ if len(tokens) > 1:
+ # ``VARCHAR(64)`` arrives as one token; the family is the part
+ # before the parenthesis.
+ declared = tokens[1].split("(")[0]
+ family = _family(declared)
+ if match := re.search(r"\((\d+)\)", tokens[1]):
+ length = int(match.group(1))
+ out.append((name, family, length))
+ return out
+
+
+def _record(
+ store: dict[str, dict[str, tuple[str | None, int | None]]],
+ table: str,
+ column: str,
+ family: str | None,
+ length: int | None,
+) -> None:
+ store.setdefault(table, {})[column] = (family, length)
+
+
+def _scan_sql(text: str, store: dict[str, dict[str, tuple[str | None, int | None]]], dropped: set[tuple[str, str]]) -> None:
+ for table, body in _CREATE_TABLE.findall(text):
+ for column, family, length in _split_columns(body):
+ _record(store, table, column, family, length)
+ for table, column, type_name, length in _ADD_COLUMN.findall(text):
+ _record(store, table, column, _family(type_name), int(length) if length else None)
+ for table, column, type_name, length in _ALTER_TYPE.findall(text):
+ # Only widens what is already known; an ALTER TYPE on a column this
+ # scan never saw created is still a column it never saw created.
+ if column in store.get(table, {}):
+ _record(store, table, column, _family(type_name), int(length) if length else None)
+ for table, column in _DROP_COLUMN.findall(text):
+ dropped.add((table, column))
+
+
+def _scan_alembic_module(tree: ast.AST, store: dict[str, dict[str, tuple[str | None, int | None]]], dropped: set[tuple[str, str]]) -> None:
+ """``op.*`` table and column operations, plus the SQL they execute."""
+ for node in ast.walk(tree):
+ if not isinstance(node, ast.Call) or not isinstance(node.func, ast.Attribute):
+ continue
+ op_name = node.func.attr
+
+ if op_name == "execute":
+ for arg in node.args:
+ for literal in ast.walk(arg):
+ if (sql := _string(literal)) and "TABLE" in sql.upper():
+ _scan_sql(sql, store, dropped)
+ continue
+
+ if op_name == "create_table" and node.args:
+ table = _string(node.args[0])
+ if not table:
+ continue
+ for arg in node.args[1:]:
+ if isinstance(arg, ast.Call) and (column := _string(arg.args[0]) if arg.args else None):
+ _record(store, table, column, *_declared_type(arg))
+ continue
+
+ if op_name == "add_column" and len(node.args) >= 2:
+ table = _string(node.args[0])
+ spec = node.args[1]
+ if table and isinstance(spec, ast.Call) and spec.args and (column := _string(spec.args[0])):
+ _record(store, table, column, *_declared_type(spec))
+ continue
+
+ if op_name == "alter_column" and len(node.args) >= 2:
+ table, column = _string(node.args[0]), _string(node.args[1])
+ if not table or not column or column not in store.get(table, {}):
+ continue
+ for kw in node.keywords:
+ if kw.arg == "type_" and isinstance(kw.value, ast.Call):
+ _record(store, table, column, *_declared_type(kw.value))
+ continue
+
+ if op_name == "drop_column" and len(node.args) >= 2:
+ table, column = _string(node.args[0]), _string(node.args[1])
+ if table and column:
+ dropped.add((table, column))
+
+
+def _upgrade_only(path: pathlib.Path) -> ast.Module | None:
+ """The revision's ``upgrade()`` body, with ``downgrade()`` removed.
+
+ ``downgrade`` drops what ``upgrade`` creates. Scanning the whole module
+ made every revision cancel itself out — the parser credited a column and
+ then recorded it as dropped, so a correct chain reported every column
+ missing. Caught by reading ``--credits``, not by the exit code.
+ """
+ try:
+ tree = ast.parse(path.read_text(encoding="utf-8", errors="replace"))
+ except (SyntaxError, OSError):
+ return None
+ tree.body = [node for node in tree.body if not (isinstance(node, ast.FunctionDef) and node.name.startswith("downgrade"))]
+ return tree
+
+
+def migration_columns(service: pathlib.Path) -> tuple[dict[str, dict[str, tuple[str | None, int | None]]], set[tuple[str, str]], list[str]]:
+ """Columns this service's migrations create, columns they drop, files read."""
+ store: dict[str, dict[str, tuple[str | None, int | None]]] = {}
+ dropped: set[tuple[str, str]] = set()
+ read: list[str] = []
+
+ # Alembic chain, located from alembic.ini rather than assumed: osquery-tls
+ # keeps its chain under app/db, and a gate that globbed for a directory
+ # named `alembic` would have reported that service as having no migrations
+ # at all — which is a clean result, wrongly.
+ ini = service / "alembic.ini"
+ if ini.is_file():
+ match = re.search(r"^\s*script_location\s*=\s*(.+?)\s*$", ini.read_text(encoding="utf-8"), re.M)
+ if match:
+ versions = service / match.group(1).strip() / "versions"
+ for path in sorted(versions.glob("*.py")):
+ tree = _upgrade_only(path)
+ if tree is not None:
+ _scan_alembic_module(tree, store, dropped)
+ read.append(str(path.relative_to(REPO_ROOT)))
+
+ for name in SQL_MIGRATION_DIRS:
+ for path in sorted((service / name).glob("*.sql")):
+ _scan_sql(path.read_text(encoding="utf-8", errors="replace"), store, dropped)
+ read.append(str(path.relative_to(REPO_ROOT)))
+
+ return store, dropped, read
+
+
+# ---------------------------------------------------------------------------
+# Comparison
+# ---------------------------------------------------------------------------
+
+
+def model_paths(service: pathlib.Path) -> list[pathlib.Path]:
+ return [path for path in sorted((service / "app").rglob("*.py")) if not _SKIP_PARTS & set(path.relative_to(service).parts)]
+
+
+def creates_tables_from_metadata(paths: list[pathlib.Path]) -> str | None:
+ """The file where this service calls ``metadata.create_all``, if it does.
+
+ This is the line that decides whether the check applies, and why it is
+ read structurally rather than assumed per service.
+
+ A service that calls ``create_all`` on the way up builds its tables from
+ the models, so a column the models declare and no migration creates is
+ *created anyway* — the migrations are supplemental there, adding columns
+ to tables the ORM already owns. ``services/api`` is the one such service
+ in this tree, and it accounts for 173 of the 173 disagreements found
+ before this distinction existed. Holding it to the same rule would have
+ meant either 173 allowlist entries or a gate nobody could turn on.
+
+ A migration-managed service has no such fallback, which is exactly why
+ UEBA's missing column was fatal rather than cosmetic.
+
+ Those services are still scanned, counted, and printed — ``--all`` shows
+ their findings — because "excluded" has to be visible. An exclusion that
+ prints nothing is the same defect one level up.
+ """
+ for path in paths:
+ try:
+ tree = ast.parse(path.read_text(encoding="utf-8", errors="replace"))
+ except (SyntaxError, OSError):
+ continue
+ for node in ast.walk(tree):
+ if isinstance(node, ast.Attribute) and node.attr == "create_all":
+ return str(path.relative_to(REPO_ROOT))
+ return None
+
+
+def scan() -> dict:
+ if not SERVICES_DIR.is_dir():
+ return {"error": f"no services directory at {SERVICES_DIR}"}
+
+ services: list[dict] = []
+ for service in sorted(p for p in SERVICES_DIR.iterdir() if p.is_dir()):
+ if not (service / "app").is_dir():
+ continue
+ paths = model_paths(service)
+ models = model_tables(paths)
+ if not models:
+ continue
+ create_all = creates_tables_from_metadata(paths)
+ created, dropped, files = migration_columns(service)
+ if not files:
+ # A service with models and no migration chain manages its schema
+ # elsewhere (create_all, or another service's chain). Not a
+ # finding, but counted, so `--credits` shows it was considered.
+ services.append(
+ {
+ "service": service.name,
+ "schema_source": "orm" if create_all else "none",
+ "tables": len(models),
+ "columns": sum(len(t["columns"]) for t in models.values()),
+ "migration_files": 0,
+ "findings": [],
+ "credits": {},
+ }
+ )
+ continue
+
+ findings: list[dict] = []
+ credits: dict[str, dict] = {}
+ for table, spec in sorted(models.items()):
+ table_columns = {c: spec_t for c, spec_t in created.get(table, {}).items() if (table, c) not in dropped}
+ credits[table] = {
+ "model": spec["model"],
+ "model_source": spec["source"],
+ "migration_columns": sorted(table_columns),
+ }
+ if not table_columns:
+ if table in UNSOURCED_TABLES:
+ credits[table]["recorded"] = UNSOURCED_TABLES[table]
+ continue
+ findings.append(
+ {
+ "kind": "no-migration-source",
+ "table": table,
+ "column": "*",
+ "detail": f"{spec['source']} maps {spec['model']} to {table}; no migration in this service creates it",
+ }
+ )
+ continue
+
+ for column, (model_family, model_length) in sorted(spec["columns"].items()):
+ if column not in table_columns:
+ findings.append(
+ {
+ "kind": "missing-in-migration",
+ "table": table,
+ "column": column,
+ "detail": f"{spec['model']} declares it; no migration creates it. Every query naming this table will fail.",
+ }
+ )
+ continue
+ table_family, table_length = table_columns[column]
+
+ if model_family and table_family and model_family != table_family:
+ findings.append(
+ {
+ "kind": "type-family-mismatch",
+ "table": table,
+ "column": column,
+ "detail": f"{spec['model']} declares {model_family}; the table is {table_family}",
+ }
+ )
+ # The width question is meaningless across families.
+ continue
+
+ if model_length is not None and table_length is not None and table_length < model_length:
+ findings.append(
+ {
+ "kind": "type-narrower-in-migration",
+ "table": table,
+ "column": column,
+ "detail": f"{spec['model']} declares String({model_length}); the table is {table_length} wide",
+ }
+ )
+
+ for column in sorted(table_columns):
+ if column in spec["columns"] or (table, column) in UNMAPPED_COLUMNS:
+ continue
+ findings.append(
+ {
+ "kind": "missing-in-model",
+ "table": table,
+ "column": column,
+ "detail": f"a migration creates it; {spec['model']} does not map it",
+ }
+ )
+
+ services.append(
+ {
+ "service": service.name,
+ "schema_source": "orm" if create_all else "migrations",
+ "create_all_at": create_all,
+ "tables": len(models),
+ "columns": sum(len(t["columns"]) for t in models.values()),
+ "migration_files": len(files),
+ "findings": findings,
+ "credits": credits,
+ }
+ )
+
+ gated = [s for s in services if s["schema_source"] == "migrations"]
+ return {
+ "root": str(REPO_ROOT),
+ "services": services,
+ "gated_services": [s["service"] for s in gated],
+ "advisory_services": [s["service"] for s in services if s["schema_source"] != "migrations"],
+ "tables": sum(s["tables"] for s in services),
+ "columns": sum(s["columns"] for s in services),
+ # Only the gated services: quoting the total next to the verdict would
+ # credit this check with 832 columns in `api` that it deliberately
+ # does not gate.
+ "gated_columns": sum(s["columns"] for s in gated),
+ "compared": sum(len(s["credits"]) for s in gated),
+ "findings": [dict(f, service=s["service"]) for s in gated for f in s["findings"]],
+ "advisory_findings": [dict(f, service=s["service"]) for s in services if s["schema_source"] != "migrations" for f in s["findings"]],
+ }
+
+
+def unused_exceptions(result: dict) -> list[str]:
+ """Recorded exceptions nothing needs any more.
+
+ The other direction of the same check: an allowlist verified only when it
+ forgives something keeps growing, and every stale entry is a column the
+ gate has stopped looking at for a reason that no longer exists.
+ """
+ seen_columns: set[tuple[str, str]] = set()
+ seen_tables: set[str] = set()
+ for service in result["services"]:
+ if service["schema_source"] != "migrations":
+ continue
+ for table, credit in service["credits"].items():
+ seen_tables.add(table)
+ for column in credit["migration_columns"]:
+ seen_columns.add((table, column))
+
+ stale = [
+ f"UNMAPPED_COLUMNS[{key!r}] — {key[1]} is no longer created by any migration" for key in UNMAPPED_COLUMNS if key not in seen_columns
+ ]
+ stale += [f"UNSOURCED_TABLES[{name!r}] — {name} is no longer a mapped table" for name in UNSOURCED_TABLES if name not in seen_tables]
+ return stale
+
+
+# ---------------------------------------------------------------------------
+# Self-test
+# ---------------------------------------------------------------------------
+
+_INJECTED_MODEL = """
+from sqlalchemy.orm import Mapped, mapped_column
+from sqlalchemy import String
+
+class Widget:
+ __tablename__ = "widgets"
+ id: Mapped[str] = mapped_column(String(16), primary_key=True)
+ ghost: Mapped[str] = mapped_column(String(32))
+ squeezed: Mapped[str] = mapped_column(String(128))
+ mistyped: Mapped[str] = mapped_column(String(64))
+"""
+
+_INJECTED_MIGRATION = """
+from alembic import op
+import sqlalchemy as sa
+from sqlalchemy.dialects import postgresql
+revision = "0001"
+down_revision = None
+
+def upgrade() -> None:
+ op.create_table(
+ "widgets",
+ sa.Column("id", sa.String(16), primary_key=True),
+ sa.Column("squeezed", sa.String(64)),
+ sa.Column("mistyped", postgresql.UUID(as_uuid=True)),
+ sa.Column("orphan", sa.String(8), server_default=""),
+ )
+
+def downgrade() -> None:
+ op.drop_table("widgets")
+"""
+
+
+def _detects_injected_drift() -> list[tuple[str, bool]]:
+ """Prove the gate still catches each kind, on a tree built for the purpose.
+
+ Three synthetic disagreements in one service, one per kind. Run before the
+ gate renders any verdict about the real tree, which is the point of a
+ self-test: a gate whose parser has quietly stopped matching anything
+ reports a clean repository in exactly the same words as a clean one.
+ """
+ import tempfile
+
+ global REPO_ROOT, SERVICES_DIR # noqa: PLW0603 — restored below
+ saved_root, saved_services = REPO_ROOT, SERVICES_DIR
+ try:
+ with tempfile.TemporaryDirectory() as raw:
+ root = pathlib.Path(raw)
+ service = root / "services" / "widgetry"
+ (service / "app" / "models").mkdir(parents=True)
+ (service / "app" / "models" / "widget.py").write_text(_INJECTED_MODEL)
+ (service / "alembic" / "versions").mkdir(parents=True)
+ (service / "alembic.ini").write_text("[alembic]\nscript_location = alembic\n")
+ (service / "alembic" / "versions" / "0001_init.py").write_text(_INJECTED_MIGRATION)
+
+ REPO_ROOT, SERVICES_DIR = root, root / "services"
+ kinds = {f["kind"] for f in scan()["findings"]}
+ finally:
+ REPO_ROOT, SERVICES_DIR = saved_root, saved_services
+
+ return [
+ ("detects a model column no migration creates", "missing-in-migration" in kinds),
+ ("detects a migration column no model maps", "missing-in-model" in kinds),
+ ("detects a column the table is too narrow for", "type-narrower-in-migration" in kinds),
+ ("detects a column whose type family disagrees", "type-family-mismatch" in kinds),
+ ]
+
+
+def _detects_unsourced_table() -> list[tuple[str, bool]]:
+ """And a mapped table whose creation the parser cannot find at all.
+
+ Separate from the three above because it is the failure mode of this gate
+ rather than of the tree: pairing nothing has to be a finding, or a parser
+ that matches no migration statement at all reports every service clean.
+ """
+ import tempfile
+
+ global REPO_ROOT, SERVICES_DIR # noqa: PLW0603 — restored below
+ saved_root, saved_services = REPO_ROOT, SERVICES_DIR
+ try:
+ with tempfile.TemporaryDirectory() as raw:
+ root = pathlib.Path(raw)
+ service = root / "services" / "widgetry"
+ (service / "app" / "models").mkdir(parents=True)
+ (service / "app" / "models" / "widget.py").write_text(_INJECTED_MODEL)
+ (service / "alembic" / "versions").mkdir(parents=True)
+ (service / "alembic.ini").write_text("[alembic]\nscript_location = alembic\n")
+ # A revision that creates a different table, so the chain is real
+ # and the mapped table is still unaccounted for.
+ (service / "alembic" / "versions" / "0001_init.py").write_text(_INJECTED_MIGRATION.replace('"widgets"', '"sprockets"'))
+ REPO_ROOT, SERVICES_DIR = root, root / "services"
+ kinds = {f["kind"] for f in scan()["findings"]}
+ finally:
+ REPO_ROOT, SERVICES_DIR = saved_root, saved_services
+
+ return [("detects a mapped table no migration mentions", "no-migration-source" in kinds)]
+
+
+# ---------------------------------------------------------------------------
+# Entry point
+# ---------------------------------------------------------------------------
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description="ORM/migration column parity")
+ parser.add_argument("--check", action="store_true", help="exit non-zero on findings")
+ parser.add_argument("--all", action="store_true", help="also list findings for services that build tables from the models")
+ parser.add_argument("--credits", action="store_true", help="print what each verdict was based on")
+ parser.add_argument("--json", action="store_true", help="machine-readable output")
+ parser.add_argument("--self-test", action="store_true", help="prove this gate still detects drift")
+ args = parser.parse_args()
+
+ if args.self_test:
+ return self_test_main(
+ pathlib.Path(__file__).name,
+ ["--check"],
+ extra=[*_detects_injected_drift(), *_detects_unsourced_table()],
+ )
+
+ result = scan()
+ if "error" in result:
+ print(f"orm-parity: {result['error']}", file=sys.stderr)
+ return 2
+
+ if args.json:
+ print(json.dumps(result, indent=2, sort_keys=True))
+
+ # Naming the root and the corpus size is not decoration. A gate that
+ # printed "OK" without them could be reporting on a different checkout,
+ # or on none: `found nothing` and `scanned nothing` print the same word.
+ if not args.json:
+ print(f"orm-parity: root {result['root']}")
+ print(f"{'service':<14} {'schema from':<11} {'tables':>6} {'columns':>8} {'migrations':>11} {'findings':>9}")
+ print(f"{'-' * 14} {'-' * 11} {'-' * 6} {'-' * 8} {'-' * 11} {'-' * 9}")
+ for service in result["services"]:
+ print(
+ f"{service['service']:<14} {service['schema_source']:<11} {service['tables']:>6} "
+ f"{service['columns']:>8} {service['migration_files']:>11} {len(service['findings']):>9}"
+ )
+ for service in result["services"]:
+ if service["schema_source"] == "orm":
+ print(
+ f"\n{service['service']}: advisory only — builds its tables from the models at "
+ f"{service['create_all_at']}, so a column no migration creates is created anyway. "
+ f"{len(service['findings'])} disagreement(s); see --all."
+ )
+
+ if args.all and result["advisory_findings"]:
+ print(f"\nAdvisory ({len(result['advisory_findings'])}), in services whose tables come from the models:")
+ for finding in result["advisory_findings"]:
+ print(f" [{finding['kind']}] {finding['service']}: {finding['table']}.{finding['column']}")
+
+ if args.credits:
+ print("\nWhat each table's verdict was based on:")
+ for service in result["services"]:
+ for table, credit in sorted(service["credits"].items()):
+ columns = ", ".join(credit["migration_columns"]) or "(nothing — this is a finding)"
+ print(f" {service['service']}/{table} ← {credit['model_source']}::{credit['model']}")
+ print(f" migrations credit: {columns}")
+
+ if not result["gated_services"] or not result["compared"]:
+ # The corpus refusal. A tree with no migration-managed table produces
+ # no findings, and that is not a clean repository — it is a scan that
+ # read nothing, which this must never report as a pass. It also
+ # catches the scoping above going wrong: if every service were
+ # classified as building its tables from the models, this gate would
+ # have nothing left to judge and has to say so rather than pass.
+ print(
+ f"\norm-parity: paired 0 migration-managed tables under {SERVICES_DIR}. Zero tables "
+ f"compared is not zero drift; check the root above is the tree you meant.",
+ file=sys.stderr,
+ )
+ return 1
+
+ stale = unused_exceptions(result)
+ findings = result["findings"]
+
+ if not args.json:
+ print(
+ f"\ngated: {result['compared']} migration-managed table(s), "
+ f"{result['gated_columns']} declared column(s) across {len(result['gated_services'])} service(s)"
+ )
+
+ if findings:
+ print(f"\nFAIL: {len(findings)} ORM/migration disagreement(s):", file=sys.stderr)
+ for finding in findings:
+ print(
+ f" [{finding['kind']}] {finding['service']}: {finding['table']}.{finding['column']}\n" f" {finding['detail']}",
+ file=sys.stderr,
+ )
+ print(
+ "\nFix: add the column in a new migration (the direction that breaks at runtime), "
+ "or record it in UNMAPPED_COLUMNS in this file with the reason it is harmless.",
+ file=sys.stderr,
+ )
+ if stale:
+ print("\nFAIL: recorded exceptions that nothing needs any more:", file=sys.stderr)
+ for entry in stale:
+ print(f" {entry}", file=sys.stderr)
+ print("\nFix: delete them. This list only shrinks.", file=sys.stderr)
+
+ if args.check:
+ return 1 if (findings or stale) else 0
+ if findings or stale:
+ print("\n(advisory run — use --check to make this a gate)")
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/scripts/check_service_migration_bootstrap.py b/scripts/check_service_migration_bootstrap.py
new file mode 100644
index 000000000..2aed085d8
--- /dev/null
+++ b/scripts/check_service_migration_bootstrap.py
@@ -0,0 +1,304 @@
+#!/usr/bin/env python3
+"""A migration chain nothing invokes is not a migration chain.
+
+Why this exists
+---------------
+
+Four services own their schema through alembic — ``ueba``, ``honeytokens``,
+``purple-team`` and ``osquery-tls``. Their chains were correct: run by hand
+they applied cleanly, nine tables each with a row-level-security policy, each
+with its own version table. **No compose path invoked any of them.** Every
+container command was a plain ``uvicorn``, so on the documented quickstart
+those services booted healthy with empty schemas and no policies.
+
+That is how two services shipped with no tables while every command in the
+sequence reported success. The chain existing, the chain being correct, and
+the chain having run are three different facts, and only the third one
+matters to a running deployment.
+
+What it checks
+--------------
+
+For every service with an ``alembic.ini`` — discovered from the file, not
+from a list kept here, so a fifth service cannot be added without being
+held to this:
+
+``no-bootstrap-module``
+ ``app/_migrate.py`` is missing. That is the module that applies the
+ chain and verifies it reached head.
+
+``bootstrap-not-invoked``
+ The Dockerfile never runs it. This is the defect as it actually was: the
+ module could have been present and the ``CMD`` still a bare ``uvicorn``.
+ Checked against the ``CMD``/``ENTRYPOINT`` lines rather than the whole
+ file, so a mention in a comment does not satisfy it.
+
+``bootstrap-module-drift``
+ The copy differs from the reference one. The four are vendored because
+ services share no Python path, and a wrapper that verifies "reached
+ head" in one tree and not in another is worse than none: the fleet-wide
+ claim becomes unreadable.
+
+``no-owner-credential``
+ ``docker-compose.yml`` does not give the service a ``*DATABASE_MIGRATION_URL``.
+ Alembic issues DDL and the runtime role deliberately holds no ``CREATE``
+ on schema public — that is what stops it turning RLS off — so a chain
+ applied as the runtime credential dies on the first ``CREATE TABLE``.
+
+``no-healthcheck``
+ The service has no compose healthcheck, so a container that refused to
+ start still reads as ``running``. Invisibility is the whole reason both
+ of these defects survived a full acceptance pass.
+
+And the other direction: a service shipping ``app/_migrate.py`` with no
+``alembic.ini`` is reported too, because a bootstrap for a chain that is not
+there will exit 1 on every start of a service that has nothing to migrate.
+
+Usage
+-----
+
+::
+
+ python3 scripts/check_service_migration_bootstrap.py # table
+ python3 scripts/check_service_migration_bootstrap.py --check # gate
+ python3 scripts/check_service_migration_bootstrap.py --json
+ python3 scripts/check_service_migration_bootstrap.py --self-test
+
+Exit codes: 0 clean, 1 findings, 2 the scan itself could not run.
+"""
+
+from __future__ import annotations
+
+import argparse
+import json
+import pathlib
+import re
+import sys
+
+sys.path.insert(0, str(pathlib.Path(__file__).resolve().parent))
+
+from gate_toolkit import repo_root, self_test_main
+
+REPO_ROOT = repo_root()
+SERVICES_DIR = REPO_ROOT / "services"
+COMPOSE = REPO_ROOT / "docker-compose.yml"
+
+#: The module each service runs before its server, and the reference copy the
+#: others are compared against.
+BOOTSTRAP = pathlib.Path("app") / "_migrate.py"
+REFERENCE_SERVICE = "ueba"
+
+#: How the Dockerfile must invoke it. Matched on the module path so the shell
+#: spelling is free to change.
+INVOCATION = "app._migrate"
+
+
+def _compose_block(text: str, service: str) -> str:
+ """The ``docker-compose.yml`` stanza for one service.
+
+ Sliced on indentation rather than parsed with a YAML library: this gate
+ runs on a bare interpreter in CI before any ``pip install``, and pulling
+ in PyYAML to read two keys would take that away from it.
+ """
+ match = re.search(rf"^ {re.escape(service)}:$", text, re.M)
+ if not match:
+ return ""
+ rest = text[match.end() :]
+ end = re.search(r"^ \S", rest, re.M)
+ return rest[: end.start()] if end else rest
+
+
+def _command_lines(dockerfile: str) -> str:
+ """Just the CMD / ENTRYPOINT lines, so a comment cannot satisfy the check."""
+ return "\n".join(line for line in dockerfile.splitlines() if line.strip().upper().startswith(("CMD", "ENTRYPOINT")))
+
+
+def scan() -> dict:
+ if not SERVICES_DIR.is_dir():
+ return {"error": f"no services directory at {SERVICES_DIR}"}
+
+ compose_text = COMPOSE.read_text(encoding="utf-8") if COMPOSE.is_file() else ""
+ reference_path = SERVICES_DIR / REFERENCE_SERVICE / BOOTSTRAP
+ reference = reference_path.read_bytes() if reference_path.is_file() else None
+
+ services: list[dict] = []
+ findings: list[dict] = []
+
+ candidates = sorted(p for p in SERVICES_DIR.iterdir() if p.is_dir())
+
+ for service in candidates:
+ has_chain = (service / "alembic.ini").is_file()
+ module = service / BOOTSTRAP
+
+ # The other direction: a bootstrap with no chain to apply.
+ if module.is_file() and not has_chain:
+ findings.append(
+ {
+ "service": service.name,
+ "kind": "bootstrap-without-chain",
+ "detail": f"ships {BOOTSTRAP} but has no alembic.ini; it would exit 1 on every start",
+ }
+ )
+ if not has_chain:
+ continue
+
+ dockerfile = service / "Dockerfile"
+ commands = _command_lines(dockerfile.read_text(encoding="utf-8")) if dockerfile.is_file() else ""
+ block = _compose_block(compose_text, service.name)
+
+ row = {
+ "service": service.name,
+ "module": module.is_file(),
+ "in_sync": reference is not None and module.is_file() and module.read_bytes() == reference,
+ "invoked": INVOCATION in commands,
+ "owner_credential": bool(re.search(r"^\s+\w*DATABASE_MIGRATION_URL:", block, re.M)),
+ "healthcheck": "healthcheck:" in block,
+ "in_compose": bool(block),
+ }
+ services.append(row)
+
+ if not row["module"]:
+ findings.append(
+ {
+ "service": service.name,
+ "kind": "no-bootstrap-module",
+ "detail": f"{BOOTSTRAP} is missing; nothing applies this service's chain",
+ }
+ )
+ elif not row["in_sync"]:
+ findings.append(
+ {
+ "service": service.name,
+ "kind": "bootstrap-module-drift",
+ "detail": f"{BOOTSTRAP} differs from services/{REFERENCE_SERVICE}/{BOOTSTRAP}",
+ }
+ )
+ if not row["invoked"]:
+ findings.append(
+ {
+ "service": service.name,
+ "kind": "bootstrap-not-invoked",
+ "detail": f"no CMD/ENTRYPOINT runs {INVOCATION}; this service would boot on whatever schema is there",
+ }
+ )
+ if row["in_compose"] and not row["owner_credential"]:
+ findings.append(
+ {
+ "service": service.name,
+ "kind": "no-owner-credential",
+ "detail": (
+ "no *DATABASE_MIGRATION_URL in its compose environment; the chain would be "
+ "applied as the runtime role and fail on the first CREATE TABLE"
+ ),
+ }
+ )
+ if row["in_compose"] and not row["healthcheck"]:
+ findings.append(
+ {
+ "service": service.name,
+ "kind": "no-healthcheck",
+ "detail": "no compose healthcheck, so a container that refused to start still reads as running",
+ }
+ )
+
+ return {"root": str(REPO_ROOT), "services": services, "findings": findings}
+
+
+# ---------------------------------------------------------------------------
+# Self-test
+# ---------------------------------------------------------------------------
+
+
+def _detects_uninvoked_chain() -> list[tuple[str, bool]]:
+ """Prove each kind is still detected, on a tree built for the purpose."""
+ import tempfile
+
+ global REPO_ROOT, SERVICES_DIR, COMPOSE # noqa: PLW0603 — restored below
+ saved = (REPO_ROOT, SERVICES_DIR, COMPOSE)
+ try:
+ with tempfile.TemporaryDirectory() as raw:
+ root = pathlib.Path(raw)
+ service = root / "services" / "widgetry"
+ (service / "app").mkdir(parents=True)
+ (service / "alembic.ini").write_text("[alembic]\nscript_location = alembic\n")
+ # The defect exactly as it was: a chain, and a bare uvicorn CMD.
+ (service / "Dockerfile").write_text('FROM python:3.11-slim\nCMD ["uvicorn", "app.main:app", "--port", "8000"]\n')
+ (root / "docker-compose.yml").write_text("services:\n widgetry:\n image: x\n")
+
+ # A second service with the bootstrap and no chain — the other
+ # direction.
+ stray = root / "services" / "strayville"
+ (stray / "app").mkdir(parents=True)
+ (stray / "app" / "_migrate.py").write_text("# stub\n")
+
+ REPO_ROOT, SERVICES_DIR, COMPOSE = root, root / "services", root / "docker-compose.yml"
+ kinds = {f["kind"] for f in scan()["findings"]}
+ finally:
+ REPO_ROOT, SERVICES_DIR, COMPOSE = saved
+
+ return [
+ ("detects a chain with no bootstrap module", "no-bootstrap-module" in kinds),
+ ("detects a bootstrap module the Dockerfile never runs", "bootstrap-not-invoked" in kinds),
+ ("detects a service with no owner credential", "no-owner-credential" in kinds),
+ ("detects a service with no compose healthcheck", "no-healthcheck" in kinds),
+ ("detects a bootstrap module with no chain to apply", "bootstrap-without-chain" in kinds),
+ ]
+
+
+def main() -> int:
+ parser = argparse.ArgumentParser(description="alembic chains must be invoked on the way up")
+ parser.add_argument("--check", action="store_true", help="exit non-zero on findings")
+ parser.add_argument("--json", action="store_true", help="machine-readable output")
+ parser.add_argument("--self-test", action="store_true", help="prove this gate still detects drift")
+ args = parser.parse_args()
+
+ if args.self_test:
+ return self_test_main(pathlib.Path(__file__).name, ["--check"], extra=_detects_uninvoked_chain())
+
+ result = scan()
+ if "error" in result:
+ print(f"migration-bootstrap: {result['error']}", file=sys.stderr)
+ return 2
+
+ if args.json:
+ print(json.dumps(result, indent=2, sort_keys=True))
+ else:
+ print(f"migration-bootstrap: root {result['root']}")
+ print(f"{'service':<14} {'_migrate':<9} {'in-sync':<8} {'invoked':<8} {'owner':<6} {'health':<7}")
+ print(f"{'-' * 14} {'-' * 9} {'-' * 8} {'-' * 8} {'-' * 6} {'-' * 7}")
+ for row in result["services"]:
+ mark = lambda flag: "yes" if flag else "NO" # noqa: E731 — table formatting only
+ print(
+ f"{row['service']:<14} {mark(row['module']):<9} {mark(row['in_sync']):<8} "
+ f"{mark(row['invoked']):<8} {mark(row['owner_credential']):<6} {mark(row['healthcheck']):<7}"
+ )
+
+ if not result["services"]:
+ # The corpus refusal. No service with an alembic.ini means this gate
+ # inspected nothing, and "nothing to check" must not print the same
+ # verdict as "everything checked out".
+ print(
+ f"\nmigration-bootstrap: found no service with an alembic.ini under {SERVICES_DIR}. "
+ f"Four services own their schema that way; zero found means the root above is not "
+ f"the tree you meant.",
+ file=sys.stderr,
+ )
+ return 1
+
+ if not result["findings"]:
+ print(f"\nOK: {len(result['services'])} alembic-managed service(s) apply their chain before serving")
+ return 0
+
+ print(f"\nFAIL: {len(result['findings'])} finding(s):", file=sys.stderr)
+ for finding in result["findings"]:
+ print(f" [{finding['kind']}] {finding['service']}: {finding['detail']}", file=sys.stderr)
+ print(
+ f"\nFix: copy services/{REFERENCE_SERVICE}/{BOOTSTRAP} into the service, and make its "
+ f"Dockerfile CMD run `python -m {INVOCATION} -- <server command>`.",
+ file=sys.stderr,
+ )
+ return 1 if args.check else 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/scripts/mypy_baseline.json b/scripts/mypy_baseline.json
index 06122f20a..a34a48c4e 100644
--- a/scripts/mypy_baseline.json
+++ b/scripts/mypy_baseline.json
@@ -510,10 +510,6 @@
"tests/test_event_warehouse_registry.py": {
"arg-type": 3
},
- "tests/test_health_probes.py": {
- "misc": 6,
- "valid-type": 1
- },
"tests/test_hunts_tenant_isolation.py": {
"misc": 1
},
diff --git a/services/actions/app/_health.py b/services/actions/app/_health.py
index 974269731..0acb9e4da 100644
--- a/services/actions/app/_health.py
+++ b/services/actions/app/_health.py
@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /
``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /
``mark_not_ready()`` from those handlers respectively.
+Readiness is a question, not a latch
+------------------------------------
+
+``mark_ready()`` answers "did startup finish", and for a service
+whose only job is to answer HTTP that is the whole story. For a
+service whose job is to *consume a topic* it is not: three
+services in this repository created the consumer as a
+fire-and-forget ``asyncio.create_task`` and called
+``mark_ready()`` on the next line, so readiness recorded that the
+task had been **created**, never that it was still running.
+
+UEBA is what that cost. Its handler raised
+``UndefinedColumnError`` on the first scoreable event, the
+exception left the ``async for``, the ``finally`` stopped the
+consumer, and the task ended — holding the exception, which
+nothing retrieved because the task was parked in a module global
+and so never garbage-collected. The container stayed ``running``
+with restarts 0, ``/health`` returned 200 and ``/readyz``
+returned 200, permanently. A consumer that has silently detached
+is indistinguishable from an idle one, and the whole point of a
+readiness probe is to tell those two apart.
+
+So a service that owns a subscription registers a probe for it
+with :func:`register_subscription`, and ``/readyz`` evaluates
+every probe on each request. A probe that returns false — or
+raises — is reported as detached and ``/readyz`` answers 503 with
+the names, which is a readiness failure rather than a liveness
+one on purpose: the orchestrator should stop routing to a
+half-working replica, and killing the process would only hide a
+schema or broker fault behind a restart loop.
+
+``/livez`` deliberately does **not** consult the probes. It
+answers "is the interpreter responsive", and wiring a detached
+consumer into it would make every orchestrator restart the pod on
+a fault a restart cannot fix.
+
This module is intentionally dependency-free (only FastAPI) so it
can be vendored into every service tree without import-graph
side-effects. Each service ships its own copy under
``services/<svc>/app/_health.py`` because services don't share a
-Python path — keep the file in sync via
-``scripts/sync_health_module.py`` (CI gate).
+Python path; ``scripts/audit_health_probes.py --check`` is the CI
+gate that every service has one and wires it in.
"""
from __future__ import annotations
+import json
from collections.abc import Callable
from typing import Any
from fastapi import FastAPI, Response
+#: Where the probes registered against an app are kept. Read by ``/readyz``,
+#: written by :func:`register_subscription`. On ``app.state`` rather than in a
+#: module global because a module global is shared by every app in the
+#: process, which makes two test clients interfere.
+_STATE_ATTR = "health_subscriptions"
+
+
+def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:
+ """Names of the subscriptions currently reporting not-attached.
+
+ A probe that raises counts as detached. It is asking a live object
+ whether it is still working; if that question itself fails, the honest
+ answer is not "yes".
+ """
+ detached: list[str] = []
+ for name, probe in sorted(probes.items()):
+ try:
+ attached = bool(probe())
+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one
+ attached = False
+ if not attached:
+ detached.append(name)
+ return detached
+
+
+def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:
+ """Make ``/readyz`` depend on *probe* returning true.
+
+ Call this once per subscription the service owns, after
+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —
+ it runs on every ``/readyz`` request — so it should read a flag the
+ consumer maintains or check ``task.done()``, not round-trip to a broker.
+ """
+ probes = getattr(app.state, _STATE_ATTR, None)
+ if probes is None: # pragma: no cover — install_health_routes was not called
+ raise RuntimeError("register_subscription requires install_health_routes(app) first")
+ probes[name] = probe
+
def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:
"""Install ``/livez`` and ``/readyz`` on ``app``.
@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
``mark_ready()`` is called — exactly the right behaviour for a
rolling deploy where the orchestrator should hold traffic
until the pod's dependencies are connected.
+
+ Startup having finished is necessary and not sufficient: any
+ subscription registered with :func:`register_subscription`
+ must also still be attached. See the module docstring.
"""
state: dict[str, bool] = {"ready": False}
+ probes: dict[str, Callable[[], bool]] = {}
+ setattr(app.state, _STATE_ATTR, probes)
@app.get(
"/livez",
@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
include_in_schema=False,
)
async def _readyz() -> Response:
- if state["ready"]:
- return Response(
- content='{"status":"ready","service":"' + service_name + '"}',
- media_type="application/json",
- status_code=200,
- )
+ body: dict[str, Any] = {"status": "ready", "service": service_name}
+ status_code = 200
+
+ if not state["ready"]:
+ body["status"] = "starting"
+ status_code = 503
+ else:
+ detached = _probe_detached(probes)
+ if detached:
+ body["status"] = "degraded"
+ body["detached"] = detached
+ status_code = 503
+
+ # Named subscriptions are reported either way, so a 200 says which
+ # ones were checked rather than only that nothing was wrong. A probe
+ # list that is unexpectedly empty is then visible in the same place
+ # the verdict is.
+ if probes:
+ body["subscriptions"] = sorted(probes)
+
return Response(
- content='{"status":"starting","service":"' + service_name + '"}',
+ content=json.dumps(body),
media_type="application/json",
- status_code=503,
+ status_code=status_code,
)
def mark_ready() -> None:
diff --git a/services/agents/app/_health.py b/services/agents/app/_health.py
index 974269731..0acb9e4da 100644
--- a/services/agents/app/_health.py
+++ b/services/agents/app/_health.py
@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /
``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /
``mark_not_ready()`` from those handlers respectively.
+Readiness is a question, not a latch
+------------------------------------
+
+``mark_ready()`` answers "did startup finish", and for a service
+whose only job is to answer HTTP that is the whole story. For a
+service whose job is to *consume a topic* it is not: three
+services in this repository created the consumer as a
+fire-and-forget ``asyncio.create_task`` and called
+``mark_ready()`` on the next line, so readiness recorded that the
+task had been **created**, never that it was still running.
+
+UEBA is what that cost. Its handler raised
+``UndefinedColumnError`` on the first scoreable event, the
+exception left the ``async for``, the ``finally`` stopped the
+consumer, and the task ended — holding the exception, which
+nothing retrieved because the task was parked in a module global
+and so never garbage-collected. The container stayed ``running``
+with restarts 0, ``/health`` returned 200 and ``/readyz``
+returned 200, permanently. A consumer that has silently detached
+is indistinguishable from an idle one, and the whole point of a
+readiness probe is to tell those two apart.
+
+So a service that owns a subscription registers a probe for it
+with :func:`register_subscription`, and ``/readyz`` evaluates
+every probe on each request. A probe that returns false — or
+raises — is reported as detached and ``/readyz`` answers 503 with
+the names, which is a readiness failure rather than a liveness
+one on purpose: the orchestrator should stop routing to a
+half-working replica, and killing the process would only hide a
+schema or broker fault behind a restart loop.
+
+``/livez`` deliberately does **not** consult the probes. It
+answers "is the interpreter responsive", and wiring a detached
+consumer into it would make every orchestrator restart the pod on
+a fault a restart cannot fix.
+
This module is intentionally dependency-free (only FastAPI) so it
can be vendored into every service tree without import-graph
side-effects. Each service ships its own copy under
``services/<svc>/app/_health.py`` because services don't share a
-Python path — keep the file in sync via
-``scripts/sync_health_module.py`` (CI gate).
+Python path; ``scripts/audit_health_probes.py --check`` is the CI
+gate that every service has one and wires it in.
"""
from __future__ import annotations
+import json
from collections.abc import Callable
from typing import Any
from fastapi import FastAPI, Response
+#: Where the probes registered against an app are kept. Read by ``/readyz``,
+#: written by :func:`register_subscription`. On ``app.state`` rather than in a
+#: module global because a module global is shared by every app in the
+#: process, which makes two test clients interfere.
+_STATE_ATTR = "health_subscriptions"
+
+
+def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:
+ """Names of the subscriptions currently reporting not-attached.
+
+ A probe that raises counts as detached. It is asking a live object
+ whether it is still working; if that question itself fails, the honest
+ answer is not "yes".
+ """
+ detached: list[str] = []
+ for name, probe in sorted(probes.items()):
+ try:
+ attached = bool(probe())
+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one
+ attached = False
+ if not attached:
+ detached.append(name)
+ return detached
+
+
+def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:
+ """Make ``/readyz`` depend on *probe* returning true.
+
+ Call this once per subscription the service owns, after
+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —
+ it runs on every ``/readyz`` request — so it should read a flag the
+ consumer maintains or check ``task.done()``, not round-trip to a broker.
+ """
+ probes = getattr(app.state, _STATE_ATTR, None)
+ if probes is None: # pragma: no cover — install_health_routes was not called
+ raise RuntimeError("register_subscription requires install_health_routes(app) first")
+ probes[name] = probe
+
def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:
"""Install ``/livez`` and ``/readyz`` on ``app``.
@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
``mark_ready()`` is called — exactly the right behaviour for a
rolling deploy where the orchestrator should hold traffic
until the pod's dependencies are connected.
+
+ Startup having finished is necessary and not sufficient: any
+ subscription registered with :func:`register_subscription`
+ must also still be attached. See the module docstring.
"""
state: dict[str, bool] = {"ready": False}
+ probes: dict[str, Callable[[], bool]] = {}
+ setattr(app.state, _STATE_ATTR, probes)
@app.get(
"/livez",
@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
include_in_schema=False,
)
async def _readyz() -> Response:
- if state["ready"]:
- return Response(
- content='{"status":"ready","service":"' + service_name + '"}',
- media_type="application/json",
- status_code=200,
- )
+ body: dict[str, Any] = {"status": "ready", "service": service_name}
+ status_code = 200
+
+ if not state["ready"]:
+ body["status"] = "starting"
+ status_code = 503
+ else:
+ detached = _probe_detached(probes)
+ if detached:
+ body["status"] = "degraded"
+ body["detached"] = detached
+ status_code = 503
+
+ # Named subscriptions are reported either way, so a 200 says which
+ # ones were checked rather than only that nothing was wrong. A probe
+ # list that is unexpectedly empty is then visible in the same place
+ # the verdict is.
+ if probes:
+ body["subscriptions"] = sorted(probes)
+
return Response(
- content='{"status":"starting","service":"' + service_name + '"}',
+ content=json.dumps(body),
media_type="application/json",
- status_code=503,
+ status_code=status_code,
)
def mark_ready() -> None:
diff --git a/services/agents/app/main.py b/services/agents/app/main.py
index f434c64ec..5de9c6269 100644
--- a/services/agents/app/main.py
+++ b/services/agents/app/main.py
@@ -7,7 +7,7 @@ import structlog
from fastapi import FastAPI
from fastapi.middleware.cors import CORSMiddleware
-from app._health import install_health_routes
+from app._health import install_health_routes, register_subscription
from app.api.contextual import router as contextual_router
from app.api.copilot import router as copilot_router
from app.api.explain import router as explain_router
@@ -32,6 +32,18 @@ from app.workers.fused_alert_consumer import FusedAlertTriageWorker, worker_enab
logger = structlog.get_logger()
+def _log_triage_worker_exit(task: asyncio.Task) -> None:
+ """Report the auto-triage worker's exit instead of losing it in the task."""
+ if task.cancelled():
+ logger.info("auto_triage_worker.cancelled")
+ return
+ exc = task.exception()
+ if exc is None:
+ logger.error("auto_triage_worker.exited", detail="consume loop returned; fused alerts are no longer triaged")
+ return
+ logger.error("auto_triage_worker.died", error=str(exc), error_type=type(exc).__name__, exc_info=exc)
+
+
@asynccontextmanager
async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
"""Load resources on startup."""
@@ -100,7 +112,15 @@ async def lifespan(app: FastAPI) -> AsyncGenerator[None, None]:
business_context=applier,
)
app.state.triage_worker = triage_worker
- app.state.triage_worker_task = asyncio.create_task(triage_worker.start())
+ task = asyncio.create_task(triage_worker.start())
+ # Same pair as fusion and ueba: retrieve the task's outcome so a
+ # dead worker is in the log, and make readiness depend on the
+ # subscription rather than on this block having run. Without the
+ # probe, /readyz answered 200 for a worker whose loop had exited —
+ # and nothing auto-triages a fused alert after that.
+ task.add_done_callback(_log_triage_worker_exit)
+ app.state.triage_worker_task = task
+ register_subscription(app, triage_worker.topic, lambda: triage_worker.attached)
logger.info("auto_triage_worker.enabled")
except Exception as exc: # noqa: BLE001 — never block API startup
logger.warning("auto_triage_worker.start_failed", error=str(exc))
diff --git a/services/agents/app/workers/fused_alert_consumer.py b/services/agents/app/workers/fused_alert_consumer.py
index 620d95e0f..6a3df7abf 100644
--- a/services/agents/app/workers/fused_alert_consumer.py
+++ b/services/agents/app/workers/fused_alert_consumer.py
@@ -241,6 +241,7 @@ class FusedAlertTriageWorker:
self._consumer: Any | None = None
self._producer: Any | None = None # lazily created for the DLQ
self._running = False
+ self._attached = False
# Phase B4 — environment-specific noise reduction applied post-fusion →
# pre-triage. None = disabled (no rules file / flag off).
self._business_context = business_context
@@ -260,6 +261,7 @@ class FusedAlertTriageWorker:
)
await self._consumer.start()
self._running = True
+ self._attached = True
logger.info("auto_triage_worker.started", topic=self._topic, group=self._group_id)
try:
async for msg in self._consumer:
@@ -269,12 +271,33 @@ class FusedAlertTriageWorker:
# or after the poison alert is dead-lettered — so a crash between
# inference and persistence replays the alert instead of losing
# it, and offsets advance only on durable completion (issue #571).
- await self._process_with_retry(msg.value)
+ #
+ # ``_process_with_retry`` catches and dead-letters rather than
+ # raising, which is the reason this loop survives a poison
+ # alert. The guard is here as well because that is a property
+ # of another method: the loop must not depend on a callee
+ # staying total, which is exactly how the ueba consumer came
+ # to have no ``except`` on its only handler.
+ try:
+ await self._process_with_retry(msg.value)
+ except Exception as exc: # noqa: BLE001 — one alert must not end the subscription
+ _METRICS["errors"] += 1
+ logger.error(
+ "auto_triage_worker.handler_escaped",
+ error=str(exc),
+ error_type=type(exc).__name__,
+ partition=getattr(msg, "partition", None),
+ offset=getattr(msg, "offset", None),
+ exc_info=True,
+ )
try:
await self._consumer.commit()
except Exception as commit_exc: # noqa: BLE001 — reprocess on restart
logger.warning("auto_triage_worker.commit_failed", error=str(commit_exc))
finally:
+ # Cleared before the teardown await, which can itself raise on a
+ # broker that has gone away.
+ self._attached = False
await self._consumer.stop()
async def _process_with_retry(self, message: Any) -> bool:
@@ -349,8 +372,19 @@ class FusedAlertTriageWorker:
except Exception as exc: # noqa: BLE001 — never wedge the loop on a DLQ failure
logger.error("auto_triage_worker.dlq_send_failed", alert_id=alert_id, error=str(exc))
+ @property
+ def topic(self) -> str:
+ """The topic this worker subscribes to (names its readiness probe)."""
+ return self._topic
+
+ @property
+ def attached(self) -> bool:
+ """Whether the consume loop is currently iterating the subscription."""
+ return self._attached
+
async def stop(self) -> None:
self._running = False
+ self._attached = False
if self._consumer is not None:
await self._consumer.stop()
self._consumer = None
diff --git a/services/api/app/_health.py b/services/api/app/_health.py
index 974269731..0acb9e4da 100644
--- a/services/api/app/_health.py
+++ b/services/api/app/_health.py
@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /
``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /
``mark_not_ready()`` from those handlers respectively.
+Readiness is a question, not a latch
+------------------------------------
+
+``mark_ready()`` answers "did startup finish", and for a service
+whose only job is to answer HTTP that is the whole story. For a
+service whose job is to *consume a topic* it is not: three
+services in this repository created the consumer as a
+fire-and-forget ``asyncio.create_task`` and called
+``mark_ready()`` on the next line, so readiness recorded that the
+task had been **created**, never that it was still running.
+
+UEBA is what that cost. Its handler raised
+``UndefinedColumnError`` on the first scoreable event, the
+exception left the ``async for``, the ``finally`` stopped the
+consumer, and the task ended — holding the exception, which
+nothing retrieved because the task was parked in a module global
+and so never garbage-collected. The container stayed ``running``
+with restarts 0, ``/health`` returned 200 and ``/readyz``
+returned 200, permanently. A consumer that has silently detached
+is indistinguishable from an idle one, and the whole point of a
+readiness probe is to tell those two apart.
+
+So a service that owns a subscription registers a probe for it
+with :func:`register_subscription`, and ``/readyz`` evaluates
+every probe on each request. A probe that returns false — or
+raises — is reported as detached and ``/readyz`` answers 503 with
+the names, which is a readiness failure rather than a liveness
+one on purpose: the orchestrator should stop routing to a
+half-working replica, and killing the process would only hide a
+schema or broker fault behind a restart loop.
+
+``/livez`` deliberately does **not** consult the probes. It
+answers "is the interpreter responsive", and wiring a detached
+consumer into it would make every orchestrator restart the pod on
+a fault a restart cannot fix.
+
This module is intentionally dependency-free (only FastAPI) so it
can be vendored into every service tree without import-graph
side-effects. Each service ships its own copy under
``services/<svc>/app/_health.py`` because services don't share a
-Python path — keep the file in sync via
-``scripts/sync_health_module.py`` (CI gate).
+Python path; ``scripts/audit_health_probes.py --check`` is the CI
+gate that every service has one and wires it in.
"""
from __future__ import annotations
+import json
from collections.abc import Callable
from typing import Any
from fastapi import FastAPI, Response
+#: Where the probes registered against an app are kept. Read by ``/readyz``,
+#: written by :func:`register_subscription`. On ``app.state`` rather than in a
+#: module global because a module global is shared by every app in the
+#: process, which makes two test clients interfere.
+_STATE_ATTR = "health_subscriptions"
+
+
+def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:
+ """Names of the subscriptions currently reporting not-attached.
+
+ A probe that raises counts as detached. It is asking a live object
+ whether it is still working; if that question itself fails, the honest
+ answer is not "yes".
+ """
+ detached: list[str] = []
+ for name, probe in sorted(probes.items()):
+ try:
+ attached = bool(probe())
+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one
+ attached = False
+ if not attached:
+ detached.append(name)
+ return detached
+
+
+def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:
+ """Make ``/readyz`` depend on *probe* returning true.
+
+ Call this once per subscription the service owns, after
+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —
+ it runs on every ``/readyz`` request — so it should read a flag the
+ consumer maintains or check ``task.done()``, not round-trip to a broker.
+ """
+ probes = getattr(app.state, _STATE_ATTR, None)
+ if probes is None: # pragma: no cover — install_health_routes was not called
+ raise RuntimeError("register_subscription requires install_health_routes(app) first")
+ probes[name] = probe
+
def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:
"""Install ``/livez`` and ``/readyz`` on ``app``.
@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
``mark_ready()`` is called — exactly the right behaviour for a
rolling deploy where the orchestrator should hold traffic
until the pod's dependencies are connected.
+
+ Startup having finished is necessary and not sufficient: any
+ subscription registered with :func:`register_subscription`
+ must also still be attached. See the module docstring.
"""
state: dict[str, bool] = {"ready": False}
+ probes: dict[str, Callable[[], bool]] = {}
+ setattr(app.state, _STATE_ATTR, probes)
@app.get(
"/livez",
@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
include_in_schema=False,
)
async def _readyz() -> Response:
- if state["ready"]:
- return Response(
- content='{"status":"ready","service":"' + service_name + '"}',
- media_type="application/json",
- status_code=200,
- )
+ body: dict[str, Any] = {"status": "ready", "service": service_name}
+ status_code = 200
+
+ if not state["ready"]:
+ body["status"] = "starting"
+ status_code = 503
+ else:
+ detached = _probe_detached(probes)
+ if detached:
+ body["status"] = "degraded"
+ body["detached"] = detached
+ status_code = 503
+
+ # Named subscriptions are reported either way, so a 200 says which
+ # ones were checked rather than only that nothing was wrong. A probe
+ # list that is unexpectedly empty is then visible in the same place
+ # the verdict is.
+ if probes:
+ body["subscriptions"] = sorted(probes)
+
return Response(
- content='{"status":"starting","service":"' + service_name + '"}',
+ content=json.dumps(body),
media_type="application/json",
- status_code=503,
+ status_code=status_code,
)
def mark_ready() -> None:
diff --git a/services/api/tests/test_health_probes.py b/services/api/tests/test_health_probes.py
index 37f51e968..d0f83b69a 100644
--- a/services/api/tests/test_health_probes.py
+++ b/services/api/tests/test_health_probes.py
@@ -19,12 +19,20 @@ canonical copy.
from __future__ import annotations
-from app._health import install_health_routes
+from collections.abc import Callable
+
+import pytest
+from app._health import install_health_routes, register_subscription
from fastapi import FastAPI
from fastapi.testclient import TestClient
-def _build_app() -> tuple[FastAPI, callable, callable]:
+def _build_app() -> tuple[FastAPI, Callable[[], None], Callable[[], None]]:
+ # `callable` (the builtin) was used here as a type, which is not one:
+ # mypy read every `mark_ready()` in this file as calling a value of type
+ # `callable?` and recorded six `misc` findings plus a `valid-type` for
+ # the annotation. Adding tests below made the count grow rather than the
+ # cause get fixed.
app = FastAPI()
mark_ready, mark_not_ready = install_health_routes(app, service_name="aisoc-test")
return app, mark_ready, mark_not_ready
@@ -89,6 +97,97 @@ def test_readyz_can_recover_after_not_ready() -> None:
assert client.get("/readyz").status_code == 200
+class TestSubscriptionProbes:
+ """Readiness has to answer "is it running", not "was it started".
+
+ UEBA is what made this necessary. Its consumer was created as a
+ fire-and-forget ``asyncio.create_task`` and ``mark_ready()`` was called
+ on the next line, so readiness recorded that the task had been created.
+ The first scoreable event raised ``UndefinedColumnError``, the loop
+ exited, and ``/readyz`` answered 200 for as long as the container was
+ left up. Two other services had the same pairing.
+ """
+
+ def test_ready_stays_200_while_the_subscription_is_attached(self) -> None:
+ app, mark_ready, _ = _build_app()
+ register_subscription(app, "aisoc.raw_events", lambda: True)
+ mark_ready()
+
+ response = TestClient(app).get("/readyz")
+
+ assert response.status_code == 200
+ # Named on the way past, so a 200 says what it checked rather than
+ # only that nothing was wrong.
+ assert response.json()["subscriptions"] == ["aisoc.raw_events"]
+
+ def test_detached_subscription_makes_readiness_503(self) -> None:
+ app, mark_ready, _ = _build_app()
+ attached = {"value": True}
+ register_subscription(app, "aisoc.raw_events", lambda: attached["value"])
+ mark_ready()
+ client = TestClient(app)
+ assert client.get("/readyz").status_code == 200
+
+ attached["value"] = False
+
+ response = client.get("/readyz")
+ assert response.status_code == 503
+ assert response.json()["status"] == "degraded"
+ assert response.json()["detached"] == ["aisoc.raw_events"]
+
+ def test_a_probe_that_raises_counts_as_detached(self) -> None:
+ """Fail closed. The probe asks a live object whether it still
+ works; if that question itself raises, the answer is not "yes"."""
+
+ def explode() -> bool:
+ raise RuntimeError("consumer object is gone")
+
+ app, mark_ready, _ = _build_app()
+ register_subscription(app, "aisoc.alerts.fused", explode)
+ mark_ready()
+
+ response = TestClient(app).get("/readyz")
+
+ assert response.status_code == 503
+ assert response.json()["detached"] == ["aisoc.alerts.fused"]
+
+ def test_livez_ignores_a_detached_subscription(self) -> None:
+ """Deliberate. Liveness failing tells an orchestrator to restart the
+ pod, and a restart does not fix a missing column or a revoked grant
+ — it just hides it behind a crash loop."""
+ app, mark_ready, _ = _build_app()
+ register_subscription(app, "aisoc.raw_events", lambda: False)
+ mark_ready()
+
+ assert TestClient(app).get("/livez").status_code == 200
+
+ def test_probes_are_not_consulted_before_startup_finishes(self) -> None:
+ """An attached consumer does not make a half-started service ready."""
+ app, _, _ = _build_app()
+ register_subscription(app, "aisoc.raw_events", lambda: True)
+
+ response = TestClient(app).get("/readyz")
+
+ assert response.status_code == 503
+ assert response.json()["status"] == "starting"
+
+ def test_registration_requires_the_routes_to_be_installed(self) -> None:
+ with pytest.raises(RuntimeError):
+ register_subscription(FastAPI(), "aisoc.raw_events", lambda: True)
+
+ def test_two_apps_do_not_share_a_registry(self) -> None:
+ """The registry hangs off ``app.state``, not a module global: two
+ test clients in one process must not see each other's probes."""
+ first, first_ready, _ = _build_app()
+ second, second_ready, _ = _build_app()
+ register_subscription(first, "first.topic", lambda: False)
+ first_ready()
+ second_ready()
+
+ assert TestClient(first).get("/readyz").status_code == 503
+ assert TestClient(second).get("/readyz").status_code == 200
+
+
def test_probes_excluded_from_openapi() -> None:
"""``/livez`` and ``/readyz`` are operator-facing; we don't want
them spamming the SDK / OpenAPI surface that the rest of the
diff --git a/services/connectors/app/_health.py b/services/connectors/app/_health.py
index 974269731..0acb9e4da 100644
--- a/services/connectors/app/_health.py
+++ b/services/connectors/app/_health.py
@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /
``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /
``mark_not_ready()`` from those handlers respectively.
+Readiness is a question, not a latch
+------------------------------------
+
+``mark_ready()`` answers "did startup finish", and for a service
+whose only job is to answer HTTP that is the whole story. For a
+service whose job is to *consume a topic* it is not: three
+services in this repository created the consumer as a
+fire-and-forget ``asyncio.create_task`` and called
+``mark_ready()`` on the next line, so readiness recorded that the
+task had been **created**, never that it was still running.
+
+UEBA is what that cost. Its handler raised
+``UndefinedColumnError`` on the first scoreable event, the
+exception left the ``async for``, the ``finally`` stopped the
+consumer, and the task ended — holding the exception, which
+nothing retrieved because the task was parked in a module global
+and so never garbage-collected. The container stayed ``running``
+with restarts 0, ``/health`` returned 200 and ``/readyz``
+returned 200, permanently. A consumer that has silently detached
+is indistinguishable from an idle one, and the whole point of a
+readiness probe is to tell those two apart.
+
+So a service that owns a subscription registers a probe for it
+with :func:`register_subscription`, and ``/readyz`` evaluates
+every probe on each request. A probe that returns false — or
+raises — is reported as detached and ``/readyz`` answers 503 with
+the names, which is a readiness failure rather than a liveness
+one on purpose: the orchestrator should stop routing to a
+half-working replica, and killing the process would only hide a
+schema or broker fault behind a restart loop.
+
+``/livez`` deliberately does **not** consult the probes. It
+answers "is the interpreter responsive", and wiring a detached
+consumer into it would make every orchestrator restart the pod on
+a fault a restart cannot fix.
+
This module is intentionally dependency-free (only FastAPI) so it
can be vendored into every service tree without import-graph
side-effects. Each service ships its own copy under
``services/<svc>/app/_health.py`` because services don't share a
-Python path — keep the file in sync via
-``scripts/sync_health_module.py`` (CI gate).
+Python path; ``scripts/audit_health_probes.py --check`` is the CI
+gate that every service has one and wires it in.
"""
from __future__ import annotations
+import json
from collections.abc import Callable
from typing import Any
from fastapi import FastAPI, Response
+#: Where the probes registered against an app are kept. Read by ``/readyz``,
+#: written by :func:`register_subscription`. On ``app.state`` rather than in a
+#: module global because a module global is shared by every app in the
+#: process, which makes two test clients interfere.
+_STATE_ATTR = "health_subscriptions"
+
+
+def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:
+ """Names of the subscriptions currently reporting not-attached.
+
+ A probe that raises counts as detached. It is asking a live object
+ whether it is still working; if that question itself fails, the honest
+ answer is not "yes".
+ """
+ detached: list[str] = []
+ for name, probe in sorted(probes.items()):
+ try:
+ attached = bool(probe())
+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one
+ attached = False
+ if not attached:
+ detached.append(name)
+ return detached
+
+
+def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:
+ """Make ``/readyz`` depend on *probe* returning true.
+
+ Call this once per subscription the service owns, after
+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —
+ it runs on every ``/readyz`` request — so it should read a flag the
+ consumer maintains or check ``task.done()``, not round-trip to a broker.
+ """
+ probes = getattr(app.state, _STATE_ATTR, None)
+ if probes is None: # pragma: no cover — install_health_routes was not called
+ raise RuntimeError("register_subscription requires install_health_routes(app) first")
+ probes[name] = probe
+
def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:
"""Install ``/livez`` and ``/readyz`` on ``app``.
@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
``mark_ready()`` is called — exactly the right behaviour for a
rolling deploy where the orchestrator should hold traffic
until the pod's dependencies are connected.
+
+ Startup having finished is necessary and not sufficient: any
+ subscription registered with :func:`register_subscription`
+ must also still be attached. See the module docstring.
"""
state: dict[str, bool] = {"ready": False}
+ probes: dict[str, Callable[[], bool]] = {}
+ setattr(app.state, _STATE_ATTR, probes)
@app.get(
"/livez",
@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
include_in_schema=False,
)
async def _readyz() -> Response:
- if state["ready"]:
- return Response(
- content='{"status":"ready","service":"' + service_name + '"}',
- media_type="application/json",
- status_code=200,
- )
+ body: dict[str, Any] = {"status": "ready", "service": service_name}
+ status_code = 200
+
+ if not state["ready"]:
+ body["status"] = "starting"
+ status_code = 503
+ else:
+ detached = _probe_detached(probes)
+ if detached:
+ body["status"] = "degraded"
+ body["detached"] = detached
+ status_code = 503
+
+ # Named subscriptions are reported either way, so a 200 says which
+ # ones were checked rather than only that nothing was wrong. A probe
+ # list that is unexpectedly empty is then visible in the same place
+ # the verdict is.
+ if probes:
+ body["subscriptions"] = sorted(probes)
+
return Response(
- content='{"status":"starting","service":"' + service_name + '"}',
+ content=json.dumps(body),
media_type="application/json",
- status_code=503,
+ status_code=status_code,
)
def mark_ready() -> None:
diff --git a/services/fusion/app/_health.py b/services/fusion/app/_health.py
index 974269731..0acb9e4da 100644
--- a/services/fusion/app/_health.py
+++ b/services/fusion/app/_health.py
@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /
``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /
``mark_not_ready()`` from those handlers respectively.
+Readiness is a question, not a latch
+------------------------------------
+
+``mark_ready()`` answers "did startup finish", and for a service
+whose only job is to answer HTTP that is the whole story. For a
+service whose job is to *consume a topic* it is not: three
+services in this repository created the consumer as a
+fire-and-forget ``asyncio.create_task`` and called
+``mark_ready()`` on the next line, so readiness recorded that the
+task had been **created**, never that it was still running.
+
+UEBA is what that cost. Its handler raised
+``UndefinedColumnError`` on the first scoreable event, the
+exception left the ``async for``, the ``finally`` stopped the
+consumer, and the task ended — holding the exception, which
+nothing retrieved because the task was parked in a module global
+and so never garbage-collected. The container stayed ``running``
+with restarts 0, ``/health`` returned 200 and ``/readyz``
+returned 200, permanently. A consumer that has silently detached
+is indistinguishable from an idle one, and the whole point of a
+readiness probe is to tell those two apart.
+
+So a service that owns a subscription registers a probe for it
+with :func:`register_subscription`, and ``/readyz`` evaluates
+every probe on each request. A probe that returns false — or
+raises — is reported as detached and ``/readyz`` answers 503 with
+the names, which is a readiness failure rather than a liveness
+one on purpose: the orchestrator should stop routing to a
+half-working replica, and killing the process would only hide a
+schema or broker fault behind a restart loop.
+
+``/livez`` deliberately does **not** consult the probes. It
+answers "is the interpreter responsive", and wiring a detached
+consumer into it would make every orchestrator restart the pod on
+a fault a restart cannot fix.
+
This module is intentionally dependency-free (only FastAPI) so it
can be vendored into every service tree without import-graph
side-effects. Each service ships its own copy under
``services/<svc>/app/_health.py`` because services don't share a
-Python path — keep the file in sync via
-``scripts/sync_health_module.py`` (CI gate).
+Python path; ``scripts/audit_health_probes.py --check`` is the CI
+gate that every service has one and wires it in.
"""
from __future__ import annotations
+import json
from collections.abc import Callable
from typing import Any
from fastapi import FastAPI, Response
+#: Where the probes registered against an app are kept. Read by ``/readyz``,
+#: written by :func:`register_subscription`. On ``app.state`` rather than in a
+#: module global because a module global is shared by every app in the
+#: process, which makes two test clients interfere.
+_STATE_ATTR = "health_subscriptions"
+
+
+def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:
+ """Names of the subscriptions currently reporting not-attached.
+
+ A probe that raises counts as detached. It is asking a live object
+ whether it is still working; if that question itself fails, the honest
+ answer is not "yes".
+ """
+ detached: list[str] = []
+ for name, probe in sorted(probes.items()):
+ try:
+ attached = bool(probe())
+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one
+ attached = False
+ if not attached:
+ detached.append(name)
+ return detached
+
+
+def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:
+ """Make ``/readyz`` depend on *probe* returning true.
+
+ Call this once per subscription the service owns, after
+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —
+ it runs on every ``/readyz`` request — so it should read a flag the
+ consumer maintains or check ``task.done()``, not round-trip to a broker.
+ """
+ probes = getattr(app.state, _STATE_ATTR, None)
+ if probes is None: # pragma: no cover — install_health_routes was not called
+ raise RuntimeError("register_subscription requires install_health_routes(app) first")
+ probes[name] = probe
+
def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:
"""Install ``/livez`` and ``/readyz`` on ``app``.
@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
``mark_ready()`` is called — exactly the right behaviour for a
rolling deploy where the orchestrator should hold traffic
until the pod's dependencies are connected.
+
+ Startup having finished is necessary and not sufficient: any
+ subscription registered with :func:`register_subscription`
+ must also still be attached. See the module docstring.
"""
state: dict[str, bool] = {"ready": False}
+ probes: dict[str, Callable[[], bool]] = {}
+ setattr(app.state, _STATE_ATTR, probes)
@app.get(
"/livez",
@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
include_in_schema=False,
)
async def _readyz() -> Response:
- if state["ready"]:
- return Response(
- content='{"status":"ready","service":"' + service_name + '"}',
- media_type="application/json",
- status_code=200,
- )
+ body: dict[str, Any] = {"status": "ready", "service": service_name}
+ status_code = 200
+
+ if not state["ready"]:
+ body["status"] = "starting"
+ status_code = 503
+ else:
+ detached = _probe_detached(probes)
+ if detached:
+ body["status"] = "degraded"
+ body["detached"] = detached
+ status_code = 503
+
+ # Named subscriptions are reported either way, so a 200 says which
+ # ones were checked rather than only that nothing was wrong. A probe
+ # list that is unexpectedly empty is then visible in the same place
+ # the verdict is.
+ if probes:
+ body["subscriptions"] = sorted(probes)
+
return Response(
- content='{"status":"starting","service":"' + service_name + '"}',
+ content=json.dumps(body),
media_type="application/json",
- status_code=503,
+ status_code=status_code,
)
def mark_ready() -> None:
diff --git a/services/fusion/app/main.py b/services/fusion/app/main.py
index fd7e4ba3c..9a9164168 100644
--- a/services/fusion/app/main.py
+++ b/services/fusion/app/main.py
@@ -5,7 +5,7 @@ from contextlib import asynccontextmanager
import redis.asyncio as aioredis
from fastapi import FastAPI
-from app._health import install_health_routes
+from app._health import install_health_routes, register_subscription
from app.api.router import router, set_worker
from app.core.config import settings
from app.core.logging import configure_logging, logger
@@ -26,6 +26,23 @@ from app.services.windowed_detection import WindowedDetectionEngine
from app.workers.consumer import FusionWorker
+def _log_worker_exit(task: asyncio.Task) -> None:
+ """Say why the consume loop stopped, whatever the reason.
+
+ A worker that ends cleanly is as much of a problem as one that raises:
+ either way nothing is consuming ``raw_events`` any more, and both used to
+ be silent.
+ """
+ if task.cancelled():
+ logger.info("fusion.worker_cancelled")
+ return
+ exc = task.exception()
+ if exc is None:
+ logger.error("fusion.worker_exited", detail="consume loop returned; the subscription is gone")
+ return
+ logger.error("fusion.worker_died", error=str(exc), error_type=type(exc).__name__, exc_info=exc)
+
+
@asynccontextmanager
async def lifespan(app: FastAPI):
configure_logging()
@@ -114,8 +131,16 @@ async def lifespan(app: FastAPI):
)
set_worker(worker)
- # Start Kafka worker as a background task
+ # Start Kafka worker as a background task.
+ #
+ # The done-callback is what makes a dead worker findable. The task is held
+ # on app.state, so it is never garbage-collected and asyncio never emits
+ # its "Task exception was never retrieved" warning — without this, a
+ # consume loop that died took the reason with it. The matching readiness
+ # probe below is what stops /readyz answering 200 afterwards.
worker_task = asyncio.create_task(worker.start())
+ worker_task.add_done_callback(_log_worker_exit)
+ register_subscription(app, "alerts+raw_events", lambda: worker.attached)
app.state.worker_task = worker_task
app.state.redis = redis_client
diff --git a/services/fusion/app/workers/consumer.py b/services/fusion/app/workers/consumer.py
index 9c22423c1..dcc76a78d 100644
--- a/services/fusion/app/workers/consumer.py
+++ b/services/fusion/app/workers/consumer.py
@@ -67,12 +67,24 @@ class FusionWorker:
self._consumer: AIOKafkaConsumer | None = None
self._producer: AIOKafkaProducer | None = None
self._running = False
+ self._attached = False
self._flush_task: asyncio.Task | None = None
@property
def engine(self) -> FusionEngine:
return self._engine
+ @property
+ def attached(self) -> bool:
+ """Whether the consume loop is currently iterating the subscription.
+
+ ``_running`` is a request ("keep going"); this is an observation ("it
+ is going"). Readiness needs the second one: the worker was started as
+ a fire-and-forget task and ``/readyz`` latched to 200 next to it, so a
+ loop that exited left the service reporting ready forever.
+ """
+ return self._attached
+
async def start(self) -> None:
topics = [settings.kafka_topic_alerts_raw]
# Subscribe to raw_events when EITHER promotion or lake archival needs
@@ -155,27 +167,44 @@ class FusionWorker:
consumer = self._consumer
if consumer is None: # pragma: no cover — unreachable via start()
raise RuntimeError("fusion consume loop started before the Kafka consumer")
- async for msg in consumer:
- if not self._running:
- break
- try:
- await self._process_message(msg.value, topic=msg.topic)
- except Exception as exc:
- _METRICS["errors"] += 1
- logger.error("Failed to process message", error=str(exc), exc_info=True)
- else:
- # At-least-once: commit only after the message is handled
- # (validated + laked + promoted, or dead-lettered). A crash
- # before this line re-delivers the in-flight message on restart
- # instead of losing it. Commit failure => reprocess on restart.
+ # Attached for as long as this loop is iterating; read by the
+ # readiness probe app/main.py registers. Cleared in the ``finally`` so
+ # every exit path reports the same way, including one that raises.
+ self._attached = True
+ try:
+ async for msg in consumer:
+ if not self._running:
+ break
try:
- await consumer.commit()
- except Exception as commit_exc: # noqa: BLE001
- logger.warning("fusion.commit_failed", error=str(commit_exc))
- # Flush any stale lake batch so archival isn't stranded during a
- # low-traffic window (batch fills by size OR age).
- if self._lake is not None:
- await self._lake.flush_if_stale()
+ await self._process_message(msg.value, topic=msg.topic)
+ except Exception as exc:
+ _METRICS["errors"] += 1
+ logger.error("Failed to process message", error=str(exc), exc_info=True)
+ else:
+ # At-least-once: commit only after the message is handled
+ # (validated + laked + promoted, or dead-lettered). A crash
+ # before this line re-delivers the in-flight message on restart
+ # instead of losing it. Commit failure => reprocess on restart.
+ try:
+ await consumer.commit()
+ except Exception as commit_exc: # noqa: BLE001
+ logger.warning("fusion.commit_failed", error=str(commit_exc))
+ # Flush any stale lake batch so archival isn't stranded during a
+ # low-traffic window (batch fills by size OR age).
+ #
+ # Guarded, because this await sat outside the per-message
+ # try/except: a ClickHouse outage raising here left the
+ # ``async for`` entirely and stopped fusion consuming, with the
+ # archive fault reported as nothing at all. Archival falling
+ # behind must not cost the alert path.
+ if self._lake is not None:
+ try:
+ await self._lake.flush_if_stale()
+ except Exception as flush_exc: # noqa: BLE001 — the lake is not the alert path
+ _METRICS["errors"] += 1
+ logger.error("fusion.lake_flush_failed", error=str(flush_exc), exc_info=True)
+ finally:
+ self._attached = False
async def _dead_letter(
self,
diff --git a/services/honeytokens/Dockerfile b/services/honeytokens/Dockerfile
index fe6f67cda..8280f7cb2 100644
--- a/services/honeytokens/Dockerfile
+++ b/services/honeytokens/Dockerfile
@@ -25,4 +25,17 @@ COPY app/ app/
EXPOSE 8005
-CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8005"]
+# The chain runs on the way up, then this process is replaced by the server.
+#
+# Nothing invoked these four alembic chains. This was a bare `uvicorn` line, so
+# the documented quickstart booted the service against an empty schema with
+# none of its row-level-security policies installed, and every command in that
+# sequence reported success. `app/_migrate.py` applies the chain as the owner
+# credential, verifies it reached head, and refuses to start the server if it
+# did not.
+#
+# `sh -c` because an exec-form CMD cannot run two things. Neither shell nor
+# wrapper stays in the process tree: `sh -c` with a single command execs it,
+# and `_migrate` hands over with `os.execvp`, so uvicorn ends up as PID 1 and
+# receives SIGTERM directly rather than through a parent that would swallow it.
+CMD ["sh", "-c", "python -m app._migrate -- uvicorn app.main:app --host 0.0.0.0 --port 8005"]
diff --git a/services/honeytokens/alembic/env.py b/services/honeytokens/alembic/env.py
index b8d41012e..f27d751cf 100644
--- a/services/honeytokens/alembic/env.py
+++ b/services/honeytokens/alembic/env.py
@@ -157,9 +157,27 @@ def run_migrations_offline() -> None:
context.run_migrations()
+#: Serialises the four chains that share one database against each other.
+#:
+#: They are started simultaneously by ``docker compose up``, and while each
+#: now owns its own version table, they do not own their own *database*: the
+#: legacy-version probe below reads a table a sibling may be creating, and
+#: ``000N_runtime_role_grants`` issues ``GRANT`` statements that take
+#: catalog locks on ``pg_class``. Two chains arriving together deadlocked
+#: rather than queued.
+#:
+#: ``pg_advisory_xact_lock`` and not the session form: it is released by the
+#: commit or rollback that ends the upgrade, so a chain that dies mid-apply
+#: cannot leave the other three waiting on a lock nobody holds a connection
+#: for. The key is an arbitrary constant, shared by every chain on purpose —
+#: the point is mutual exclusion, not per-chain exclusion.
+MIGRATION_LOCK_KEY = 0x4149_5354
+
+
def do_run_migrations(connection): # type: ignore[no-untyped-def]
context.configure(connection=connection, target_metadata=target_metadata, version_table=VERSION_TABLE)
with context.begin_transaction():
+ connection.execute(sa.text("SELECT pg_advisory_xact_lock(:key)"), {"key": MIGRATION_LOCK_KEY})
# Inside ``begin_transaction()``, not before it. Alembic's
# ``begin_transaction`` is a no-op when the connection is *already* in a
# transaction, and in SQLAlchemy 2.0 any execute on a plain ``connect()``
diff --git a/services/honeytokens/app/_health.py b/services/honeytokens/app/_health.py
index 974269731..0acb9e4da 100644
--- a/services/honeytokens/app/_health.py
+++ b/services/honeytokens/app/_health.py
@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /
``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /
``mark_not_ready()`` from those handlers respectively.
+Readiness is a question, not a latch
+------------------------------------
+
+``mark_ready()`` answers "did startup finish", and for a service
+whose only job is to answer HTTP that is the whole story. For a
+service whose job is to *consume a topic* it is not: three
+services in this repository created the consumer as a
+fire-and-forget ``asyncio.create_task`` and called
+``mark_ready()`` on the next line, so readiness recorded that the
+task had been **created**, never that it was still running.
+
+UEBA is what that cost. Its handler raised
+``UndefinedColumnError`` on the first scoreable event, the
+exception left the ``async for``, the ``finally`` stopped the
+consumer, and the task ended — holding the exception, which
+nothing retrieved because the task was parked in a module global
+and so never garbage-collected. The container stayed ``running``
+with restarts 0, ``/health`` returned 200 and ``/readyz``
+returned 200, permanently. A consumer that has silently detached
+is indistinguishable from an idle one, and the whole point of a
+readiness probe is to tell those two apart.
+
+So a service that owns a subscription registers a probe for it
+with :func:`register_subscription`, and ``/readyz`` evaluates
+every probe on each request. A probe that returns false — or
+raises — is reported as detached and ``/readyz`` answers 503 with
+the names, which is a readiness failure rather than a liveness
+one on purpose: the orchestrator should stop routing to a
+half-working replica, and killing the process would only hide a
+schema or broker fault behind a restart loop.
+
+``/livez`` deliberately does **not** consult the probes. It
+answers "is the interpreter responsive", and wiring a detached
+consumer into it would make every orchestrator restart the pod on
+a fault a restart cannot fix.
+
This module is intentionally dependency-free (only FastAPI) so it
can be vendored into every service tree without import-graph
side-effects. Each service ships its own copy under
``services/<svc>/app/_health.py`` because services don't share a
-Python path — keep the file in sync via
-``scripts/sync_health_module.py`` (CI gate).
+Python path; ``scripts/audit_health_probes.py --check`` is the CI
+gate that every service has one and wires it in.
"""
from __future__ import annotations
+import json
from collections.abc import Callable
from typing import Any
from fastapi import FastAPI, Response
+#: Where the probes registered against an app are kept. Read by ``/readyz``,
+#: written by :func:`register_subscription`. On ``app.state`` rather than in a
+#: module global because a module global is shared by every app in the
+#: process, which makes two test clients interfere.
+_STATE_ATTR = "health_subscriptions"
+
+
+def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:
+ """Names of the subscriptions currently reporting not-attached.
+
+ A probe that raises counts as detached. It is asking a live object
+ whether it is still working; if that question itself fails, the honest
+ answer is not "yes".
+ """
+ detached: list[str] = []
+ for name, probe in sorted(probes.items()):
+ try:
+ attached = bool(probe())
+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one
+ attached = False
+ if not attached:
+ detached.append(name)
+ return detached
+
+
+def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:
+ """Make ``/readyz`` depend on *probe* returning true.
+
+ Call this once per subscription the service owns, after
+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —
+ it runs on every ``/readyz`` request — so it should read a flag the
+ consumer maintains or check ``task.done()``, not round-trip to a broker.
+ """
+ probes = getattr(app.state, _STATE_ATTR, None)
+ if probes is None: # pragma: no cover — install_health_routes was not called
+ raise RuntimeError("register_subscription requires install_health_routes(app) first")
+ probes[name] = probe
+
def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:
"""Install ``/livez`` and ``/readyz`` on ``app``.
@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
``mark_ready()`` is called — exactly the right behaviour for a
rolling deploy where the orchestrator should hold traffic
until the pod's dependencies are connected.
+
+ Startup having finished is necessary and not sufficient: any
+ subscription registered with :func:`register_subscription`
+ must also still be attached. See the module docstring.
"""
state: dict[str, bool] = {"ready": False}
+ probes: dict[str, Callable[[], bool]] = {}
+ setattr(app.state, _STATE_ATTR, probes)
@app.get(
"/livez",
@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
include_in_schema=False,
)
async def _readyz() -> Response:
- if state["ready"]:
- return Response(
- content='{"status":"ready","service":"' + service_name + '"}',
- media_type="application/json",
- status_code=200,
- )
+ body: dict[str, Any] = {"status": "ready", "service": service_name}
+ status_code = 200
+
+ if not state["ready"]:
+ body["status"] = "starting"
+ status_code = 503
+ else:
+ detached = _probe_detached(probes)
+ if detached:
+ body["status"] = "degraded"
+ body["detached"] = detached
+ status_code = 503
+
+ # Named subscriptions are reported either way, so a 200 says which
+ # ones were checked rather than only that nothing was wrong. A probe
+ # list that is unexpectedly empty is then visible in the same place
+ # the verdict is.
+ if probes:
+ body["subscriptions"] = sorted(probes)
+
return Response(
- content='{"status":"starting","service":"' + service_name + '"}',
+ content=json.dumps(body),
media_type="application/json",
- status_code=503,
+ status_code=status_code,
)
def mark_ready() -> None:
diff --git a/services/honeytokens/app/_migrate.py b/services/honeytokens/app/_migrate.py
new file mode 100644
index 000000000..1297f113b
--- /dev/null
+++ b/services/honeytokens/app/_migrate.py
@@ -0,0 +1,228 @@
+"""Apply this service's alembic chain, then exec what comes after it.
+
+Why this exists
+---------------
+
+Four services in this repository own their schema through alembic —
+``ueba``, ``honeytokens``, ``purple-team`` and ``osquery-tls`` — and **no
+compose path invoked any of them**. Their container command was a plain
+``uvicorn``, so on the documented quickstart they booted healthy against
+empty schemas with none of their row-level-security policies installed. The
+only documented invocation was a manual step in a docs-portal quickstart
+whose own "start the stack" step does not start three of the four services,
+and whose first line failed outright.
+
+That is how two services shipped with no tables while every command
+reported success. The API does not have this problem because its chain runs
+from its own process on the way up; this is the same idea for the four that
+use alembic.
+
+What it guarantees
+------------------
+
+**Idempotent.** ``alembic upgrade head`` on a database already at head
+applies nothing. Running this on every container start is the point: there
+is no separate "have you migrated yet" state for an operator to get wrong.
+
+**Race-free.** Four chains start at once against one database. The chains
+take a shared transaction-scoped advisory lock (``MIGRATION_LOCK_KEY`` in
+each ``env.py``), so they queue instead of deadlocking on catalog locks.
+This wrapper deliberately holds no lock of its own — a lock held here would
+be a second mechanism to reason about, and it would not cover the window
+that actually matters.
+
+**Loud.** The service does not start unless the chain reached head. Not "the
+command exited 0": alembic exits 0 when it applies *nothing*, which is
+exactly what happened when four chains shared one ``alembic_version`` table
+and the second one to run believed it was already done. So ``current`` is
+read back and compared to ``heads``, and a mismatch — or an empty
+``current``, meaning the chain never ran — stops the container with the
+reason on stderr. A service booting on an empty schema is the failure this
+replaces, and it must not be reachable by a wrapper that only checks an exit
+code.
+
+Why the owner credential, and why a subprocess
+----------------------------------------------
+
+Alembic issues DDL. The runtime role every service serves requests as holds
+no ``CREATE`` on schema public, by design — that is what stops it turning
+row-level security off — so the chain must be applied as the owner. Each
+``env.py`` already resolves that (its own ``*_DATABASE_MIGRATION_URL``
+first, then ``DATABASE_MIGRATION_URL``), and this runs alembic as a
+subprocess precisely so ``env.py`` stays the single answer to "which
+credential". A second resolution here would be a second thing to keep in
+step, and the two would disagree the first time either changed.
+
+This module is vendored per service because services do not share a Python
+path. ``scripts/check_service_migration_bootstrap.py`` is the gate that each
+of the four ships a copy, keeps them identical, and runs one on the way up.
+
+Usage
+-----
+
+::
+
+ python -m app._migrate # apply the chain, exit
+ python -m app._migrate -- uvicorn app.main:app --port 8004
+"""
+
+from __future__ import annotations
+
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+#: Where ``alembic.ini`` sits relative to the working directory in every one
+#: of the four images (``WORKDIR /app``, and the ini is copied to ``/app``).
+ALEMBIC_INI = "alembic.ini"
+
+#: Suffix that marks a DSN as the migration (owner) credential. Matches the
+#: convention each ``env.py`` documents and ``scripts/check_runtime_db_role.py``
+#: enforces on the deployment surfaces: a service-prefixed
+#: ``<SVC>_DATABASE_MIGRATION_URL`` wins over the bare form.
+MIGRATION_URL_SUFFIX = "DATABASE_MIGRATION_URL"
+
+
+def migration_credential_present(env: dict[str, str] | None = None) -> bool:
+ """Whether an owner DSN is configured.
+
+ Used only for the warning below. The chain is applied either way — a
+ deployment that has not split the roles yet behaves as it did before —
+ but it is announced, because a chain silently applied as the runtime
+ credential fails on the first ``CREATE TABLE`` and the message names a
+ permission rather than a configuration.
+ """
+ source = os.environ if env is None else env
+ return any(name.endswith(MIGRATION_URL_SUFFIX) and value.strip() for name, value in source.items())
+
+
+def _alembic(args: list[str], *, capture: bool) -> subprocess.CompletedProcess[str]:
+ return subprocess.run( # noqa: S603 — fixed argv, no shell
+ [sys.executable, "-m", "alembic", "-c", ALEMBIC_INI, *args],
+ capture_output=capture,
+ text=True,
+ check=False,
+ )
+
+
+def _revisions(output: str) -> set[str]:
+ """Revision ids in ``alembic current`` / ``alembic heads`` output.
+
+ Both print ``<rev> (head)`` or ``<rev>``, one per line, with informational
+ lines alembic sends to stderr. An empty set means the version table holds
+ nothing — the chain has not run — which is the case this exists to catch
+ and is therefore never treated as agreement.
+ """
+ found: set[str] = set()
+ for line in output.splitlines():
+ token = line.strip().split(" ")[0].strip()
+ if token and not token.startswith(("INFO", "WARNING", "ERROR", "alembic:")):
+ found.add(token)
+ return found
+
+
+def apply_chain() -> int:
+ """Apply the chain and verify it reached head. Returns an exit code."""
+ if not Path(ALEMBIC_INI).is_file():
+ print(
+ f"migrate: {ALEMBIC_INI} not found in {Path.cwd()}. This service's image must "
+ f"copy its alembic config and chain; see its Dockerfile.",
+ file=sys.stderr,
+ )
+ return 1
+
+ if not migration_credential_present():
+ print(
+ f"migrate: no *{MIGRATION_URL_SUFFIX} is set, so the chain will be applied as "
+ f"the runtime credential. That works only while the two are the same role; "
+ f"under the DML-only runtime role the first CREATE TABLE fails with "
+ f"'permission denied for schema public'.",
+ file=sys.stderr,
+ )
+
+ upgrade = _alembic(["upgrade", "head"], capture=False)
+ if upgrade.returncode != 0:
+ print(
+ f"migrate: 'alembic upgrade head' failed (exit {upgrade.returncode}). Refusing to "
+ f"start: this service would serve requests against whatever schema is there now.",
+ file=sys.stderr,
+ )
+ return upgrade.returncode
+
+ # Read back rather than trusting the exit code. `alembic upgrade head`
+ # exits 0 having applied nothing when it believes the database is already
+ # at head, and believing that wrongly is the documented way these four
+ # services ended up with no tables.
+ current = _alembic(["current"], capture=True)
+ heads = _alembic(["heads"], capture=True)
+ if current.returncode != 0 or heads.returncode != 0:
+ print(
+ "migrate: could not read back the applied revision "
+ f"(current exit {current.returncode}, heads exit {heads.returncode}). "
+ "Refusing to start: an unverified migration is the state this check exists "
+ "to reject.\n"
+ f"{(current.stderr or '') + (heads.stderr or '')}",
+ file=sys.stderr,
+ )
+ return 1
+
+ applied = _revisions(current.stdout)
+ expected = _revisions(heads.stdout)
+
+ if not expected:
+ print(
+ "migrate: this chain declares no head revision, so there is nothing to verify "
+ "against. Refusing to start rather than reporting an empty chain as applied.",
+ file=sys.stderr,
+ )
+ return 1
+
+ if not applied:
+ print(
+ f"migrate: 'alembic upgrade head' reported success but the version table is "
+ f"empty — the chain never ran. Expected {sorted(expected)}. Refusing to start "
+ f"on an unmigrated schema.",
+ file=sys.stderr,
+ )
+ return 1
+
+ if not expected.issubset(applied):
+ print(
+ f"migrate: chain did not reach head. At {sorted(applied)}, head is " f"{sorted(expected)}. Refusing to start.",
+ file=sys.stderr,
+ )
+ return 1
+
+ print(f"migrate: schema at head {sorted(expected)}", file=sys.stderr)
+ return 0
+
+
+def main(argv: list[str] | None = None) -> int:
+ args = list(sys.argv[1:] if argv is None else argv)
+
+ # Everything after `--` is the real service command. Exec'd rather than
+ # spawned so the server keeps PID 1 and receives SIGTERM directly — a
+ # Python parent waiting on a child swallows the orchestrator's stop
+ # signal and turns every shutdown into a ten-second kill.
+ command: list[str] = []
+ if "--" in args:
+ split = args.index("--")
+ command = args[split + 1 :]
+ args = args[:split]
+
+ if args:
+ print(f"migrate: unexpected arguments {args}; usage: python -m app._migrate [-- CMD ...]", file=sys.stderr)
+ return 2
+
+ code = apply_chain()
+ if code != 0:
+ return code
+
+ if command:
+ os.execvp(command[0], command) # noqa: S606 — replaces this process by design
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/services/mesh/app/_health.py b/services/mesh/app/_health.py
index 974269731..0acb9e4da 100644
--- a/services/mesh/app/_health.py
+++ b/services/mesh/app/_health.py
@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /
``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /
``mark_not_ready()`` from those handlers respectively.
+Readiness is a question, not a latch
+------------------------------------
+
+``mark_ready()`` answers "did startup finish", and for a service
+whose only job is to answer HTTP that is the whole story. For a
+service whose job is to *consume a topic* it is not: three
+services in this repository created the consumer as a
+fire-and-forget ``asyncio.create_task`` and called
+``mark_ready()`` on the next line, so readiness recorded that the
+task had been **created**, never that it was still running.
+
+UEBA is what that cost. Its handler raised
+``UndefinedColumnError`` on the first scoreable event, the
+exception left the ``async for``, the ``finally`` stopped the
+consumer, and the task ended — holding the exception, which
+nothing retrieved because the task was parked in a module global
+and so never garbage-collected. The container stayed ``running``
+with restarts 0, ``/health`` returned 200 and ``/readyz``
+returned 200, permanently. A consumer that has silently detached
+is indistinguishable from an idle one, and the whole point of a
+readiness probe is to tell those two apart.
+
+So a service that owns a subscription registers a probe for it
+with :func:`register_subscription`, and ``/readyz`` evaluates
+every probe on each request. A probe that returns false — or
+raises — is reported as detached and ``/readyz`` answers 503 with
+the names, which is a readiness failure rather than a liveness
+one on purpose: the orchestrator should stop routing to a
+half-working replica, and killing the process would only hide a
+schema or broker fault behind a restart loop.
+
+``/livez`` deliberately does **not** consult the probes. It
+answers "is the interpreter responsive", and wiring a detached
+consumer into it would make every orchestrator restart the pod on
+a fault a restart cannot fix.
+
This module is intentionally dependency-free (only FastAPI) so it
can be vendored into every service tree without import-graph
side-effects. Each service ships its own copy under
``services/<svc>/app/_health.py`` because services don't share a
-Python path — keep the file in sync via
-``scripts/sync_health_module.py`` (CI gate).
+Python path; ``scripts/audit_health_probes.py --check`` is the CI
+gate that every service has one and wires it in.
"""
from __future__ import annotations
+import json
from collections.abc import Callable
from typing import Any
from fastapi import FastAPI, Response
+#: Where the probes registered against an app are kept. Read by ``/readyz``,
+#: written by :func:`register_subscription`. On ``app.state`` rather than in a
+#: module global because a module global is shared by every app in the
+#: process, which makes two test clients interfere.
+_STATE_ATTR = "health_subscriptions"
+
+
+def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:
+ """Names of the subscriptions currently reporting not-attached.
+
+ A probe that raises counts as detached. It is asking a live object
+ whether it is still working; if that question itself fails, the honest
+ answer is not "yes".
+ """
+ detached: list[str] = []
+ for name, probe in sorted(probes.items()):
+ try:
+ attached = bool(probe())
+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one
+ attached = False
+ if not attached:
+ detached.append(name)
+ return detached
+
+
+def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:
+ """Make ``/readyz`` depend on *probe* returning true.
+
+ Call this once per subscription the service owns, after
+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —
+ it runs on every ``/readyz`` request — so it should read a flag the
+ consumer maintains or check ``task.done()``, not round-trip to a broker.
+ """
+ probes = getattr(app.state, _STATE_ATTR, None)
+ if probes is None: # pragma: no cover — install_health_routes was not called
+ raise RuntimeError("register_subscription requires install_health_routes(app) first")
+ probes[name] = probe
+
def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:
"""Install ``/livez`` and ``/readyz`` on ``app``.
@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
``mark_ready()`` is called — exactly the right behaviour for a
rolling deploy where the orchestrator should hold traffic
until the pod's dependencies are connected.
+
+ Startup having finished is necessary and not sufficient: any
+ subscription registered with :func:`register_subscription`
+ must also still be attached. See the module docstring.
"""
state: dict[str, bool] = {"ready": False}
+ probes: dict[str, Callable[[], bool]] = {}
+ setattr(app.state, _STATE_ATTR, probes)
@app.get(
"/livez",
@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
include_in_schema=False,
)
async def _readyz() -> Response:
- if state["ready"]:
- return Response(
- content='{"status":"ready","service":"' + service_name + '"}',
- media_type="application/json",
- status_code=200,
- )
+ body: dict[str, Any] = {"status": "ready", "service": service_name}
+ status_code = 200
+
+ if not state["ready"]:
+ body["status"] = "starting"
+ status_code = 503
+ else:
+ detached = _probe_detached(probes)
+ if detached:
+ body["status"] = "degraded"
+ body["detached"] = detached
+ status_code = 503
+
+ # Named subscriptions are reported either way, so a 200 says which
+ # ones were checked rather than only that nothing was wrong. A probe
+ # list that is unexpectedly empty is then visible in the same place
+ # the verdict is.
+ if probes:
+ body["subscriptions"] = sorted(probes)
+
return Response(
- content='{"status":"starting","service":"' + service_name + '"}',
+ content=json.dumps(body),
media_type="application/json",
- status_code=503,
+ status_code=status_code,
)
def mark_ready() -> None:
diff --git a/services/osquery-tls/Dockerfile b/services/osquery-tls/Dockerfile
index 24191e267..bedcf7846 100644
--- a/services/osquery-tls/Dockerfile
+++ b/services/osquery-tls/Dockerfile
@@ -35,4 +35,17 @@ COPY . .
EXPOSE 9001
-CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "9001"]
+# The chain runs on the way up, then this process is replaced by the server.
+#
+# Nothing invoked these four alembic chains. This was a bare `uvicorn` line, so
+# the documented quickstart booted the service against an empty schema with
+# none of its row-level-security policies installed, and every command in that
+# sequence reported success. `app/_migrate.py` applies the chain as the owner
+# credential, verifies it reached head, and refuses to start the server if it
+# did not.
+#
+# `sh -c` because an exec-form CMD cannot run two things. Neither shell nor
+# wrapper stays in the process tree: `sh -c` with a single command execs it,
+# and `_migrate` hands over with `os.execvp`, so uvicorn ends up as PID 1 and
+# receives SIGTERM directly rather than through a parent that would swallow it.
+CMD ["sh", "-c", "python -m app._migrate -- uvicorn app.main:app --host 0.0.0.0 --port 9001"]
diff --git a/services/osquery-tls/app/_health.py b/services/osquery-tls/app/_health.py
index 974269731..0acb9e4da 100644
--- a/services/osquery-tls/app/_health.py
+++ b/services/osquery-tls/app/_health.py
@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /
``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /
``mark_not_ready()`` from those handlers respectively.
+Readiness is a question, not a latch
+------------------------------------
+
+``mark_ready()`` answers "did startup finish", and for a service
+whose only job is to answer HTTP that is the whole story. For a
+service whose job is to *consume a topic* it is not: three
+services in this repository created the consumer as a
+fire-and-forget ``asyncio.create_task`` and called
+``mark_ready()`` on the next line, so readiness recorded that the
+task had been **created**, never that it was still running.
+
+UEBA is what that cost. Its handler raised
+``UndefinedColumnError`` on the first scoreable event, the
+exception left the ``async for``, the ``finally`` stopped the
+consumer, and the task ended — holding the exception, which
+nothing retrieved because the task was parked in a module global
+and so never garbage-collected. The container stayed ``running``
+with restarts 0, ``/health`` returned 200 and ``/readyz``
+returned 200, permanently. A consumer that has silently detached
+is indistinguishable from an idle one, and the whole point of a
+readiness probe is to tell those two apart.
+
+So a service that owns a subscription registers a probe for it
+with :func:`register_subscription`, and ``/readyz`` evaluates
+every probe on each request. A probe that returns false — or
+raises — is reported as detached and ``/readyz`` answers 503 with
+the names, which is a readiness failure rather than a liveness
+one on purpose: the orchestrator should stop routing to a
+half-working replica, and killing the process would only hide a
+schema or broker fault behind a restart loop.
+
+``/livez`` deliberately does **not** consult the probes. It
+answers "is the interpreter responsive", and wiring a detached
+consumer into it would make every orchestrator restart the pod on
+a fault a restart cannot fix.
+
This module is intentionally dependency-free (only FastAPI) so it
can be vendored into every service tree without import-graph
side-effects. Each service ships its own copy under
``services/<svc>/app/_health.py`` because services don't share a
-Python path — keep the file in sync via
-``scripts/sync_health_module.py`` (CI gate).
+Python path; ``scripts/audit_health_probes.py --check`` is the CI
+gate that every service has one and wires it in.
"""
from __future__ import annotations
+import json
from collections.abc import Callable
from typing import Any
from fastapi import FastAPI, Response
+#: Where the probes registered against an app are kept. Read by ``/readyz``,
+#: written by :func:`register_subscription`. On ``app.state`` rather than in a
+#: module global because a module global is shared by every app in the
+#: process, which makes two test clients interfere.
+_STATE_ATTR = "health_subscriptions"
+
+
+def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:
+ """Names of the subscriptions currently reporting not-attached.
+
+ A probe that raises counts as detached. It is asking a live object
+ whether it is still working; if that question itself fails, the honest
+ answer is not "yes".
+ """
+ detached: list[str] = []
+ for name, probe in sorted(probes.items()):
+ try:
+ attached = bool(probe())
+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one
+ attached = False
+ if not attached:
+ detached.append(name)
+ return detached
+
+
+def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:
+ """Make ``/readyz`` depend on *probe* returning true.
+
+ Call this once per subscription the service owns, after
+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —
+ it runs on every ``/readyz`` request — so it should read a flag the
+ consumer maintains or check ``task.done()``, not round-trip to a broker.
+ """
+ probes = getattr(app.state, _STATE_ATTR, None)
+ if probes is None: # pragma: no cover — install_health_routes was not called
+ raise RuntimeError("register_subscription requires install_health_routes(app) first")
+ probes[name] = probe
+
def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:
"""Install ``/livez`` and ``/readyz`` on ``app``.
@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
``mark_ready()`` is called — exactly the right behaviour for a
rolling deploy where the orchestrator should hold traffic
until the pod's dependencies are connected.
+
+ Startup having finished is necessary and not sufficient: any
+ subscription registered with :func:`register_subscription`
+ must also still be attached. See the module docstring.
"""
state: dict[str, bool] = {"ready": False}
+ probes: dict[str, Callable[[], bool]] = {}
+ setattr(app.state, _STATE_ATTR, probes)
@app.get(
"/livez",
@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
include_in_schema=False,
)
async def _readyz() -> Response:
- if state["ready"]:
- return Response(
- content='{"status":"ready","service":"' + service_name + '"}',
- media_type="application/json",
- status_code=200,
- )
+ body: dict[str, Any] = {"status": "ready", "service": service_name}
+ status_code = 200
+
+ if not state["ready"]:
+ body["status"] = "starting"
+ status_code = 503
+ else:
+ detached = _probe_detached(probes)
+ if detached:
+ body["status"] = "degraded"
+ body["detached"] = detached
+ status_code = 503
+
+ # Named subscriptions are reported either way, so a 200 says which
+ # ones were checked rather than only that nothing was wrong. A probe
+ # list that is unexpectedly empty is then visible in the same place
+ # the verdict is.
+ if probes:
+ body["subscriptions"] = sorted(probes)
+
return Response(
- content='{"status":"starting","service":"' + service_name + '"}',
+ content=json.dumps(body),
media_type="application/json",
- status_code=503,
+ status_code=status_code,
)
def mark_ready() -> None:
diff --git a/services/osquery-tls/app/_migrate.py b/services/osquery-tls/app/_migrate.py
new file mode 100644
index 000000000..1297f113b
--- /dev/null
+++ b/services/osquery-tls/app/_migrate.py
@@ -0,0 +1,228 @@
+"""Apply this service's alembic chain, then exec what comes after it.
+
+Why this exists
+---------------
+
+Four services in this repository own their schema through alembic —
+``ueba``, ``honeytokens``, ``purple-team`` and ``osquery-tls`` — and **no
+compose path invoked any of them**. Their container command was a plain
+``uvicorn``, so on the documented quickstart they booted healthy against
+empty schemas with none of their row-level-security policies installed. The
+only documented invocation was a manual step in a docs-portal quickstart
+whose own "start the stack" step does not start three of the four services,
+and whose first line failed outright.
+
+That is how two services shipped with no tables while every command
+reported success. The API does not have this problem because its chain runs
+from its own process on the way up; this is the same idea for the four that
+use alembic.
+
+What it guarantees
+------------------
+
+**Idempotent.** ``alembic upgrade head`` on a database already at head
+applies nothing. Running this on every container start is the point: there
+is no separate "have you migrated yet" state for an operator to get wrong.
+
+**Race-free.** Four chains start at once against one database. The chains
+take a shared transaction-scoped advisory lock (``MIGRATION_LOCK_KEY`` in
+each ``env.py``), so they queue instead of deadlocking on catalog locks.
+This wrapper deliberately holds no lock of its own — a lock held here would
+be a second mechanism to reason about, and it would not cover the window
+that actually matters.
+
+**Loud.** The service does not start unless the chain reached head. Not "the
+command exited 0": alembic exits 0 when it applies *nothing*, which is
+exactly what happened when four chains shared one ``alembic_version`` table
+and the second one to run believed it was already done. So ``current`` is
+read back and compared to ``heads``, and a mismatch — or an empty
+``current``, meaning the chain never ran — stops the container with the
+reason on stderr. A service booting on an empty schema is the failure this
+replaces, and it must not be reachable by a wrapper that only checks an exit
+code.
+
+Why the owner credential, and why a subprocess
+----------------------------------------------
+
+Alembic issues DDL. The runtime role every service serves requests as holds
+no ``CREATE`` on schema public, by design — that is what stops it turning
+row-level security off — so the chain must be applied as the owner. Each
+``env.py`` already resolves that (its own ``*_DATABASE_MIGRATION_URL``
+first, then ``DATABASE_MIGRATION_URL``), and this runs alembic as a
+subprocess precisely so ``env.py`` stays the single answer to "which
+credential". A second resolution here would be a second thing to keep in
+step, and the two would disagree the first time either changed.
+
+This module is vendored per service because services do not share a Python
+path. ``scripts/check_service_migration_bootstrap.py`` is the gate that each
+of the four ships a copy, keeps them identical, and runs one on the way up.
+
+Usage
+-----
+
+::
+
+ python -m app._migrate # apply the chain, exit
+ python -m app._migrate -- uvicorn app.main:app --port 8004
+"""
+
+from __future__ import annotations
+
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+#: Where ``alembic.ini`` sits relative to the working directory in every one
+#: of the four images (``WORKDIR /app``, and the ini is copied to ``/app``).
+ALEMBIC_INI = "alembic.ini"
+
+#: Suffix that marks a DSN as the migration (owner) credential. Matches the
+#: convention each ``env.py`` documents and ``scripts/check_runtime_db_role.py``
+#: enforces on the deployment surfaces: a service-prefixed
+#: ``<SVC>_DATABASE_MIGRATION_URL`` wins over the bare form.
+MIGRATION_URL_SUFFIX = "DATABASE_MIGRATION_URL"
+
+
+def migration_credential_present(env: dict[str, str] | None = None) -> bool:
+ """Whether an owner DSN is configured.
+
+ Used only for the warning below. The chain is applied either way — a
+ deployment that has not split the roles yet behaves as it did before —
+ but it is announced, because a chain silently applied as the runtime
+ credential fails on the first ``CREATE TABLE`` and the message names a
+ permission rather than a configuration.
+ """
+ source = os.environ if env is None else env
+ return any(name.endswith(MIGRATION_URL_SUFFIX) and value.strip() for name, value in source.items())
+
+
+def _alembic(args: list[str], *, capture: bool) -> subprocess.CompletedProcess[str]:
+ return subprocess.run( # noqa: S603 — fixed argv, no shell
+ [sys.executable, "-m", "alembic", "-c", ALEMBIC_INI, *args],
+ capture_output=capture,
+ text=True,
+ check=False,
+ )
+
+
+def _revisions(output: str) -> set[str]:
+ """Revision ids in ``alembic current`` / ``alembic heads`` output.
+
+ Both print ``<rev> (head)`` or ``<rev>``, one per line, with informational
+ lines alembic sends to stderr. An empty set means the version table holds
+ nothing — the chain has not run — which is the case this exists to catch
+ and is therefore never treated as agreement.
+ """
+ found: set[str] = set()
+ for line in output.splitlines():
+ token = line.strip().split(" ")[0].strip()
+ if token and not token.startswith(("INFO", "WARNING", "ERROR", "alembic:")):
+ found.add(token)
+ return found
+
+
+def apply_chain() -> int:
+ """Apply the chain and verify it reached head. Returns an exit code."""
+ if not Path(ALEMBIC_INI).is_file():
+ print(
+ f"migrate: {ALEMBIC_INI} not found in {Path.cwd()}. This service's image must "
+ f"copy its alembic config and chain; see its Dockerfile.",
+ file=sys.stderr,
+ )
+ return 1
+
+ if not migration_credential_present():
+ print(
+ f"migrate: no *{MIGRATION_URL_SUFFIX} is set, so the chain will be applied as "
+ f"the runtime credential. That works only while the two are the same role; "
+ f"under the DML-only runtime role the first CREATE TABLE fails with "
+ f"'permission denied for schema public'.",
+ file=sys.stderr,
+ )
+
+ upgrade = _alembic(["upgrade", "head"], capture=False)
+ if upgrade.returncode != 0:
+ print(
+ f"migrate: 'alembic upgrade head' failed (exit {upgrade.returncode}). Refusing to "
+ f"start: this service would serve requests against whatever schema is there now.",
+ file=sys.stderr,
+ )
+ return upgrade.returncode
+
+ # Read back rather than trusting the exit code. `alembic upgrade head`
+ # exits 0 having applied nothing when it believes the database is already
+ # at head, and believing that wrongly is the documented way these four
+ # services ended up with no tables.
+ current = _alembic(["current"], capture=True)
+ heads = _alembic(["heads"], capture=True)
+ if current.returncode != 0 or heads.returncode != 0:
+ print(
+ "migrate: could not read back the applied revision "
+ f"(current exit {current.returncode}, heads exit {heads.returncode}). "
+ "Refusing to start: an unverified migration is the state this check exists "
+ "to reject.\n"
+ f"{(current.stderr or '') + (heads.stderr or '')}",
+ file=sys.stderr,
+ )
+ return 1
+
+ applied = _revisions(current.stdout)
+ expected = _revisions(heads.stdout)
+
+ if not expected:
+ print(
+ "migrate: this chain declares no head revision, so there is nothing to verify "
+ "against. Refusing to start rather than reporting an empty chain as applied.",
+ file=sys.stderr,
+ )
+ return 1
+
+ if not applied:
+ print(
+ f"migrate: 'alembic upgrade head' reported success but the version table is "
+ f"empty — the chain never ran. Expected {sorted(expected)}. Refusing to start "
+ f"on an unmigrated schema.",
+ file=sys.stderr,
+ )
+ return 1
+
+ if not expected.issubset(applied):
+ print(
+ f"migrate: chain did not reach head. At {sorted(applied)}, head is " f"{sorted(expected)}. Refusing to start.",
+ file=sys.stderr,
+ )
+ return 1
+
+ print(f"migrate: schema at head {sorted(expected)}", file=sys.stderr)
+ return 0
+
+
+def main(argv: list[str] | None = None) -> int:
+ args = list(sys.argv[1:] if argv is None else argv)
+
+ # Everything after `--` is the real service command. Exec'd rather than
+ # spawned so the server keeps PID 1 and receives SIGTERM directly — a
+ # Python parent waiting on a child swallows the orchestrator's stop
+ # signal and turns every shutdown into a ten-second kill.
+ command: list[str] = []
+ if "--" in args:
+ split = args.index("--")
+ command = args[split + 1 :]
+ args = args[:split]
+
+ if args:
+ print(f"migrate: unexpected arguments {args}; usage: python -m app._migrate [-- CMD ...]", file=sys.stderr)
+ return 2
+
+ code = apply_chain()
+ if code != 0:
+ return code
+
+ if command:
+ os.execvp(command[0], command) # noqa: S606 — replaces this process by design
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/services/osquery-tls/app/db/env.py b/services/osquery-tls/app/db/env.py
index 32b2a400c..fbaab4e57 100644
--- a/services/osquery-tls/app/db/env.py
+++ b/services/osquery-tls/app/db/env.py
@@ -154,9 +154,27 @@ def run_migrations_offline() -> None:
context.run_migrations()
+#: Serialises the four chains that share one database against each other.
+#:
+#: They are started simultaneously by ``docker compose up``, and while each
+#: now owns its own version table, they do not own their own *database*: the
+#: legacy-version probe below reads a table a sibling may be creating, and
+#: ``000N_runtime_role_grants`` issues ``GRANT`` statements that take
+#: catalog locks on ``pg_class``. Two chains arriving together deadlocked
+#: rather than queued.
+#:
+#: ``pg_advisory_xact_lock`` and not the session form: it is released by the
+#: commit or rollback that ends the upgrade, so a chain that dies mid-apply
+#: cannot leave the other three waiting on a lock nobody holds a connection
+#: for. The key is an arbitrary constant, shared by every chain on purpose —
+#: the point is mutual exclusion, not per-chain exclusion.
+MIGRATION_LOCK_KEY = 0x4149_5354
+
+
def do_run_migrations(connection):
context.configure(connection=connection, target_metadata=target_metadata, version_table=VERSION_TABLE)
with context.begin_transaction():
+ connection.execute(sa.text("SELECT pg_advisory_xact_lock(:key)"), {"key": MIGRATION_LOCK_KEY})
# Inside ``begin_transaction()``, not before it. Alembic's
# ``begin_transaction`` is a no-op when the connection is *already* in a
# transaction, and in SQLAlchemy 2.0 any execute on a plain ``connect()``
diff --git a/services/purple-team/Dockerfile b/services/purple-team/Dockerfile
index 023b60fae..1adb44ab0 100644
--- a/services/purple-team/Dockerfile
+++ b/services/purple-team/Dockerfile
@@ -25,4 +25,17 @@ COPY app/ app/
EXPOSE 8006
-CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8006"]
+# The chain runs on the way up, then this process is replaced by the server.
+#
+# Nothing invoked these four alembic chains. This was a bare `uvicorn` line, so
+# the documented quickstart booted the service against an empty schema with
+# none of its row-level-security policies installed, and every command in that
+# sequence reported success. `app/_migrate.py` applies the chain as the owner
+# credential, verifies it reached head, and refuses to start the server if it
+# did not.
+#
+# `sh -c` because an exec-form CMD cannot run two things. Neither shell nor
+# wrapper stays in the process tree: `sh -c` with a single command execs it,
+# and `_migrate` hands over with `os.execvp`, so uvicorn ends up as PID 1 and
+# receives SIGTERM directly rather than through a parent that would swallow it.
+CMD ["sh", "-c", "python -m app._migrate -- uvicorn app.main:app --host 0.0.0.0 --port 8006"]
diff --git a/services/purple-team/alembic/env.py b/services/purple-team/alembic/env.py
index be505bc8e..0e1a5452b 100644
--- a/services/purple-team/alembic/env.py
+++ b/services/purple-team/alembic/env.py
@@ -157,9 +157,27 @@ def run_migrations_offline() -> None:
context.run_migrations()
+#: Serialises the four chains that share one database against each other.
+#:
+#: They are started simultaneously by ``docker compose up``, and while each
+#: now owns its own version table, they do not own their own *database*: the
+#: legacy-version probe below reads a table a sibling may be creating, and
+#: ``000N_runtime_role_grants`` issues ``GRANT`` statements that take
+#: catalog locks on ``pg_class``. Two chains arriving together deadlocked
+#: rather than queued.
+#:
+#: ``pg_advisory_xact_lock`` and not the session form: it is released by the
+#: commit or rollback that ends the upgrade, so a chain that dies mid-apply
+#: cannot leave the other three waiting on a lock nobody holds a connection
+#: for. The key is an arbitrary constant, shared by every chain on purpose —
+#: the point is mutual exclusion, not per-chain exclusion.
+MIGRATION_LOCK_KEY = 0x4149_5354
+
+
def do_run_migrations(connection): # type: ignore[no-untyped-def]
context.configure(connection=connection, target_metadata=target_metadata, version_table=VERSION_TABLE)
with context.begin_transaction():
+ connection.execute(sa.text("SELECT pg_advisory_xact_lock(:key)"), {"key": MIGRATION_LOCK_KEY})
# Inside ``begin_transaction()``, not before it. Alembic's
# ``begin_transaction`` is a no-op when the connection is *already* in a
# transaction, and in SQLAlchemy 2.0 any execute on a plain ``connect()``
diff --git a/services/purple-team/app/_health.py b/services/purple-team/app/_health.py
index 974269731..0acb9e4da 100644
--- a/services/purple-team/app/_health.py
+++ b/services/purple-team/app/_health.py
@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /
``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /
``mark_not_ready()`` from those handlers respectively.
+Readiness is a question, not a latch
+------------------------------------
+
+``mark_ready()`` answers "did startup finish", and for a service
+whose only job is to answer HTTP that is the whole story. For a
+service whose job is to *consume a topic* it is not: three
+services in this repository created the consumer as a
+fire-and-forget ``asyncio.create_task`` and called
+``mark_ready()`` on the next line, so readiness recorded that the
+task had been **created**, never that it was still running.
+
+UEBA is what that cost. Its handler raised
+``UndefinedColumnError`` on the first scoreable event, the
+exception left the ``async for``, the ``finally`` stopped the
+consumer, and the task ended — holding the exception, which
+nothing retrieved because the task was parked in a module global
+and so never garbage-collected. The container stayed ``running``
+with restarts 0, ``/health`` returned 200 and ``/readyz``
+returned 200, permanently. A consumer that has silently detached
+is indistinguishable from an idle one, and the whole point of a
+readiness probe is to tell those two apart.
+
+So a service that owns a subscription registers a probe for it
+with :func:`register_subscription`, and ``/readyz`` evaluates
+every probe on each request. A probe that returns false — or
+raises — is reported as detached and ``/readyz`` answers 503 with
+the names, which is a readiness failure rather than a liveness
+one on purpose: the orchestrator should stop routing to a
+half-working replica, and killing the process would only hide a
+schema or broker fault behind a restart loop.
+
+``/livez`` deliberately does **not** consult the probes. It
+answers "is the interpreter responsive", and wiring a detached
+consumer into it would make every orchestrator restart the pod on
+a fault a restart cannot fix.
+
This module is intentionally dependency-free (only FastAPI) so it
can be vendored into every service tree without import-graph
side-effects. Each service ships its own copy under
``services/<svc>/app/_health.py`` because services don't share a
-Python path — keep the file in sync via
-``scripts/sync_health_module.py`` (CI gate).
+Python path; ``scripts/audit_health_probes.py --check`` is the CI
+gate that every service has one and wires it in.
"""
from __future__ import annotations
+import json
from collections.abc import Callable
from typing import Any
from fastapi import FastAPI, Response
+#: Where the probes registered against an app are kept. Read by ``/readyz``,
+#: written by :func:`register_subscription`. On ``app.state`` rather than in a
+#: module global because a module global is shared by every app in the
+#: process, which makes two test clients interfere.
+_STATE_ATTR = "health_subscriptions"
+
+
+def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:
+ """Names of the subscriptions currently reporting not-attached.
+
+ A probe that raises counts as detached. It is asking a live object
+ whether it is still working; if that question itself fails, the honest
+ answer is not "yes".
+ """
+ detached: list[str] = []
+ for name, probe in sorted(probes.items()):
+ try:
+ attached = bool(probe())
+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one
+ attached = False
+ if not attached:
+ detached.append(name)
+ return detached
+
+
+def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:
+ """Make ``/readyz`` depend on *probe* returning true.
+
+ Call this once per subscription the service owns, after
+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —
+ it runs on every ``/readyz`` request — so it should read a flag the
+ consumer maintains or check ``task.done()``, not round-trip to a broker.
+ """
+ probes = getattr(app.state, _STATE_ATTR, None)
+ if probes is None: # pragma: no cover — install_health_routes was not called
+ raise RuntimeError("register_subscription requires install_health_routes(app) first")
+ probes[name] = probe
+
def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:
"""Install ``/livez`` and ``/readyz`` on ``app``.
@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
``mark_ready()`` is called — exactly the right behaviour for a
rolling deploy where the orchestrator should hold traffic
until the pod's dependencies are connected.
+
+ Startup having finished is necessary and not sufficient: any
+ subscription registered with :func:`register_subscription`
+ must also still be attached. See the module docstring.
"""
state: dict[str, bool] = {"ready": False}
+ probes: dict[str, Callable[[], bool]] = {}
+ setattr(app.state, _STATE_ATTR, probes)
@app.get(
"/livez",
@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
include_in_schema=False,
)
async def _readyz() -> Response:
- if state["ready"]:
- return Response(
- content='{"status":"ready","service":"' + service_name + '"}',
- media_type="application/json",
- status_code=200,
- )
+ body: dict[str, Any] = {"status": "ready", "service": service_name}
+ status_code = 200
+
+ if not state["ready"]:
+ body["status"] = "starting"
+ status_code = 503
+ else:
+ detached = _probe_detached(probes)
+ if detached:
+ body["status"] = "degraded"
+ body["detached"] = detached
+ status_code = 503
+
+ # Named subscriptions are reported either way, so a 200 says which
+ # ones were checked rather than only that nothing was wrong. A probe
+ # list that is unexpectedly empty is then visible in the same place
+ # the verdict is.
+ if probes:
+ body["subscriptions"] = sorted(probes)
+
return Response(
- content='{"status":"starting","service":"' + service_name + '"}',
+ content=json.dumps(body),
media_type="application/json",
- status_code=503,
+ status_code=status_code,
)
def mark_ready() -> None:
diff --git a/services/purple-team/app/_migrate.py b/services/purple-team/app/_migrate.py
new file mode 100644
index 000000000..1297f113b
--- /dev/null
+++ b/services/purple-team/app/_migrate.py
@@ -0,0 +1,228 @@
+"""Apply this service's alembic chain, then exec what comes after it.
+
+Why this exists
+---------------
+
+Four services in this repository own their schema through alembic —
+``ueba``, ``honeytokens``, ``purple-team`` and ``osquery-tls`` — and **no
+compose path invoked any of them**. Their container command was a plain
+``uvicorn``, so on the documented quickstart they booted healthy against
+empty schemas with none of their row-level-security policies installed. The
+only documented invocation was a manual step in a docs-portal quickstart
+whose own "start the stack" step does not start three of the four services,
+and whose first line failed outright.
+
+That is how two services shipped with no tables while every command
+reported success. The API does not have this problem because its chain runs
+from its own process on the way up; this is the same idea for the four that
+use alembic.
+
+What it guarantees
+------------------
+
+**Idempotent.** ``alembic upgrade head`` on a database already at head
+applies nothing. Running this on every container start is the point: there
+is no separate "have you migrated yet" state for an operator to get wrong.
+
+**Race-free.** Four chains start at once against one database. The chains
+take a shared transaction-scoped advisory lock (``MIGRATION_LOCK_KEY`` in
+each ``env.py``), so they queue instead of deadlocking on catalog locks.
+This wrapper deliberately holds no lock of its own — a lock held here would
+be a second mechanism to reason about, and it would not cover the window
+that actually matters.
+
+**Loud.** The service does not start unless the chain reached head. Not "the
+command exited 0": alembic exits 0 when it applies *nothing*, which is
+exactly what happened when four chains shared one ``alembic_version`` table
+and the second one to run believed it was already done. So ``current`` is
+read back and compared to ``heads``, and a mismatch — or an empty
+``current``, meaning the chain never ran — stops the container with the
+reason on stderr. A service booting on an empty schema is the failure this
+replaces, and it must not be reachable by a wrapper that only checks an exit
+code.
+
+Why the owner credential, and why a subprocess
+----------------------------------------------
+
+Alembic issues DDL. The runtime role every service serves requests as holds
+no ``CREATE`` on schema public, by design — that is what stops it turning
+row-level security off — so the chain must be applied as the owner. Each
+``env.py`` already resolves that (its own ``*_DATABASE_MIGRATION_URL``
+first, then ``DATABASE_MIGRATION_URL``), and this runs alembic as a
+subprocess precisely so ``env.py`` stays the single answer to "which
+credential". A second resolution here would be a second thing to keep in
+step, and the two would disagree the first time either changed.
+
+This module is vendored per service because services do not share a Python
+path. ``scripts/check_service_migration_bootstrap.py`` is the gate that each
+of the four ships a copy, keeps them identical, and runs one on the way up.
+
+Usage
+-----
+
+::
+
+ python -m app._migrate # apply the chain, exit
+ python -m app._migrate -- uvicorn app.main:app --port 8004
+"""
+
+from __future__ import annotations
+
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+#: Where ``alembic.ini`` sits relative to the working directory in every one
+#: of the four images (``WORKDIR /app``, and the ini is copied to ``/app``).
+ALEMBIC_INI = "alembic.ini"
+
+#: Suffix that marks a DSN as the migration (owner) credential. Matches the
+#: convention each ``env.py`` documents and ``scripts/check_runtime_db_role.py``
+#: enforces on the deployment surfaces: a service-prefixed
+#: ``<SVC>_DATABASE_MIGRATION_URL`` wins over the bare form.
+MIGRATION_URL_SUFFIX = "DATABASE_MIGRATION_URL"
+
+
+def migration_credential_present(env: dict[str, str] | None = None) -> bool:
+ """Whether an owner DSN is configured.
+
+ Used only for the warning below. The chain is applied either way — a
+ deployment that has not split the roles yet behaves as it did before —
+ but it is announced, because a chain silently applied as the runtime
+ credential fails on the first ``CREATE TABLE`` and the message names a
+ permission rather than a configuration.
+ """
+ source = os.environ if env is None else env
+ return any(name.endswith(MIGRATION_URL_SUFFIX) and value.strip() for name, value in source.items())
+
+
+def _alembic(args: list[str], *, capture: bool) -> subprocess.CompletedProcess[str]:
+ return subprocess.run( # noqa: S603 — fixed argv, no shell
+ [sys.executable, "-m", "alembic", "-c", ALEMBIC_INI, *args],
+ capture_output=capture,
+ text=True,
+ check=False,
+ )
+
+
+def _revisions(output: str) -> set[str]:
+ """Revision ids in ``alembic current`` / ``alembic heads`` output.
+
+ Both print ``<rev> (head)`` or ``<rev>``, one per line, with informational
+ lines alembic sends to stderr. An empty set means the version table holds
+ nothing — the chain has not run — which is the case this exists to catch
+ and is therefore never treated as agreement.
+ """
+ found: set[str] = set()
+ for line in output.splitlines():
+ token = line.strip().split(" ")[0].strip()
+ if token and not token.startswith(("INFO", "WARNING", "ERROR", "alembic:")):
+ found.add(token)
+ return found
+
+
+def apply_chain() -> int:
+ """Apply the chain and verify it reached head. Returns an exit code."""
+ if not Path(ALEMBIC_INI).is_file():
+ print(
+ f"migrate: {ALEMBIC_INI} not found in {Path.cwd()}. This service's image must "
+ f"copy its alembic config and chain; see its Dockerfile.",
+ file=sys.stderr,
+ )
+ return 1
+
+ if not migration_credential_present():
+ print(
+ f"migrate: no *{MIGRATION_URL_SUFFIX} is set, so the chain will be applied as "
+ f"the runtime credential. That works only while the two are the same role; "
+ f"under the DML-only runtime role the first CREATE TABLE fails with "
+ f"'permission denied for schema public'.",
+ file=sys.stderr,
+ )
+
+ upgrade = _alembic(["upgrade", "head"], capture=False)
+ if upgrade.returncode != 0:
+ print(
+ f"migrate: 'alembic upgrade head' failed (exit {upgrade.returncode}). Refusing to "
+ f"start: this service would serve requests against whatever schema is there now.",
+ file=sys.stderr,
+ )
+ return upgrade.returncode
+
+ # Read back rather than trusting the exit code. `alembic upgrade head`
+ # exits 0 having applied nothing when it believes the database is already
+ # at head, and believing that wrongly is the documented way these four
+ # services ended up with no tables.
+ current = _alembic(["current"], capture=True)
+ heads = _alembic(["heads"], capture=True)
+ if current.returncode != 0 or heads.returncode != 0:
+ print(
+ "migrate: could not read back the applied revision "
+ f"(current exit {current.returncode}, heads exit {heads.returncode}). "
+ "Refusing to start: an unverified migration is the state this check exists "
+ "to reject.\n"
+ f"{(current.stderr or '') + (heads.stderr or '')}",
+ file=sys.stderr,
+ )
+ return 1
+
+ applied = _revisions(current.stdout)
+ expected = _revisions(heads.stdout)
+
+ if not expected:
+ print(
+ "migrate: this chain declares no head revision, so there is nothing to verify "
+ "against. Refusing to start rather than reporting an empty chain as applied.",
+ file=sys.stderr,
+ )
+ return 1
+
+ if not applied:
+ print(
+ f"migrate: 'alembic upgrade head' reported success but the version table is "
+ f"empty — the chain never ran. Expected {sorted(expected)}. Refusing to start "
+ f"on an unmigrated schema.",
+ file=sys.stderr,
+ )
+ return 1
+
+ if not expected.issubset(applied):
+ print(
+ f"migrate: chain did not reach head. At {sorted(applied)}, head is " f"{sorted(expected)}. Refusing to start.",
+ file=sys.stderr,
+ )
+ return 1
+
+ print(f"migrate: schema at head {sorted(expected)}", file=sys.stderr)
+ return 0
+
+
+def main(argv: list[str] | None = None) -> int:
+ args = list(sys.argv[1:] if argv is None else argv)
+
+ # Everything after `--` is the real service command. Exec'd rather than
+ # spawned so the server keeps PID 1 and receives SIGTERM directly — a
+ # Python parent waiting on a child swallows the orchestrator's stop
+ # signal and turns every shutdown into a ten-second kill.
+ command: list[str] = []
+ if "--" in args:
+ split = args.index("--")
+ command = args[split + 1 :]
+ args = args[:split]
+
+ if args:
+ print(f"migrate: unexpected arguments {args}; usage: python -m app._migrate [-- CMD ...]", file=sys.stderr)
+ return 2
+
+ code = apply_chain()
+ if code != 0:
+ return code
+
+ if command:
+ os.execvp(command[0], command) # noqa: S606 — replaces this process by design
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/services/realtime/src/index.ts b/services/realtime/src/index.ts
index 82cff5417..daf6c1ec0 100644
--- a/services/realtime/src/index.ts
+++ b/services/realtime/src/index.ts
@@ -414,9 +414,89 @@ const kafka = new Kafka({
retry: { retries: 5 },
});
+// --- Subscription state, reported on /health ---
+//
+// `/health` returned a hardcoded `status: 'healthy'` with a client count and
+// nothing about Kafka, so a realtime service whose consumers had never
+// connected was indistinguishable from one fanning out every alert. The two
+// consumers below register here and `reportHealth` answers 503 while either
+// is detached — the same contract the Python services answer on /readyz.
+type SubscriptionState = {
+ attached: boolean;
+ lastError?: string;
+ attempts: number;
+};
+const subscriptions = new Map<string, SubscriptionState>();
+
+// Bounded backoff, capped. A detached consumer is retried because the usual
+// cause is a broker or topic that is not there *yet*, which a retry fixes;
+// the cap stops a permanently-broken cluster becoming a busy loop, and the
+// state above stops a permanent failure being silent while it retries.
+const RETRY_BASE_MS = 2_000;
+const RETRY_MAX_MS = 60_000;
+
+/**
+ * Keep `start` running, and record whether it is.
+ *
+ * The previous code called each starter inside a try/catch that logged
+ * "Kafka consumer failed to start (will retry)" and then returned. Nothing
+ * retried. The message described behaviour the code did not have, which is
+ * worse than no message: it tells a reader the gap is already handled.
+ */
+function superviseConsumer(
+ name: string,
+ start: (onCrash: (err: unknown) => void) => Promise<void>,
+): void {
+ subscriptions.set(name, { attached: false, attempts: 0 });
+
+ const run = async (): Promise<void> => {
+ const state = subscriptions.get(name)!;
+ for (;;) {
+ state.attempts += 1;
+ try {
+ let crashed = false;
+ await start((err: unknown) => {
+ // kafkajs surfaces a dead consumer loop through its CRASH event,
+ // not by rejecting the promise `run()` returned — so without this
+ // hook a consumer that died after a successful start stayed
+ // recorded as attached for the life of the process.
+ if (crashed) return;
+ crashed = true;
+ state.attached = false;
+ state.lastError = err instanceof Error ? err.message : String(err);
+ log.error({ err, name }, 'Kafka consumer crashed; reconnecting');
+ });
+ state.attached = true;
+ state.lastError = undefined;
+ log.info({ name, attempts: state.attempts }, 'Kafka consumer attached');
+ // Wait for a crash rather than returning: this function owns the
+ // retry, so it has to still be here when the consumer stops.
+ while (!crashed) {
+ await new Promise((resolve) => setTimeout(resolve, 5_000));
+ }
+ } catch (err) {
+ state.attached = false;
+ state.lastError = err instanceof Error ? err.message : String(err);
+ const delay = Math.min(
+ RETRY_BASE_MS * 2 ** Math.min(state.attempts - 1, 5),
+ RETRY_MAX_MS,
+ );
+ log.warn(
+ { err, name, attempts: state.attempts, retryInMs: delay },
+ 'Kafka consumer not attached; retrying',
+ );
+ await new Promise((resolve) => setTimeout(resolve, delay));
+ }
+ }
+ };
+
+ void run();
+}
+
// --- Kafka consumer: bridge fused alerts to WebSocket clients ---
-async function startKafkaConsumer() {
+async function startKafkaConsumer(onCrash: (err: unknown) => void) {
const consumer = kafka.consumer({ groupId: 'aisoc-realtime-ws' });
+ consumer.on(consumer.events.CRASH, (event) => onCrash(event.payload.error));
await consumer.connect();
await consumer.subscribe({ topic: KAFKA_TOPIC_FUSED, fromBeginning: false });
@@ -486,13 +566,14 @@ async function startKafkaConsumer() {
// versa. A failure to start (Kafka unreachable, topic missing, etc.) is
// logged and surfaced via the outer retry wrapper; it does NOT crash the
// process, because the alert fan-out path is the higher-priority surface.
-async function startGraphUpdateConsumer() {
+async function startGraphUpdateConsumer(onCrash: (err: unknown) => void) {
if (!KAFKA_TOPIC_GRAPH_UPDATES) {
log.info('Graph update consumer disabled (empty KAFKA_TOPIC_GRAPH_UPDATES)');
return;
}
const consumer = kafka.consumer({ groupId: 'aisoc-realtime-graph' });
+ consumer.on(consumer.events.CRASH, (event) => onCrash(event.payload.error));
await consumer.connect();
await consumer.subscribe({
@@ -708,10 +789,22 @@ app.post('/internal/agent-event', internalEventRateLimit, (req, res) => {
// Expose both `/health` (canonical) and `/healthz` (k8s + frontend default) so
// callers don't have to guess.
const reportHealth = (_req: express.Request, res: express.Response) => {
- res.json({
- status: 'healthy',
+ const detached = [...subscriptions.entries()]
+ .filter(([, state]) => !state.attached)
+ .map(([name]) => name);
+
+ res.status(detached.length > 0 ? 503 : 200).json({
+ status: detached.length > 0 ? 'degraded' : 'healthy',
service: 'aisoc-realtime',
clients: wss.clients.size,
+ // Reported whether or not anything is wrong, so a healthy answer says
+ // which subscriptions were checked rather than only that none failed.
+ subscriptions: [...subscriptions.entries()].map(([name, state]) => ({
+ topic: name,
+ attached: state.attached,
+ attempts: state.attempts,
+ ...(state.lastError ? { last_error: state.lastError } : {}),
+ })),
});
};
app.get('/health', reportHealth);
@@ -743,30 +836,19 @@ server.listen(PORT, '::', () => {
}
})();
- // Start the two Kafka consumers concurrently. Each is wrapped in its own
- // try/catch so a failure on the graph topic (e.g. it doesn't exist yet on
- // a brand-new cluster) does NOT block the higher-priority fused-alerts
- // fan-out. We deliberately fire them in parallel rather than awaiting
- // sequentially so the HTTP/WS listener is fully up before either Kafka
- // round-trip completes.
- void (async () => {
- try {
- await startKafkaConsumer();
- } catch (err) {
- log.warn({ err }, 'Kafka consumer failed to start (will retry)');
- }
- })();
-
- void (async () => {
- try {
- await startGraphUpdateConsumer();
- } catch (err) {
- // T1.4 fan-out is best-effort: if the graph topic isn't available the
- // alerts/cases/agents/insights channels still work, the graph panel
- // just won't light up in real time.
- log.warn({ err }, 'Graph update consumer failed to start (will retry)');
- }
- })();
+ // Start the two Kafka consumers under their own supervisors, so a failure
+ // on the graph topic (e.g. it doesn't exist yet on a brand-new cluster)
+ // does NOT block the higher-priority fused-alerts fan-out, and neither one
+ // stays silently detached. Fired rather than awaited so the HTTP/WS
+ // listener is fully up before either Kafka round-trip completes.
+ superviseConsumer(KAFKA_TOPIC_FUSED, startKafkaConsumer);
+
+ // Only supervised when it is configured. An empty KAFKA_TOPIC_GRAPH_UPDATES
+ // turns this fan-out off deliberately, and reporting a switched-off consumer
+ // as detached would make /health red for a supported configuration.
+ if (KAFKA_TOPIC_GRAPH_UPDATES) {
+ superviseConsumer(KAFKA_TOPIC_GRAPH_UPDATES, startGraphUpdateConsumer);
+ }
});
// Flush spans on the way out. Without this the exporter drops the spans
diff --git a/services/slack-bot/app/_health.py b/services/slack-bot/app/_health.py
index 974269731..0acb9e4da 100644
--- a/services/slack-bot/app/_health.py
+++ b/services/slack-bot/app/_health.py
@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /
``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /
``mark_not_ready()`` from those handlers respectively.
+Readiness is a question, not a latch
+------------------------------------
+
+``mark_ready()`` answers "did startup finish", and for a service
+whose only job is to answer HTTP that is the whole story. For a
+service whose job is to *consume a topic* it is not: three
+services in this repository created the consumer as a
+fire-and-forget ``asyncio.create_task`` and called
+``mark_ready()`` on the next line, so readiness recorded that the
+task had been **created**, never that it was still running.
+
+UEBA is what that cost. Its handler raised
+``UndefinedColumnError`` on the first scoreable event, the
+exception left the ``async for``, the ``finally`` stopped the
+consumer, and the task ended — holding the exception, which
+nothing retrieved because the task was parked in a module global
+and so never garbage-collected. The container stayed ``running``
+with restarts 0, ``/health`` returned 200 and ``/readyz``
+returned 200, permanently. A consumer that has silently detached
+is indistinguishable from an idle one, and the whole point of a
+readiness probe is to tell those two apart.
+
+So a service that owns a subscription registers a probe for it
+with :func:`register_subscription`, and ``/readyz`` evaluates
+every probe on each request. A probe that returns false — or
+raises — is reported as detached and ``/readyz`` answers 503 with
+the names, which is a readiness failure rather than a liveness
+one on purpose: the orchestrator should stop routing to a
+half-working replica, and killing the process would only hide a
+schema or broker fault behind a restart loop.
+
+``/livez`` deliberately does **not** consult the probes. It
+answers "is the interpreter responsive", and wiring a detached
+consumer into it would make every orchestrator restart the pod on
+a fault a restart cannot fix.
+
This module is intentionally dependency-free (only FastAPI) so it
can be vendored into every service tree without import-graph
side-effects. Each service ships its own copy under
``services/<svc>/app/_health.py`` because services don't share a
-Python path — keep the file in sync via
-``scripts/sync_health_module.py`` (CI gate).
+Python path; ``scripts/audit_health_probes.py --check`` is the CI
+gate that every service has one and wires it in.
"""
from __future__ import annotations
+import json
from collections.abc import Callable
from typing import Any
from fastapi import FastAPI, Response
+#: Where the probes registered against an app are kept. Read by ``/readyz``,
+#: written by :func:`register_subscription`. On ``app.state`` rather than in a
+#: module global because a module global is shared by every app in the
+#: process, which makes two test clients interfere.
+_STATE_ATTR = "health_subscriptions"
+
+
+def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:
+ """Names of the subscriptions currently reporting not-attached.
+
+ A probe that raises counts as detached. It is asking a live object
+ whether it is still working; if that question itself fails, the honest
+ answer is not "yes".
+ """
+ detached: list[str] = []
+ for name, probe in sorted(probes.items()):
+ try:
+ attached = bool(probe())
+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one
+ attached = False
+ if not attached:
+ detached.append(name)
+ return detached
+
+
+def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:
+ """Make ``/readyz`` depend on *probe* returning true.
+
+ Call this once per subscription the service owns, after
+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —
+ it runs on every ``/readyz`` request — so it should read a flag the
+ consumer maintains or check ``task.done()``, not round-trip to a broker.
+ """
+ probes = getattr(app.state, _STATE_ATTR, None)
+ if probes is None: # pragma: no cover — install_health_routes was not called
+ raise RuntimeError("register_subscription requires install_health_routes(app) first")
+ probes[name] = probe
+
def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:
"""Install ``/livez`` and ``/readyz`` on ``app``.
@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
``mark_ready()`` is called — exactly the right behaviour for a
rolling deploy where the orchestrator should hold traffic
until the pod's dependencies are connected.
+
+ Startup having finished is necessary and not sufficient: any
+ subscription registered with :func:`register_subscription`
+ must also still be attached. See the module docstring.
"""
state: dict[str, bool] = {"ready": False}
+ probes: dict[str, Callable[[], bool]] = {}
+ setattr(app.state, _STATE_ATTR, probes)
@app.get(
"/livez",
@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
include_in_schema=False,
)
async def _readyz() -> Response:
- if state["ready"]:
- return Response(
- content='{"status":"ready","service":"' + service_name + '"}',
- media_type="application/json",
- status_code=200,
- )
+ body: dict[str, Any] = {"status": "ready", "service": service_name}
+ status_code = 200
+
+ if not state["ready"]:
+ body["status"] = "starting"
+ status_code = 503
+ else:
+ detached = _probe_detached(probes)
+ if detached:
+ body["status"] = "degraded"
+ body["detached"] = detached
+ status_code = 503
+
+ # Named subscriptions are reported either way, so a 200 says which
+ # ones were checked rather than only that nothing was wrong. A probe
+ # list that is unexpectedly empty is then visible in the same place
+ # the verdict is.
+ if probes:
+ body["subscriptions"] = sorted(probes)
+
return Response(
- content='{"status":"starting","service":"' + service_name + '"}',
+ content=json.dumps(body),
media_type="application/json",
- status_code=503,
+ status_code=status_code,
)
def mark_ready() -> None:
diff --git a/services/slack-bot/tests/test_health_probes.py b/services/slack-bot/tests/test_health_probes.py
new file mode 100644
index 000000000..00bd7914e
--- /dev/null
+++ b/services/slack-bot/tests/test_health_probes.py
@@ -0,0 +1,98 @@
+"""The shared probe helper, exercised in this tree too.
+
+``app/_health.py`` is vendored byte-identical into thirteen services, and its
+whole point is that every copy answers the same contract — which is only
+worth something if more than one copy is ever run. ``services/api`` owns the
+exhaustive suite; this asserts the parts an operator reads off a dashboard,
+against *this* service's copy.
+
+Worth keeping rather than deduplicating: when the helper grew
+``register_subscription``, the api tree's tests covered the new branches and
+none of the other twelve did. Here that showed up as this service's coverage
+falling below its floor, which is a coarse signal and, that time, the only
+one.
+"""
+
+from __future__ import annotations
+
+from collections.abc import Callable
+
+import pytest
+from app._health import install_health_routes, register_subscription
+from fastapi import FastAPI
+from fastapi.testclient import TestClient
+
+
+def _build_app() -> tuple[FastAPI, Callable[[], None], Callable[[], None]]:
+ app = FastAPI()
+ mark_ready, mark_not_ready = install_health_routes(app, service_name="aisoc-slack-bot")
+ return app, mark_ready, mark_not_ready
+
+
+def test_livez_is_200_before_startup_finishes() -> None:
+ app, _, _ = _build_app()
+
+ response = TestClient(app).get("/livez")
+
+ assert response.status_code == 200
+ assert response.json()["service"] == "aisoc-slack-bot"
+
+
+def test_readyz_is_503_until_marked_ready() -> None:
+ app, mark_ready, _ = _build_app()
+ client = TestClient(app)
+
+ assert client.get("/readyz").status_code == 503
+ mark_ready()
+ assert client.get("/readyz").status_code == 200
+
+
+def test_readyz_is_503_again_after_draining() -> None:
+ app, mark_ready, mark_not_ready = _build_app()
+ client = TestClient(app)
+
+ mark_ready()
+ mark_not_ready()
+
+ assert client.get("/readyz").status_code == 503
+
+
+def test_a_detached_subscription_is_named_in_the_503() -> None:
+ app, mark_ready, _ = _build_app()
+ attached = {"value": True}
+ register_subscription(app, "aisoc.alerts.fused", lambda: attached["value"])
+ mark_ready()
+ client = TestClient(app)
+ assert client.get("/readyz").json()["subscriptions"] == ["aisoc.alerts.fused"]
+
+ attached["value"] = False
+
+ response = client.get("/readyz")
+ assert response.status_code == 503
+ assert response.json()["detached"] == ["aisoc.alerts.fused"]
+
+
+def test_a_probe_that_raises_is_not_a_healthy_answer() -> None:
+ def explode() -> bool:
+ raise RuntimeError("gone")
+
+ app, mark_ready, _ = _build_app()
+ register_subscription(app, "aisoc.alerts.fused", explode)
+ mark_ready()
+
+ assert TestClient(app).get("/readyz").status_code == 503
+
+
+def test_livez_ignores_a_detached_subscription() -> None:
+ """Liveness failing means "restart me", and a restart does not fix a
+ missing column or a revoked grant."""
+ app, mark_ready, _ = _build_app()
+ register_subscription(app, "aisoc.alerts.fused", lambda: False)
+ mark_ready()
+
+ assert TestClient(app).get("/livez").status_code == 200
+
+
+def test_registration_requires_the_routes_first() -> None:
+ with pytest.raises(RuntimeError):
+ register_subscription(FastAPI(), "aisoc.alerts.fused", lambda: True)
diff --git a/services/teams-bot/app/_health.py b/services/teams-bot/app/_health.py
index 974269731..0acb9e4da 100644
--- a/services/teams-bot/app/_health.py
+++ b/services/teams-bot/app/_health.py
@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /
``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /
``mark_not_ready()`` from those handlers respectively.
+Readiness is a question, not a latch
+------------------------------------
+
+``mark_ready()`` answers "did startup finish", and for a service
+whose only job is to answer HTTP that is the whole story. For a
+service whose job is to *consume a topic* it is not: three
+services in this repository created the consumer as a
+fire-and-forget ``asyncio.create_task`` and called
+``mark_ready()`` on the next line, so readiness recorded that the
+task had been **created**, never that it was still running.
+
+UEBA is what that cost. Its handler raised
+``UndefinedColumnError`` on the first scoreable event, the
+exception left the ``async for``, the ``finally`` stopped the
+consumer, and the task ended — holding the exception, which
+nothing retrieved because the task was parked in a module global
+and so never garbage-collected. The container stayed ``running``
+with restarts 0, ``/health`` returned 200 and ``/readyz``
+returned 200, permanently. A consumer that has silently detached
+is indistinguishable from an idle one, and the whole point of a
+readiness probe is to tell those two apart.
+
+So a service that owns a subscription registers a probe for it
+with :func:`register_subscription`, and ``/readyz`` evaluates
+every probe on each request. A probe that returns false — or
+raises — is reported as detached and ``/readyz`` answers 503 with
+the names, which is a readiness failure rather than a liveness
+one on purpose: the orchestrator should stop routing to a
+half-working replica, and killing the process would only hide a
+schema or broker fault behind a restart loop.
+
+``/livez`` deliberately does **not** consult the probes. It
+answers "is the interpreter responsive", and wiring a detached
+consumer into it would make every orchestrator restart the pod on
+a fault a restart cannot fix.
+
This module is intentionally dependency-free (only FastAPI) so it
can be vendored into every service tree without import-graph
side-effects. Each service ships its own copy under
``services/<svc>/app/_health.py`` because services don't share a
-Python path — keep the file in sync via
-``scripts/sync_health_module.py`` (CI gate).
+Python path; ``scripts/audit_health_probes.py --check`` is the CI
+gate that every service has one and wires it in.
"""
from __future__ import annotations
+import json
from collections.abc import Callable
from typing import Any
from fastapi import FastAPI, Response
+#: Where the probes registered against an app are kept. Read by ``/readyz``,
+#: written by :func:`register_subscription`. On ``app.state`` rather than in a
+#: module global because a module global is shared by every app in the
+#: process, which makes two test clients interfere.
+_STATE_ATTR = "health_subscriptions"
+
+
+def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:
+ """Names of the subscriptions currently reporting not-attached.
+
+ A probe that raises counts as detached. It is asking a live object
+ whether it is still working; if that question itself fails, the honest
+ answer is not "yes".
+ """
+ detached: list[str] = []
+ for name, probe in sorted(probes.items()):
+ try:
+ attached = bool(probe())
+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one
+ attached = False
+ if not attached:
+ detached.append(name)
+ return detached
+
+
+def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:
+ """Make ``/readyz`` depend on *probe* returning true.
+
+ Call this once per subscription the service owns, after
+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —
+ it runs on every ``/readyz`` request — so it should read a flag the
+ consumer maintains or check ``task.done()``, not round-trip to a broker.
+ """
+ probes = getattr(app.state, _STATE_ATTR, None)
+ if probes is None: # pragma: no cover — install_health_routes was not called
+ raise RuntimeError("register_subscription requires install_health_routes(app) first")
+ probes[name] = probe
+
def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:
"""Install ``/livez`` and ``/readyz`` on ``app``.
@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
``mark_ready()`` is called — exactly the right behaviour for a
rolling deploy where the orchestrator should hold traffic
until the pod's dependencies are connected.
+
+ Startup having finished is necessary and not sufficient: any
+ subscription registered with :func:`register_subscription`
+ must also still be attached. See the module docstring.
"""
state: dict[str, bool] = {"ready": False}
+ probes: dict[str, Callable[[], bool]] = {}
+ setattr(app.state, _STATE_ATTR, probes)
@app.get(
"/livez",
@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
include_in_schema=False,
)
async def _readyz() -> Response:
- if state["ready"]:
- return Response(
- content='{"status":"ready","service":"' + service_name + '"}',
- media_type="application/json",
- status_code=200,
- )
+ body: dict[str, Any] = {"status": "ready", "service": service_name}
+ status_code = 200
+
+ if not state["ready"]:
+ body["status"] = "starting"
+ status_code = 503
+ else:
+ detached = _probe_detached(probes)
+ if detached:
+ body["status"] = "degraded"
+ body["detached"] = detached
+ status_code = 503
+
+ # Named subscriptions are reported either way, so a 200 says which
+ # ones were checked rather than only that nothing was wrong. A probe
+ # list that is unexpectedly empty is then visible in the same place
+ # the verdict is.
+ if probes:
+ body["subscriptions"] = sorted(probes)
+
return Response(
- content='{"status":"starting","service":"' + service_name + '"}',
+ content=json.dumps(body),
media_type="application/json",
- status_code=503,
+ status_code=status_code,
)
def mark_ready() -> None:
diff --git a/services/threatintel/app/_health.py b/services/threatintel/app/_health.py
index 974269731..0acb9e4da 100644
--- a/services/threatintel/app/_health.py
+++ b/services/threatintel/app/_health.py
@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /
``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /
``mark_not_ready()`` from those handlers respectively.
+Readiness is a question, not a latch
+------------------------------------
+
+``mark_ready()`` answers "did startup finish", and for a service
+whose only job is to answer HTTP that is the whole story. For a
+service whose job is to *consume a topic* it is not: three
+services in this repository created the consumer as a
+fire-and-forget ``asyncio.create_task`` and called
+``mark_ready()`` on the next line, so readiness recorded that the
+task had been **created**, never that it was still running.
+
+UEBA is what that cost. Its handler raised
+``UndefinedColumnError`` on the first scoreable event, the
+exception left the ``async for``, the ``finally`` stopped the
+consumer, and the task ended — holding the exception, which
+nothing retrieved because the task was parked in a module global
+and so never garbage-collected. The container stayed ``running``
+with restarts 0, ``/health`` returned 200 and ``/readyz``
+returned 200, permanently. A consumer that has silently detached
+is indistinguishable from an idle one, and the whole point of a
+readiness probe is to tell those two apart.
+
+So a service that owns a subscription registers a probe for it
+with :func:`register_subscription`, and ``/readyz`` evaluates
+every probe on each request. A probe that returns false — or
+raises — is reported as detached and ``/readyz`` answers 503 with
+the names, which is a readiness failure rather than a liveness
+one on purpose: the orchestrator should stop routing to a
+half-working replica, and killing the process would only hide a
+schema or broker fault behind a restart loop.
+
+``/livez`` deliberately does **not** consult the probes. It
+answers "is the interpreter responsive", and wiring a detached
+consumer into it would make every orchestrator restart the pod on
+a fault a restart cannot fix.
+
This module is intentionally dependency-free (only FastAPI) so it
can be vendored into every service tree without import-graph
side-effects. Each service ships its own copy under
``services/<svc>/app/_health.py`` because services don't share a
-Python path — keep the file in sync via
-``scripts/sync_health_module.py`` (CI gate).
+Python path; ``scripts/audit_health_probes.py --check`` is the CI
+gate that every service has one and wires it in.
"""
from __future__ import annotations
+import json
from collections.abc import Callable
from typing import Any
from fastapi import FastAPI, Response
+#: Where the probes registered against an app are kept. Read by ``/readyz``,
+#: written by :func:`register_subscription`. On ``app.state`` rather than in a
+#: module global because a module global is shared by every app in the
+#: process, which makes two test clients interfere.
+_STATE_ATTR = "health_subscriptions"
+
+
+def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:
+ """Names of the subscriptions currently reporting not-attached.
+
+ A probe that raises counts as detached. It is asking a live object
+ whether it is still working; if that question itself fails, the honest
+ answer is not "yes".
+ """
+ detached: list[str] = []
+ for name, probe in sorted(probes.items()):
+ try:
+ attached = bool(probe())
+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one
+ attached = False
+ if not attached:
+ detached.append(name)
+ return detached
+
+
+def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:
+ """Make ``/readyz`` depend on *probe* returning true.
+
+ Call this once per subscription the service owns, after
+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —
+ it runs on every ``/readyz`` request — so it should read a flag the
+ consumer maintains or check ``task.done()``, not round-trip to a broker.
+ """
+ probes = getattr(app.state, _STATE_ATTR, None)
+ if probes is None: # pragma: no cover — install_health_routes was not called
+ raise RuntimeError("register_subscription requires install_health_routes(app) first")
+ probes[name] = probe
+
def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:
"""Install ``/livez`` and ``/readyz`` on ``app``.
@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
``mark_ready()`` is called — exactly the right behaviour for a
rolling deploy where the orchestrator should hold traffic
until the pod's dependencies are connected.
+
+ Startup having finished is necessary and not sufficient: any
+ subscription registered with :func:`register_subscription`
+ must also still be attached. See the module docstring.
"""
state: dict[str, bool] = {"ready": False}
+ probes: dict[str, Callable[[], bool]] = {}
+ setattr(app.state, _STATE_ATTR, probes)
@app.get(
"/livez",
@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
include_in_schema=False,
)
async def _readyz() -> Response:
- if state["ready"]:
- return Response(
- content='{"status":"ready","service":"' + service_name + '"}',
- media_type="application/json",
- status_code=200,
- )
+ body: dict[str, Any] = {"status": "ready", "service": service_name}
+ status_code = 200
+
+ if not state["ready"]:
+ body["status"] = "starting"
+ status_code = 503
+ else:
+ detached = _probe_detached(probes)
+ if detached:
+ body["status"] = "degraded"
+ body["detached"] = detached
+ status_code = 503
+
+ # Named subscriptions are reported either way, so a 200 says which
+ # ones were checked rather than only that nothing was wrong. A probe
+ # list that is unexpectedly empty is then visible in the same place
+ # the verdict is.
+ if probes:
+ body["subscriptions"] = sorted(probes)
+
return Response(
- content='{"status":"starting","service":"' + service_name + '"}',
+ content=json.dumps(body),
media_type="application/json",
- status_code=503,
+ status_code=status_code,
)
def mark_ready() -> None:
diff --git a/services/ueba/Dockerfile b/services/ueba/Dockerfile
index 810b570f0..7ce8f612f 100644
--- a/services/ueba/Dockerfile
+++ b/services/ueba/Dockerfile
@@ -25,4 +25,17 @@ COPY app/ app/
EXPOSE 8004
-CMD ["uvicorn", "app.main:app", "--host", "0.0.0.0", "--port", "8004"]
+# The chain runs on the way up, then this process is replaced by the server.
+#
+# Nothing invoked these four alembic chains. This was a bare `uvicorn` line, so
+# the documented quickstart booted the service against an empty schema with
+# none of its row-level-security policies installed, and every command in that
+# sequence reported success. `app/_migrate.py` applies the chain as the owner
+# credential, verifies it reached head, and refuses to start the server if it
+# did not.
+#
+# `sh -c` because an exec-form CMD cannot run two things. Neither shell nor
+# wrapper stays in the process tree: `sh -c` with a single command execs it,
+# and `_migrate` hands over with `os.execvp`, so uvicorn ends up as PID 1 and
+# receives SIGTERM directly rather than through a parent that would swallow it.
+CMD ["sh", "-c", "python -m app._migrate -- uvicorn app.main:app --host 0.0.0.0 --port 8004"]
diff --git a/services/ueba/alembic/env.py b/services/ueba/alembic/env.py
index b7318aeda..12e7567c2 100644
--- a/services/ueba/alembic/env.py
+++ b/services/ueba/alembic/env.py
@@ -157,9 +157,27 @@ def run_migrations_offline() -> None:
context.run_migrations()
+#: Serialises the four chains that share one database against each other.
+#:
+#: They are started simultaneously by ``docker compose up``, and while each
+#: now owns its own version table, they do not own their own *database*: the
+#: legacy-version probe below reads a table a sibling may be creating, and
+#: ``000N_runtime_role_grants`` issues ``GRANT`` statements that take
+#: catalog locks on ``pg_class``. Two chains arriving together deadlocked
+#: rather than queued.
+#:
+#: ``pg_advisory_xact_lock`` and not the session form: it is released by the
+#: commit or rollback that ends the upgrade, so a chain that dies mid-apply
+#: cannot leave the other three waiting on a lock nobody holds a connection
+#: for. The key is an arbitrary constant, shared by every chain on purpose —
+#: the point is mutual exclusion, not per-chain exclusion.
+MIGRATION_LOCK_KEY = 0x4149_5354
+
+
def do_run_migrations(connection): # type: ignore[no-untyped-def]
context.configure(connection=connection, target_metadata=target_metadata, version_table=VERSION_TABLE)
with context.begin_transaction():
+ connection.execute(sa.text("SELECT pg_advisory_xact_lock(:key)"), {"key": MIGRATION_LOCK_KEY})
# Inside ``begin_transaction()``, not before it. Alembic's
# ``begin_transaction`` is a no-op when the connection is *already* in a
# transaction, and in SQLAlchemy 2.0 any execute on a plain ``connect()``
diff --git a/services/ueba/alembic/versions/0004_baseline_peer_group_and_column_parity.py b/services/ueba/alembic/versions/0004_baseline_peer_group_and_column_parity.py
new file mode 100644
index 000000000..d4293a7f5
--- /dev/null
+++ b/services/ueba/alembic/versions/0004_baseline_peer_group_and_column_parity.py
@@ -0,0 +1,90 @@
+"""Bring the UEBA tables back in line with the models that read them.
+
+Revision ID: 0004
+Revises: 0003
+Create Date: 2026-09-24
+
+``0001`` and ``app/models/ueba.py`` disagreed in three places, and every one of
+them was reached on the first scoreable event rather than on some rare path:
+
+``ueba_entity_baselines.peer_group_id``
+ Declared by :class:`~app.models.ueba.EntityBaseline`, created by no
+ migration. SQLAlchemy names every mapped column in its ``SELECT``, so
+ ``BaselineService.get_or_create`` — the first database call
+ ``score_event`` makes — raised ``UndefinedColumnError``. UEBA could
+ therefore never write a baseline or an anomaly, on any deployment, and
+ the container stayed healthy throughout.
+
+``ueba_peer_groups.id``
+ Created as ``UUID DEFAULT gen_random_uuid()``, declared as
+ ``String(64)``, and written with values like ``dept:engineering`` — the
+ ids are externally-supplied group names, not surrogate keys, which is
+ why ``PeerGroupService.get_or_create`` passes one explicitly. An insert
+ raised ``InvalidTextRepresentation``. The column follows the model
+ because the model matches the data; a UUID here would mean the caller
+ could not look a group up by the name it knows it by.
+
+``ueba_anomalies.event_type``
+ ``String(64)`` in the table against ``String(128)`` in the model, so an
+ event type between the two lengths passed every test and failed at
+ insert time in production. Widened to the declared length rather than
+ narrowing the model, which would only move the failure.
+
+The remaining differences are the harmless direction — ``observation_count``,
+``created_at`` and ``z_scores`` exist in the tables and are not mapped — and
+are recorded as such in ``scripts/check_orm_migration_parity.py`` rather than
+dropped. A column the models never name costs a default and nothing else;
+removing it would be an irreversible change made for tidiness.
+"""
+
+from __future__ import annotations
+
+import sqlalchemy as sa
+from alembic import op # type: ignore[attr-defined] # re-exported at runtime, absent from the stub
+
+revision = "0004"
+down_revision = "0003"
+branch_labels = None
+depends_on = None
+
+
+def upgrade() -> None:
+ # Guarded so the revision is safe on a database whose tables were created
+ # from the models instead of from this chain (``create_all`` in a test
+ # fixture, and the sandbox), where the column is already present.
+ op.execute("ALTER TABLE ueba_entity_baselines ADD COLUMN IF NOT EXISTS peer_group_id VARCHAR(64)")
+
+ # ``USING id::text`` rather than a drop-and-add: an existing row holds a
+ # group whose members' baselines point at it, and a generated UUID is not
+ # a name anything can look up again.
+ op.execute("ALTER TABLE ueba_peer_groups ALTER COLUMN id DROP DEFAULT")
+ op.execute("ALTER TABLE ueba_peer_groups ALTER COLUMN id TYPE VARCHAR(64) USING id::text")
+
+ op.alter_column(
+ "ueba_anomalies",
+ "event_type",
+ existing_type=sa.String(64),
+ type_=sa.String(128),
+ existing_nullable=False,
+ )
+
+
+def downgrade() -> None:
+ op.alter_column(
+ "ueba_anomalies",
+ "event_type",
+ existing_type=sa.String(128),
+ type_=sa.String(64),
+ existing_nullable=False,
+ )
+
+ # A group id that is not a UUID cannot be cast back to one, so the
+ # downgrade discards the names rather than failing halfway: anything this
+ # revision made storable is not representable in the old column. Stated
+ # here because a downgrade that silently drops rows is worse than one that
+ # says it will.
+ op.execute("DELETE FROM ueba_peer_groups WHERE id !~* '^[0-9a-f]{8}-([0-9a-f]{4}-){3}[0-9a-f]{12}$'")
+ op.execute("ALTER TABLE ueba_peer_groups ALTER COLUMN id TYPE UUID USING id::uuid")
+ op.execute("ALTER TABLE ueba_peer_groups ALTER COLUMN id SET DEFAULT gen_random_uuid()")
+
+ op.execute("ALTER TABLE ueba_entity_baselines DROP COLUMN IF EXISTS peer_group_id")
diff --git a/services/ueba/app/_health.py b/services/ueba/app/_health.py
index 974269731..0acb9e4da 100644
--- a/services/ueba/app/_health.py
+++ b/services/ueba/app/_health.py
@@ -48,21 +48,96 @@ Services using the deprecated ``@app.on_event("startup")`` /
``@app.on_event("shutdown")`` hooks call ``mark_ready()`` /
``mark_not_ready()`` from those handlers respectively.
+Readiness is a question, not a latch
+------------------------------------
+
+``mark_ready()`` answers "did startup finish", and for a service
+whose only job is to answer HTTP that is the whole story. For a
+service whose job is to *consume a topic* it is not: three
+services in this repository created the consumer as a
+fire-and-forget ``asyncio.create_task`` and called
+``mark_ready()`` on the next line, so readiness recorded that the
+task had been **created**, never that it was still running.
+
+UEBA is what that cost. Its handler raised
+``UndefinedColumnError`` on the first scoreable event, the
+exception left the ``async for``, the ``finally`` stopped the
+consumer, and the task ended — holding the exception, which
+nothing retrieved because the task was parked in a module global
+and so never garbage-collected. The container stayed ``running``
+with restarts 0, ``/health`` returned 200 and ``/readyz``
+returned 200, permanently. A consumer that has silently detached
+is indistinguishable from an idle one, and the whole point of a
+readiness probe is to tell those two apart.
+
+So a service that owns a subscription registers a probe for it
+with :func:`register_subscription`, and ``/readyz`` evaluates
+every probe on each request. A probe that returns false — or
+raises — is reported as detached and ``/readyz`` answers 503 with
+the names, which is a readiness failure rather than a liveness
+one on purpose: the orchestrator should stop routing to a
+half-working replica, and killing the process would only hide a
+schema or broker fault behind a restart loop.
+
+``/livez`` deliberately does **not** consult the probes. It
+answers "is the interpreter responsive", and wiring a detached
+consumer into it would make every orchestrator restart the pod on
+a fault a restart cannot fix.
+
This module is intentionally dependency-free (only FastAPI) so it
can be vendored into every service tree without import-graph
side-effects. Each service ships its own copy under
``services/<svc>/app/_health.py`` because services don't share a
-Python path — keep the file in sync via
-``scripts/sync_health_module.py`` (CI gate).
+Python path; ``scripts/audit_health_probes.py --check`` is the CI
+gate that every service has one and wires it in.
"""
from __future__ import annotations
+import json
from collections.abc import Callable
from typing import Any
from fastapi import FastAPI, Response
+#: Where the probes registered against an app are kept. Read by ``/readyz``,
+#: written by :func:`register_subscription`. On ``app.state`` rather than in a
+#: module global because a module global is shared by every app in the
+#: process, which makes two test clients interfere.
+_STATE_ATTR = "health_subscriptions"
+
+
+def _probe_detached(probes: dict[str, Callable[[], bool]]) -> list[str]:
+ """Names of the subscriptions currently reporting not-attached.
+
+ A probe that raises counts as detached. It is asking a live object
+ whether it is still working; if that question itself fails, the honest
+ answer is not "yes".
+ """
+ detached: list[str] = []
+ for name, probe in sorted(probes.items()):
+ try:
+ attached = bool(probe())
+ except Exception: # noqa: BLE001 — an unanswerable probe is not a healthy one
+ attached = False
+ if not attached:
+ detached.append(name)
+ return detached
+
+
+def register_subscription(app: FastAPI, name: str, probe: Callable[[], bool]) -> None:
+ """Make ``/readyz`` depend on *probe* returning true.
+
+ Call this once per subscription the service owns, after
+ :func:`install_health_routes`. *probe* must be cheap and non-blocking —
+ it runs on every ``/readyz`` request — so it should read a flag the
+ consumer maintains or check ``task.done()``, not round-trip to a broker.
+ """
+ probes = getattr(app.state, _STATE_ATTR, None)
+ if probes is None: # pragma: no cover — install_health_routes was not called
+ raise RuntimeError("register_subscription requires install_health_routes(app) first")
+ probes[name] = probe
+
def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[[], None], Callable[[], None]]:
"""Install ``/livez`` and ``/readyz`` on ``app``.
@@ -73,8 +148,14 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
``mark_ready()`` is called — exactly the right behaviour for a
rolling deploy where the orchestrator should hold traffic
until the pod's dependencies are connected.
+
+ Startup having finished is necessary and not sufficient: any
+ subscription registered with :func:`register_subscription`
+ must also still be attached. See the module docstring.
"""
state: dict[str, bool] = {"ready": False}
+ probes: dict[str, Callable[[], bool]] = {}
+ setattr(app.state, _STATE_ATTR, probes)
@app.get(
"/livez",
@@ -90,16 +171,30 @@ def install_health_routes(app: FastAPI, *, service_name: str) -> tuple[Callable[
include_in_schema=False,
)
async def _readyz() -> Response:
- if state["ready"]:
- return Response(
- content='{"status":"ready","service":"' + service_name + '"}',
- media_type="application/json",
- status_code=200,
- )
+ body: dict[str, Any] = {"status": "ready", "service": service_name}
+ status_code = 200
+
+ if not state["ready"]:
+ body["status"] = "starting"
+ status_code = 503
+ else:
+ detached = _probe_detached(probes)
+ if detached:
+ body["status"] = "degraded"
+ body["detached"] = detached
+ status_code = 503
+
+ # Named subscriptions are reported either way, so a 200 says which
+ # ones were checked rather than only that nothing was wrong. A probe
+ # list that is unexpectedly empty is then visible in the same place
+ # the verdict is.
+ if probes:
+ body["subscriptions"] = sorted(probes)
+
return Response(
- content='{"status":"starting","service":"' + service_name + '"}',
+ content=json.dumps(body),
media_type="application/json",
- status_code=503,
+ status_code=status_code,
)
def mark_ready() -> None:
diff --git a/services/ueba/app/_migrate.py b/services/ueba/app/_migrate.py
new file mode 100644
index 000000000..1297f113b
--- /dev/null
+++ b/services/ueba/app/_migrate.py
@@ -0,0 +1,228 @@
+"""Apply this service's alembic chain, then exec what comes after it.
+
+Why this exists
+---------------
+
+Four services in this repository own their schema through alembic —
+``ueba``, ``honeytokens``, ``purple-team`` and ``osquery-tls`` — and **no
+compose path invoked any of them**. Their container command was a plain
+``uvicorn``, so on the documented quickstart they booted healthy against
+empty schemas with none of their row-level-security policies installed. The
+only documented invocation was a manual step in a docs-portal quickstart
+whose own "start the stack" step does not start three of the four services,
+and whose first line failed outright.
+
+That is how two services shipped with no tables while every command
+reported success. The API does not have this problem because its chain runs
+from its own process on the way up; this is the same idea for the four that
+use alembic.
+
+What it guarantees
+------------------
+
+**Idempotent.** ``alembic upgrade head`` on a database already at head
+applies nothing. Running this on every container start is the point: there
+is no separate "have you migrated yet" state for an operator to get wrong.
+
+**Race-free.** Four chains start at once against one database. The chains
+take a shared transaction-scoped advisory lock (``MIGRATION_LOCK_KEY`` in
+each ``env.py``), so they queue instead of deadlocking on catalog locks.
+This wrapper deliberately holds no lock of its own — a lock held here would
+be a second mechanism to reason about, and it would not cover the window
+that actually matters.
+
+**Loud.** The service does not start unless the chain reached head. Not "the
+command exited 0": alembic exits 0 when it applies *nothing*, which is
+exactly what happened when four chains shared one ``alembic_version`` table
+and the second one to run believed it was already done. So ``current`` is
+read back and compared to ``heads``, and a mismatch — or an empty
+``current``, meaning the chain never ran — stops the container with the
+reason on stderr. A service booting on an empty schema is the failure this
+replaces, and it must not be reachable by a wrapper that only checks an exit
+code.
+
+Why the owner credential, and why a subprocess
+----------------------------------------------
+
+Alembic issues DDL. The runtime role every service serves requests as holds
+no ``CREATE`` on schema public, by design — that is what stops it turning
+row-level security off — so the chain must be applied as the owner. Each
+``env.py`` already resolves that (its own ``*_DATABASE_MIGRATION_URL``
+first, then ``DATABASE_MIGRATION_URL``), and this runs alembic as a
+subprocess precisely so ``env.py`` stays the single answer to "which
+credential". A second resolution here would be a second thing to keep in
+step, and the two would disagree the first time either changed.
+
+This module is vendored per service because services do not share a Python
+path. ``scripts/check_service_migration_bootstrap.py`` is the gate that each
+of the four ships a copy, keeps them identical, and runs one on the way up.
+
+Usage
+-----
+
+::
+
+ python -m app._migrate # apply the chain, exit
+ python -m app._migrate -- uvicorn app.main:app --port 8004
+"""
+
+from __future__ import annotations
+
+import os
+import subprocess
+import sys
+from pathlib import Path
+
+#: Where ``alembic.ini`` sits relative to the working directory in every one
+#: of the four images (``WORKDIR /app``, and the ini is copied to ``/app``).
+ALEMBIC_INI = "alembic.ini"
+
+#: Suffix that marks a DSN as the migration (owner) credential. Matches the
+#: convention each ``env.py`` documents and ``scripts/check_runtime_db_role.py``
+#: enforces on the deployment surfaces: a service-prefixed
+#: ``<SVC>_DATABASE_MIGRATION_URL`` wins over the bare form.
+MIGRATION_URL_SUFFIX = "DATABASE_MIGRATION_URL"
+
+
+def migration_credential_present(env: dict[str, str] | None = None) -> bool:
+ """Whether an owner DSN is configured.
+
+ Used only for the warning below. The chain is applied either way — a
+ deployment that has not split the roles yet behaves as it did before —
+ but it is announced, because a chain silently applied as the runtime
+ credential fails on the first ``CREATE TABLE`` and the message names a
+ permission rather than a configuration.
+ """
+ source = os.environ if env is None else env
+ return any(name.endswith(MIGRATION_URL_SUFFIX) and value.strip() for name, value in source.items())
+
+
+def _alembic(args: list[str], *, capture: bool) -> subprocess.CompletedProcess[str]:
+ return subprocess.run( # noqa: S603 — fixed argv, no shell
+ [sys.executable, "-m", "alembic", "-c", ALEMBIC_INI, *args],
+ capture_output=capture,
+ text=True,
+ check=False,
+ )
+
+
+def _revisions(output: str) -> set[str]:
+ """Revision ids in ``alembic current`` / ``alembic heads`` output.
+
+ Both print ``<rev> (head)`` or ``<rev>``, one per line, with informational
+ lines alembic sends to stderr. An empty set means the version table holds
+ nothing — the chain has not run — which is the case this exists to catch
+ and is therefore never treated as agreement.
+ """
+ found: set[str] = set()
+ for line in output.splitlines():
+ token = line.strip().split(" ")[0].strip()
+ if token and not token.startswith(("INFO", "WARNING", "ERROR", "alembic:")):
+ found.add(token)
+ return found
+
+
+def apply_chain() -> int:
+ """Apply the chain and verify it reached head. Returns an exit code."""
+ if not Path(ALEMBIC_INI).is_file():
+ print(
+ f"migrate: {ALEMBIC_INI} not found in {Path.cwd()}. This service's image must "
+ f"copy its alembic config and chain; see its Dockerfile.",
+ file=sys.stderr,
+ )
+ return 1
+
+ if not migration_credential_present():
+ print(
+ f"migrate: no *{MIGRATION_URL_SUFFIX} is set, so the chain will be applied as "
+ f"the runtime credential. That works only while the two are the same role; "
+ f"under the DML-only runtime role the first CREATE TABLE fails with "
+ f"'permission denied for schema public'.",
+ file=sys.stderr,
+ )
+
+ upgrade = _alembic(["upgrade", "head"], capture=False)
+ if upgrade.returncode != 0:
+ print(
+ f"migrate: 'alembic upgrade head' failed (exit {upgrade.returncode}). Refusing to "
+ f"start: this service would serve requests against whatever schema is there now.",
+ file=sys.stderr,
+ )
+ return upgrade.returncode
+
+ # Read back rather than trusting the exit code. `alembic upgrade head`
+ # exits 0 having applied nothing when it believes the database is already
+ # at head, and believing that wrongly is the documented way these four
+ # services ended up with no tables.
+ current = _alembic(["current"], capture=True)
+ heads = _alembic(["heads"], capture=True)
+ if current.returncode != 0 or heads.returncode != 0:
+ print(
+ "migrate: could not read back the applied revision "
+ f"(current exit {current.returncode}, heads exit {heads.returncode}). "
+ "Refusing to start: an unverified migration is the state this check exists "
+ "to reject.\n"
+ f"{(current.stderr or '') + (heads.stderr or '')}",
+ file=sys.stderr,
+ )
+ return 1
+
+ applied = _revisions(current.stdout)
+ expected = _revisions(heads.stdout)
+
+ if not expected:
+ print(
+ "migrate: this chain declares no head revision, so there is nothing to verify "
+ "against. Refusing to start rather than reporting an empty chain as applied.",
+ file=sys.stderr,
+ )
+ return 1
+
+ if not applied:
+ print(
+ f"migrate: 'alembic upgrade head' reported success but the version table is "
+ f"empty — the chain never ran. Expected {sorted(expected)}. Refusing to start "
+ f"on an unmigrated schema.",
+ file=sys.stderr,
+ )
+ return 1
+
+ if not expected.issubset(applied):
+ print(
+ f"migrate: chain did not reach head. At {sorted(applied)}, head is " f"{sorted(expected)}. Refusing to start.",
+ file=sys.stderr,
+ )
+ return 1
+
+ print(f"migrate: schema at head {sorted(expected)}", file=sys.stderr)
+ return 0
+
+
+def main(argv: list[str] | None = None) -> int:
+ args = list(sys.argv[1:] if argv is None else argv)
+
+ # Everything after `--` is the real service command. Exec'd rather than
+ # spawned so the server keeps PID 1 and receives SIGTERM directly — a
+ # Python parent waiting on a child swallows the orchestrator's stop
+ # signal and turns every shutdown into a ten-second kill.
+ command: list[str] = []
+ if "--" in args:
+ split = args.index("--")
+ command = args[split + 1 :]
+ args = args[:split]
+
+ if args:
+ print(f"migrate: unexpected arguments {args}; usage: python -m app._migrate [-- CMD ...]", file=sys.stderr)
+ return 2
+
+ code = apply_chain()
+ if code != 0:
+ return code
+
+ if command:
+ os.execvp(command[0], command) # noqa: S606 — replaces this process by design
+ return 0
+
+
+if __name__ == "__main__":
+ sys.exit(main())
diff --git a/services/ueba/app/main.py b/services/ueba/app/main.py
index a9166d041..bec289484 100644
--- a/services/ueba/app/main.py
+++ b/services/ueba/app/main.py
@@ -3,12 +3,14 @@
from __future__ import annotations
import asyncio
+import json
import logging
+from typing import Any
-from fastapi import FastAPI
+from fastapi import FastAPI, Response
from fastapi.middleware.cors import CORSMiddleware
-from app._health import install_health_routes
+from app._health import install_health_routes, register_subscription
from app.core.config import settings
from app.core.cors import build_cors_kwargs
@@ -53,8 +55,12 @@ app = FastAPI(
)
# Phase 2.6 — k8s liveness + readiness probes (see app/_health.py).
-# /readyz flips on once the Kafka consumer task is created in the
-# startup hook.
+#
+# /readyz used to flip on the moment the Kafka consumer task was *created*,
+# which is not the same thing as it running: the first scoreable event raised
+# UndefinedColumnError, the task ended, and this endpoint kept answering 200.
+# The startup hook now registers a probe over the consumer itself, so
+# readiness is re-evaluated per request instead of latched once.
_mark_ready, _mark_not_ready = install_health_routes(app, service_name="aisoc-ueba")
app.state.mark_ready = _mark_ready
app.state.mark_not_ready = _mark_not_ready
@@ -77,15 +83,49 @@ if _otel_enabled:
# Kafka consumer lifecycle
# ---------------------------------------------------------------------------
_consumer_task: asyncio.Task | None = None # type: ignore[type-arg]
+_consumer: Any = None
+
+
+def _log_consumer_exit(task: asyncio.Task) -> None: # type: ignore[type-arg]
+ """Retrieve the consumer task's outcome and say what it was.
+
+ This callback is the reason a dead consumer is now findable. The task was
+ parked in a module global, so it was never garbage-collected, so asyncio
+ never emitted its "Task exception was never retrieved" warning — the
+ exception that stopped UEBA from ever writing an anomaly existed only
+ inside an object nobody read.
+ """
+ if task.cancelled():
+ LOG.info("UEBA Kafka consumer task cancelled (shutdown).")
+ return
+ exc = task.exception()
+ if exc is None:
+ LOG.error(
+ "UEBA Kafka consumer task ended without an error. The subscription is gone "
+ "and no further events will be scored; /readyz now reports 503."
+ )
+ return
+ LOG.error(
+ "UEBA Kafka consumer task died: %s: %s. The subscription is gone and no further " "events will be scored; /readyz now reports 503.",
+ type(exc).__name__,
+ exc,
+ exc_info=exc,
+ )
@app.on_event("startup")
async def _start_kafka() -> None:
- global _consumer_task
+ global _consumer_task, _consumer
from app.services.kafka_consumer import UEBAKafkaConsumer
- consumer = UEBAKafkaConsumer()
- _consumer_task = asyncio.create_task(consumer.run(), name="ueba-kafka-consumer")
+ _consumer = UEBAKafkaConsumer()
+ _consumer_task = asyncio.create_task(_consumer.run(), name="ueba-kafka-consumer")
+ _consumer_task.add_done_callback(_log_consumer_exit)
+
+ # Readiness depends on the consumer still being attached, not on this
+ # function having reached its last line.
+ register_subscription(app, settings.kafka_input_topic, lambda: bool(_consumer and _consumer.attached))
+
LOG.info("UEBA service started (OTel=%s)", _otel_enabled)
# Phase 2.6 — Kafka consumer is up; HTTP surface is serving.
app.state.mark_ready()
@@ -105,5 +145,33 @@ async def _stop_kafka() -> None:
@app.get("/health")
-async def health() -> dict:
- return {"status": "ok", "service": settings.service_name}
+async def health() -> Response:
+ """Service status, including whether the consumer is actually consuming.
+
+ This endpoint returned a hardcoded ``{"status": "ok"}`` and so reported
+ exactly the same thing whether UEBA was scoring every event or had been
+ detached from its topic since the first one. It now answers 503 when the
+ consumer is not attached, and carries the counters — including the last
+ handler error — that say which of the two it is.
+ """
+ attached = bool(_consumer and _consumer.attached)
+ body: dict[str, Any] = {
+ "status": "ok" if attached else "degraded",
+ "service": settings.service_name,
+ "consumer": {
+ "topic": settings.kafka_input_topic,
+ "attached": attached,
+ **(_consumer.stats() if _consumer else {}),
+ },
+ }
+ if not attached:
+ body["consumer"]["hint"] = (
+ "the subscription is not attached — see this service's logs for the "
+ "consumer task's exception, and GET /api/v1/health/dead-letters for "
+ "events it refused"
+ )
+ return Response(
+ content=json.dumps(body),
+ media_type="application/json",
+ status_code=200 if attached else 503,
+ )
diff --git a/services/ueba/app/services/baseline.py b/services/ueba/app/services/baseline.py
index a209e4e9f..f0ed35e9b 100644
--- a/services/ueba/app/services/baseline.py
+++ b/services/ueba/app/services/baseline.py
@@ -27,7 +27,7 @@ def _welford_update(
feature: str,
value: float,
) -> dict[str, dict[str, float]]:
- """Update incremental mean/variance for *feature* with new *value*.
+ """Return *stats* with *feature* advanced by one observation of *value*.
Uses Welford's online algorithm:
n ← n + 1
@@ -36,11 +36,32 @@ def _welford_update(
δ2 ← x - mean
M2 ← M2 + δ·δ2
variance = M2 / (n−1) for n > 1
+
+ Returns new dictionaries and mutates neither *stats* nor the per-feature
+ dictionaries inside it. That is not a style preference — it is the
+ difference between a baseline that accumulates and one that does not.
+
+ This used to edit ``stats[feature]`` in place. Its caller passed a
+ *shallow* copy of ``EntityBaseline.feature_stats``, so the per-feature
+ dictionary being edited was the very object SQLAlchemy had loaded from
+ the column. By the time the caller assigned the result back, the loaded
+ value and the new value were equal, and SQLAlchemy decides whether to
+ include a column in an ``UPDATE`` by comparing exactly those two — a
+ ``JSON``/``JSONB`` column has no change tracking of its own unless it is
+ wrapped in ``MutableDict``. So the statement went out without the column.
+
+ Measured on a live stack: 36 events for one entity, every one processed
+ without error, and the stored baseline stayed at ``count: 1`` with the
+ first event's mean while ``window_end`` — an ordinary timestamp column,
+ with a value that genuinely differed — advanced on every one. No entity
+ could ever reach ``min_baseline_samples``, so ``compute_z_score``
+ returned ``None`` for every feature of every event, so no anomaly was
+ ever scored. The service logged ``features_unscoreable`` 36 times, which
+ reads as "this entity is too quiet to score" rather than as a bug.
"""
- if feature not in stats:
- stats[feature] = {"mean": 0.0, "M2": 0.0, "count": 0}
+ current = stats.get(feature)
+ s: dict[str, float] = dict(current) if current else {"mean": 0.0, "M2": 0.0, "count": 0}
- s = stats[feature]
s["count"] += 1
n = s["count"]
delta = value - s["mean"]
@@ -52,7 +73,7 @@ def _welford_update(
variance = s["M2"] / (n - 1) if n > 1 else 0.0
s["std"] = math.sqrt(variance)
- return stats
+ return {**stats, feature: s}
def compute_z_score(
@@ -145,7 +166,10 @@ class BaselineService:
) -> EntityBaseline:
"""Incrementally update the baseline with new feature observations."""
baseline = await self.get_or_create(tenant_id, entity_type, entity_id)
- stats = dict(baseline.feature_stats) # copy so SQLAlchemy detects mutation
+ # A shallow copy is enough only because ``_welford_update`` replaces
+ # the per-feature dictionary rather than editing it; see its docstring
+ # for what happens when it does not.
+ stats = dict(baseline.feature_stats)
for feature, value in features.items():
stats = _welford_update(stats, feature, value)
diff --git a/services/ueba/app/services/dead_letters.py b/services/ueba/app/services/dead_letters.py
new file mode 100644
index 000000000..52076669e
--- /dev/null
+++ b/services/ueba/app/services/dead_letters.py
@@ -0,0 +1,125 @@
+"""Record events this service refused, where an operator already looks.
+
+The UEBA consumer had no ``except`` around its handler at all, so the first
+event that raised ended the ``async for``, ran the ``finally`` that stopped
+the consumer, and left the container ``running`` with ``/health`` answering
+200. Nothing was logged, because the task holding the exception lived in a
+module global and was never garbage-collected, so asyncio's
+"exception was never retrieved" warning never fired either.
+
+Guarding the handler is half the fix; the other half is that a guarded
+failure has to go somewhere. It goes to ``aisoc_dead_letters`` — the table
+``services/fusion`` already writes and ``GET /api/v1/health/dead-letters``
+already reads, with a per-reason breakdown that turns fifty identical rows
+into one finding. A second dead-letter path would mean an operator had to
+know to check two.
+
+Two properties, both inherited deliberately from
+``services/fusion/app/services/dlq_sink.py``:
+
+**A failure to record must not become a second failure.** The sink exists to
+let the consumer keep going; if writing the record can raise, one bad event
+stops the stream anyway. Every error here is logged and swallowed.
+
+**The payload is truncated, never stored whole.** It is the thing that was
+refused, so it is untrusted by definition and may be large.
+"""
+
+from __future__ import annotations
+
+import json
+from typing import Any
+
+import structlog
+from sqlalchemy import text
+from sqlalchemy.ext.asyncio import async_sessionmaker
+
+logger = structlog.get_logger(__name__)
+
+#: Characters of payload kept. Matches the fusion sink so the same column
+#: holds comparable rows: enough to recognise the shape and find the
+#: producer, far short of enough to replay it.
+MAX_EXCERPT = 2000
+
+#: This service's schema version marker on the shared table. The column is
+#: ``NOT NULL DEFAULT 'unknown'`` and "unknown" would be a worse answer than
+#: naming the envelope the consumer was reading.
+SCHEMA_VERSION = "ueba.raw_events.v1"
+
+_INSERT_SQL = text(
+ """
+ INSERT INTO aisoc_dead_letters
+ (tenant_id, topic, reason, schema_version, payload_excerpt, source_event_id)
+ VALUES (:tenant_id, :topic, :reason, :schema_version, :payload_excerpt, :source_event_id)
+ """
+)
+
+
+def _excerpt(payload: Any) -> str:
+ try:
+ return json.dumps(payload, default=str)[:MAX_EXCERPT]
+ except (TypeError, ValueError):
+ return str(payload)[:MAX_EXCERPT]
+
+
+class DeadLetterSink:
+ """Persist a refused event to the shared dead-letter table."""
+
+ def __init__(self, session_factory: async_sessionmaker) -> None:
+ self._session_factory = session_factory
+
+ async def record(
+ self,
+ *,
+ topic: str,
+ reason: str,
+ payload: Any,
+ tenant_id: str | None = None,
+ source_event_id: str | None = None,
+ ) -> bool:
+ """Write one dead letter. Returns whether it landed.
+
+ The return value is for the caller's counters, not for control flow —
+ there is no branch here a consumer should take differently.
+ """
+ params = {
+ "tenant_id": tenant_id,
+ "topic": topic,
+ "reason": reason[:1000],
+ "schema_version": SCHEMA_VERSION,
+ "payload_excerpt": _excerpt(payload),
+ "source_event_id": source_event_id,
+ }
+
+ # Two attempts, and the second is not a retry of the same statement.
+ # ``tenant_id`` carries a foreign key to ``tenants``, and UEBA's
+ # tenants come from the event envelope rather than from a join — an
+ # event naming a tenant this database has never seen would otherwise
+ # lose the dead letter to a constraint violation, and that event is
+ # precisely the evidence that something upstream is producing them.
+ # The reason string already names the tenant, so nothing is lost by
+ # detaching the column.
+ last_error: Exception | None = None
+ for attempt_params in (params, {**params, "tenant_id": None}):
+ try:
+ async with self._session_factory() as session:
+ async with session.begin():
+ await session.execute(_INSERT_SQL, attempt_params)
+ return True
+ except Exception as exc: # noqa: BLE001 — see the module docstring
+ last_error = exc
+ if attempt_params["tenant_id"] is None:
+ break
+
+ logger.warning(
+ "ueba.dead_letter_persist_failed",
+ topic=topic,
+ reason=reason[:200],
+ error=str(last_error)[:300],
+ hint=(
+ "the refused event is in the log line above this one; "
+ "aisoc_dead_letters is created by services/api migration "
+ "053_dead_letters.sql"
+ ),
+ )
+ return False
diff --git a/services/ueba/app/services/kafka_consumer.py b/services/ueba/app/services/kafka_consumer.py
index 609b6c9cd..4acf5e08f 100644
--- a/services/ueba/app/services/kafka_consumer.py
+++ b/services/ueba/app/services/kafka_consumer.py
@@ -34,6 +34,31 @@ Anomalies are published to ``ueba.anomalies``:
"features": {...},
"detected_at": "2026-05-03T12:00:01Z"
}
+
+What happens when the handler raises
+------------------------------------
+
+It logs at ``error``, dead-letters the event, and carries on. A single
+unprocessable event must not stop the stream, which is already the policy in
+``services/fusion`` and ``services/agents`` — one answer across the three
+consumers is worth more than a cleverer answer here.
+
+What must never happen again is the previous behaviour. There was no
+``except`` at all: the first event that raised left the ``async for``, the
+``finally`` stopped the consumer, and the exception went into the task
+object. Because the task was held in a module global it was never
+garbage-collected, so asyncio never emitted "Task exception was never
+retrieved". The container stayed ``running`` with restarts 0, ``/health``
+answered 200, and ``/readyz`` answered 200, for as long as anyone left it up.
+
+Continuing is only safe if a subscription that *does* end says so, so this
+class tracks whether it is attached and ``app/main.py`` registers that as a
+readiness probe. Detaching is not treated as recoverable in place: there is
+no reconnect loop here, because a loop would turn a permanent fault — a
+missing column, a revoked grant — into an indefinite retry that reads as
+ordinary churn. The honest report is a 503 on ``/readyz``, the exception in
+the log, and the restart decision left to whatever is supervising the
+container.
"""
from __future__ import annotations
@@ -46,6 +71,7 @@ from typing import Any
from sqlalchemy.ext.asyncio import async_sessionmaker, create_async_engine
from app.core.config import settings
+from app.services.dead_letters import DeadLetterSink
from app.services.feature_extraction import extract
from app.services.peer_group import PeerGroupService
from app.services.scoring import ScoringService
@@ -58,6 +84,27 @@ class UEBAKafkaConsumer:
self._engine = create_async_engine(settings.database_url, pool_size=5)
self._session_factory = async_sessionmaker(self._engine, expire_on_commit=False)
self._running = False
+ # True only while the ``async for`` is iterating a started consumer.
+ # Read through ``attached`` by the readiness probe; set False in the
+ # loop's ``finally`` so every exit path — clean stop, broker loss,
+ # unhandled error — is reported the same way.
+ self._attached = False
+ self._dlq = DeadLetterSink(self._session_factory)
+ self._stats: dict[str, Any] = {
+ "processed": 0,
+ "handler_failures": 0,
+ "dead_lettered": 0,
+ "last_error": None,
+ }
+
+ @property
+ def attached(self) -> bool:
+ """Whether this consumer is currently subscribed and iterating."""
+ return self._attached
+
+ def stats(self) -> dict[str, Any]:
+ """Counters for ``/health``. A copy, so a caller cannot edit them."""
+ return dict(self._stats)
async def _process_message(self, raw: bytes, producer: Any) -> None:
try:
@@ -151,17 +198,74 @@ class UEBAKafkaConsumer:
await consumer.start()
await producer.start()
self._running = True
+ self._attached = True
LOG.info("UEBA Kafka consumer started (topic=%s)", settings.kafka_input_topic)
try:
async for msg in consumer:
if not self._running:
break
- await self._process_message(msg.value, producer)
+ try:
+ await self._process_message(msg.value, producer)
+ except Exception as exc: # noqa: BLE001 — one event must not end the subscription
+ await self._handler_failed(msg, exc)
+ else:
+ self._stats["processed"] += 1
finally:
+ # Before the awaits below, not after: ``consumer.stop()`` can
+ # itself raise on a broker that has gone away, and readiness must
+ # already be reporting detached by then rather than staying 200
+ # through a failed teardown.
+ self._attached = False
await consumer.stop()
await producer.stop()
LOG.info("UEBA Kafka consumer stopped.")
+ async def _handler_failed(self, msg: Any, exc: Exception) -> None:
+ """Report one unprocessable event and keep the subscription.
+
+ Loud on purpose. This is the exact point the old code exited from
+ without writing anything anywhere, so the log line carries the
+ partition and offset needed to find the event again, and
+ ``exc_info`` carries the traceback that says which column or grant
+ was missing.
+ """
+ self._stats["handler_failures"] += 1
+ self._stats["last_error"] = f"{type(exc).__name__}: {str(exc)[:300]}"
+
+ topic = getattr(msg, "topic", settings.kafka_input_topic)
+ partition = getattr(msg, "partition", None)
+ offset = getattr(msg, "offset", None)
+ raw = getattr(msg, "value", msg)
+
+ LOG.error(
+ "UEBA handler failed; event dead-lettered and the subscription kept. " "topic=%s partition=%s offset=%s error=%s: %s",
+ topic,
+ partition,
+ offset,
+ type(exc).__name__,
+ exc,
+ exc_info=True,
+ )
+
+ payload: Any
+ try:
+ payload = json.loads(raw) if isinstance(raw, bytes | bytearray) else raw
+ except (json.JSONDecodeError, UnicodeDecodeError):
+ payload = {"undecodable": str(raw)[:500]}
+
+ tenant_id = payload.get("tenant_id") if isinstance(payload, dict) else None
+ source_event_id = payload.get("event_id") if isinstance(payload, dict) else None
+
+ recorded = await self._dlq.record(
+ topic=str(topic),
+ reason=f"{type(exc).__name__}: {str(exc)[:400]}",
+ payload=payload,
+ tenant_id=str(tenant_id) if tenant_id else None,
+ source_event_id=str(source_event_id) if source_event_id else None,
+ )
+ if recorded:
+ self._stats["dead_lettered"] += 1
+
def stop(self) -> None:
self._running = False
diff --git a/services/ueba/tests/test_baseline_scoring.py b/services/ueba/tests/test_baseline_scoring.py
index e83d1b478..e3116b8a5 100644
--- a/services/ueba/tests/test_baseline_scoring.py
+++ b/services/ueba/tests/test_baseline_scoring.py
@@ -15,6 +15,7 @@ raise an anomaly.
from __future__ import annotations
+import json
import math
import pytest
@@ -192,6 +193,72 @@ class TestUnscoreableFeatures:
assert _unscoreable_features({"a": {"z_score": 1.5}}) == []
+class TestWelfordDoesNotMutateItsInput:
+ """The property that decides whether a baseline accumulates at all.
+
+ ``_welford_update`` is correct arithmetic either way, which is why the
+ existing tests above passed while no entity in any deployment ever got
+ past one observation. The caller hands it a shallow copy of the JSONB
+ column SQLAlchemy loaded; editing the per-feature dictionary in place
+ therefore edited the loaded value too, so the new value compared equal to
+ the old one and SQLAlchemy left the column out of the ``UPDATE``
+ entirely — a ``JSON`` column has no change tracking unless it is wrapped
+ in ``MutableDict``.
+
+ These assert the shape of the return value rather than the numbers, so
+ they fail if anyone reintroduces in-place editing even if the statistics
+ stay right.
+ """
+
+ def test_leaves_the_dict_it_was_given_untouched(self):
+ original = {"f": {"mean": 10.0, "M2": 0.0, "count": 1, "std": 0.0}}
+ before = json.dumps(original, sort_keys=True)
+
+ _welford_update(original, "f", 20.0)
+
+ assert json.dumps(original, sort_keys=True) == before
+
+ def test_returns_a_distinct_per_feature_dict(self):
+ # Identity, not equality: a returned dict that *is* the loaded one is
+ # exactly what made the change invisible to the flush.
+ original = {"f": {"mean": 10.0, "M2": 0.0, "count": 1, "std": 0.0}}
+
+ updated = _welford_update(original, "f", 20.0)
+
+ assert updated is not original
+ assert updated["f"] is not original["f"]
+
+ def test_a_shallow_copy_of_the_previous_value_still_differs(self):
+ """The caller's exact pattern, asserted end to end.
+
+ ``BaselineService.update`` does ``dict(baseline.feature_stats)`` and
+ assigns the result back. If the two compare equal, the column is not
+ written and the baseline is frozen at its first observation.
+ """
+ loaded = {"f": {"mean": 10.0, "M2": 0.0, "count": 1, "std": 0.0}}
+
+ new_value = _welford_update(dict(loaded), "f", 20.0)
+
+ assert new_value != loaded
+ assert new_value["f"]["count"] == 2
+
+ def test_accumulates_across_repeated_round_trips(self):
+ """Thirty observations must produce a baseline of thirty.
+
+ ``min_baseline_samples`` defaults to 30, so an updater that does not
+ accumulate makes ``compute_z_score`` return ``None`` forever and no
+ anomaly is ever scored. Measured on a live stack before the fix: 36
+ events, ``count: 1``.
+ """
+ stored: dict[str, dict[str, float]] = {}
+ for value in range(30):
+ # Round-tripped through JSON each time, as a JSONB column does.
+ stored = json.loads(json.dumps(_welford_update(dict(stored), "f", float(value))))
+
+ assert stored["f"]["count"] == 30
+ assert stored["f"]["std"] > 0
+
+
class TestRiskLevel:
"""`critical` and `high` are hard-coded; only `medium` follows settings."""