Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

feat(deploy): ship the production compose the docs have always pointed at (#946)

Beenu Arora01f0a4af2a72
diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml
index e4e07121d..ec37cb05e 100644
--- a/.github/workflows/integration.yml
+++ b/.github/workflows/integration.yml
@@ -190,10 +190,23 @@ jobs:
printf '\nAISOC_VERSION=%s\n' "${AISOC_VERSION}" >> .env
fi
+ # Copying .env.example leaves every generated secret empty, which is
+ # what `make up` runs ensure_env.py to fix. Without it this job
+ # provisions an environment no real deployment ever has: v12.0.0 made
+ # an empty AISOC_REALTIME_JWT_SECRET fatal — the ticket endpoint
+ # answers 503 and the realtime edge rejects every connection, by
+ # design — and the spine test failed at mint_ws_ticket. It was not
+ # caught on main because this job is path-filtered and had not run on
+ # a commit that triggers it since that release.
+ python3 scripts/ensure_env.py
+ python3 scripts/ensure_env.py --check
+
token="$(openssl rand -hex 32)"
echo "::add-mask::$token"
# Written into .env so compose substitutes it into the ingest and
# connectors services; exported so the spine test presents it.
+ # Overwrites the value ensure_env just generated, deliberately: the
+ # test has to present the same token the services verify.
sed -i "s|^AISOC_SERVICE_TOKEN=.*|AISOC_SERVICE_TOKEN=${token}|" .env
grep -q "^AISOC_SERVICE_TOKEN=${token}$" .env || {
echo "::error::AISOC_SERVICE_TOKEN was not written into .env — check that .env.example still declares it"
diff --git a/apps/docs/docs/deployment/docker.md b/apps/docs/docs/deployment/docker.md
index a3e8b84ce..af845db84 100644
--- a/apps/docs/docs/deployment/docker.md
+++ b/apps/docs/docs/deployment/docker.md
@@ -10,7 +10,20 @@ AiSOC ships three Compose flavors. Pick the one that matches what you are doing.
|------|---------|-------------|
| `infra/compose/docker-compose.demo.yml` | Streamlined demo with seeded data | Trying AiSOC for the first time |
| `docker-compose.yml` | Full developer stack | Active development against real source |
-| `docker-compose.prod.yml` | Production-leaning stack | Self-hosting on a single VM |
+| `docker-compose.prod.yml` | Production stack | Self-hosting on a single VM |
+
+:::warning The development stack has no effective authentication
+
+`docker-compose.yml` defaults `ENVIRONMENT` to `development`, and in a
+development-class environment `services/api/app/api/v1/dev_auth.py` resolves a
+request that arrives without a bearer token to a demo user whose role is
+`admin`. That is deliberate for a laptop and unsafe for anything else.
+
+Use `docker-compose.prod.yml` for anything network-reachable. It fixes
+`ENVIRONMENT` to `production` so the bypass cannot be switched back on by
+configuration.
+
+:::
> If you don't already have Docker installed, the simplest path is the
> [one-click installer](../installation) — it installs Docker (Engine + Compose
@@ -103,20 +116,56 @@ COMPOSE_PROFILES=connectors,osquery,slack docker compose up -d
## Production
-Use the production compose file:
-
```bash
+make env # generates the six service secrets
docker compose -f docker-compose.prod.yml up -d
```
+The production file `include`s `docker-compose.yml` and overrides what
+production changes, so there is one definition of every service and the two
+cannot drift apart. It differs from the development stack in three ways that
+matter:
+
+- **The auth bypass is unreachable.** `ENVIRONMENT` and `AISOC_DEV_MODE` are
+ fixed values, not interpolated, so no `.env` can re-enable the demo-admin
+ shim described above.
+- **Nothing starts on a default credential.** Every secret is declared
+ `${VAR:?...}`, so Compose refuses to start and names the variable rather than
+ booting on a literal published in this repository and looking healthy.
+- **No datastore is bound to the host.** Postgres, Redis, Kafka, Qdrant, Neo4j,
+ ClickHouse and OpenSearch publish no ports. `kafka-ui` browses every topic
+ with no authentication of its own, so `--profile full` does not start it —
+ ask for it by name with `--profile debug-ui` if you want it.
+
+CORE is 16 long-running services here, the same as development. `--profile full`
+is 21 rather than 22, the difference being `kafka-ui`.
+
+### Required variables
+
+`make env` generates the first six. The rest are yours to choose, and
+`docker compose ... config` will name any you miss before anything starts.
+
+| Variable | Generated by `make env` | Used for |
+|---|---|---|
+| `SECRET_KEY` | yes | Console session signing |
+| `AISOC_CREDENTIAL_KEY` | yes | Connector credential vault |
+| `AISOC_SERVICE_TOKEN` | yes | Service-to-service calls |
+| `AISOC_ACTIONS_SERVICE_TOKEN` | yes | Response actions |
+| `AISOC_REALTIME_JWT_SECRET` | yes | Realtime WebSocket/SSE tickets |
+| `REALTIME_INTERNAL_TOKEN` | yes | Realtime `/internal/*` fan-out |
+| `POSTGRES_PASSWORD` | no | Postgres owner |
+| `AISOC_APP_DB_PASSWORD` | no | DML-only runtime role |
+| `REDIS_PASSWORD` | no | Redis |
+| `JWT_SECRET` | no | Ingest token signing |
+| `NEO4J_PASSWORD`, `CLICKHOUSE_PASSWORD` | no | `full` profile stores |
+
+Put a TLS-terminating reverse proxy in front: the console and ingest bind to
+loopback unless you set `AISOC_BIND_ADDR` deliberately.
+
Before going live, walk through the
[Hardening Runbook](https://github.com/beenuar/AiSOC/blob/main/docs/runbooks/HARDENING.md)
— TLS termination, secret rotation, network policies, audit log forwarding,
-and tenant-scoped backups all need to be in place.
-
-### Environment variables
-
-Copy `.env.example` to `.env` and fill in every required value before starting.
+and tenant-scoped backups are not things a compose file can do for you.
See [Environment Variables](./env-vars) for the full reference.
## Building images
diff --git a/docker-compose.prod.yml b/docker-compose.prod.yml
new file mode 100644
index 000000000..ed6229bba
--- /dev/null
+++ b/docker-compose.prod.yml
@@ -0,0 +1,213 @@
+# AiSOC — production compose.
+#
+# docker compose -f docker-compose.prod.yml up -d
+#
+# This is the file `apps/docs/docs/deployment/docker.md` has pointed at since
+# the deployment page was written. It did not exist, which is what discussion
+# #629 reported: the documented production command failed on a missing file,
+# and the only stack an operator could actually start was the development one,
+# where `ENVIRONMENT` defaults to `development` and
+# `services/api/app/api/v1/dev_auth.py` resolves an unauthenticated request to
+# a demo user whose role is `admin`.
+#
+# It uses the Compose Spec `include` directive, the same way
+# `infra/compose/docker-compose.dev.yml` does, so there is exactly one
+# definition of every service and this file holds only what production changes.
+# A standalone copy would have drifted from the base within a release, which is
+# the failure this repository keeps finding.
+#
+# WHAT THIS FILE GUARANTEES
+#
+# * The authentication bypass is unreachable. `ENVIRONMENT` and `AISOC_DEV_MODE`
+# are fixed values here, not interpolated, so no `.env` can re-enable them.
+# * Nothing starts on a default credential. Every secret uses `${VAR:?...}`,
+# which makes Compose refuse to start and name the variable, rather than
+# booting on `aisoc_dev_secret` and looking healthy.
+# * No datastore is reachable from the host. Postgres, Redis, Kafka and
+# Qdrant publish no ports at all; the console and the ingest endpoint are
+# the only things bound, and they bind to loopback unless you set
+# AISOC_BIND_ADDR deliberately (put a TLS-terminating proxy in front —
+# see docs/runbooks/HARDENING.md section 2).
+#
+# WHAT IT DOES NOT DO
+#
+# TLS, secret rotation, network policies, audit-log forwarding and backups
+# are not in a compose file. `docs/runbooks/HARDENING.md` is the checklist
+# and this file is the starting posture, not the finished one.
+
+include:
+ - ./docker-compose.yml
+
+services:
+ # ---------------------------------------------------------------------------
+ # Data plane. No published ports: in the development stack these are bound to
+ # 127.0.0.1 so a developer can attach a client, which is reasonable there and
+ # is an attack surface here. `ports: []` overrides the base list rather than
+ # appending to it.
+ # ---------------------------------------------------------------------------
+ postgres:
+ ports: !reset []
+ restart: unless-stopped
+ environment:
+ POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD — the development default is a published literal}
+ # The DML-only runtime role the services connect as, so the row-level
+ # security policies actually apply to them. Separate from the owner
+ # password above and rotated the same way.
+ AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:?set AISOC_APP_DB_PASSWORD — the development default is a published literal}
+
+ redis:
+ ports: !reset []
+ restart: unless-stopped
+
+ zookeeper:
+ ports: !reset []
+ restart: unless-stopped
+
+ kafka:
+ ports: !reset []
+ restart: unless-stopped
+
+ qdrant:
+ ports: !reset []
+ restart: unless-stopped
+
+ # ---------------------------------------------------------------------------
+ # Application services. `ENVIRONMENT` and `AISOC_DEV_MODE` are literals on
+ # purpose: an interpolated value here would let a stray `.env` put a
+ # production deployment back into the environment where the auth bypass
+ # lives, which is the defect this file exists to make impossible.
+ # ---------------------------------------------------------------------------
+ api:
+ restart: unless-stopped
+ environment:
+ ENVIRONMENT: production
+ AISOC_DEV_MODE: 0
+ POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}
+ AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:?set AISOC_APP_DB_PASSWORD}
+ DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:?set AISOC_APP_DB_PASSWORD}@postgres:5432/aisoc
+ DATABASE_MIGRATION_URL: postgresql+asyncpg://aisoc:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}@postgres:5432/aisoc
+ REDIS_URL: redis://:${REDIS_PASSWORD:?set REDIS_PASSWORD}@redis:6379/0
+ # `make env` generates all six of these. They are required rather than
+ # defaulted because a service that starts without one either serves
+ # unauthenticated or signs with a key published in this repository —
+ # both were reported vulnerabilities, fixed in v12.0.0.
+ SECRET_KEY: ${SECRET_KEY:?set SECRET_KEY — run `make env` to generate it}
+ AISOC_CREDENTIAL_KEY: ${AISOC_CREDENTIAL_KEY:?set AISOC_CREDENTIAL_KEY — run `make env` to generate it}
+ AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:?set AISOC_SERVICE_TOKEN — run `make env` to generate it}
+ AISOC_ACTIONS_SERVICE_TOKEN: ${AISOC_ACTIONS_SERVICE_TOKEN:?set AISOC_ACTIONS_SERVICE_TOKEN — run `make env` to generate it}
+ AISOC_REALTIME_JWT_SECRET: ${AISOC_REALTIME_JWT_SECRET:?set AISOC_REALTIME_JWT_SECRET — run `make env` to generate it}
+ REALTIME_INTERNAL_TOKEN: ${REALTIME_INTERNAL_TOKEN:?set REALTIME_INTERNAL_TOKEN — run `make env` to generate it}
+ NEO4J_PASSWORD: ${NEO4J_PASSWORD:?set NEO4J_PASSWORD}
+ CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:?set CLICKHOUSE_PASSWORD}
+
+ ingest-worker:
+ restart: unless-stopped
+ environment:
+ # Both keys, because `envmode.Current()` reads ENV first and the base
+ # file's ENV is what shadowed an operator's ENVIRONMENT before this.
+ ENV: production
+ ENVIRONMENT: production
+ # Required outside development by
+ # services/ingest/internal/config/config.go. This is the service that
+ # accepts events off the network, so it is the last one that should be
+ # allowed to start without a signing secret.
+ JWT_SECRET: ${JWT_SECRET:?set JWT_SECRET — ingest refuses to start without it outside development}
+ AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:?set AISOC_SERVICE_TOKEN}
+ AISOC_NEO4J_PASSWORD: ${NEO4J_PASSWORD:?set NEO4J_PASSWORD}
+
+ fusion:
+ restart: unless-stopped
+ environment:
+ ENVIRONMENT: production
+ AISOC_DEV_MODE: 0
+ AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:?set AISOC_SERVICE_TOKEN}
+ CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:?set CLICKHOUSE_PASSWORD}
+
+ agents:
+ restart: unless-stopped
+ environment:
+ ENVIRONMENT: production
+ AISOC_DEV_MODE: 0
+ AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:?set AISOC_SERVICE_TOKEN}
+ AISOC_ACTIONS_SERVICE_TOKEN: ${AISOC_ACTIONS_SERVICE_TOKEN:?set AISOC_ACTIONS_SERVICE_TOKEN}
+ NEO4J_PASSWORD: ${NEO4J_PASSWORD:?set NEO4J_PASSWORD}
+
+ actions:
+ restart: unless-stopped
+ environment:
+ ENVIRONMENT: production
+ AISOC_DEV_MODE: 0
+ # Without this the service answers 503 on every mutating route, which is
+ # the intended failure introduced in v12.0.0 — it previously served
+ # isolate_host and run_script to anything that could reach the port.
+ AISOC_ACTIONS_SERVICE_TOKEN: ${AISOC_ACTIONS_SERVICE_TOKEN:?set AISOC_ACTIONS_SERVICE_TOKEN}
+
+ connectors:
+ restart: unless-stopped
+ environment:
+ ENVIRONMENT: production
+ AISOC_DEV_MODE: 0
+ AISOC_CREDENTIAL_KEY: ${AISOC_CREDENTIAL_KEY:?set AISOC_CREDENTIAL_KEY}
+ AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:?set AISOC_SERVICE_TOKEN}
+ SECRET_KEY: ${SECRET_KEY:?set SECRET_KEY}
+
+ threatintel:
+ restart: unless-stopped
+ environment:
+ ENVIRONMENT: production
+ AISOC_DEV_MODE: 0
+ NEO4J_PASSWORD: ${NEO4J_PASSWORD:?set NEO4J_PASSWORD}
+
+ realtime:
+ restart: unless-stopped
+ environment:
+ # Unset, the edge rejects every connection and /internal/* answers 503.
+ # Both used to fall back: the ticket verifier to a constant committed to
+ # this repository, and the internal guard to "authorized".
+ AISOC_REALTIME_JWT_SECRET: ${AISOC_REALTIME_JWT_SECRET:?set AISOC_REALTIME_JWT_SECRET — run `make env` to generate it}
+ REALTIME_INTERNAL_TOKEN: ${REALTIME_INTERNAL_TOKEN:?set REALTIME_INTERNAL_TOKEN — run `make env` to generate it}
+
+ litellm:
+ restart: unless-stopped
+
+ ollama:
+ restart: unless-stopped
+
+ # The console. Demo mode is inlined at build time by Next, so this asserts
+ # the runtime side only; pull the `latest` tag rather than a `-demo` one.
+ web:
+ restart: unless-stopped
+ environment:
+ NODE_ENV: production
+ AISOC_DEMO_MODE: "false"
+
+ # ---------------------------------------------------------------------------
+ # `full`-profile stores. They do not run in a CORE deployment, but the base
+ # file gives every service a default that points at the published dev
+ # credential, so `--profile full` would start Neo4j and ClickHouse on a
+ # password anyone can read. Required here rather than defaulted, and required
+ # on the services that dial them so the two cannot disagree.
+ # ---------------------------------------------------------------------------
+ neo4j:
+ ports: !reset []
+ restart: unless-stopped
+ environment:
+ NEO4J_AUTH: neo4j/${NEO4J_PASSWORD:?set NEO4J_PASSWORD — required for the full profile}
+
+ clickhouse:
+ ports: !reset []
+ restart: unless-stopped
+ environment:
+ CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:?set CLICKHOUSE_PASSWORD — required for the full profile}
+
+ opensearch:
+ ports: !reset []
+ restart: unless-stopped
+
+ # Kafka UI browses every topic with no authentication of its own. It is
+ # useful while developing and is not something to run beside production data,
+ # so it is moved onto a profile nothing here activates: `--profile full` no
+ # longer starts it. An operator who wants it can still ask for it by name.
+ kafka-ui:
+ profiles: !override ["debug-ui"]
+ ports: !reset []
diff --git a/docker-compose.yml b/docker-compose.yml
index 8a8bc3be6..354dfabe7 100644
--- a/docker-compose.yml
+++ b/docker-compose.yml
@@ -128,7 +128,7 @@ services:
redis:
image: redis:7-alpine
container_name: aisoc-redis
- command: redis-server --appendonly yes --requirepass redis_dev_secret
+ command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD:-redis_dev_secret}
ports:
- "${AISOC_BIND_ADDR:-127.0.0.1}:6379:6379"
volumes:
@@ -136,7 +136,7 @@ services:
networks:
- aisoc
healthcheck:
- test: ["CMD", "redis-cli", "-a", "redis_dev_secret", "ping"]
+ test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD:-redis_dev_secret}", "ping"]
interval: 10s
timeout: 5s
retries: 5
@@ -233,7 +233,7 @@ services:
environment:
CLICKHOUSE_DB: aisoc
CLICKHOUSE_USER: aisoc
- CLICKHOUSE_PASSWORD: clickhouse_dev_secret
+ CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse_dev_secret}
CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1
networks:
- aisoc
@@ -324,7 +324,7 @@ services:
- "${AISOC_BIND_ADDR:-127.0.0.1}:7474:7474" # HTTP browser
- "${AISOC_BIND_ADDR:-127.0.0.1}:7687:7687" # Bolt
environment:
- NEO4J_AUTH: neo4j/neo4j_dev_secret
+ NEO4J_AUTH: neo4j/${NEO4J_PASSWORD:-neo4j_dev_secret}
NEO4J_PLUGINS: '["apoc"]'
NEO4J_dbms_security_procedures_unrestricted: apoc.*
NEO4J_dbms_memory_heap_initial__size: 256m
@@ -371,7 +371,7 @@ services:
# Read by the migration runner to (re)apply the runtime role's password
# on an existing data volume, where the postgres init hook never fires.
AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}
- REDIS_URL: redis://:redis_dev_secret@redis:6379/0
+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/0
KAFKA_BOOTSTRAP_SERVERS: kafka:29092
# The lake client reads these five, not CLICKHOUSE_URL — nothing under
# services/api/app reads that name at all. Supplying only the URL left
@@ -383,7 +383,7 @@ services:
CLICKHOUSE_PORT: 9000
CLICKHOUSE_DATABASE: aisoc
CLICKHOUSE_USER: aisoc
- CLICKHOUSE_PASSWORD: clickhouse_dev_secret
+ CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse_dev_secret}
OPENSEARCH_URL: http://opensearch:9200
QDRANT_URL: http://qdrant:6333
# The API owns the vault write path; the connectors service reads with
@@ -393,7 +393,7 @@ services:
AISOC_CREDENTIAL_KEY_ROTATION_FROM: ${AISOC_CREDENTIAL_KEY_ROTATION_FROM:-}
NEO4J_URI: bolt://neo4j:7687
NEO4J_USER: neo4j
- NEO4J_PASSWORD: neo4j_dev_secret
+ NEO4J_PASSWORD: ${NEO4J_PASSWORD:-neo4j_dev_secret}
# Was a bare literal, so a SECRET_KEY set in .env was ignored and the
# JWT signing key silently stayed on the dev default. It also has to
# match whatever the connectors service verifies with, or the console
@@ -486,11 +486,18 @@ services:
- "${AISOC_BIND_ADDR:-127.0.0.1}:8081:8080"
- "${AISOC_BIND_ADDR:-127.0.0.1}:9090:9090"
environment:
- ENV: development
+ # Interpolated, not a literal. `.env.example` tells an operator that
+ # setting ENVIRONMENT=production is enough, and for every other service
+ # it is. This one pinned `development`, and `envmode.Current()` reads
+ # ENV *before* ENVIRONMENT — so ingest stayed dev-mode however the
+ # operator set it, and `JWT_SECRET must be set in non-development
+ # environments` never fired on the one service taking events off the
+ # network.
+ ENV: ${ENVIRONMENT:-development}
HTTP_PORT: 8080
METRICS_PORT: 9090
KAFKA_BOOTSTRAP_SERVERS: kafka:29092
- REDIS_URL: redis://:redis_dev_secret@redis:6379/1
+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/1
# The inbox webhook routes (/v1/inbox/*) mount only when this is set,
# and no compose file set it, so the documented "push events straight
# from your SIEM" path was unreachable on every local and CI stack.
@@ -515,7 +522,7 @@ services:
AISOC_GRAPH_ENABLED: "${AISOC_GRAPH_ENABLED:-false}"
AISOC_NEO4J_URI: bolt://neo4j:7687
AISOC_NEO4J_USER: neo4j
- AISOC_NEO4J_PASSWORD: neo4j_dev_secret
+ AISOC_NEO4J_PASSWORD: ${NEO4J_PASSWORD:-neo4j_dev_secret}
networks:
- aisoc
restart: unless-stopped
@@ -538,7 +545,7 @@ services:
- "${AISOC_BIND_ADDR:-127.0.0.1}:8080:8082"
environment:
KAFKA_BOOTSTRAP_SERVERS: kafka:29092
- REDIS_URL: redis://:redis_dev_secret@redis:6379/2
+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/2
VIRUSTOTAL_API_KEY: ${VIRUSTOTAL_API_KEY:-}
ABUSEIPDB_API_KEY: ${ABUSEIPDB_API_KEY:-}
GREYNOISE_API_KEY: ${GREYNOISE_API_KEY:-}
@@ -568,7 +575,7 @@ services:
- "${AISOC_BIND_ADDR:-127.0.0.1}:8003:8003"
environment:
KAFKA_BOOTSTRAP_SERVERS: kafka:29092
- REDIS_URL: redis://:redis_dev_secret@redis:6379/3
+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/3
DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisoc
# The ClickHouse event lake is a `full`-profile capability: it backs
# /lake/sql and the hunt surface, not alerting. CORE runs without it and
@@ -578,7 +585,7 @@ services:
CLICKHOUSE_PORT: "9000"
CLICKHOUSE_DATABASE: aisoc
CLICKHOUSE_USER: aisoc
- CLICKHOUSE_PASSWORD: clickhouse_dev_secret
+ CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse_dev_secret}
# Fusion defaulted to http://localhost:8082, which inside its own
# container is fusion itself — so fuse-time enrichment always failed and
# the failure was swallowed at DEBUG. `enrichment` is a `full`-profile
@@ -607,7 +614,7 @@ services:
- "${AISOC_BIND_ADDR:-127.0.0.1}:8001:8084"
environment:
DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisoc
- REDIS_URL: redis://:redis_dev_secret@redis:6379/4
+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/4
CORE_API_URL: http://api:8000
# Phase B1 — auto-triage every fused alert off the Kafka stream (copilot:
# read-only triage, no response dispatch). Degrades to deterministic
@@ -617,7 +624,7 @@ services:
QDRANT_URL: http://qdrant:6333
NEO4J_URI: bolt://neo4j:7687
NEO4J_USER: neo4j
- NEO4J_PASSWORD: neo4j_dev_secret
+ NEO4J_PASSWORD: ${NEO4J_PASSWORD:-neo4j_dev_secret}
OPENAI_API_KEY: ${OPENAI_API_KEY:-}
ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}
# BYOK / "explain this alert" only — never a task role. A task role asks
@@ -858,7 +865,7 @@ services:
- "${AISOC_BIND_ADDR:-127.0.0.1}:8002:8085"
environment:
KAFKA_BOOTSTRAP_SERVERS: kafka:29092
- REDIS_URL: redis://:redis_dev_secret@redis:6379/5
+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/5
DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisoc
CORE_API_URL: http://api:8000
# Must match AISOC_ACTIONS_SERVICE_TOKEN on the api and slack-bot
@@ -922,7 +929,7 @@ services:
AISOC_DEV_MODE: ${AISOC_DEV_MODE:-1}
ENVIRONMENT: ${ENVIRONMENT:-development}
KAFKA_BOOTSTRAP_SERVERS: kafka:29092
- REDIS_URL: redis://:redis_dev_secret@redis:6379/6
+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/6
DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisoc
CORE_API_URL: http://api:8000
# The scheduler decrypts stored connector credentials through the vault
@@ -981,7 +988,7 @@ services:
- "${AISOC_BIND_ADDR:-127.0.0.1}:8005:8005"
environment:
KAFKA_BOOTSTRAP_SERVERS: kafka:29092
- REDIS_URL: redis://:redis_dev_secret@redis:6379/7
+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/7
# This service reads OPENSEARCH_HOST/PORT. It was given OPENSEARCH_URL,
# which its Settings has no field for and `extra="ignore"` discards, so
# the connection worked only because the defaults happened to match the
@@ -991,7 +998,7 @@ services:
QDRANT_URL: http://qdrant:6333
NEO4J_URI: bolt://neo4j:7687
NEO4J_USER: neo4j
- NEO4J_PASSWORD: neo4j_dev_secret
+ NEO4J_PASSWORD: ${NEO4J_PASSWORD:-neo4j_dev_secret}
# Same credential the API presents when it proxies /threat-intel/indicators.
SECRET_KEY: ${SECRET_KEY:-dev_secret_key_change_in_production}
AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:-}
@@ -1036,9 +1043,9 @@ services:
# migration runner otherwise. Passed here too so this service's own
# DATABASE_URL and the role's actual credential come from one value.
AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}
- REDIS_URL: redis://:redis_dev_secret@redis:6379/8
+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/8
KAFKA_BOOTSTRAP_SERVERS: kafka:29092
- CLICKHOUSE_URL: http://aisoc:clickhouse_dev_secret@clickhouse:8123/aisoc
+ CLICKHOUSE_URL: http://aisoc:${CLICKHOUSE_PASSWORD:-clickhouse_dev_secret}@clickhouse:8123/aisoc
CORE_API_URL: http://api:8000
AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:-}
AISOC_DEV_MODE: ${AISOC_DEV_MODE:-1}
@@ -1092,7 +1099,7 @@ services:
# migration runner otherwise. Passed here too so this service's own
# DATABASE_URL and the role's actual credential come from one value.
AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}
- REDIS_URL: redis://:redis_dev_secret@redis:6379/9
+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/9
KAFKA_BOOTSTRAP_SERVERS: kafka:29092
CORE_API_URL: http://api:8000
AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:-}
@@ -1145,7 +1152,7 @@ services:
# migration runner otherwise. Passed here too so this service's own
# DATABASE_URL and the role's actual credential come from one value.
AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}
- REDIS_URL: redis://:redis_dev_secret@redis:6379/10
+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/10
CORE_API_URL: http://api:8000
ATTCK_DATA_PATH: /app/data/enterprise-attack.json
AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:-}
@@ -1181,8 +1188,8 @@ services:
- "${AISOC_BIND_ADDR:-127.0.0.1}:8086:4000"
environment:
KAFKA_BOOTSTRAP_SERVERS: kafka:29092
- REDIS_URL: redis://:redis_dev_secret@redis:6379
- REDIS_PASSWORD: redis_dev_secret
+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379
+ REDIS_PASSWORD: ${REDIS_PASSWORD:-redis_dev_secret}
PORT: 4000
# Both are generated by `make up`. Neither was passed to this container
# before, which is what made the ticket verifier fall back to a constant
diff --git a/scripts/check_profile_service_counts.py b/scripts/check_profile_service_counts.py
index f6fc446d5..4bb95415b 100644
--- a/scripts/check_profile_service_counts.py
+++ b/scripts/check_profile_service_counts.py
@@ -49,6 +49,14 @@ self_test_if_requested(__file__)
REPO_ROOT = repo_root()
COMPOSE = REPO_ROOT / "docker-compose.yml"
+#: The production stack. It `include`s the base and then differs from it
+#: deliberately — `kafka-ui` browses every topic with no authentication of its
+#: own and is moved off the `full` profile there — so its `full` count is one
+#: lower. Scanned and reported beside the development figures rather than left
+#: for an operator to discover: two stacks that differ by design should differ
+#: on the page too.
+PROD_COMPOSE = REPO_ROOT / "docker-compose.prod.yml"
+
#: One-shot containers that run to completion and exit. Counted separately
#: because "long-running services" is the figure the documents publish, and
#: folding a container that exits into it would overstate what is resident.
@@ -60,6 +68,13 @@ ONE_SHOT: frozenset[str] = frozenset({"ollama-pull"})
#: integer near the word "services": a gate that guesses which numbers are
#: claims would either miss the one that matters or flag prose forever.
CLAIM_SITES: tuple[tuple[str, str, str], ...] = (
+ # The production deployment page. `prod:full` is a separate figure from
+ # `full` because the production stack deliberately does not start kafka-ui,
+ # and a reader comparing the two pages would otherwise find a discrepancy
+ # with nothing explaining it.
+ ("apps/docs/docs/deployment/docker.md", "core", r"CORE is (\d+) long-running services here"),
+ ("apps/docs/docs/deployment/docker.md", "prod:full", r"`--profile full`\s*\n?is (\d+) rather than"),
+ ("apps/docs/docs/deployment/docker.md", "full", r"is \d+ rather than (\d+), the difference being"),
("README.md", "core", r"\|\s*\*\*core\*\*\s*\|\s*`make up`\s*\|\s*(\d+)\s*\|"),
("README.md", "full", r"\|\s*\*\*full\*\*\s*\|\s*`make up-full`\s*\|\s*(\d+)\s*\|"),
("README.md", "core", r"\|\s*\*\*demo\*\*\s*\|\s*`make up && make demo`\s*\|\s*(\d+)\s*\|"),
@@ -199,7 +214,10 @@ def _parse_services(text: str) -> dict[str, list[str] | None]:
to keep that true. The shape being read is narrow: two-space-indented
service keys under a top-level ``services:``, and a ``profiles:`` key
four spaces in, in either inline (``["a", "b"]``) or block (``- a``)
- form. Both forms appear in this file.
+ form. Both forms appear in this file, and either may carry a Compose Spec
+ merge tag (``!override``) in the production overlay — untagged, that line
+ read as "no profiles declared" and the gate counted a service the
+ production stack does not start.
"""
services: dict[str, list[str] | None] = {}
in_services = False
@@ -226,13 +244,13 @@ def _parse_services(text: str) -> dict[str, list[str] | None]:
if current is None:
continue
- inline = re.match(r"^ profiles:\s*\[(.*)\]\s*$", raw)
+ inline = re.match(r"^ profiles:\s*(?:![a-z]+\s+)?\[(.*)\]\s*$", raw)
if inline:
services[current] = re.findall(r"[A-Za-z0-9._-]+", inline.group(1))
collecting_block = False
continue
- if re.match(r"^ profiles:\s*$", raw):
+ if re.match(r"^ profiles:\s*(?:![a-z]+\s*)?$", raw):
services[current] = []
collecting_block = True
continue
@@ -271,6 +289,29 @@ def _declared_profiles(services: dict[str, list[str] | None]) -> set[str]:
return {p for profiles in services.values() if profiles for p in profiles}
+def _production_services(base: dict[str, list[str] | None]) -> dict[str, list[str] | None]:
+ """The base service set with the production file's profile overrides applied.
+
+ Computed rather than shelled out to ``docker compose config``, because a
+ gate that needs a Docker daemon is a gate that skips on the runners that
+ do not have one — and a skip here reports nothing while looking green.
+
+ Only ``profiles:`` is applied, because that is the only key that changes
+ *which* services a run starts. The production file's other overrides change
+ how a service is configured, which the counts do not describe.
+ """
+ if not PROD_COMPOSE.is_file():
+ return base
+ overrides = _parse_services(PROD_COMPOSE.read_text(encoding="utf-8"))
+ merged = dict(base)
+ for name, profiles in overrides.items():
+ # `_parse_services` returns None for a service that names no profiles,
+ # which in the overlay means "not overridden" rather than "no profile".
+ if profiles is not None:
+ merged[name] = profiles
+ return merged
+
+
def scan() -> tuple[dict[str, int], dict[str, set[str]], list[str]]:
"""Returns (counts by profile, members by profile, errors)."""
if not COMPOSE.is_file():
@@ -285,6 +326,14 @@ def scan() -> tuple[dict[str, int], dict[str, set[str]], list[str]]:
members = {p: _profile_members(services, p) for p in profiles}
counts = {p: _long_running(members[p]) for p in profiles}
+ # The production stack, under `prod:` keys so a claim site can name either
+ # and the two cannot be confused for one another.
+ prod_services = _production_services(services)
+ for profile in profiles:
+ prod_members = _profile_members(prod_services, profile)
+ members[f"prod:{profile}"] = prod_members
+ counts[f"prod:{profile}"] = _long_running(prod_members)
+
for rel, profile, pattern in CLAIM_SITES:
path = REPO_ROOT / rel
if not path.is_file():
@@ -352,6 +401,12 @@ def main(argv: list[str] | None = None) -> int:
f"profile-service-counts: OK, core {counts['core']}, full {counts.get('full', 0)}, "
f"{len(CLAIM_SITES)} published figures agree with docker-compose.yml"
)
+ # Printed every run rather than only when it differs: a figure that appears
+ # only on disagreement is one nobody knows the value of.
+ print(
+ f" production stack: core {counts.get('prod:core', 0)}, full {counts.get('prod:full', 0)}"
+ f" ({counts.get('full', 0) - counts.get('prod:full', 0)} fewer on full — kafka-ui is not run beside production data)"
+ )
return 0
diff --git a/scripts/check_published_images.py b/scripts/check_published_images.py
index 1ace1ea3b..d36ea1dc1 100644
--- a/scripts/check_published_images.py
+++ b/scripts/check_published_images.py
@@ -99,6 +99,18 @@ sys.path.insert(0, str(Path(__file__).resolve().parent))
from gate_toolkit import refuses_an_empty_tree, repo_root, scratch_tree # noqa: E402
COMPOSE_REL = Path("docker-compose.yml")
+
+#: Compose files that name images an operator is told to pull. The production
+#: file was added after this gate and would have shipped unread: it `include`s
+#: the base rather than repeating it, so it contributes no `image:` of its own
+#: today — but a tag pinned there later would be invisible to this scan, which
+#: is exactly the shape the gate exists to catch. Reading the whole set costs
+#: nothing and removes the question.
+COMPOSE_FILES: tuple[Path, ...] = (
+ COMPOSE_REL,
+ Path("docker-compose.prod.yml"),
+)
+
CHART_REL = Path("infra/helm/aisoc")
VERSION_REL = Path("VERSION")
@@ -205,24 +217,68 @@ def _split(image: str) -> tuple[str, str]:
return image, "latest"
+class _ComposeLoader(yaml.SafeLoader):
+ """SafeLoader that understands the Compose Spec's merge-control tags.
+
+ ``!reset`` and ``!override`` tell Compose to clear or replace an inherited
+ value instead of merging it, and `docker-compose.prod.yml` needs both to
+ unpublish ports the base file binds. PyYAML's safe loader raises
+ ``ConstructorError`` on an unknown tag, so without this the gate would
+ crash on a file it was extended to read — a scan that cannot run is worse
+ than one that reports nothing, because it looks like a broken gate rather
+ than a finding.
+ """
+
+
+def _strip_tag(loader: yaml.Loader, node: yaml.Node) -> object:
+ """Return the tagged value itself; the tag is a merge instruction only."""
+ if isinstance(node, yaml.SequenceNode):
+ return loader.construct_sequence(node, deep=True)
+ if isinstance(node, yaml.MappingNode):
+ return loader.construct_mapping(node, deep=True)
+ return loader.construct_scalar(node)
+
+
+for _tag in ("!reset", "!override"):
+ _ComposeLoader.add_constructor(_tag, _strip_tag)
+
+
+def _load_compose(path: Path) -> dict:
+ return yaml.load(path.read_text(encoding="utf-8"), Loader=_ComposeLoader) or {} # noqa: S506
+
+
def collect_compose_references(root: Path) -> list[Reference]:
- """Every ``image:`` under ``services:``, at the tag ``docker compose up`` requests."""
- path = root / COMPOSE_REL
- if not path.is_file():
- raise ScanError(f"no {COMPOSE_REL} to read: {path}")
- try:
- document = yaml.safe_load(path.read_text(encoding="utf-8")) or {}
- except yaml.YAMLError as exc:
- raise ScanError(f"{COMPOSE_REL} is not readable as YAML: {exc}") from exc
+ """Every ``image:`` under ``services:``, at the tag ``docker compose up`` requests.
+
+ Reads every file in :data:`COMPOSE_FILES`. The base file must exist — its
+ absence means the scan cannot answer and is an error rather than a clean
+ verdict — while the others are read when present, so adding or removing one
+ does not require editing this function.
+ """
+ base = root / COMPOSE_REL
+ if not base.is_file():
+ raise ScanError(f"no {COMPOSE_REL} to read: {base}")
references: list[Reference] = []
- for name, service in sorted((document.get("services") or {}).items()):
- if not isinstance(service, dict) or not isinstance(service.get("image"), str):
+ for relative in COMPOSE_FILES:
+ path = root / relative
+ if not path.is_file():
continue
- expanded, interpolated = _expand(service["image"])
- repository, tag = _split(expanded)
- origin = "the default in ${AISOC_VERSION:-…}" if interpolated else "written literally"
- references.append(Reference(str(COMPOSE_REL), name, repository, tag, origin))
+ try:
+ # `!reset` / `!override` are Compose Spec tags. PyYAML's safe
+ # loader raises on an unknown tag, so a production overlay using
+ # them would make this gate crash rather than scan.
+ document = _load_compose(path)
+ except yaml.YAMLError as exc:
+ raise ScanError(f"{relative} is not readable as YAML: {exc}") from exc
+
+ for name, service in sorted((document.get("services") or {}).items()):
+ if not isinstance(service, dict) or not isinstance(service.get("image"), str):
+ continue
+ expanded, interpolated = _expand(service["image"])
+ repository, tag = _split(expanded)
+ origin = "the default in ${AISOC_VERSION:-…}" if interpolated else "written literally"
+ references.append(Reference(str(relative), name, repository, tag, origin))
return references
diff --git a/services/api/tests/test_production_environment_requires_auth.py b/services/api/tests/test_production_environment_requires_auth.py
new file mode 100644
index 000000000..8056020a1
--- /dev/null
+++ b/services/api/tests/test_production_environment_requires_auth.py
@@ -0,0 +1,87 @@
+"""An unauthenticated request must not be admin outside development.
+
+This is the assertion discussion #629 was really about. `docker-compose.yml`
+defaults `ENVIRONMENT` to `development`, `development` is in
+`AUTH_BYPASS_ENVIRONMENTS`, and `app/api/v1/dev_auth.py` then resolves a
+request carrying no bearer token to a demo user whose role is `admin`. The
+documented production compose file did not exist, so that was the only stack an
+operator could start.
+
+The bypass itself is intentional and stays: a contributor running the stack on
+a laptop should not have to seed a user and log in. What was missing is
+anything asserting the other half — that naming a non-development environment
+actually turns it off at the point a request is served, rather than only in the
+module's docstring.
+
+Both directions are asserted. A test that only checked production would pass
+against a build where the shim had been deleted, which would be a different
+product and a silently broken developer experience.
+"""
+
+from __future__ import annotations
+
+import importlib
+
+import pytest
+from fastapi import Depends, FastAPI
+from fastapi.testclient import TestClient
+
+
+def _probe_app(get_current_user) -> FastAPI: # noqa: ANN001 - dependency callable
+ """A one-route app whose only job is to resolve the current user.
+
+ The dependency is a default value rather than `Annotated[..., Depends(...)]`
+ because this module uses postponed annotations: the annotation would be a
+ string, FastAPI would resolve it against this module's globals where the
+ dependency is a local, and `user` would degrade to a query parameter —
+ giving 422 on every request and testing nothing about authentication.
+ """
+ app = FastAPI()
+
+ @app.get("/probe")
+ async def probe(user=Depends(get_current_user)): # noqa: ANN001, B008
+ return {"user_id": str(getattr(user, "user_id", None)), "role": getattr(user, "role", None)}
+
+ return app
+
+
+def client_in(monkeypatch: pytest.MonkeyPatch):
+ def _build(environment: str) -> TestClient:
+ # `current_env_from_os` reads ENV first, so both are set; a stale ENV
+ # is precisely what shadowed an operator's ENVIRONMENT on ingest.
+ monkeypatch.setenv("ENVIRONMENT", environment)
+ monkeypatch.delenv("ENV", raising=False)
+ from app.api.v1 import deps, dev_auth
+
+ # Both, and in this order. Reloading `dev_auth` alone leaves
+ # `deps.get_current_user` closed over the previous module, and the
+ # route then rejects on validation (422) instead of exercising the
+ # authentication path this test is about.
+ importlib.reload(dev_auth)
+ importlib.reload(deps)
+ return TestClient(_probe_app(deps.get_current_user), raise_server_exceptions=False)
+
+ return _build
+
+
+class TestTheBypassIsEnvironmentGated:
+ def test_production_refuses_an_unauthenticated_request(self, client_in) -> None:
+ response = client_in("production").get("/probe")
+ assert response.status_code == 401, (
+ "an unauthenticated request was served outside development — this is the state "
+ "every stock `docker compose up` ran in, because the production compose file "
+ "the docs pointed at did not exist (discussion #629)"
+ )
+
+ @pytest.mark.parametrize("environment", ["staging", "prod", "production"])
+ def test_no_non_development_environment_hands_back_a_user(self, client_in, environment: str) -> None:
+ """`staging` included deliberately: it is not in the allow-list, and a
+ reader could reasonably assume any named environment is safe."""
+ assert client_in(environment).get("/probe").status_code == 401
+
+ def test_development_still_resolves_a_user(self, client_in) -> None:
+ """The other direction, so this cannot pass on a build with no shim."""
+ response = client_in("development").get("/probe")
+ assert response.status_code == 200
+ assert response.json()["role"] == "admin"
diff --git a/tests/test_compose_environment_gate.py b/tests/test_compose_environment_gate.py
new file mode 100644
index 000000000..0553fb979
--- /dev/null
+++ b/tests/test_compose_environment_gate.py
@@ -0,0 +1,107 @@
+"""A service must not pin its environment to a dev-class literal.
+
+`.env.example` tells an operator that setting `ENVIRONMENT=production` is
+enough, and `docker-compose.yml` reads that value. That promise only holds if
+every service takes the variable rather than hardcoding a value.
+
+One did not. `ingest-worker` carried `ENV: development` as a literal, and
+`envmode.Current()` reads `ENV` *before* `ENVIRONMENT`
+(`services/ingest/internal/envmode/envmode.go`), so the operator's setting was
+shadowed on the one service that accepts events off the network — leaving
+`JWT_SECRET must be set in non-development environments`
+(`services/ingest/internal/config/config.go`) unreachable.
+
+The failure shape is worth naming, because it is the one this repository keeps
+finding: the escape hatch was documented, and the documentation was true of
+every service but one. Nothing compared the two.
+"""
+
+from __future__ import annotations
+
+import pathlib
+
+import pytest
+import yaml
+
+REPO = pathlib.Path(__file__).resolve().parents[1]
+
+#: Mirrors DEV_ENVIRONMENTS in services/api/app/core/config.py and
+#: devEnvironments in services/ingest/internal/envmode/envmode.go. A value in
+#: this set relaxes an auth or secret requirement somewhere in the stack.
+DEV_ENVIRONMENTS = frozenset({"development", "dev", "local", "demo", "test"})
+
+#: The keys any service reads to decide whether it is in a dev-class
+#: environment. `current_env_from_os()` reads ENV or ENVIRONMENT; the Go
+#: mirror reads the same two; APP_ENV is checked by the realtime edge.
+ENV_KEYS = ("ENV", "ENVIRONMENT", "APP_ENV")
+
+COMPOSE_FILES = [
+ REPO / "docker-compose.yml",
+ REPO / "infra" / "compose" / "docker-compose.dev.yml",
+]
+
+
+def _services(path: pathlib.Path) -> dict:
+ if not path.is_file():
+ return {}
+ return (yaml.safe_load(path.read_text(encoding="utf-8")) or {}).get("services") or {}
+
+
+def _literal_dev_envs(path: pathlib.Path) -> list[str]:
+ """Services pinning a dev-class environment as a literal, not a variable."""
+ offenders: list[str] = []
+ for name, service in _services(path).items():
+ env = (service or {}).get("environment") or {}
+ if not isinstance(env, dict):
+ continue
+ for key in ENV_KEYS:
+ value = str(env.get(key, "")).strip()
+ # `${...}` is interpolated and therefore follows the operator's
+ # .env; only a bare value shadows it.
+ if value and "${" not in value and value.lower() in DEV_ENVIRONMENTS:
+ offenders.append(f"{path.name}:{name}:{key}={value}")
+ return offenders
+
+
+class TestTheDocumentedSwitchReachesEveryService:
+ def test_the_shipped_compose_pins_no_dev_environment(self) -> None:
+ offenders = _literal_dev_envs(REPO / "docker-compose.yml")
+ assert not offenders, (
+ f"these services hardcode a dev-class environment, so `ENVIRONMENT=production` in .env does not reach them: {offenders}"
+ )
+
+ @pytest.mark.parametrize("path", COMPOSE_FILES, ids=lambda p: p.name)
+ def test_no_tracked_compose_file_pins_one(self, path: pathlib.Path) -> None:
+ """The dev overlay is allowed to *be* development — by interpolation.
+
+ Included because the next service to hardcode one is as likely to land
+ in the overlay as in the root file, and a gate that reads one file is
+ the shape that let this through.
+ """
+ assert not _literal_dev_envs(path)
+
+ def test_the_gate_detects_the_literal_it_was_written_for(self, tmp_path: pathlib.Path) -> None:
+ """Proven against the defect, not only against the fixed tree.
+
+ Without this, a refactor that stopped reading `environment:` at all
+ would report every file clean.
+ """
+ broken = tmp_path / "docker-compose.yml"
+ broken.write_text(
+ yaml.safe_dump({"services": {"ingest-worker": {"environment": {"ENV": "development"}}}}),
+ encoding="utf-8",
+ )
+ assert _literal_dev_envs(broken) == ["docker-compose.yml:ingest-worker:ENV=development"]
+
+ def test_an_interpolated_default_is_not_an_offender(self, tmp_path: pathlib.Path) -> None:
+ """`${ENVIRONMENT:-development}` is the fix, so it must pass.
+
+ A gate that rejected the default too would force the dev stack to
+ require a variable nobody sets locally, and would be reverted.
+ """
+ ok = tmp_path / "docker-compose.yml"
+ ok.write_text(
+ yaml.safe_dump({"services": {"api": {"environment": {"ENVIRONMENT": "${ENVIRONMENT:-development}"}}}}),
+ encoding="utf-8",
+ )
+ assert _literal_dev_envs(ok) == []
diff --git a/tests/test_production_compose_gate.py b/tests/test_production_compose_gate.py
new file mode 100644
index 000000000..ef61dcc9a
--- /dev/null
+++ b/tests/test_production_compose_gate.py
@@ -0,0 +1,198 @@
+"""`docker-compose.prod.yml` must be production by construction.
+
+Discussion #629 reported that the deployment page pointed at a production
+compose file that did not exist, so the only stack an operator could start was
+the development one — where `ENVIRONMENT` defaults to `development`,
+`development` is in `AUTH_BYPASS_ENVIRONMENTS`, and `dev_auth.py` resolves an
+unauthenticated request to a demo user whose role is `admin`.
+
+A file that merely *documents* the right settings would have the same problem
+one release later. These assertions are about the file rather than the docs:
+the bypass must be unreachable, no service may start on a credential published
+in this repository, and no datastore may be bound to the host.
+
+Every assertion is run against `docker-compose.yml` as well, which must fail
+them. That is what distinguishes these from a test that would pass on an empty
+file.
+"""
+
+from __future__ import annotations
+
+import pathlib
+import re
+
+import pytest
+import yaml
+
+REPO = pathlib.Path(__file__).resolve().parents[1]
+PROD = REPO / "docker-compose.prod.yml"
+DEV = REPO / "docker-compose.yml"
+
+#: Mirrors AUTH_BYPASS_ENVIRONMENTS and DEV_ENVIRONMENTS. Any of these in
+#: ENV/ENVIRONMENT/APP_ENV relaxes an auth or secret requirement somewhere.
+DEV_ENVIRONMENTS = frozenset({"development", "dev", "local", "demo", "test"})
+
+#: Stores that must never be reachable from the host in production. The
+#: development file binds each to 127.0.0.1, which is right there and wrong
+#: here.
+DATASTORES = ("postgres", "redis", "kafka", "zookeeper", "qdrant", "neo4j", "clickhouse", "opensearch")
+
+
+class _ComposeLoader(yaml.SafeLoader):
+ """Understands `!reset` and `!override`, which the production file needs."""
+
+
+def _strip_tag(loader, node): # noqa: ANN001, ANN202 - loader plumbing
+ if isinstance(node, yaml.SequenceNode):
+ return loader.construct_sequence(node, deep=True)
+ if isinstance(node, yaml.MappingNode):
+ return loader.construct_mapping(node, deep=True)
+ return loader.construct_scalar(node)
+
+
+for _tag in ("!reset", "!override"):
+ _ComposeLoader.add_constructor(_tag, _strip_tag)
+
+
+def _services(path: pathlib.Path) -> dict:
+ return (yaml.load(path.read_text(encoding="utf-8"), Loader=_ComposeLoader) or {}).get("services") or {} # noqa: S506
+
+
+def _env_items(service: dict) -> list[tuple[str, str]]:
+ """`environment:` as pairs, accepting either shape Compose allows."""
+ env = (service or {}).get("environment") or {}
+ if isinstance(env, dict):
+ return [(str(k), str(v)) for k, v in env.items()]
+ pairs = []
+ for entry in env:
+ key, _, value = str(entry).partition("=")
+ pairs.append((key, value))
+ return pairs
+
+
+#: `${VAR:-default}` — the default is what a deployment with no `.env` gets.
+_DEFAULTED = re.compile(r"^\$\{[A-Z_]+:-([^}]*)\}$")
+
+
+def _resolves_to_dev(value: str) -> bool:
+ """True if this value is dev-class with no environment set.
+
+ Both spellings count. A literal `development` is dev-class, and so is
+ `${ENVIRONMENT:-development}` on a host with nothing exported — which is
+ the state the development stack is designed to start in, and the state a
+ production stack must not be able to reach.
+ """
+ raw = value.strip()
+ match = _DEFAULTED.match(raw)
+ candidate = match.group(1) if match else raw
+ return candidate.strip().lower() in DEV_ENVIRONMENTS
+
+
+def _dev_environments(services: dict) -> list[str]:
+ found = []
+ for name, service in services.items():
+ for key, value in _env_items(service):
+ if key in ("ENV", "ENVIRONMENT", "APP_ENV") and _resolves_to_dev(value):
+ found.append(f"{name}:{key}={value}")
+ return found
+
+
+def _published_datastores(services: dict) -> list[str]:
+ return [name for name in DATASTORES if (services.get(name) or {}).get("ports")]
+
+
+def _unguarded_secrets(services: dict) -> list[str]:
+ """Secret-shaped variables that carry a default instead of demanding a value.
+
+ `${VAR:?message}` makes Compose refuse to start and name the variable.
+ `${VAR:-default}` starts on the default, which for this repository means a
+ literal anyone can read in the development file.
+ """
+ pattern = re.compile(r"\$\{([A-Z_]*(?:PASSWORD|SECRET|TOKEN|KEY)[A-Z_]*):-")
+ found = []
+ for name, service in services.items():
+ for key, value in _env_items(service):
+ for variable in pattern.findall(str(value)):
+ found.append(f"{name}:{key} defaults {variable}")
+ command = (service or {}).get("command")
+ if command:
+ for variable in pattern.findall(str(command)):
+ found.append(f"{name}:command defaults {variable}")
+ return found
+
+
[email protected](scope="module")
+def prod() -> dict:
+ assert PROD.is_file(), (
+ "docker-compose.prod.yml does not exist. apps/docs/docs/deployment/docker.md "
+ "has told operators to run it since the page was written (discussion #629)."
+ )
+ return _services(PROD)
+
+
+class TestTheBypassIsUnreachable:
+ def test_no_service_runs_in_a_dev_class_environment(self, prod: dict) -> None:
+ assert not _dev_environments(prod)
+
+ def test_the_development_file_would_fail_this(self) -> None:
+ """The contrast is the point: this is what production differs from."""
+ assert _dev_environments(_services(DEV)), (
+ "docker-compose.yml no longer resolves to a dev-class environment with an empty "
+ ".env, so this gate is comparing production against nothing and proves less than "
+ "it claims"
+ )
+
+ def test_environment_and_dev_mode_are_literals_not_variables(self, prod: dict) -> None:
+ """An interpolated value here could be re-enabled from a stray `.env`.
+
+ The whole point of the file is that the bypass cannot be switched back
+ on by configuration, so these two keys are fixed rather than defaulted.
+ """
+ for name, service in prod.items():
+ env = (service or {}).get("environment") or {}
+ for key in ("ENVIRONMENT", "AISOC_DEV_MODE"):
+ if key in env:
+ assert "${" not in str(env[key]), f"{name}:{key} is interpolated; it must be a fixed value"
+
+
+class TestNothingStartsOnAPublishedCredential:
+ def test_no_secret_carries_a_default(self, prod: dict) -> None:
+ assert not _unguarded_secrets(prod), (
+ "these would start on a default rather than refusing: a deployment that boots on "
+ "a credential published in this repository looks healthy and is not"
+ )
+
+ def test_no_dev_secret_literal_reaches_a_service(self, prod: dict) -> None:
+ """Asserted on values, not on the file text.
+
+ A first attempt grepped the raw file and matched the comment that
+ explains the defect, which is the difference between reading what
+ ships and reading what it says about itself.
+ """
+ leaked = [
+ f"{name}:{key}"
+ for name, service in prod.items()
+ for key, value in _env_items(service)
+ if "_dev_secret" in value and ":-" not in value
+ ]
+ assert not leaked
+
+
+class TestNoDatastoreIsBoundToTheHost:
+ def test_production_publishes_no_datastore(self, prod: dict) -> None:
+ assert not _published_datastores(prod)
+
+ def test_the_development_file_would_fail_this(self) -> None:
+ assert _published_datastores(_services(DEV)), (
+ "docker-compose.yml no longer publishes a datastore port, so this gate is comparing production against nothing"
+ )
+
+ def test_the_unauthenticated_kafka_ui_is_off_the_full_profile(self, prod: dict) -> None:
+ """It browses every topic and has no authentication of its own.
+
+ Moved rather than deleted, so an operator can still ask for it by name
+ — the objection is to it starting beside production data unasked.
+ """
+ profiles = (prod.get("kafka-ui") or {}).get("profiles") or []
+ assert "full" not in profiles, "`--profile full` would start kafka-ui beside production data"
+ assert profiles, "kafka-ui has no profile at all, so it now starts in the default stack"