Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat(deploy): ship the production compose the docs have always pointed at (#946)
Beenu Arora01f0a4af2a72
diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.ymlindex e4e07121d..ec37cb05e 100644--- a/.github/workflows/integration.yml+++ b/.github/workflows/integration.yml@@ -190,10 +190,23 @@ jobs:printf '\nAISOC_VERSION=%s\n' "${AISOC_VERSION}" >> .envfi+ # Copying .env.example leaves every generated secret empty, which is+ # what `make up` runs ensure_env.py to fix. Without it this job+ # provisions an environment no real deployment ever has: v12.0.0 made+ # an empty AISOC_REALTIME_JWT_SECRET fatal — the ticket endpoint+ # answers 503 and the realtime edge rejects every connection, by+ # design — and the spine test failed at mint_ws_ticket. It was not+ # caught on main because this job is path-filtered and had not run on+ # a commit that triggers it since that release.+ python3 scripts/ensure_env.py+ python3 scripts/ensure_env.py --check+token="$(openssl rand -hex 32)"echo "::add-mask::$token"# Written into .env so compose substitutes it into the ingest and# connectors services; exported so the spine test presents it.+ # Overwrites the value ensure_env just generated, deliberately: the+ # test has to present the same token the services verify.sed -i "s|^AISOC_SERVICE_TOKEN=.*|AISOC_SERVICE_TOKEN=${token}|" .envgrep -q "^AISOC_SERVICE_TOKEN=${token}$" .env || {echo "::error::AISOC_SERVICE_TOKEN was not written into .env — check that .env.example still declares it"diff --git a/apps/docs/docs/deployment/docker.md b/apps/docs/docs/deployment/docker.mdindex a3e8b84ce..af845db84 100644--- a/apps/docs/docs/deployment/docker.md+++ b/apps/docs/docs/deployment/docker.md@@ -10,7 +10,20 @@ AiSOC ships three Compose flavors. Pick the one that matches what you are doing.|------|---------|-------------|| `infra/compose/docker-compose.demo.yml` | Streamlined demo with seeded data | Trying AiSOC for the first time || `docker-compose.yml` | Full developer stack | Active development against real source |-| `docker-compose.prod.yml` | Production-leaning stack | Self-hosting on a single VM |+| `docker-compose.prod.yml` | Production stack | Self-hosting on a single VM |++:::warning The development stack has no effective authentication++`docker-compose.yml` defaults `ENVIRONMENT` to `development`, and in a+development-class environment `services/api/app/api/v1/dev_auth.py` resolves a+request that arrives without a bearer token to a demo user whose role is+`admin`. That is deliberate for a laptop and unsafe for anything else.++Use `docker-compose.prod.yml` for anything network-reachable. It fixes+`ENVIRONMENT` to `production` so the bypass cannot be switched back on by+configuration.++:::> If you don't already have Docker installed, the simplest path is the> [one-click installer](../installation) — it installs Docker (Engine + Compose@@ -103,20 +116,56 @@ COMPOSE_PROFILES=connectors,osquery,slack docker compose up -d## Production-Use the production compose file:-```bash+make env # generates the six service secretsdocker compose -f docker-compose.prod.yml up -d```+The production file `include`s `docker-compose.yml` and overrides what+production changes, so there is one definition of every service and the two+cannot drift apart. It differs from the development stack in three ways that+matter:++- **The auth bypass is unreachable.** `ENVIRONMENT` and `AISOC_DEV_MODE` are+ fixed values, not interpolated, so no `.env` can re-enable the demo-admin+ shim described above.+- **Nothing starts on a default credential.** Every secret is declared+ `${VAR:?...}`, so Compose refuses to start and names the variable rather than+ booting on a literal published in this repository and looking healthy.+- **No datastore is bound to the host.** Postgres, Redis, Kafka, Qdrant, Neo4j,+ ClickHouse and OpenSearch publish no ports. `kafka-ui` browses every topic+ with no authentication of its own, so `--profile full` does not start it —+ ask for it by name with `--profile debug-ui` if you want it.++CORE is 16 long-running services here, the same as development. `--profile full`+is 21 rather than 22, the difference being `kafka-ui`.++### Required variables++`make env` generates the first six. The rest are yours to choose, and+`docker compose ... config` will name any you miss before anything starts.++| Variable | Generated by `make env` | Used for |+|---|---|---|+| `SECRET_KEY` | yes | Console session signing |+| `AISOC_CREDENTIAL_KEY` | yes | Connector credential vault |+| `AISOC_SERVICE_TOKEN` | yes | Service-to-service calls |+| `AISOC_ACTIONS_SERVICE_TOKEN` | yes | Response actions |+| `AISOC_REALTIME_JWT_SECRET` | yes | Realtime WebSocket/SSE tickets |+| `REALTIME_INTERNAL_TOKEN` | yes | Realtime `/internal/*` fan-out |+| `POSTGRES_PASSWORD` | no | Postgres owner |+| `AISOC_APP_DB_PASSWORD` | no | DML-only runtime role |+| `REDIS_PASSWORD` | no | Redis |+| `JWT_SECRET` | no | Ingest token signing |+| `NEO4J_PASSWORD`, `CLICKHOUSE_PASSWORD` | no | `full` profile stores |++Put a TLS-terminating reverse proxy in front: the console and ingest bind to+loopback unless you set `AISOC_BIND_ADDR` deliberately.+Before going live, walk through the[Hardening Runbook](https://github.com/beenuar/AiSOC/blob/main/docs/runbooks/HARDENING.md)— TLS termination, secret rotation, network policies, audit log forwarding,-and tenant-scoped backups all need to be in place.--### Environment variables--Copy `.env.example` to `.env` and fill in every required value before starting.+and tenant-scoped backups are not things a compose file can do for you.See [Environment Variables](./env-vars) for the full reference.## Building imagesdiff --git a/docker-compose.prod.yml b/docker-compose.prod.ymlnew file mode 100644index 000000000..ed6229bba--- /dev/null+++ b/docker-compose.prod.yml@@ -0,0 +1,213 @@+# AiSOC — production compose.+#+# docker compose -f docker-compose.prod.yml up -d+#+# This is the file `apps/docs/docs/deployment/docker.md` has pointed at since+# the deployment page was written. It did not exist, which is what discussion+# #629 reported: the documented production command failed on a missing file,+# and the only stack an operator could actually start was the development one,+# where `ENVIRONMENT` defaults to `development` and+# `services/api/app/api/v1/dev_auth.py` resolves an unauthenticated request to+# a demo user whose role is `admin`.+#+# It uses the Compose Spec `include` directive, the same way+# `infra/compose/docker-compose.dev.yml` does, so there is exactly one+# definition of every service and this file holds only what production changes.+# A standalone copy would have drifted from the base within a release, which is+# the failure this repository keeps finding.+#+# WHAT THIS FILE GUARANTEES+#+# * The authentication bypass is unreachable. `ENVIRONMENT` and `AISOC_DEV_MODE`+# are fixed values here, not interpolated, so no `.env` can re-enable them.+# * Nothing starts on a default credential. Every secret uses `${VAR:?...}`,+# which makes Compose refuse to start and name the variable, rather than+# booting on `aisoc_dev_secret` and looking healthy.+# * No datastore is reachable from the host. Postgres, Redis, Kafka and+# Qdrant publish no ports at all; the console and the ingest endpoint are+# the only things bound, and they bind to loopback unless you set+# AISOC_BIND_ADDR deliberately (put a TLS-terminating proxy in front —+# see docs/runbooks/HARDENING.md section 2).+#+# WHAT IT DOES NOT DO+#+# TLS, secret rotation, network policies, audit-log forwarding and backups+# are not in a compose file. `docs/runbooks/HARDENING.md` is the checklist+# and this file is the starting posture, not the finished one.++include:+ - ./docker-compose.yml++services:+ # ---------------------------------------------------------------------------+ # Data plane. No published ports: in the development stack these are bound to+ # 127.0.0.1 so a developer can attach a client, which is reasonable there and+ # is an attack surface here. `ports: []` overrides the base list rather than+ # appending to it.+ # ---------------------------------------------------------------------------+ postgres:+ ports: !reset []+ restart: unless-stopped+ environment:+ POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD — the development default is a published literal}+ # The DML-only runtime role the services connect as, so the row-level+ # security policies actually apply to them. Separate from the owner+ # password above and rotated the same way.+ AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:?set AISOC_APP_DB_PASSWORD — the development default is a published literal}++ redis:+ ports: !reset []+ restart: unless-stopped++ zookeeper:+ ports: !reset []+ restart: unless-stopped++ kafka:+ ports: !reset []+ restart: unless-stopped++ qdrant:+ ports: !reset []+ restart: unless-stopped++ # ---------------------------------------------------------------------------+ # Application services. `ENVIRONMENT` and `AISOC_DEV_MODE` are literals on+ # purpose: an interpolated value here would let a stray `.env` put a+ # production deployment back into the environment where the auth bypass+ # lives, which is the defect this file exists to make impossible.+ # ---------------------------------------------------------------------------+ api:+ restart: unless-stopped+ environment:+ ENVIRONMENT: production+ AISOC_DEV_MODE: 0+ POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}+ AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:?set AISOC_APP_DB_PASSWORD}+ DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:?set AISOC_APP_DB_PASSWORD}@postgres:5432/aisoc+ DATABASE_MIGRATION_URL: postgresql+asyncpg://aisoc:${POSTGRES_PASSWORD:?set POSTGRES_PASSWORD}@postgres:5432/aisoc+ REDIS_URL: redis://:${REDIS_PASSWORD:?set REDIS_PASSWORD}@redis:6379/0+ # `make env` generates all six of these. They are required rather than+ # defaulted because a service that starts without one either serves+ # unauthenticated or signs with a key published in this repository —+ # both were reported vulnerabilities, fixed in v12.0.0.+ SECRET_KEY: ${SECRET_KEY:?set SECRET_KEY — run `make env` to generate it}+ AISOC_CREDENTIAL_KEY: ${AISOC_CREDENTIAL_KEY:?set AISOC_CREDENTIAL_KEY — run `make env` to generate it}+ AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:?set AISOC_SERVICE_TOKEN — run `make env` to generate it}+ AISOC_ACTIONS_SERVICE_TOKEN: ${AISOC_ACTIONS_SERVICE_TOKEN:?set AISOC_ACTIONS_SERVICE_TOKEN — run `make env` to generate it}+ AISOC_REALTIME_JWT_SECRET: ${AISOC_REALTIME_JWT_SECRET:?set AISOC_REALTIME_JWT_SECRET — run `make env` to generate it}+ REALTIME_INTERNAL_TOKEN: ${REALTIME_INTERNAL_TOKEN:?set REALTIME_INTERNAL_TOKEN — run `make env` to generate it}+ NEO4J_PASSWORD: ${NEO4J_PASSWORD:?set NEO4J_PASSWORD}+ CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:?set CLICKHOUSE_PASSWORD}++ ingest-worker:+ restart: unless-stopped+ environment:+ # Both keys, because `envmode.Current()` reads ENV first and the base+ # file's ENV is what shadowed an operator's ENVIRONMENT before this.+ ENV: production+ ENVIRONMENT: production+ # Required outside development by+ # services/ingest/internal/config/config.go. This is the service that+ # accepts events off the network, so it is the last one that should be+ # allowed to start without a signing secret.+ JWT_SECRET: ${JWT_SECRET:?set JWT_SECRET — ingest refuses to start without it outside development}+ AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:?set AISOC_SERVICE_TOKEN}+ AISOC_NEO4J_PASSWORD: ${NEO4J_PASSWORD:?set NEO4J_PASSWORD}++ fusion:+ restart: unless-stopped+ environment:+ ENVIRONMENT: production+ AISOC_DEV_MODE: 0+ AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:?set AISOC_SERVICE_TOKEN}+ CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:?set CLICKHOUSE_PASSWORD}++ agents:+ restart: unless-stopped+ environment:+ ENVIRONMENT: production+ AISOC_DEV_MODE: 0+ AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:?set AISOC_SERVICE_TOKEN}+ AISOC_ACTIONS_SERVICE_TOKEN: ${AISOC_ACTIONS_SERVICE_TOKEN:?set AISOC_ACTIONS_SERVICE_TOKEN}+ NEO4J_PASSWORD: ${NEO4J_PASSWORD:?set NEO4J_PASSWORD}++ actions:+ restart: unless-stopped+ environment:+ ENVIRONMENT: production+ AISOC_DEV_MODE: 0+ # Without this the service answers 503 on every mutating route, which is+ # the intended failure introduced in v12.0.0 — it previously served+ # isolate_host and run_script to anything that could reach the port.+ AISOC_ACTIONS_SERVICE_TOKEN: ${AISOC_ACTIONS_SERVICE_TOKEN:?set AISOC_ACTIONS_SERVICE_TOKEN}++ connectors:+ restart: unless-stopped+ environment:+ ENVIRONMENT: production+ AISOC_DEV_MODE: 0+ AISOC_CREDENTIAL_KEY: ${AISOC_CREDENTIAL_KEY:?set AISOC_CREDENTIAL_KEY}+ AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:?set AISOC_SERVICE_TOKEN}+ SECRET_KEY: ${SECRET_KEY:?set SECRET_KEY}++ threatintel:+ restart: unless-stopped+ environment:+ ENVIRONMENT: production+ AISOC_DEV_MODE: 0+ NEO4J_PASSWORD: ${NEO4J_PASSWORD:?set NEO4J_PASSWORD}++ realtime:+ restart: unless-stopped+ environment:+ # Unset, the edge rejects every connection and /internal/* answers 503.+ # Both used to fall back: the ticket verifier to a constant committed to+ # this repository, and the internal guard to "authorized".+ AISOC_REALTIME_JWT_SECRET: ${AISOC_REALTIME_JWT_SECRET:?set AISOC_REALTIME_JWT_SECRET — run `make env` to generate it}+ REALTIME_INTERNAL_TOKEN: ${REALTIME_INTERNAL_TOKEN:?set REALTIME_INTERNAL_TOKEN — run `make env` to generate it}++ litellm:+ restart: unless-stopped++ ollama:+ restart: unless-stopped++ # The console. Demo mode is inlined at build time by Next, so this asserts+ # the runtime side only; pull the `latest` tag rather than a `-demo` one.+ web:+ restart: unless-stopped+ environment:+ NODE_ENV: production+ AISOC_DEMO_MODE: "false"++ # ---------------------------------------------------------------------------+ # `full`-profile stores. They do not run in a CORE deployment, but the base+ # file gives every service a default that points at the published dev+ # credential, so `--profile full` would start Neo4j and ClickHouse on a+ # password anyone can read. Required here rather than defaulted, and required+ # on the services that dial them so the two cannot disagree.+ # ---------------------------------------------------------------------------+ neo4j:+ ports: !reset []+ restart: unless-stopped+ environment:+ NEO4J_AUTH: neo4j/${NEO4J_PASSWORD:?set NEO4J_PASSWORD — required for the full profile}++ clickhouse:+ ports: !reset []+ restart: unless-stopped+ environment:+ CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:?set CLICKHOUSE_PASSWORD — required for the full profile}++ opensearch:+ ports: !reset []+ restart: unless-stopped++ # Kafka UI browses every topic with no authentication of its own. It is+ # useful while developing and is not something to run beside production data,+ # so it is moved onto a profile nothing here activates: `--profile full` no+ # longer starts it. An operator who wants it can still ask for it by name.+ kafka-ui:+ profiles: !override ["debug-ui"]+ ports: !reset []diff --git a/docker-compose.yml b/docker-compose.ymlindex 8a8bc3be6..354dfabe7 100644--- a/docker-compose.yml+++ b/docker-compose.yml@@ -128,7 +128,7 @@ services:redis:image: redis:7-alpinecontainer_name: aisoc-redis- command: redis-server --appendonly yes --requirepass redis_dev_secret+ command: redis-server --appendonly yes --requirepass ${REDIS_PASSWORD:-redis_dev_secret}ports:- "${AISOC_BIND_ADDR:-127.0.0.1}:6379:6379"volumes:@@ -136,7 +136,7 @@ services:networks:- aisochealthcheck:- test: ["CMD", "redis-cli", "-a", "redis_dev_secret", "ping"]+ test: ["CMD", "redis-cli", "-a", "${REDIS_PASSWORD:-redis_dev_secret}", "ping"]interval: 10stimeout: 5sretries: 5@@ -233,7 +233,7 @@ services:environment:CLICKHOUSE_DB: aisocCLICKHOUSE_USER: aisoc- CLICKHOUSE_PASSWORD: clickhouse_dev_secret+ CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse_dev_secret}CLICKHOUSE_DEFAULT_ACCESS_MANAGEMENT: 1networks:- aisoc@@ -324,7 +324,7 @@ services:- "${AISOC_BIND_ADDR:-127.0.0.1}:7474:7474" # HTTP browser- "${AISOC_BIND_ADDR:-127.0.0.1}:7687:7687" # Boltenvironment:- NEO4J_AUTH: neo4j/neo4j_dev_secret+ NEO4J_AUTH: neo4j/${NEO4J_PASSWORD:-neo4j_dev_secret}NEO4J_PLUGINS: '["apoc"]'NEO4J_dbms_security_procedures_unrestricted: apoc.*NEO4J_dbms_memory_heap_initial__size: 256m@@ -371,7 +371,7 @@ services:# Read by the migration runner to (re)apply the runtime role's password# on an existing data volume, where the postgres init hook never fires.AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}- REDIS_URL: redis://:redis_dev_secret@redis:6379/0+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/0KAFKA_BOOTSTRAP_SERVERS: kafka:29092# The lake client reads these five, not CLICKHOUSE_URL — nothing under# services/api/app reads that name at all. Supplying only the URL left@@ -383,7 +383,7 @@ services:CLICKHOUSE_PORT: 9000CLICKHOUSE_DATABASE: aisocCLICKHOUSE_USER: aisoc- CLICKHOUSE_PASSWORD: clickhouse_dev_secret+ CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse_dev_secret}OPENSEARCH_URL: http://opensearch:9200QDRANT_URL: http://qdrant:6333# The API owns the vault write path; the connectors service reads with@@ -393,7 +393,7 @@ services:AISOC_CREDENTIAL_KEY_ROTATION_FROM: ${AISOC_CREDENTIAL_KEY_ROTATION_FROM:-}NEO4J_URI: bolt://neo4j:7687NEO4J_USER: neo4j- NEO4J_PASSWORD: neo4j_dev_secret+ NEO4J_PASSWORD: ${NEO4J_PASSWORD:-neo4j_dev_secret}# Was a bare literal, so a SECRET_KEY set in .env was ignored and the# JWT signing key silently stayed on the dev default. It also has to# match whatever the connectors service verifies with, or the console@@ -486,11 +486,18 @@ services:- "${AISOC_BIND_ADDR:-127.0.0.1}:8081:8080"- "${AISOC_BIND_ADDR:-127.0.0.1}:9090:9090"environment:- ENV: development+ # Interpolated, not a literal. `.env.example` tells an operator that+ # setting ENVIRONMENT=production is enough, and for every other service+ # it is. This one pinned `development`, and `envmode.Current()` reads+ # ENV *before* ENVIRONMENT — so ingest stayed dev-mode however the+ # operator set it, and `JWT_SECRET must be set in non-development+ # environments` never fired on the one service taking events off the+ # network.+ ENV: ${ENVIRONMENT:-development}HTTP_PORT: 8080METRICS_PORT: 9090KAFKA_BOOTSTRAP_SERVERS: kafka:29092- REDIS_URL: redis://:redis_dev_secret@redis:6379/1+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/1# The inbox webhook routes (/v1/inbox/*) mount only when this is set,# and no compose file set it, so the documented "push events straight# from your SIEM" path was unreachable on every local and CI stack.@@ -515,7 +522,7 @@ services:AISOC_GRAPH_ENABLED: "${AISOC_GRAPH_ENABLED:-false}"AISOC_NEO4J_URI: bolt://neo4j:7687AISOC_NEO4J_USER: neo4j- AISOC_NEO4J_PASSWORD: neo4j_dev_secret+ AISOC_NEO4J_PASSWORD: ${NEO4J_PASSWORD:-neo4j_dev_secret}networks:- aisocrestart: unless-stopped@@ -538,7 +545,7 @@ services:- "${AISOC_BIND_ADDR:-127.0.0.1}:8080:8082"environment:KAFKA_BOOTSTRAP_SERVERS: kafka:29092- REDIS_URL: redis://:redis_dev_secret@redis:6379/2+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/2VIRUSTOTAL_API_KEY: ${VIRUSTOTAL_API_KEY:-}ABUSEIPDB_API_KEY: ${ABUSEIPDB_API_KEY:-}GREYNOISE_API_KEY: ${GREYNOISE_API_KEY:-}@@ -568,7 +575,7 @@ services:- "${AISOC_BIND_ADDR:-127.0.0.1}:8003:8003"environment:KAFKA_BOOTSTRAP_SERVERS: kafka:29092- REDIS_URL: redis://:redis_dev_secret@redis:6379/3+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/3DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisoc# The ClickHouse event lake is a `full`-profile capability: it backs# /lake/sql and the hunt surface, not alerting. CORE runs without it and@@ -578,7 +585,7 @@ services:CLICKHOUSE_PORT: "9000"CLICKHOUSE_DATABASE: aisocCLICKHOUSE_USER: aisoc- CLICKHOUSE_PASSWORD: clickhouse_dev_secret+ CLICKHOUSE_PASSWORD: ${CLICKHOUSE_PASSWORD:-clickhouse_dev_secret}# Fusion defaulted to http://localhost:8082, which inside its own# container is fusion itself — so fuse-time enrichment always failed and# the failure was swallowed at DEBUG. `enrichment` is a `full`-profile@@ -607,7 +614,7 @@ services:- "${AISOC_BIND_ADDR:-127.0.0.1}:8001:8084"environment:DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisoc- REDIS_URL: redis://:redis_dev_secret@redis:6379/4+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/4CORE_API_URL: http://api:8000# Phase B1 — auto-triage every fused alert off the Kafka stream (copilot:# read-only triage, no response dispatch). Degrades to deterministic@@ -617,7 +624,7 @@ services:QDRANT_URL: http://qdrant:6333NEO4J_URI: bolt://neo4j:7687NEO4J_USER: neo4j- NEO4J_PASSWORD: neo4j_dev_secret+ NEO4J_PASSWORD: ${NEO4J_PASSWORD:-neo4j_dev_secret}OPENAI_API_KEY: ${OPENAI_API_KEY:-}ANTHROPIC_API_KEY: ${ANTHROPIC_API_KEY:-}# BYOK / "explain this alert" only — never a task role. A task role asks@@ -858,7 +865,7 @@ services:- "${AISOC_BIND_ADDR:-127.0.0.1}:8002:8085"environment:KAFKA_BOOTSTRAP_SERVERS: kafka:29092- REDIS_URL: redis://:redis_dev_secret@redis:6379/5+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/5DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisocCORE_API_URL: http://api:8000# Must match AISOC_ACTIONS_SERVICE_TOKEN on the api and slack-bot@@ -922,7 +929,7 @@ services:AISOC_DEV_MODE: ${AISOC_DEV_MODE:-1}ENVIRONMENT: ${ENVIRONMENT:-development}KAFKA_BOOTSTRAP_SERVERS: kafka:29092- REDIS_URL: redis://:redis_dev_secret@redis:6379/6+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/6DATABASE_URL: postgresql+asyncpg://aisoc_app:${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}@postgres:5432/aisocCORE_API_URL: http://api:8000# The scheduler decrypts stored connector credentials through the vault@@ -981,7 +988,7 @@ services:- "${AISOC_BIND_ADDR:-127.0.0.1}:8005:8005"environment:KAFKA_BOOTSTRAP_SERVERS: kafka:29092- REDIS_URL: redis://:redis_dev_secret@redis:6379/7+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/7# This service reads OPENSEARCH_HOST/PORT. It was given OPENSEARCH_URL,# which its Settings has no field for and `extra="ignore"` discards, so# the connection worked only because the defaults happened to match the@@ -991,7 +998,7 @@ services:QDRANT_URL: http://qdrant:6333NEO4J_URI: bolt://neo4j:7687NEO4J_USER: neo4j- NEO4J_PASSWORD: neo4j_dev_secret+ NEO4J_PASSWORD: ${NEO4J_PASSWORD:-neo4j_dev_secret}# Same credential the API presents when it proxies /threat-intel/indicators.SECRET_KEY: ${SECRET_KEY:-dev_secret_key_change_in_production}AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:-}@@ -1036,9 +1043,9 @@ services:# migration runner otherwise. Passed here too so this service's own# DATABASE_URL and the role's actual credential come from one value.AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}- REDIS_URL: redis://:redis_dev_secret@redis:6379/8+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/8KAFKA_BOOTSTRAP_SERVERS: kafka:29092- CLICKHOUSE_URL: http://aisoc:clickhouse_dev_secret@clickhouse:8123/aisoc+ CLICKHOUSE_URL: http://aisoc:${CLICKHOUSE_PASSWORD:-clickhouse_dev_secret}@clickhouse:8123/aisocCORE_API_URL: http://api:8000AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:-}AISOC_DEV_MODE: ${AISOC_DEV_MODE:-1}@@ -1092,7 +1099,7 @@ services:# migration runner otherwise. Passed here too so this service's own# DATABASE_URL and the role's actual credential come from one value.AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}- REDIS_URL: redis://:redis_dev_secret@redis:6379/9+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/9KAFKA_BOOTSTRAP_SERVERS: kafka:29092CORE_API_URL: http://api:8000AISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:-}@@ -1145,7 +1152,7 @@ services:# migration runner otherwise. Passed here too so this service's own# DATABASE_URL and the role's actual credential come from one value.AISOC_APP_DB_PASSWORD: ${AISOC_APP_DB_PASSWORD:-aisoc_app_dev_secret}- REDIS_URL: redis://:redis_dev_secret@redis:6379/10+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379/10CORE_API_URL: http://api:8000ATTCK_DATA_PATH: /app/data/enterprise-attack.jsonAISOC_SERVICE_TOKEN: ${AISOC_SERVICE_TOKEN:-}@@ -1181,8 +1188,8 @@ services:- "${AISOC_BIND_ADDR:-127.0.0.1}:8086:4000"environment:KAFKA_BOOTSTRAP_SERVERS: kafka:29092- REDIS_URL: redis://:redis_dev_secret@redis:6379- REDIS_PASSWORD: redis_dev_secret+ REDIS_URL: redis://:${REDIS_PASSWORD:-redis_dev_secret}@redis:6379+ REDIS_PASSWORD: ${REDIS_PASSWORD:-redis_dev_secret}PORT: 4000# Both are generated by `make up`. Neither was passed to this container# before, which is what made the ticket verifier fall back to a constantdiff --git a/scripts/check_profile_service_counts.py b/scripts/check_profile_service_counts.pyindex f6fc446d5..4bb95415b 100644--- a/scripts/check_profile_service_counts.py+++ b/scripts/check_profile_service_counts.py@@ -49,6 +49,14 @@ self_test_if_requested(__file__)REPO_ROOT = repo_root()COMPOSE = REPO_ROOT / "docker-compose.yml"+#: The production stack. It `include`s the base and then differs from it+#: deliberately — `kafka-ui` browses every topic with no authentication of its+#: own and is moved off the `full` profile there — so its `full` count is one+#: lower. Scanned and reported beside the development figures rather than left+#: for an operator to discover: two stacks that differ by design should differ+#: on the page too.+PROD_COMPOSE = REPO_ROOT / "docker-compose.prod.yml"+#: One-shot containers that run to completion and exit. Counted separately#: because "long-running services" is the figure the documents publish, and#: folding a container that exits into it would overstate what is resident.@@ -60,6 +68,13 @@ ONE_SHOT: frozenset[str] = frozenset({"ollama-pull"})#: integer near the word "services": a gate that guesses which numbers are#: claims would either miss the one that matters or flag prose forever.CLAIM_SITES: tuple[tuple[str, str, str], ...] = (+ # The production deployment page. `prod:full` is a separate figure from+ # `full` because the production stack deliberately does not start kafka-ui,+ # and a reader comparing the two pages would otherwise find a discrepancy+ # with nothing explaining it.+ ("apps/docs/docs/deployment/docker.md", "core", r"CORE is (\d+) long-running services here"),+ ("apps/docs/docs/deployment/docker.md", "prod:full", r"`--profile full`\s*\n?is (\d+) rather than"),+ ("apps/docs/docs/deployment/docker.md", "full", r"is \d+ rather than (\d+), the difference being"),("README.md", "core", r"\|\s*\*\*core\*\*\s*\|\s*`make up`\s*\|\s*(\d+)\s*\|"),("README.md", "full", r"\|\s*\*\*full\*\*\s*\|\s*`make up-full`\s*\|\s*(\d+)\s*\|"),("README.md", "core", r"\|\s*\*\*demo\*\*\s*\|\s*`make up && make demo`\s*\|\s*(\d+)\s*\|"),@@ -199,7 +214,10 @@ def _parse_services(text: str) -> dict[str, list[str] | None]:to keep that true. The shape being read is narrow: two-space-indentedservice keys under a top-level ``services:``, and a ``profiles:`` keyfour spaces in, in either inline (``["a", "b"]``) or block (``- a``)- form. Both forms appear in this file.+ form. Both forms appear in this file, and either may carry a Compose Spec+ merge tag (``!override``) in the production overlay — untagged, that line+ read as "no profiles declared" and the gate counted a service the+ production stack does not start."""services: dict[str, list[str] | None] = {}in_services = False@@ -226,13 +244,13 @@ def _parse_services(text: str) -> dict[str, list[str] | None]:if current is None:continue- inline = re.match(r"^ profiles:\s*\[(.*)\]\s*$", raw)+ inline = re.match(r"^ profiles:\s*(?:![a-z]+\s+)?\[(.*)\]\s*$", raw)if inline:services[current] = re.findall(r"[A-Za-z0-9._-]+", inline.group(1))collecting_block = Falsecontinue- if re.match(r"^ profiles:\s*$", raw):+ if re.match(r"^ profiles:\s*(?:![a-z]+\s*)?$", raw):services[current] = []collecting_block = Truecontinue@@ -271,6 +289,29 @@ def _declared_profiles(services: dict[str, list[str] | None]) -> set[str]:return {p for profiles in services.values() if profiles for p in profiles}+def _production_services(base: dict[str, list[str] | None]) -> dict[str, list[str] | None]:+ """The base service set with the production file's profile overrides applied.++ Computed rather than shelled out to ``docker compose config``, because a+ gate that needs a Docker daemon is a gate that skips on the runners that+ do not have one — and a skip here reports nothing while looking green.++ Only ``profiles:`` is applied, because that is the only key that changes+ *which* services a run starts. The production file's other overrides change+ how a service is configured, which the counts do not describe.+ """+ if not PROD_COMPOSE.is_file():+ return base+ overrides = _parse_services(PROD_COMPOSE.read_text(encoding="utf-8"))+ merged = dict(base)+ for name, profiles in overrides.items():+ # `_parse_services` returns None for a service that names no profiles,+ # which in the overlay means "not overridden" rather than "no profile".+ if profiles is not None:+ merged[name] = profiles+ return merged++def scan() -> tuple[dict[str, int], dict[str, set[str]], list[str]]:"""Returns (counts by profile, members by profile, errors)."""if not COMPOSE.is_file():@@ -285,6 +326,14 @@ def scan() -> tuple[dict[str, int], dict[str, set[str]], list[str]]:members = {p: _profile_members(services, p) for p in profiles}counts = {p: _long_running(members[p]) for p in profiles}+ # The production stack, under `prod:` keys so a claim site can name either+ # and the two cannot be confused for one another.+ prod_services = _production_services(services)+ for profile in profiles:+ prod_members = _profile_members(prod_services, profile)+ members[f"prod:{profile}"] = prod_members+ counts[f"prod:{profile}"] = _long_running(prod_members)+for rel, profile, pattern in CLAIM_SITES:path = REPO_ROOT / relif not path.is_file():@@ -352,6 +401,12 @@ def main(argv: list[str] | None = None) -> int:f"profile-service-counts: OK, core {counts['core']}, full {counts.get('full', 0)}, "f"{len(CLAIM_SITES)} published figures agree with docker-compose.yml")+ # Printed every run rather than only when it differs: a figure that appears+ # only on disagreement is one nobody knows the value of.+ print(+ f" production stack: core {counts.get('prod:core', 0)}, full {counts.get('prod:full', 0)}"+ f" ({counts.get('full', 0) - counts.get('prod:full', 0)} fewer on full — kafka-ui is not run beside production data)"+ )return 0diff --git a/scripts/check_published_images.py b/scripts/check_published_images.pyindex 1ace1ea3b..d36ea1dc1 100644--- a/scripts/check_published_images.py+++ b/scripts/check_published_images.py@@ -99,6 +99,18 @@ sys.path.insert(0, str(Path(__file__).resolve().parent))from gate_toolkit import refuses_an_empty_tree, repo_root, scratch_tree # noqa: E402COMPOSE_REL = Path("docker-compose.yml")++#: Compose files that name images an operator is told to pull. The production+#: file was added after this gate and would have shipped unread: it `include`s+#: the base rather than repeating it, so it contributes no `image:` of its own+#: today — but a tag pinned there later would be invisible to this scan, which+#: is exactly the shape the gate exists to catch. Reading the whole set costs+#: nothing and removes the question.+COMPOSE_FILES: tuple[Path, ...] = (+ COMPOSE_REL,+ Path("docker-compose.prod.yml"),+)+CHART_REL = Path("infra/helm/aisoc")VERSION_REL = Path("VERSION")@@ -205,24 +217,68 @@ def _split(image: str) -> tuple[str, str]:return image, "latest"+class _ComposeLoader(yaml.SafeLoader):+ """SafeLoader that understands the Compose Spec's merge-control tags.++ ``!reset`` and ``!override`` tell Compose to clear or replace an inherited+ value instead of merging it, and `docker-compose.prod.yml` needs both to+ unpublish ports the base file binds. PyYAML's safe loader raises+ ``ConstructorError`` on an unknown tag, so without this the gate would+ crash on a file it was extended to read — a scan that cannot run is worse+ than one that reports nothing, because it looks like a broken gate rather+ than a finding.+ """+++def _strip_tag(loader: yaml.Loader, node: yaml.Node) -> object:+ """Return the tagged value itself; the tag is a merge instruction only."""+ if isinstance(node, yaml.SequenceNode):+ return loader.construct_sequence(node, deep=True)+ if isinstance(node, yaml.MappingNode):+ return loader.construct_mapping(node, deep=True)+ return loader.construct_scalar(node)+++for _tag in ("!reset", "!override"):+ _ComposeLoader.add_constructor(_tag, _strip_tag)+++def _load_compose(path: Path) -> dict:+ return yaml.load(path.read_text(encoding="utf-8"), Loader=_ComposeLoader) or {} # noqa: S506++def collect_compose_references(root: Path) -> list[Reference]:- """Every ``image:`` under ``services:``, at the tag ``docker compose up`` requests."""- path = root / COMPOSE_REL- if not path.is_file():- raise ScanError(f"no {COMPOSE_REL} to read: {path}")- try:- document = yaml.safe_load(path.read_text(encoding="utf-8")) or {}- except yaml.YAMLError as exc:- raise ScanError(f"{COMPOSE_REL} is not readable as YAML: {exc}") from exc+ """Every ``image:`` under ``services:``, at the tag ``docker compose up`` requests.++ Reads every file in :data:`COMPOSE_FILES`. The base file must exist — its+ absence means the scan cannot answer and is an error rather than a clean+ verdict — while the others are read when present, so adding or removing one+ does not require editing this function.+ """+ base = root / COMPOSE_REL+ if not base.is_file():+ raise ScanError(f"no {COMPOSE_REL} to read: {base}")references: list[Reference] = []- for name, service in sorted((document.get("services") or {}).items()):- if not isinstance(service, dict) or not isinstance(service.get("image"), str):+ for relative in COMPOSE_FILES:+ path = root / relative+ if not path.is_file():continue- expanded, interpolated = _expand(service["image"])- repository, tag = _split(expanded)- origin = "the default in ${AISOC_VERSION:-…}" if interpolated else "written literally"- references.append(Reference(str(COMPOSE_REL), name, repository, tag, origin))+ try:+ # `!reset` / `!override` are Compose Spec tags. PyYAML's safe+ # loader raises on an unknown tag, so a production overlay using+ # them would make this gate crash rather than scan.+ document = _load_compose(path)+ except yaml.YAMLError as exc:+ raise ScanError(f"{relative} is not readable as YAML: {exc}") from exc++ for name, service in sorted((document.get("services") or {}).items()):+ if not isinstance(service, dict) or not isinstance(service.get("image"), str):+ continue+ expanded, interpolated = _expand(service["image"])+ repository, tag = _split(expanded)+ origin = "the default in ${AISOC_VERSION:-…}" if interpolated else "written literally"+ references.append(Reference(str(relative), name, repository, tag, origin))return referencesdiff --git a/services/api/tests/test_production_environment_requires_auth.py b/services/api/tests/test_production_environment_requires_auth.pynew file mode 100644index 000000000..8056020a1--- /dev/null+++ b/services/api/tests/test_production_environment_requires_auth.py@@ -0,0 +1,87 @@+"""An unauthenticated request must not be admin outside development.++This is the assertion discussion #629 was really about. `docker-compose.yml`+defaults `ENVIRONMENT` to `development`, `development` is in+`AUTH_BYPASS_ENVIRONMENTS`, and `app/api/v1/dev_auth.py` then resolves a+request carrying no bearer token to a demo user whose role is `admin`. The+documented production compose file did not exist, so that was the only stack an+operator could start.++The bypass itself is intentional and stays: a contributor running the stack on+a laptop should not have to seed a user and log in. What was missing is+anything asserting the other half — that naming a non-development environment+actually turns it off at the point a request is served, rather than only in the+module's docstring.++Both directions are asserted. A test that only checked production would pass+against a build where the shim had been deleted, which would be a different+product and a silently broken developer experience.+"""++from __future__ import annotations++import importlib++import pytest+from fastapi import Depends, FastAPI+from fastapi.testclient import TestClient+++def _probe_app(get_current_user) -> FastAPI: # noqa: ANN001 - dependency callable+ """A one-route app whose only job is to resolve the current user.++ The dependency is a default value rather than `Annotated[..., Depends(...)]`+ because this module uses postponed annotations: the annotation would be a+ string, FastAPI would resolve it against this module's globals where the+ dependency is a local, and `user` would degrade to a query parameter —+ giving 422 on every request and testing nothing about authentication.+ """+ app = FastAPI()++ @app.get("/probe")+ async def probe(user=Depends(get_current_user)): # noqa: ANN001, B008+ return {"user_id": str(getattr(user, "user_id", None)), "role": getattr(user, "role", None)}++ return app+++def client_in(monkeypatch: pytest.MonkeyPatch):+ def _build(environment: str) -> TestClient:+ # `current_env_from_os` reads ENV first, so both are set; a stale ENV+ # is precisely what shadowed an operator's ENVIRONMENT on ingest.+ monkeypatch.setenv("ENVIRONMENT", environment)+ monkeypatch.delenv("ENV", raising=False)+ from app.api.v1 import deps, dev_auth++ # Both, and in this order. Reloading `dev_auth` alone leaves+ # `deps.get_current_user` closed over the previous module, and the+ # route then rejects on validation (422) instead of exercising the+ # authentication path this test is about.+ importlib.reload(dev_auth)+ importlib.reload(deps)+ return TestClient(_probe_app(deps.get_current_user), raise_server_exceptions=False)++ return _build+++class TestTheBypassIsEnvironmentGated:+ def test_production_refuses_an_unauthenticated_request(self, client_in) -> None:+ response = client_in("production").get("/probe")+ assert response.status_code == 401, (+ "an unauthenticated request was served outside development — this is the state "+ "every stock `docker compose up` ran in, because the production compose file "+ "the docs pointed at did not exist (discussion #629)"+ )++ @pytest.mark.parametrize("environment", ["staging", "prod", "production"])+ def test_no_non_development_environment_hands_back_a_user(self, client_in, environment: str) -> None:+ """`staging` included deliberately: it is not in the allow-list, and a+ reader could reasonably assume any named environment is safe."""+ assert client_in(environment).get("/probe").status_code == 401++ def test_development_still_resolves_a_user(self, client_in) -> None:+ """The other direction, so this cannot pass on a build with no shim."""+ response = client_in("development").get("/probe")+ assert response.status_code == 200+ assert response.json()["role"] == "admin"diff --git a/tests/test_compose_environment_gate.py b/tests/test_compose_environment_gate.pynew file mode 100644index 000000000..0553fb979--- /dev/null+++ b/tests/test_compose_environment_gate.py@@ -0,0 +1,107 @@+"""A service must not pin its environment to a dev-class literal.++`.env.example` tells an operator that setting `ENVIRONMENT=production` is+enough, and `docker-compose.yml` reads that value. That promise only holds if+every service takes the variable rather than hardcoding a value.++One did not. `ingest-worker` carried `ENV: development` as a literal, and+`envmode.Current()` reads `ENV` *before* `ENVIRONMENT`+(`services/ingest/internal/envmode/envmode.go`), so the operator's setting was+shadowed on the one service that accepts events off the network — leaving+`JWT_SECRET must be set in non-development environments`+(`services/ingest/internal/config/config.go`) unreachable.++The failure shape is worth naming, because it is the one this repository keeps+finding: the escape hatch was documented, and the documentation was true of+every service but one. Nothing compared the two.+"""++from __future__ import annotations++import pathlib++import pytest+import yaml++REPO = pathlib.Path(__file__).resolve().parents[1]++#: Mirrors DEV_ENVIRONMENTS in services/api/app/core/config.py and+#: devEnvironments in services/ingest/internal/envmode/envmode.go. A value in+#: this set relaxes an auth or secret requirement somewhere in the stack.+DEV_ENVIRONMENTS = frozenset({"development", "dev", "local", "demo", "test"})++#: The keys any service reads to decide whether it is in a dev-class+#: environment. `current_env_from_os()` reads ENV or ENVIRONMENT; the Go+#: mirror reads the same two; APP_ENV is checked by the realtime edge.+ENV_KEYS = ("ENV", "ENVIRONMENT", "APP_ENV")++COMPOSE_FILES = [+ REPO / "docker-compose.yml",+ REPO / "infra" / "compose" / "docker-compose.dev.yml",+]+++def _services(path: pathlib.Path) -> dict:+ if not path.is_file():+ return {}+ return (yaml.safe_load(path.read_text(encoding="utf-8")) or {}).get("services") or {}+++def _literal_dev_envs(path: pathlib.Path) -> list[str]:+ """Services pinning a dev-class environment as a literal, not a variable."""+ offenders: list[str] = []+ for name, service in _services(path).items():+ env = (service or {}).get("environment") or {}+ if not isinstance(env, dict):+ continue+ for key in ENV_KEYS:+ value = str(env.get(key, "")).strip()+ # `${...}` is interpolated and therefore follows the operator's+ # .env; only a bare value shadows it.+ if value and "${" not in value and value.lower() in DEV_ENVIRONMENTS:+ offenders.append(f"{path.name}:{name}:{key}={value}")+ return offenders+++class TestTheDocumentedSwitchReachesEveryService:+ def test_the_shipped_compose_pins_no_dev_environment(self) -> None:+ offenders = _literal_dev_envs(REPO / "docker-compose.yml")+ assert not offenders, (+ f"these services hardcode a dev-class environment, so `ENVIRONMENT=production` in .env does not reach them: {offenders}"+ )++ @pytest.mark.parametrize("path", COMPOSE_FILES, ids=lambda p: p.name)+ def test_no_tracked_compose_file_pins_one(self, path: pathlib.Path) -> None:+ """The dev overlay is allowed to *be* development — by interpolation.++ Included because the next service to hardcode one is as likely to land+ in the overlay as in the root file, and a gate that reads one file is+ the shape that let this through.+ """+ assert not _literal_dev_envs(path)++ def test_the_gate_detects_the_literal_it_was_written_for(self, tmp_path: pathlib.Path) -> None:+ """Proven against the defect, not only against the fixed tree.++ Without this, a refactor that stopped reading `environment:` at all+ would report every file clean.+ """+ broken = tmp_path / "docker-compose.yml"+ broken.write_text(+ yaml.safe_dump({"services": {"ingest-worker": {"environment": {"ENV": "development"}}}}),+ encoding="utf-8",+ )+ assert _literal_dev_envs(broken) == ["docker-compose.yml:ingest-worker:ENV=development"]++ def test_an_interpolated_default_is_not_an_offender(self, tmp_path: pathlib.Path) -> None:+ """`${ENVIRONMENT:-development}` is the fix, so it must pass.++ A gate that rejected the default too would force the dev stack to+ require a variable nobody sets locally, and would be reverted.+ """+ ok = tmp_path / "docker-compose.yml"+ ok.write_text(+ yaml.safe_dump({"services": {"api": {"environment": {"ENVIRONMENT": "${ENVIRONMENT:-development}"}}}}),+ encoding="utf-8",+ )+ assert _literal_dev_envs(ok) == []diff --git a/tests/test_production_compose_gate.py b/tests/test_production_compose_gate.pynew file mode 100644index 000000000..ef61dcc9a--- /dev/null+++ b/tests/test_production_compose_gate.py@@ -0,0 +1,198 @@+"""`docker-compose.prod.yml` must be production by construction.++Discussion #629 reported that the deployment page pointed at a production+compose file that did not exist, so the only stack an operator could start was+the development one — where `ENVIRONMENT` defaults to `development`,+`development` is in `AUTH_BYPASS_ENVIRONMENTS`, and `dev_auth.py` resolves an+unauthenticated request to a demo user whose role is `admin`.++A file that merely *documents* the right settings would have the same problem+one release later. These assertions are about the file rather than the docs:+the bypass must be unreachable, no service may start on a credential published+in this repository, and no datastore may be bound to the host.++Every assertion is run against `docker-compose.yml` as well, which must fail+them. That is what distinguishes these from a test that would pass on an empty+file.+"""++from __future__ import annotations++import pathlib+import re++import pytest+import yaml++REPO = pathlib.Path(__file__).resolve().parents[1]+PROD = REPO / "docker-compose.prod.yml"+DEV = REPO / "docker-compose.yml"++#: Mirrors AUTH_BYPASS_ENVIRONMENTS and DEV_ENVIRONMENTS. Any of these in+#: ENV/ENVIRONMENT/APP_ENV relaxes an auth or secret requirement somewhere.+DEV_ENVIRONMENTS = frozenset({"development", "dev", "local", "demo", "test"})++#: Stores that must never be reachable from the host in production. The+#: development file binds each to 127.0.0.1, which is right there and wrong+#: here.+DATASTORES = ("postgres", "redis", "kafka", "zookeeper", "qdrant", "neo4j", "clickhouse", "opensearch")+++class _ComposeLoader(yaml.SafeLoader):+ """Understands `!reset` and `!override`, which the production file needs."""+++def _strip_tag(loader, node): # noqa: ANN001, ANN202 - loader plumbing+ if isinstance(node, yaml.SequenceNode):+ return loader.construct_sequence(node, deep=True)+ if isinstance(node, yaml.MappingNode):+ return loader.construct_mapping(node, deep=True)+ return loader.construct_scalar(node)+++for _tag in ("!reset", "!override"):+ _ComposeLoader.add_constructor(_tag, _strip_tag)+++def _services(path: pathlib.Path) -> dict:+ return (yaml.load(path.read_text(encoding="utf-8"), Loader=_ComposeLoader) or {}).get("services") or {} # noqa: S506+++def _env_items(service: dict) -> list[tuple[str, str]]:+ """`environment:` as pairs, accepting either shape Compose allows."""+ env = (service or {}).get("environment") or {}+ if isinstance(env, dict):+ return [(str(k), str(v)) for k, v in env.items()]+ pairs = []+ for entry in env:+ key, _, value = str(entry).partition("=")+ pairs.append((key, value))+ return pairs+++#: `${VAR:-default}` — the default is what a deployment with no `.env` gets.+_DEFAULTED = re.compile(r"^\$\{[A-Z_]+:-([^}]*)\}$")+++def _resolves_to_dev(value: str) -> bool:+ """True if this value is dev-class with no environment set.++ Both spellings count. A literal `development` is dev-class, and so is+ `${ENVIRONMENT:-development}` on a host with nothing exported — which is+ the state the development stack is designed to start in, and the state a+ production stack must not be able to reach.+ """+ raw = value.strip()+ match = _DEFAULTED.match(raw)+ candidate = match.group(1) if match else raw+ return candidate.strip().lower() in DEV_ENVIRONMENTS+++def _dev_environments(services: dict) -> list[str]:+ found = []+ for name, service in services.items():+ for key, value in _env_items(service):+ if key in ("ENV", "ENVIRONMENT", "APP_ENV") and _resolves_to_dev(value):+ found.append(f"{name}:{key}={value}")+ return found+++def _published_datastores(services: dict) -> list[str]:+ return [name for name in DATASTORES if (services.get(name) or {}).get("ports")]+++def _unguarded_secrets(services: dict) -> list[str]:+ """Secret-shaped variables that carry a default instead of demanding a value.++ `${VAR:?message}` makes Compose refuse to start and name the variable.+ `${VAR:-default}` starts on the default, which for this repository means a+ literal anyone can read in the development file.+ """+ pattern = re.compile(r"\$\{([A-Z_]*(?:PASSWORD|SECRET|TOKEN|KEY)[A-Z_]*):-")+ found = []+ for name, service in services.items():+ for key, value in _env_items(service):+ for variable in pattern.findall(str(value)):+ found.append(f"{name}:{key} defaults {variable}")+ command = (service or {}).get("command")+ if command:+ for variable in pattern.findall(str(command)):+ found.append(f"{name}:command defaults {variable}")+ return found++[email protected](scope="module")+def prod() -> dict:+ assert PROD.is_file(), (+ "docker-compose.prod.yml does not exist. apps/docs/docs/deployment/docker.md "+ "has told operators to run it since the page was written (discussion #629)."+ )+ return _services(PROD)+++class TestTheBypassIsUnreachable:+ def test_no_service_runs_in_a_dev_class_environment(self, prod: dict) -> None:+ assert not _dev_environments(prod)++ def test_the_development_file_would_fail_this(self) -> None:+ """The contrast is the point: this is what production differs from."""+ assert _dev_environments(_services(DEV)), (+ "docker-compose.yml no longer resolves to a dev-class environment with an empty "+ ".env, so this gate is comparing production against nothing and proves less than "+ "it claims"+ )++ def test_environment_and_dev_mode_are_literals_not_variables(self, prod: dict) -> None:+ """An interpolated value here could be re-enabled from a stray `.env`.++ The whole point of the file is that the bypass cannot be switched back+ on by configuration, so these two keys are fixed rather than defaulted.+ """+ for name, service in prod.items():+ env = (service or {}).get("environment") or {}+ for key in ("ENVIRONMENT", "AISOC_DEV_MODE"):+ if key in env:+ assert "${" not in str(env[key]), f"{name}:{key} is interpolated; it must be a fixed value"+++class TestNothingStartsOnAPublishedCredential:+ def test_no_secret_carries_a_default(self, prod: dict) -> None:+ assert not _unguarded_secrets(prod), (+ "these would start on a default rather than refusing: a deployment that boots on "+ "a credential published in this repository looks healthy and is not"+ )++ def test_no_dev_secret_literal_reaches_a_service(self, prod: dict) -> None:+ """Asserted on values, not on the file text.++ A first attempt grepped the raw file and matched the comment that+ explains the defect, which is the difference between reading what+ ships and reading what it says about itself.+ """+ leaked = [+ f"{name}:{key}"+ for name, service in prod.items()+ for key, value in _env_items(service)+ if "_dev_secret" in value and ":-" not in value+ ]+ assert not leaked+++class TestNoDatastoreIsBoundToTheHost:+ def test_production_publishes_no_datastore(self, prod: dict) -> None:+ assert not _published_datastores(prod)++ def test_the_development_file_would_fail_this(self) -> None:+ assert _published_datastores(_services(DEV)), (+ "docker-compose.yml no longer publishes a datastore port, so this gate is comparing production against nothing"+ )++ def test_the_unauthenticated_kafka_ui_is_off_the_full_profile(self, prod: dict) -> None:+ """It browses every topic and has no authentication of its own.++ Moved rather than deleted, so an operator can still ask for it by name+ — the objection is to it starting beside production data unasked.+ """+ profiles = (prod.get("kafka-ui") or {}).get("profiles") or []+ assert "full" not in profiles, "`--profile full` would start kafka-ui beside production data"+ assert profiles, "kafka-ui has no profile at all, so it now starts in the default stack"