Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
fix(deploy): demo-mode soft-fail when Postgres unreachable during release (#373)
Beenu Aroraec73f262e972
diff --git a/apps/docs/docs/deployment/env-vars.md b/apps/docs/docs/deployment/env-vars.mdindex 0a128175b..282dc4359 100644--- a/apps/docs/docs/deployment/env-vars.md+++ b/apps/docs/docs/deployment/env-vars.md@@ -36,6 +36,18 @@ The API uses bare environment variable names (no prefix). Booleans accept `true`| `REFRESH_TOKEN_EXPIRE_DAYS` | `7` | Lifetime of a refresh token || `ALGORITHM` | `HS256` | JWT signing algorithm |+### Migration runner++Source: [`services/api/app/scripts/run_migrations.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/scripts/run_migrations.py)++The migration runner is invoked by deploy orchestrators (Fly's `release_command`, Render's `preDeployCommand`, the Kubernetes `Job` in the Helm chart) the moment a new API VM boots. It races against Postgres becoming reachable — these knobs control how that race is handled.++| Variable | Default | Description |+|----------|---------|-------------|+| `AISOC_MIGRATIONS_REQUIRED` | `1` (required) | **Default is safe — leave it alone for production.** When set to `0` / `false` / `no` / `off`, the runner exits `0` instead of `1` if it cannot reach Postgres after exhausting its retry budget (6 attempts, exponential back-off, ~31 s total). This is for *demo* environments where the API code deploy must not be held hostage by a Postgres outage — the running app keeps serving the last-good schema and dataset, and the deploy log carries a loud `MIGRATION SKIPPED` line for an operator to chase. A *hard* migration failure (Postgres reachable, SQL statement failed) still aborts the deploy regardless of this flag — a schema-change PR with a bad migration must always get human attention. |++The Fly demo (`infra/fly/api/fly.toml`) sets `AISOC_MIGRATIONS_REQUIRED = "0"` and pairs it with a shell wrapper in `release_command` that uses a `/tmp/aisoc-migrations-ok` sentinel to decide whether to also run `seed_demo` — a soft-failed migration run skips the seeder (which would just hit the same connect failure through SQLAlchemy and crash the deploy anyway).+### Audit logSource: [`services/api/app/services/audit.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/services/audit.py), [`services/api/app/core/trusted_proxy.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/core/trusted_proxy.py), [`services/api/app/services/audit_redaction.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/services/audit_redaction.py). Background: [Security operations → Audit logging](../operations/security#audit-logging).diff --git a/infra/fly/api/fly.toml b/infra/fly/api/fly.tomlindex 5b5b90def..3a5b4014b 100644--- a/infra/fly/api/fly.toml+++ b/infra/fly/api/fly.toml@@ -28,19 +28,27 @@ kill_timeout = "5s"# which is exactly what we want for demo-tenant data lifecycle:# 1. `python -m app.scripts.run_migrations` applies the raw-SQL files under# services/api/migrations/ and tracks them in aisoc_schema_migrations, so-# the schema stays in step with the image. (This service does not use-# Alembic — no alembic.ini / env.py exist here; the package is listed in+# the schema stays in step with the image. On success it touches+# `/tmp/aisoc-migrations-ok`. (This service does not use Alembic —+# no alembic.ini / env.py exist here; the package is listed in# pyproject only for integration tests.)# 2. `python -m app.scripts.seed_demo` is idempotent — it short-circuits# when the canonical INC-RT-* case_numbers already exist, so it's safe# to run on every deploy. First-run boot creates all 15 incidents# including the in-flight LockBit 3.0 ransomware investigation# (INC-RT-001) that the web app's "Try the demo" CTA deeplinks to.-# Without this, hitting / on a freshly cloned Render or Fly deploy lands-# users on an empty dashboard and breaks the < 5 min clone-to-investigation-# acceptance criterion in the v1.0 plan.+#+# Why the shell wrapper instead of a plain `&&`:+# With AISOC_MIGRATIONS_REQUIRED=0 (demo policy below), the migration+# runner exits 0 even if it can't reach Postgres after the retry budget+# — that keeps the API code deploy from being held hostage by a Fly+# Postgres outage. But on that path `seed_demo` would just hit the+# same connect failure through SQLAlchemy and crash the deploy anyway.+# The wrapper uses the migrations-OK sentinel file to skip `seed_demo`+# in that scenario. A *hard* migration failure (Postgres reachable, SQL+# failed) still propagates a non-zero exit and aborts the deploy.[deploy]- release_command = "/bin/sh -c 'python -m app.scripts.run_migrations && python -m app.scripts.seed_demo'"+ release_command = "/bin/sh -c 'rm -f /tmp/aisoc-migrations-ok && python -m app.scripts.run_migrations && { [ -f /tmp/aisoc-migrations-ok ] && python -m app.scripts.seed_demo || echo \"[release_command] migrations soft-failed (DB unreachable) — seed_demo skipped, API code deploy proceeds\"; }'"[env]ENVIRONMENT = "demo"@@ -50,6 +58,15 @@ kill_timeout = "5s"AISOC_DEMO_MODE = "true"# Tenant rotation runs daily; the seeder writes the canonical INC-RT-001 caseAISOC_DEMO_TENANT = "demo"+ # Demo policy: don't hold an API code deploy hostage on a Postgres outage.+ # Migrations still RUN — they retry on transient connect errors, log every+ # attempt, and crash on any SQL-level failure — but if Postgres is fully+ # unreachable through the entire retry budget, the migration runner exits+ # 0 instead of 1 and the release_command wrapper skips `seed_demo`. The+ # last-good schema and seeded dataset stay live; an operator gets a loud+ # "MIGRATION SKIPPED" line in the deploy log to investigate. Flip this+ # back to "1" if you want demo deploys to fail-closed on DB outages.+ AISOC_MIGRATIONS_REQUIRED = "0"# Disable optional integrations (no Kafka/ClickHouse/OpenSearch on demo)AISOC_DISABLE_KAFKA = "true"AISOC_DISABLE_CLICKHOUSE = "true"diff --git a/services/api/app/scripts/run_migrations.py b/services/api/app/scripts/run_migrations.pyindex 8244a99bf..133247388 100644--- a/services/api/app/scripts/run_migrations.py+++ b/services/api/app/scripts/run_migrations.py@@ -30,6 +30,7 @@ from __future__ import annotationsimport asyncioimport logging+import osfrom pathlib import Pathfrom urllib.parse import parse_qsl, urlsplit, urlunsplit@@ -42,6 +43,13 @@ logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(messMIGRATIONS_DIR = Path(__file__).resolve().parents[2] / "migrations"+# Sentinel touched on a successful migration pass. The Fly release_command+# wrapper (see `infra/fly/api/fly.toml` → `[deploy].release_command`) keys+# off the presence of this file to decide whether to run `seed_demo`. The+# path lives under /tmp so it's local to the release VM and naturally+# disappears when the VM is reaped.+_SENTINEL_PATH = Path("/tmp/aisoc-migrations-ok")+# libpq sslmode → asyncpg ssl kwarg. Mirrors the mapping in# ``app.db.database._normalize_async_pg_url`` so this script can be run# standalone without depending on SQLAlchemy's connection plumbing.@@ -160,6 +168,26 @@ async def _apply_one(conn: asyncpg.Connection, name: str, sql: str) -> tuple[strreturn name, False, str(exc)+def _migrations_required() -> bool:+ """Whether a migration *connect* failure should crash the process.++ The default is ``True`` — a deploy must not silently ship code against+ a stale schema. Production-like environments leave this alone.++ Demo environments (the Fly ``aisoc-demo-api`` app, in particular) can+ opt out by setting ``AISOC_MIGRATIONS_REQUIRED=0``. In that mode, a+ connect-exhausted failure logs LOUDLY and exits ``0`` so the API code+ deploy still ships — preserving service for users on a demo while a+ Fly Postgres outage / SSL-handshake issue / pg_hba misconfig is being+ diagnosed. Actual SQL migration errors (server reachable, statement+ failed) are still treated as failures regardless of this flag — a+ schema-change PR that crashes mid-apply must still fail the deploy+ so it gets human eyes.+ """+ raw = os.environ.get("AISOC_MIGRATIONS_REQUIRED", "1").strip().lower()+ return raw not in {"0", "false", "no", "off"}++async def main() -> None:if not MIGRATIONS_DIR.exists():logger.warning("migrations dir not found: %s", MIGRATIONS_DIR)@@ -168,7 +196,36 @@ async def main() -> None:files = sorted(p for p in MIGRATIONS_DIR.iterdir() if p.suffix == ".sql")logger.info("Found %d migration files", len(files))- conn = await _connect()+ try:+ conn = await _connect()+ except (+ asyncpg.exceptions.ConnectionDoesNotExistError,+ asyncpg.exceptions.CannotConnectNowError,+ ConnectionResetError,+ OSError,+ ) as exc:+ if _migrations_required():+ raise+ # Demo-mode soft-fail: Postgres unreachable through every retry.+ # Log loud — a future audit must be able to find this — then exit+ # 0 so the rest of the deploy proceeds. This is intentionally+ # restricted to *connect-time* exceptions; once we have a working+ # connection, any SQL failure during a migration still raises.+ logger.error(+ "MIGRATION SKIPPED — could not reach Postgres after retry budget "+ "(%s: %s). AISOC_MIGRATIONS_REQUIRED=0 → continuing the deploy "+ "with the existing schema. Investigate the Fly Postgres app and "+ "either restore connectivity or set AISOC_MIGRATIONS_REQUIRED=1 "+ "to make this fatal again.",+ type(exc).__name__,+ exc,+ )+ # Deliberately do NOT write the sentinel — the release_command+ # wrapper uses its absence to know seed_demo should also be+ # skipped (seed_demo would just hit the same connect failure+ # through SQLAlchemy and crash).+ return+try:await conn.execute(CREATE_MIGRATIONS_TABLE)already = await _applied(conn)@@ -190,6 +247,15 @@ async def main() -> None:finally:await conn.close()+ # Sentinel for the release_command wrapper: "migrations ran against a+ # reachable database — proceed to seed_demo". A soft-failed run skips+ # this write so the wrapper also skips seeding.+ _SENTINEL_PATH.parent.mkdir(parents=True, exist_ok=True)+ _SENTINEL_PATH.write_text(+ "AiSOC run_migrations.py completed — DB reachable, seed_demo may proceed.\n",+ encoding="utf-8",+ )+if __name__ == "__main__":asyncio.run(main())diff --git a/services/api/tests/test_run_migrations_retry.py b/services/api/tests/test_run_migrations_retry.pyindex a3ad8d524..a970bec1f 100644--- a/services/api/tests/test_run_migrations_retry.py+++ b/services/api/tests/test_run_migrations_retry.py@@ -89,3 +89,26 @@ async def test_connect_does_not_retry_auth_failures(monkeypatch):await run_migrations._connect()assert call_count["connect"] == 1, "auth failures must propagate after the first attempt, no retries"+++def test_migrations_required_defaults_to_true(monkeypatch):+ """Absence of the env var means the previous strict behaviour: any+ connect failure aborts the deploy. That's the safe default."""+ monkeypatch.delenv("AISOC_MIGRATIONS_REQUIRED", raising=False)+ assert run_migrations._migrations_required() is True++[email protected]("value", ["0", "false", "no", "off", "FALSE", "Off"])+def test_migrations_required_recognises_opt_out(monkeypatch, value):+ """Demo-mode opt-out via the documented sentinel values."""+ monkeypatch.setenv("AISOC_MIGRATIONS_REQUIRED", value)+ assert run_migrations._migrations_required() is False++[email protected]("value", ["1", "true", "yes", "on", " 1 ", ""])+def test_migrations_required_truthy_values(monkeypatch, value):+ """Anything not in the opt-out vocabulary is treated as 'required'.+ Empty string included — it's an easy footgun and the safe interpretation+ is 'I didn't mean to opt out'."""+ monkeypatch.setenv("AISOC_MIGRATIONS_REQUIRED", value)+ assert run_migrations._migrations_required() is True