Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

fix(deploy): demo-mode soft-fail when Postgres unreachable during release (#373)

Beenu Aroraec73f262e972
diff --git a/apps/docs/docs/deployment/env-vars.md b/apps/docs/docs/deployment/env-vars.md
index 0a128175b..282dc4359 100644
--- a/apps/docs/docs/deployment/env-vars.md
+++ b/apps/docs/docs/deployment/env-vars.md
@@ -36,6 +36,18 @@ The API uses bare environment variable names (no prefix). Booleans accept `true`
| `REFRESH_TOKEN_EXPIRE_DAYS` | `7` | Lifetime of a refresh token |
| `ALGORITHM` | `HS256` | JWT signing algorithm |
+### Migration runner
+
+Source: [`services/api/app/scripts/run_migrations.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/scripts/run_migrations.py)
+
+The migration runner is invoked by deploy orchestrators (Fly's `release_command`, Render's `preDeployCommand`, the Kubernetes `Job` in the Helm chart) the moment a new API VM boots. It races against Postgres becoming reachable — these knobs control how that race is handled.
+
+| Variable | Default | Description |
+|----------|---------|-------------|
+| `AISOC_MIGRATIONS_REQUIRED` | `1` (required) | **Default is safe — leave it alone for production.** When set to `0` / `false` / `no` / `off`, the runner exits `0` instead of `1` if it cannot reach Postgres after exhausting its retry budget (6 attempts, exponential back-off, ~31 s total). This is for *demo* environments where the API code deploy must not be held hostage by a Postgres outage — the running app keeps serving the last-good schema and dataset, and the deploy log carries a loud `MIGRATION SKIPPED` line for an operator to chase. A *hard* migration failure (Postgres reachable, SQL statement failed) still aborts the deploy regardless of this flag — a schema-change PR with a bad migration must always get human attention. |
+
+The Fly demo (`infra/fly/api/fly.toml`) sets `AISOC_MIGRATIONS_REQUIRED = "0"` and pairs it with a shell wrapper in `release_command` that uses a `/tmp/aisoc-migrations-ok` sentinel to decide whether to also run `seed_demo` — a soft-failed migration run skips the seeder (which would just hit the same connect failure through SQLAlchemy and crash the deploy anyway).
+
### Audit log
Source: [`services/api/app/services/audit.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/services/audit.py), [`services/api/app/core/trusted_proxy.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/core/trusted_proxy.py), [`services/api/app/services/audit_redaction.py`](https://github.com/beenuar/AiSOC/blob/main/services/api/app/services/audit_redaction.py). Background: [Security operations → Audit logging](../operations/security#audit-logging).
diff --git a/infra/fly/api/fly.toml b/infra/fly/api/fly.toml
index 5b5b90def..3a5b4014b 100644
--- a/infra/fly/api/fly.toml
+++ b/infra/fly/api/fly.toml
@@ -28,19 +28,27 @@ kill_timeout = "5s"
# which is exactly what we want for demo-tenant data lifecycle:
# 1. `python -m app.scripts.run_migrations` applies the raw-SQL files under
# services/api/migrations/ and tracks them in aisoc_schema_migrations, so
-# the schema stays in step with the image. (This service does not use
-# Alembic — no alembic.ini / env.py exist here; the package is listed in
+# the schema stays in step with the image. On success it touches
+# `/tmp/aisoc-migrations-ok`. (This service does not use Alembic —
+# no alembic.ini / env.py exist here; the package is listed in
# pyproject only for integration tests.)
# 2. `python -m app.scripts.seed_demo` is idempotent — it short-circuits
# when the canonical INC-RT-* case_numbers already exist, so it's safe
# to run on every deploy. First-run boot creates all 15 incidents
# including the in-flight LockBit 3.0 ransomware investigation
# (INC-RT-001) that the web app's "Try the demo" CTA deeplinks to.
-# Without this, hitting / on a freshly cloned Render or Fly deploy lands
-# users on an empty dashboard and breaks the < 5 min clone-to-investigation
-# acceptance criterion in the v1.0 plan.
+#
+# Why the shell wrapper instead of a plain `&&`:
+# With AISOC_MIGRATIONS_REQUIRED=0 (demo policy below), the migration
+# runner exits 0 even if it can't reach Postgres after the retry budget
+# — that keeps the API code deploy from being held hostage by a Fly
+# Postgres outage. But on that path `seed_demo` would just hit the
+# same connect failure through SQLAlchemy and crash the deploy anyway.
+# The wrapper uses the migrations-OK sentinel file to skip `seed_demo`
+# in that scenario. A *hard* migration failure (Postgres reachable, SQL
+# failed) still propagates a non-zero exit and aborts the deploy.
[deploy]
- release_command = "/bin/sh -c 'python -m app.scripts.run_migrations && python -m app.scripts.seed_demo'"
+ release_command = "/bin/sh -c 'rm -f /tmp/aisoc-migrations-ok && python -m app.scripts.run_migrations && { [ -f /tmp/aisoc-migrations-ok ] && python -m app.scripts.seed_demo || echo \"[release_command] migrations soft-failed (DB unreachable) — seed_demo skipped, API code deploy proceeds\"; }'"
[env]
ENVIRONMENT = "demo"
@@ -50,6 +58,15 @@ kill_timeout = "5s"
AISOC_DEMO_MODE = "true"
# Tenant rotation runs daily; the seeder writes the canonical INC-RT-001 case
AISOC_DEMO_TENANT = "demo"
+ # Demo policy: don't hold an API code deploy hostage on a Postgres outage.
+ # Migrations still RUN — they retry on transient connect errors, log every
+ # attempt, and crash on any SQL-level failure — but if Postgres is fully
+ # unreachable through the entire retry budget, the migration runner exits
+ # 0 instead of 1 and the release_command wrapper skips `seed_demo`. The
+ # last-good schema and seeded dataset stay live; an operator gets a loud
+ # "MIGRATION SKIPPED" line in the deploy log to investigate. Flip this
+ # back to "1" if you want demo deploys to fail-closed on DB outages.
+ AISOC_MIGRATIONS_REQUIRED = "0"
# Disable optional integrations (no Kafka/ClickHouse/OpenSearch on demo)
AISOC_DISABLE_KAFKA = "true"
AISOC_DISABLE_CLICKHOUSE = "true"
diff --git a/services/api/app/scripts/run_migrations.py b/services/api/app/scripts/run_migrations.py
index 8244a99bf..133247388 100644
--- a/services/api/app/scripts/run_migrations.py
+++ b/services/api/app/scripts/run_migrations.py
@@ -30,6 +30,7 @@ from __future__ import annotations
import asyncio
import logging
+import os
from pathlib import Path
from urllib.parse import parse_qsl, urlsplit, urlunsplit
@@ -42,6 +43,13 @@ logging.basicConfig(level=logging.INFO, format="%(asctime)s %(levelname)s %(mess
MIGRATIONS_DIR = Path(__file__).resolve().parents[2] / "migrations"
+# Sentinel touched on a successful migration pass. The Fly release_command
+# wrapper (see `infra/fly/api/fly.toml` → `[deploy].release_command`) keys
+# off the presence of this file to decide whether to run `seed_demo`. The
+# path lives under /tmp so it's local to the release VM and naturally
+# disappears when the VM is reaped.
+_SENTINEL_PATH = Path("/tmp/aisoc-migrations-ok")
+
# libpq sslmode → asyncpg ssl kwarg. Mirrors the mapping in
# ``app.db.database._normalize_async_pg_url`` so this script can be run
# standalone without depending on SQLAlchemy's connection plumbing.
@@ -160,6 +168,26 @@ async def _apply_one(conn: asyncpg.Connection, name: str, sql: str) -> tuple[str
return name, False, str(exc)
+def _migrations_required() -> bool:
+ """Whether a migration *connect* failure should crash the process.
+
+ The default is ``True`` — a deploy must not silently ship code against
+ a stale schema. Production-like environments leave this alone.
+
+ Demo environments (the Fly ``aisoc-demo-api`` app, in particular) can
+ opt out by setting ``AISOC_MIGRATIONS_REQUIRED=0``. In that mode, a
+ connect-exhausted failure logs LOUDLY and exits ``0`` so the API code
+ deploy still ships — preserving service for users on a demo while a
+ Fly Postgres outage / SSL-handshake issue / pg_hba misconfig is being
+ diagnosed. Actual SQL migration errors (server reachable, statement
+ failed) are still treated as failures regardless of this flag — a
+ schema-change PR that crashes mid-apply must still fail the deploy
+ so it gets human eyes.
+ """
+ raw = os.environ.get("AISOC_MIGRATIONS_REQUIRED", "1").strip().lower()
+ return raw not in {"0", "false", "no", "off"}
+
+
async def main() -> None:
if not MIGRATIONS_DIR.exists():
logger.warning("migrations dir not found: %s", MIGRATIONS_DIR)
@@ -168,7 +196,36 @@ async def main() -> None:
files = sorted(p for p in MIGRATIONS_DIR.iterdir() if p.suffix == ".sql")
logger.info("Found %d migration files", len(files))
- conn = await _connect()
+ try:
+ conn = await _connect()
+ except (
+ asyncpg.exceptions.ConnectionDoesNotExistError,
+ asyncpg.exceptions.CannotConnectNowError,
+ ConnectionResetError,
+ OSError,
+ ) as exc:
+ if _migrations_required():
+ raise
+ # Demo-mode soft-fail: Postgres unreachable through every retry.
+ # Log loud — a future audit must be able to find this — then exit
+ # 0 so the rest of the deploy proceeds. This is intentionally
+ # restricted to *connect-time* exceptions; once we have a working
+ # connection, any SQL failure during a migration still raises.
+ logger.error(
+ "MIGRATION SKIPPED — could not reach Postgres after retry budget "
+ "(%s: %s). AISOC_MIGRATIONS_REQUIRED=0 → continuing the deploy "
+ "with the existing schema. Investigate the Fly Postgres app and "
+ "either restore connectivity or set AISOC_MIGRATIONS_REQUIRED=1 "
+ "to make this fatal again.",
+ type(exc).__name__,
+ exc,
+ )
+ # Deliberately do NOT write the sentinel — the release_command
+ # wrapper uses its absence to know seed_demo should also be
+ # skipped (seed_demo would just hit the same connect failure
+ # through SQLAlchemy and crash).
+ return
+
try:
await conn.execute(CREATE_MIGRATIONS_TABLE)
already = await _applied(conn)
@@ -190,6 +247,15 @@ async def main() -> None:
finally:
await conn.close()
+ # Sentinel for the release_command wrapper: "migrations ran against a
+ # reachable database — proceed to seed_demo". A soft-failed run skips
+ # this write so the wrapper also skips seeding.
+ _SENTINEL_PATH.parent.mkdir(parents=True, exist_ok=True)
+ _SENTINEL_PATH.write_text(
+ "AiSOC run_migrations.py completed — DB reachable, seed_demo may proceed.\n",
+ encoding="utf-8",
+ )
+
if __name__ == "__main__":
asyncio.run(main())
diff --git a/services/api/tests/test_run_migrations_retry.py b/services/api/tests/test_run_migrations_retry.py
index a3ad8d524..a970bec1f 100644
--- a/services/api/tests/test_run_migrations_retry.py
+++ b/services/api/tests/test_run_migrations_retry.py
@@ -89,3 +89,26 @@ async def test_connect_does_not_retry_auth_failures(monkeypatch):
await run_migrations._connect()
assert call_count["connect"] == 1, "auth failures must propagate after the first attempt, no retries"
+
+
+def test_migrations_required_defaults_to_true(monkeypatch):
+ """Absence of the env var means the previous strict behaviour: any
+ connect failure aborts the deploy. That's the safe default."""
+ monkeypatch.delenv("AISOC_MIGRATIONS_REQUIRED", raising=False)
+ assert run_migrations._migrations_required() is True
+
+
[email protected]("value", ["0", "false", "no", "off", "FALSE", "Off"])
+def test_migrations_required_recognises_opt_out(monkeypatch, value):
+ """Demo-mode opt-out via the documented sentinel values."""
+ monkeypatch.setenv("AISOC_MIGRATIONS_REQUIRED", value)
+ assert run_migrations._migrations_required() is False
+
+
[email protected]("value", ["1", "true", "yes", "on", " 1 ", ""])
+def test_migrations_required_truthy_values(monkeypatch, value):
+ """Anything not in the opt-out vocabulary is treated as 'required'.
+ Empty string included — it's an easy footgun and the safe interpretation
+ is 'I didn't mean to opt out'."""
+ monkeypatch.setenv("AISOC_MIGRATIONS_REQUIRED", value)
+ assert run_migrations._migrations_required() is True