Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

fix(api): reconcile detection_rules/cases schema drift on initdb installs (#492) (#493)

Aadit Singaldcd36682fbcc
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 13c6bb626..f51bb257b 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -251,6 +251,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Fixed
+- **Out-of-the-box 500 from schema drift on migration-bootstrapped installs (#492).**
+ `docker-compose.yml` mounts `services/api/migrations` into
+ `/docker-entrypoint-initdb.d`, so a fresh compose stack builds Postgres from
+ the `001_init.sql` lineage — which created `detection_rules` in its
+ pre-refactor shape (`rule_type`/`rule_content`/`hit_count`/`last_hit_at`) and
+ `cases` without `resolution`/`lessons_learned`. The current `DetectionRule`
+ and `Case` models query the refactored columns, so every default install
+ served `UndefinedColumnError` 500s (e.g. `GET /api/v1/detection/tuning`) and
+ `seed_demo` failed on `cases.resolution`. New
+ `services/api/migrations/046_detection_rules_cases_schema_drift_fix.sql`
+ reconciles both tables with the models — additive, fully idempotent
+ (`ADD COLUMN IF NOT EXISTS`), and dual-lineage safe (the `create_all` path is
+ a no-op; the legacy path backfills `rule_body`/`rule_language` from the old
+ columns and drops the stale `rule_content NOT NULL` under an
+ `information_schema` guard so ORM inserts succeed). New
+ `services/api/tests/test_schema_drift_046.py` pins the fix and adds a forward
+ guard asserting the migration lineage covers every column both models declare.
- **Phase 3.1 gates caught two latent bugs before merge** (exactly what the real-container tier is for — both would have shipped invisibly under the previous mock-only CI). (1) **`scripts/restore.sh` never restored anything.** `resolve_timestamp()` ended in a `[[ -z "$TIMESTAMP" ]] && { … }` guard that evaluates *false* once a timestamp is resolved; as the function's last command that non-zero status propagated out and, under `set -e`, aborted the script before the restore began — for **both** `--latest` and `--timestamp`. An untested backup script had been broken the whole time. Converted the guard to an explicit `if` + `return 0`; the backup → destroy → restore gate now restores 500/500 rows with a measured RTO. (2) **`services/fusion` `AlertSink` silently failed to persist every alert.** The dedup fingerprint (`$10`) was used untyped in both the `INSERT … SELECT` target (the `dedup_hash VARCHAR(64)` column) and `WHERE dedup_hash = $10` (varchar comparisons resolve through `text` operators), so asyncpg's prepare raised `inconsistent types deduced for parameter $10: text versus character varying` and the insert threw — the fused alert streamed over Kafka/WebSocket but never reached the alert store, so the spine's `GET /api/v1/alerts` assertion timed out. Pinned both uses to `::text`; the real-container spine gate now observes the alert row end to end.
### Security
diff --git a/services/api/migrations/046_detection_rules_cases_schema_drift_fix.sql b/services/api/migrations/046_detection_rules_cases_schema_drift_fix.sql
new file mode 100644
index 000000000..24bbd1bfa
--- /dev/null
+++ b/services/api/migrations/046_detection_rules_cases_schema_drift_fix.sql
@@ -0,0 +1,102 @@
+-- 046_detection_rules_cases_schema_drift_fix.sql
+--
+-- Reconcile the `detection_rules` and `cases` tables with the ORM models in
+-- services/api/app/models/detection_rule.py
+-- services/api/app/models/case.py
+--
+-- Why this exists (issue #492)
+-- ────────────────────────────
+-- docker-compose.yml mounts services/api/migrations into
+-- /docker-entrypoint-initdb.d, so on a fresh compose install Postgres builds
+-- the schema from 001_init.sql onward. That lineage created `detection_rules`
+-- in its *pre-refactor* shape (rule_type, rule_content, hit_count, last_hit_at)
+-- and never gained the columns the current DetectionRule model queries
+-- (rule_language, rule_body, category, status, confidence, fp_rate,
+-- suppression_config, threshold_config, total_hits, last_triggered,
+-- is_builtin, created_by_id). The result is a 500 out of the box on any
+-- endpoint that touches the table (e.g. GET /api/v1/detection/tuning):
+--
+-- asyncpg.exceptions.UndefinedColumnError:
+-- column detection_rules.rule_language does not exist
+--
+-- `cases` has the same divergence: the Case model (and the demo seed) use
+-- `resolution` and `lessons_learned`, neither of which the 001 lineage
+-- created — so `python -m app.scripts.seed_demo` fails with:
+--
+-- UndefinedColumnError: column "resolution" of relation "cases" does not exist
+--
+-- 036_detection_rule_provenance.sql already reconciled the single `provenance`
+-- column across both lineages; this migration finishes the job for the rest.
+--
+-- Dual-lineage safety (mirrors the 036 rationale)
+-- ───────────────────────────────────────────────
+-- Two code paths reach this table and both must survive re-running:
+-- * Dev installs that bootstrap via SQLAlchemy `Base.metadata.create_all`
+-- already have the model's columns — every `ADD COLUMN IF NOT EXISTS`
+-- below is a no-op, and the guarded legacy fix-ups are skipped because
+-- the legacy columns don't exist.
+-- * Legacy / compose installs that originated from 001_init.sql get the
+-- missing columns added without touching the legacy rule_type/rule_content
+-- columns, plus a NULL-tolerant backfill so existing rows stay usable.
+-- All clauses are idempotent, so the migration is safe on partially-migrated
+-- environments too.
+
+BEGIN;
+
+-- ── detection_rules: add the columns the current model selects ──────────────
+-- Added NULL-able (no NOT NULL) so the migration tolerates a populated legacy
+-- table; the ORM always supplies these on INSERT. Stats/flags carry the same
+-- defaults as the model so pre-existing legacy rows read back sensibly.
+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS rule_language VARCHAR(30);
+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS rule_body TEXT;
+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS category VARCHAR(100);
+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS status VARCHAR(20) DEFAULT 'testing';
+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS confidence INTEGER DEFAULT 50;
+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS fp_rate DOUBLE PRECISION DEFAULT 0;
+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS suppression_config JSONB NOT NULL DEFAULT '{}'::jsonb;
+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS threshold_config JSONB NOT NULL DEFAULT '{}'::jsonb;
+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS total_hits BIGINT DEFAULT 0;
+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS last_triggered TIMESTAMPTZ;
+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS is_builtin BOOLEAN NOT NULL DEFAULT FALSE;
+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS created_by_id UUID;
+
+CREATE INDEX IF NOT EXISTS idx_rules_category ON detection_rules(category);
+CREATE INDEX IF NOT EXISTS idx_rules_status ON detection_rules(status);
+
+-- Legacy-lineage fix-ups. Only meaningful where the pre-refactor columns still
+-- exist; on the create_all lineage the guarded block is a no-op.
+DO $$
+BEGIN
+ -- Preserve existing rule content under the new column names so legacy rows
+ -- remain readable/editable through the current model instead of surfacing
+ -- as blank rules.
+ IF EXISTS (
+ SELECT 1 FROM information_schema.columns
+ WHERE table_name = 'detection_rules' AND column_name = 'rule_content'
+ ) THEN
+ UPDATE detection_rules SET rule_body = rule_content
+ WHERE rule_body IS NULL AND rule_content IS NOT NULL;
+
+ -- The current model never writes rule_content, so its legacy NOT NULL
+ -- constraint would reject every insert through the ORM.
+ ALTER TABLE detection_rules ALTER COLUMN rule_content DROP NOT NULL;
+ END IF;
+
+ IF EXISTS (
+ SELECT 1 FROM information_schema.columns
+ WHERE table_name = 'detection_rules' AND column_name = 'rule_type'
+ ) THEN
+ UPDATE detection_rules SET rule_language = rule_type
+ WHERE rule_language IS NULL AND rule_type IS NOT NULL;
+ END IF;
+END $$;
+
+-- `category` is NOT NULL in the model; give any pre-existing (or legacy) rows a
+-- placeholder rather than leaving NULLs the read path doesn't expect.
+UPDATE detection_rules SET category = 'uncategorized' WHERE category IS NULL;
+
+-- ── cases: add the columns the current model uses ───────────────────────────
+ALTER TABLE cases ADD COLUMN IF NOT EXISTS resolution TEXT;
+ALTER TABLE cases ADD COLUMN IF NOT EXISTS lessons_learned TEXT;
+
+COMMIT;
diff --git a/services/api/tests/test_schema_drift_046.py b/services/api/tests/test_schema_drift_046.py
new file mode 100644
index 000000000..a1092ae24
--- /dev/null
+++ b/services/api/tests/test_schema_drift_046.py
@@ -0,0 +1,166 @@
+"""Regression guard for issue #492 — migration/model schema drift.
+
+A fresh compose install builds Postgres from ``services/api/migrations/*.sql``
+(mounted into ``/docker-entrypoint-initdb.d``). That lineage created
+``detection_rules`` in its pre-refactor shape and ``cases`` without
+``resolution``/``lessons_learned``, while the current ORM models query the
+refactored columns — so every install served 500s out of the box until
+``046_detection_rules_cases_schema_drift_fix.sql`` reconciled the two.
+
+These tests are pure text/parse checks (no app import, so they run without the
+full API dependency set). They pin two things:
+
+ 1. Migration 046 adds the specific columns named in #492, idempotently.
+ 2. The *whole* migration lineage now defines every column the DetectionRule
+ and Case models declare — the invariant whose violation caused the bug.
+ If someone refactors either model again without a reconciling migration,
+ this fails instead of shipping another out-of-the-box 500.
+"""
+
+from __future__ import annotations
+
+import re
+from pathlib import Path
+
+_API_ROOT = Path(__file__).resolve().parents[1]
+_MIGRATIONS = _API_ROOT / "migrations"
+_MODELS = _API_ROOT / "app" / "models"
+
+MIGRATION_046 = _MIGRATIONS / "046_detection_rules_cases_schema_drift_fix.sql"
+
+# Columns the current models query that the 001_init lineage never created.
+_EXPECTED_046_DETECTION_COLUMNS = {
+ "rule_language",
+ "rule_body",
+ "category",
+ "status",
+ "confidence",
+ "fp_rate",
+ "suppression_config",
+ "threshold_config",
+ "total_hits",
+ "last_triggered",
+ "is_builtin",
+ "created_by_id",
+}
+_EXPECTED_046_CASE_COLUMNS = {"resolution", "lessons_learned"}
+
+
+def _model_columns(model_file: str, class_name: str) -> set[str]:
+ """Column names declared on ``class_name`` in ``model_file``.
+
+ Parses the class body for ``mapped_column`` declarations. When a column is
+ remapped to a different DB name (``mapped_column("db_name", ...)``) the
+ string literal wins; otherwise the attribute name is the column name.
+ ``relationship(...)`` attributes are ignored.
+ """
+ text = (_MODELS / model_file).read_text(encoding="utf-8")
+ # Slice from the class declaration to the next top-level ``class`` (or EOF).
+ start = text.index(f"class {class_name}")
+ rest = text[start + 1 :]
+ nxt = re.search(r"\nclass \w", rest)
+ body = rest[: nxt.start()] if nxt else rest
+
+ columns: set[str] = set()
+ for attr, first_arg in re.findall(
+ r"^\s*(\w+):\s*Mapped\[[^\]]*\]\s*=\s*mapped_column\(\s*(\"[^\"]*\")?",
+ body,
+ flags=re.MULTILINE,
+ ):
+ columns.add(first_arg.strip('"') if first_arg else attr)
+ return columns
+
+
+def _lineage_columns(table: str) -> set[str]:
+ """Every column the SQL migration lineage defines for ``table``.
+
+ Scans ``CREATE TABLE`` bodies and ``ALTER TABLE ... ADD COLUMN`` clauses
+ scoped to ``table`` across all migration files (statements split on ``;``).
+ """
+ columns: set[str] = set()
+ create_re = re.compile(
+ rf"CREATE TABLE(?:\s+IF NOT EXISTS)?\s+{table}\s*\((?P<body>.*)",
+ re.IGNORECASE | re.DOTALL,
+ )
+ alter_re = re.compile(rf"ALTER TABLE\s+(?:ONLY\s+)?{table}\b", re.IGNORECASE)
+ add_col_re = re.compile(r"ADD COLUMN(?:\s+IF NOT EXISTS)?\s+(\w+)", re.IGNORECASE)
+ # A column definition line inside CREATE TABLE: leading identifier that is
+ # not a table-level constraint keyword.
+ coldef_re = re.compile(r"^\s*(\w+)\s", re.MULTILINE)
+ _skip = {
+ "constraint",
+ "primary",
+ "foreign",
+ "unique",
+ "check",
+ "like",
+ "exclude",
+ }
+
+ for sql_file in sorted(_MIGRATIONS.glob("*.sql")):
+ text = sql_file.read_text(encoding="utf-8")
+ for stmt in text.split(";"):
+ m = create_re.search(stmt)
+ if m:
+ # Trim to the matching close paren of the column list.
+ body = m.group("body")
+ depth = 1
+ cut = len(body)
+ for i, ch in enumerate(body):
+ if ch == "(":
+ depth += 1
+ elif ch == ")":
+ depth -= 1
+ if depth == 0:
+ cut = i
+ break
+ for col in coldef_re.findall(body[:cut]):
+ if col.lower() not in _skip:
+ columns.add(col)
+ if alter_re.search(stmt):
+ columns.update(add_col_re.findall(stmt))
+ return columns
+
+
+def test_migration_046_exists_and_is_transactional():
+ assert MIGRATION_046.exists(), "046 schema-drift migration is missing"
+ sql = MIGRATION_046.read_text(encoding="utf-8")
+ assert "BEGIN;" in sql and "COMMIT;" in sql
+
+
+def test_migration_046_adds_expected_columns_idempotently():
+ sql = MIGRATION_046.read_text(encoding="utf-8")
+ added = {c.lower() for c in re.findall(r"ADD COLUMN\s+IF NOT EXISTS\s+(\w+)", sql, re.IGNORECASE)}
+ missing = (_EXPECTED_046_DETECTION_COLUMNS | _EXPECTED_046_CASE_COLUMNS) - added
+ assert not missing, f"046 does not add (IF NOT EXISTS): {sorted(missing)}"
+
+ # Every ADD COLUMN in this drift-fix must be idempotent.
+ plain = re.findall(r"ADD COLUMN\s+(?!IF NOT EXISTS)(\w+)", sql, re.IGNORECASE)
+ assert not plain, f"046 has non-idempotent ADD COLUMN clauses: {plain}"
+
+
+def test_migration_046_drops_legacy_rule_content_not_null_guarded():
+ sql = MIGRATION_046.read_text(encoding="utf-8")
+ assert re.search(
+ r"ALTER COLUMN\s+rule_content\s+DROP NOT NULL", sql, re.IGNORECASE
+ ), "046 must drop the legacy rule_content NOT NULL so ORM inserts succeed"
+ # Guarded so the create_all lineage (no rule_content column) doesn't error.
+ assert "information_schema.columns" in sql
+
+
+def test_detection_rules_lineage_covers_model():
+ model_cols = _model_columns("detection_rule.py", "DetectionRule")
+ lineage_cols = _lineage_columns("detection_rules")
+ missing = model_cols - lineage_cols
+ assert not missing, (
+ "detection_rules migration lineage is missing model columns " f"{sorted(missing)} — a reconciling migration is required (see #492)"
+ )
+
+
+def test_cases_lineage_covers_model():
+ model_cols = _model_columns("case.py", "Case")
+ lineage_cols = _lineage_columns("cases")
+ missing = model_cols - lineage_cols
+ assert not missing, (
+ "cases migration lineage is missing model columns " f"{sorted(missing)} — a reconciling migration is required (see #492)"
+ )