Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
fix(api): reconcile detection_rules/cases schema drift on initdb installs (#492) (#493)
Aadit Singaldcd36682fbcc
diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 13c6bb626..f51bb257b 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -251,6 +251,23 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Fixed+- **Out-of-the-box 500 from schema drift on migration-bootstrapped installs (#492).**+ `docker-compose.yml` mounts `services/api/migrations` into+ `/docker-entrypoint-initdb.d`, so a fresh compose stack builds Postgres from+ the `001_init.sql` lineage — which created `detection_rules` in its+ pre-refactor shape (`rule_type`/`rule_content`/`hit_count`/`last_hit_at`) and+ `cases` without `resolution`/`lessons_learned`. The current `DetectionRule`+ and `Case` models query the refactored columns, so every default install+ served `UndefinedColumnError` 500s (e.g. `GET /api/v1/detection/tuning`) and+ `seed_demo` failed on `cases.resolution`. New+ `services/api/migrations/046_detection_rules_cases_schema_drift_fix.sql`+ reconciles both tables with the models — additive, fully idempotent+ (`ADD COLUMN IF NOT EXISTS`), and dual-lineage safe (the `create_all` path is+ a no-op; the legacy path backfills `rule_body`/`rule_language` from the old+ columns and drops the stale `rule_content NOT NULL` under an+ `information_schema` guard so ORM inserts succeed). New+ `services/api/tests/test_schema_drift_046.py` pins the fix and adds a forward+ guard asserting the migration lineage covers every column both models declare.- **Phase 3.1 gates caught two latent bugs before merge** (exactly what the real-container tier is for — both would have shipped invisibly under the previous mock-only CI). (1) **`scripts/restore.sh` never restored anything.** `resolve_timestamp()` ended in a `[[ -z "$TIMESTAMP" ]] && { … }` guard that evaluates *false* once a timestamp is resolved; as the function's last command that non-zero status propagated out and, under `set -e`, aborted the script before the restore began — for **both** `--latest` and `--timestamp`. An untested backup script had been broken the whole time. Converted the guard to an explicit `if` + `return 0`; the backup → destroy → restore gate now restores 500/500 rows with a measured RTO. (2) **`services/fusion` `AlertSink` silently failed to persist every alert.** The dedup fingerprint (`$10`) was used untyped in both the `INSERT … SELECT` target (the `dedup_hash VARCHAR(64)` column) and `WHERE dedup_hash = $10` (varchar comparisons resolve through `text` operators), so asyncpg's prepare raised `inconsistent types deduced for parameter $10: text versus character varying` and the insert threw — the fused alert streamed over Kafka/WebSocket but never reached the alert store, so the spine's `GET /api/v1/alerts` assertion timed out. Pinned both uses to `::text`; the real-container spine gate now observes the alert row end to end.### Securitydiff --git a/services/api/migrations/046_detection_rules_cases_schema_drift_fix.sql b/services/api/migrations/046_detection_rules_cases_schema_drift_fix.sqlnew file mode 100644index 000000000..24bbd1bfa--- /dev/null+++ b/services/api/migrations/046_detection_rules_cases_schema_drift_fix.sql@@ -0,0 +1,102 @@+-- 046_detection_rules_cases_schema_drift_fix.sql+--+-- Reconcile the `detection_rules` and `cases` tables with the ORM models in+-- services/api/app/models/detection_rule.py+-- services/api/app/models/case.py+--+-- Why this exists (issue #492)+-- ────────────────────────────+-- docker-compose.yml mounts services/api/migrations into+-- /docker-entrypoint-initdb.d, so on a fresh compose install Postgres builds+-- the schema from 001_init.sql onward. That lineage created `detection_rules`+-- in its *pre-refactor* shape (rule_type, rule_content, hit_count, last_hit_at)+-- and never gained the columns the current DetectionRule model queries+-- (rule_language, rule_body, category, status, confidence, fp_rate,+-- suppression_config, threshold_config, total_hits, last_triggered,+-- is_builtin, created_by_id). The result is a 500 out of the box on any+-- endpoint that touches the table (e.g. GET /api/v1/detection/tuning):+--+-- asyncpg.exceptions.UndefinedColumnError:+-- column detection_rules.rule_language does not exist+--+-- `cases` has the same divergence: the Case model (and the demo seed) use+-- `resolution` and `lessons_learned`, neither of which the 001 lineage+-- created — so `python -m app.scripts.seed_demo` fails with:+--+-- UndefinedColumnError: column "resolution" of relation "cases" does not exist+--+-- 036_detection_rule_provenance.sql already reconciled the single `provenance`+-- column across both lineages; this migration finishes the job for the rest.+--+-- Dual-lineage safety (mirrors the 036 rationale)+-- ───────────────────────────────────────────────+-- Two code paths reach this table and both must survive re-running:+-- * Dev installs that bootstrap via SQLAlchemy `Base.metadata.create_all`+-- already have the model's columns — every `ADD COLUMN IF NOT EXISTS`+-- below is a no-op, and the guarded legacy fix-ups are skipped because+-- the legacy columns don't exist.+-- * Legacy / compose installs that originated from 001_init.sql get the+-- missing columns added without touching the legacy rule_type/rule_content+-- columns, plus a NULL-tolerant backfill so existing rows stay usable.+-- All clauses are idempotent, so the migration is safe on partially-migrated+-- environments too.++BEGIN;++-- ── detection_rules: add the columns the current model selects ──────────────+-- Added NULL-able (no NOT NULL) so the migration tolerates a populated legacy+-- table; the ORM always supplies these on INSERT. Stats/flags carry the same+-- defaults as the model so pre-existing legacy rows read back sensibly.+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS rule_language VARCHAR(30);+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS rule_body TEXT;+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS category VARCHAR(100);+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS status VARCHAR(20) DEFAULT 'testing';+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS confidence INTEGER DEFAULT 50;+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS fp_rate DOUBLE PRECISION DEFAULT 0;+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS suppression_config JSONB NOT NULL DEFAULT '{}'::jsonb;+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS threshold_config JSONB NOT NULL DEFAULT '{}'::jsonb;+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS total_hits BIGINT DEFAULT 0;+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS last_triggered TIMESTAMPTZ;+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS is_builtin BOOLEAN NOT NULL DEFAULT FALSE;+ALTER TABLE detection_rules ADD COLUMN IF NOT EXISTS created_by_id UUID;++CREATE INDEX IF NOT EXISTS idx_rules_category ON detection_rules(category);+CREATE INDEX IF NOT EXISTS idx_rules_status ON detection_rules(status);++-- Legacy-lineage fix-ups. Only meaningful where the pre-refactor columns still+-- exist; on the create_all lineage the guarded block is a no-op.+DO $$+BEGIN+ -- Preserve existing rule content under the new column names so legacy rows+ -- remain readable/editable through the current model instead of surfacing+ -- as blank rules.+ IF EXISTS (+ SELECT 1 FROM information_schema.columns+ WHERE table_name = 'detection_rules' AND column_name = 'rule_content'+ ) THEN+ UPDATE detection_rules SET rule_body = rule_content+ WHERE rule_body IS NULL AND rule_content IS NOT NULL;++ -- The current model never writes rule_content, so its legacy NOT NULL+ -- constraint would reject every insert through the ORM.+ ALTER TABLE detection_rules ALTER COLUMN rule_content DROP NOT NULL;+ END IF;++ IF EXISTS (+ SELECT 1 FROM information_schema.columns+ WHERE table_name = 'detection_rules' AND column_name = 'rule_type'+ ) THEN+ UPDATE detection_rules SET rule_language = rule_type+ WHERE rule_language IS NULL AND rule_type IS NOT NULL;+ END IF;+END $$;++-- `category` is NOT NULL in the model; give any pre-existing (or legacy) rows a+-- placeholder rather than leaving NULLs the read path doesn't expect.+UPDATE detection_rules SET category = 'uncategorized' WHERE category IS NULL;++-- ── cases: add the columns the current model uses ───────────────────────────+ALTER TABLE cases ADD COLUMN IF NOT EXISTS resolution TEXT;+ALTER TABLE cases ADD COLUMN IF NOT EXISTS lessons_learned TEXT;++COMMIT;diff --git a/services/api/tests/test_schema_drift_046.py b/services/api/tests/test_schema_drift_046.pynew file mode 100644index 000000000..a1092ae24--- /dev/null+++ b/services/api/tests/test_schema_drift_046.py@@ -0,0 +1,166 @@+"""Regression guard for issue #492 — migration/model schema drift.++A fresh compose install builds Postgres from ``services/api/migrations/*.sql``+(mounted into ``/docker-entrypoint-initdb.d``). That lineage created+``detection_rules`` in its pre-refactor shape and ``cases`` without+``resolution``/``lessons_learned``, while the current ORM models query the+refactored columns — so every install served 500s out of the box until+``046_detection_rules_cases_schema_drift_fix.sql`` reconciled the two.++These tests are pure text/parse checks (no app import, so they run without the+full API dependency set). They pin two things:++ 1. Migration 046 adds the specific columns named in #492, idempotently.+ 2. The *whole* migration lineage now defines every column the DetectionRule+ and Case models declare — the invariant whose violation caused the bug.+ If someone refactors either model again without a reconciling migration,+ this fails instead of shipping another out-of-the-box 500.+"""++from __future__ import annotations++import re+from pathlib import Path++_API_ROOT = Path(__file__).resolve().parents[1]+_MIGRATIONS = _API_ROOT / "migrations"+_MODELS = _API_ROOT / "app" / "models"++MIGRATION_046 = _MIGRATIONS / "046_detection_rules_cases_schema_drift_fix.sql"++# Columns the current models query that the 001_init lineage never created.+_EXPECTED_046_DETECTION_COLUMNS = {+ "rule_language",+ "rule_body",+ "category",+ "status",+ "confidence",+ "fp_rate",+ "suppression_config",+ "threshold_config",+ "total_hits",+ "last_triggered",+ "is_builtin",+ "created_by_id",+}+_EXPECTED_046_CASE_COLUMNS = {"resolution", "lessons_learned"}+++def _model_columns(model_file: str, class_name: str) -> set[str]:+ """Column names declared on ``class_name`` in ``model_file``.++ Parses the class body for ``mapped_column`` declarations. When a column is+ remapped to a different DB name (``mapped_column("db_name", ...)``) the+ string literal wins; otherwise the attribute name is the column name.+ ``relationship(...)`` attributes are ignored.+ """+ text = (_MODELS / model_file).read_text(encoding="utf-8")+ # Slice from the class declaration to the next top-level ``class`` (or EOF).+ start = text.index(f"class {class_name}")+ rest = text[start + 1 :]+ nxt = re.search(r"\nclass \w", rest)+ body = rest[: nxt.start()] if nxt else rest++ columns: set[str] = set()+ for attr, first_arg in re.findall(+ r"^\s*(\w+):\s*Mapped\[[^\]]*\]\s*=\s*mapped_column\(\s*(\"[^\"]*\")?",+ body,+ flags=re.MULTILINE,+ ):+ columns.add(first_arg.strip('"') if first_arg else attr)+ return columns+++def _lineage_columns(table: str) -> set[str]:+ """Every column the SQL migration lineage defines for ``table``.++ Scans ``CREATE TABLE`` bodies and ``ALTER TABLE ... ADD COLUMN`` clauses+ scoped to ``table`` across all migration files (statements split on ``;``).+ """+ columns: set[str] = set()+ create_re = re.compile(+ rf"CREATE TABLE(?:\s+IF NOT EXISTS)?\s+{table}\s*\((?P<body>.*)",+ re.IGNORECASE | re.DOTALL,+ )+ alter_re = re.compile(rf"ALTER TABLE\s+(?:ONLY\s+)?{table}\b", re.IGNORECASE)+ add_col_re = re.compile(r"ADD COLUMN(?:\s+IF NOT EXISTS)?\s+(\w+)", re.IGNORECASE)+ # A column definition line inside CREATE TABLE: leading identifier that is+ # not a table-level constraint keyword.+ coldef_re = re.compile(r"^\s*(\w+)\s", re.MULTILINE)+ _skip = {+ "constraint",+ "primary",+ "foreign",+ "unique",+ "check",+ "like",+ "exclude",+ }++ for sql_file in sorted(_MIGRATIONS.glob("*.sql")):+ text = sql_file.read_text(encoding="utf-8")+ for stmt in text.split(";"):+ m = create_re.search(stmt)+ if m:+ # Trim to the matching close paren of the column list.+ body = m.group("body")+ depth = 1+ cut = len(body)+ for i, ch in enumerate(body):+ if ch == "(":+ depth += 1+ elif ch == ")":+ depth -= 1+ if depth == 0:+ cut = i+ break+ for col in coldef_re.findall(body[:cut]):+ if col.lower() not in _skip:+ columns.add(col)+ if alter_re.search(stmt):+ columns.update(add_col_re.findall(stmt))+ return columns+++def test_migration_046_exists_and_is_transactional():+ assert MIGRATION_046.exists(), "046 schema-drift migration is missing"+ sql = MIGRATION_046.read_text(encoding="utf-8")+ assert "BEGIN;" in sql and "COMMIT;" in sql+++def test_migration_046_adds_expected_columns_idempotently():+ sql = MIGRATION_046.read_text(encoding="utf-8")+ added = {c.lower() for c in re.findall(r"ADD COLUMN\s+IF NOT EXISTS\s+(\w+)", sql, re.IGNORECASE)}+ missing = (_EXPECTED_046_DETECTION_COLUMNS | _EXPECTED_046_CASE_COLUMNS) - added+ assert not missing, f"046 does not add (IF NOT EXISTS): {sorted(missing)}"++ # Every ADD COLUMN in this drift-fix must be idempotent.+ plain = re.findall(r"ADD COLUMN\s+(?!IF NOT EXISTS)(\w+)", sql, re.IGNORECASE)+ assert not plain, f"046 has non-idempotent ADD COLUMN clauses: {plain}"+++def test_migration_046_drops_legacy_rule_content_not_null_guarded():+ sql = MIGRATION_046.read_text(encoding="utf-8")+ assert re.search(+ r"ALTER COLUMN\s+rule_content\s+DROP NOT NULL", sql, re.IGNORECASE+ ), "046 must drop the legacy rule_content NOT NULL so ORM inserts succeed"+ # Guarded so the create_all lineage (no rule_content column) doesn't error.+ assert "information_schema.columns" in sql+++def test_detection_rules_lineage_covers_model():+ model_cols = _model_columns("detection_rule.py", "DetectionRule")+ lineage_cols = _lineage_columns("detection_rules")+ missing = model_cols - lineage_cols+ assert not missing, (+ "detection_rules migration lineage is missing model columns " f"{sorted(missing)} — a reconciling migration is required (see #492)"+ )+++def test_cases_lineage_covers_model():+ model_cols = _model_columns("case.py", "Case")+ lineage_cols = _lineage_columns("cases")+ missing = model_cols - lineage_cols+ assert not missing, (+ "cases migration lineage is missing model columns " f"{sorted(missing)} — a reconciling migration is required (see #492)"+ )