Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.
Python agentic-aiai-securityalert-triageblue-teamcybersecuritydetection-engineeringincident-responselanggraphllm-agentsmcpmitre-attackmodel-context-protocolpurple-teamsecurity-operationsself-hostedsiemsoarsocthreat-detectionthreat-intelligence
feat(phase-3.3): upgrade-safety gate (prior release -> HEAD, data survives) (#427)
Beenu Aroracf539b7f0d71
diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.ymlindex e885d39df..0e3bab306 100644--- a/.github/workflows/integration.yml+++ b/.github/workflows/integration.yml@@ -2,7 +2,7 @@ name: Integration# Phase 3 (world-class program) — prove the stack actually runs.#-# Three jobs, all against real containers, no mocks:+# Four jobs, all against real containers, no mocks:## spine — raw vendor event → ingest (OCSF normalize) → Kafka →# fusion (promote + dedup + correlate) → fused topic →@@ -21,6 +21,10 @@ name: Integration# MinIO: seed → backup → destroy → restore → assert# full data integrity. Publishes measured RTO. An# untested backup is not a backup.+# upgrade — install a prior release's migration set, seed data,+# then upgrade to HEAD and migrate: assert the seeded+# data survives (destructive-migration detection) and+# every HEAD migration applied. People upgrade; test it.## The spine job builds `fusion` and `ingest-worker` from PR source (they own# the bridges under test) and pulls the other service images from GHCR@@ -368,3 +372,132 @@ jobs:echo "| measured RTO | ${RTO_SECONDS}s |"echo "| RPO | last completed backup (event-loss window = backup cadence) |"} >> "$GITHUB_STEP_SUMMARY"++ # ── Upgrade safety: install a prior release, then upgrade to HEAD ──────────+ # "People upgrade. Nothing tested it." Apply the migration set as of a prior+ # release, seed data on that released schema, then apply HEAD's migration set+ # (the real self-host upgrade path) and assert the seeded rows survive and+ # every HEAD migration applied. The signal this adds over the fresh-apply job+ # above is destructive-migration detection: a migration that drops or rewrites+ # existing data fails here even though it applies cleanly on an empty database.+ #+ # - v7.5.0 → HEAD: the real "last release → current" self-host upgrade path.+ # - v7.3.1 → HEAD: an older baseline so several migrations land incrementally+ # on a pre-existing, populated schema (not a fresh apply in disguise).+ upgrade:+ name: Upgrade (${{ matrix.from_ref }} → HEAD, data survives)+ runs-on: ubuntu-latest+ timeout-minutes: 15+ strategy:+ fail-fast: false+ matrix:+ from_ref: ['v7.3.1', 'v7.5.0']+ services:+ postgres:+ image: postgres:16+ env:+ POSTGRES_USER: aisoc+ POSTGRES_PASSWORD: aisoc_ci_secret+ POSTGRES_DB: aisoc+ ports:+ - 5432:5432+ options: >-+ --health-cmd "pg_isready -U aisoc"+ --health-interval 5s+ --health-timeout 5s+ --health-retries 10+ env:+ DATABASE_URL: postgresql+asyncpg://aisoc:aisoc_ci_secret@localhost:5432/aisoc+ SECRET_KEY: ci-dummy-secret-key-at-least-32bytes!+ ENVIRONMENT: development+ AISOC_MIGRATIONS_STRICT: '1'+ PGPASSWORD: aisoc_ci_secret+ steps:+ - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7+ with:+ # Full history + tags so we can check out the released migration set.+ fetch-depth: 0+ fetch-tags: true++ - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6+ with:+ python-version: '3.12'+ cache: pip++ # Same importable-subset recipe as the migrations job — run_migrations+ # imports app.core.config, which needs the settings surface only.+ - name: Install API dependencies+ run: >-+ pip install --quiet+ fastapi "pydantic[email]" pydantic-settings "sqlalchemy[asyncio]" asyncpg+ structlog "python-jose[cryptography]" "passlib[bcrypt]" tenacity pyyaml+ prometheus-client "opentelemetry-sdk" "opentelemetry-api"+ "opentelemetry-exporter-otlp-proto-grpc"+ "opentelemetry-instrumentation-fastapi"+ "strawberry-graphql[fastapi]"+ neo4j redis celery httpx aiofiles email-validator PyJWT+ "sqlglot>=23,<27" aiosqlite "croniter>=2.0,<7.0"++ - name: Record HEAD sha+ id: head+ run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"++ - name: Install the released schema (${{ matrix.from_ref }})+ run: |+ set -euo pipefail+ # `git checkout <ref> -- <path>` only overwrites paths present in+ # <ref>; it will NOT delete files added since. rm -rf first so the+ # migrations dir exactly matches the released set (otherwise HEAD's+ # newer files leak in and pass 1 silently becomes a fresh apply).+ rm -rf services/api/migrations+ git checkout "${{ matrix.from_ref }}" -- services/api/migrations+ echo "migration files at ${{ matrix.from_ref }}: $(ls services/api/migrations/*.sql | wc -l)"+ ( cd services/api && python -m app.scripts.run_migrations )++ - name: Seed data on the released schema+ run: |+ set -euo pipefail+ psql -h localhost -U aisoc -d aisoc <<'SQL'+ CREATE TABLE upgrade_probe (id SERIAL PRIMARY KEY, payload TEXT NOT NULL);+ INSERT INTO upgrade_probe (payload)+ SELECT md5(g::text) FROM generate_series(1, 250) AS g;+ SQL+ COUNT=$(psql -h localhost -U aisoc -d aisoc -tAc "SELECT count(*) FROM upgrade_probe")+ echo "seeded ${COUNT} probe rows on the ${{ matrix.from_ref }} schema"+ [ "${COUNT}" = "250" ]++ - name: Upgrade to HEAD and migrate+ run: |+ set -euo pipefail+ rm -rf services/api/migrations+ git checkout "${{ steps.head.outputs.sha }}" -- services/api/migrations+ HEAD_COUNT=$(ls services/api/migrations/*.sql | wc -l)+ echo "migration files at HEAD: ${HEAD_COUNT}"+ ( cd services/api && python -m app.scripts.run_migrations )+ echo "HEAD_MIGRATION_COUNT=${HEAD_COUNT}" >> "$GITHUB_ENV"++ - name: Assert seeded data survived + every HEAD migration applied+ run: |+ set -euo pipefail+ PROBE=$(psql -h localhost -U aisoc -d aisoc -tAc "SELECT count(*) FROM upgrade_probe")+ TRACKED=$(psql -h localhost -U aisoc -d aisoc -tAc "SELECT count(*) FROM aisoc_schema_migrations")+ echo "PROBE_ROWS=${PROBE}" >> "$GITHUB_ENV"+ echo "probe rows after upgrade: ${PROBE} (expect 250)"+ echo "tracked migrations: ${TRACKED} (expect ${HEAD_MIGRATION_COUNT})"+ [ "${PROBE}" = "250" ] \+ || { echo "::error::a migration destroyed data across the ${{ matrix.from_ref }} → HEAD upgrade (${PROBE}/250 rows survived)"; exit 1; }+ [ "${TRACKED}" = "${HEAD_MIGRATION_COUNT}" ] \+ || { echo "::error::not every HEAD migration applied on upgrade (${TRACKED}/${HEAD_MIGRATION_COUNT})"; exit 1; }++ - name: Publish upgrade summary+ if: always()+ run: |+ {+ echo "## Upgrade gate — ${{ matrix.from_ref }} → HEAD"+ echo ""+ echo "| check | value |"+ echo "|-------|-------|"+ echo "| probe rows seeded on released schema | 250 |"+ echo "| probe rows after upgrade | ${PROBE_ROWS:-?} |"+ echo "| HEAD migrations applied | ${HEAD_MIGRATION_COUNT:-?} |"+ } >> "$GITHUB_STEP_SUMMARY"diff --git a/CHANGELOG.md b/CHANGELOG.mdindex 7db8bb93b..192617a40 100644--- a/CHANGELOG.md+++ b/CHANGELOG.md@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0### Added+- **Phase 3.3 — upgrade-safety gate ("people upgrade; nothing tested it").** New `upgrade` job in `integration.yml` (matrix `v7.5.0 → HEAD` and `v7.3.1 → HEAD` against a real Postgres 16): install a prior release's migration set, seed 250 probe rows on that released schema, then apply HEAD's migration set — the actual self-host upgrade path — and assert the seeded rows survive **and** every HEAD migration applied. The signal this adds beyond the existing fresh-apply job is **destructive-migration detection**: a migration that drops or rewrites existing data fails here even though it applies cleanly on an empty database. The `v7.3.1 → HEAD` leg lands 8 incremental migrations on a pre-existing populated schema (verified locally: 250/250 rows survived, 55/55 migrations tracked). Uses `git checkout <ref> -- services/api/migrations` with an `rm -rf` first so the released set is exact (a `checkout … -- path` alone leaks HEAD-only files and silently degrades to a fresh apply).- **Phase 3.2 — Postgres-outage chaos gate (fail-soft + self-heal).** Extends the `integration.yml` spine job with a second chaos scenario beyond the consumer-kill test: kill Postgres mid-stream, assert the fusion consumer keeps running (the `AlertSink` degrades to "fused alerts still stream over Kafka/WS, persistence dropped for the outage window" instead of crashing the worker), then restart Postgres and assert a *fresh* event persists — proving the sink's asyncpg pool and the API's SQLAlchemy pool both self-heal with no service restart. Distinct event titles side-step fusion's in-memory dedup so the post-restart event is a genuine fresh persist. Verified locally against a real Postgres 16 before gating (persist during outage → `None` with no raise; first persist after restart → row written). `scripts/integration/spine_test.py` polling now swallows transient HTTP errors during the recovery window so a 5xx blip while the API reconnects extends the poll rather than failing the run.- **Phase 3.1 — the event spine is now continuous, and CI proves it with real containers.** The reality audit found two silent gaps in the product's central claim: ingest published normalized OCSF events to `aisoc.raw_events` that nothing consumed, and fusion published to `aisoc.alerts.fused` that nothing persisted — a raw event could never become an alert row without a human calling the API. Both bridges now exist in `services/fusion`: a deterministic promotion policy (`app/services/promoter.py` — OCSF Findings-category events and `severity_id >= 4` telemetry become `RawAlert`s; everything else is left to the detection engine by design) and a fail-soft, idempotent Postgres `AlertSink` (`app/services/alert_sink.py` — dedup-fingerprint-guarded insert, duplicates never persisted, DB outage degrades instead of crashing the consumer). New `.github/workflows/integration.yml` gates three claims against real containers, no mocks: (1) **spine** — `POST /v1/ingest/batch` → OCSF normalize → Kafka → fusion → WebSocket `alert.fused` frame + Postgres row via `GET /api/v1/alerts`, plus duplicate suppression and a chaos step (kill the fusion consumer mid-stream, produce, restart, assert zero event loss) — driver: `scripts/integration/spine_test.py`; (2) **migrations** — the full SQL chain applies on fresh Postgres 16, re-run is a no-op (new `AISOC_MIGRATIONS_STRICT=1` mode fails CI on any failed migration), and the UEBA alembic chain round-trips upgrade → downgrade → upgrade; (3) **backup-restore** — `scripts/backup.sh`/`restore.sh` against Postgres + MinIO: seed → backup → destroy → restore → assert full integrity, with measured RTO published to the job summary. 20 new fusion unit tests cover the mapping/skip logic (`test_promoter.py`, `test_alert_sink.py`); full fusion suite 71/71.