Skip to content

armgit / AiSOC

Public Imported from GitHub
0

Open-source AI Security Operations Center: alert fusion, LLM-agent triage, MITRE ATT&CK investigation, and a replayable decision ledger for every agent step. Self-hostable, runs with no API keys, MIT licensed. Ships an MCP server for Claude, Cursor and Continue.

feat(phase-3.3): upgrade-safety gate (prior release -> HEAD, data survives) (#427)

Beenu Aroracf539b7f0d71
diff --git a/.github/workflows/integration.yml b/.github/workflows/integration.yml
index e885d39df..0e3bab306 100644
--- a/.github/workflows/integration.yml
+++ b/.github/workflows/integration.yml
@@ -2,7 +2,7 @@ name: Integration
# Phase 3 (world-class program) — prove the stack actually runs.
#
-# Three jobs, all against real containers, no mocks:
+# Four jobs, all against real containers, no mocks:
#
# spine — raw vendor event → ingest (OCSF normalize) → Kafka →
# fusion (promote + dedup + correlate) → fused topic →
@@ -21,6 +21,10 @@ name: Integration
# MinIO: seed → backup → destroy → restore → assert
# full data integrity. Publishes measured RTO. An
# untested backup is not a backup.
+# upgrade — install a prior release's migration set, seed data,
+# then upgrade to HEAD and migrate: assert the seeded
+# data survives (destructive-migration detection) and
+# every HEAD migration applied. People upgrade; test it.
#
# The spine job builds `fusion` and `ingest-worker` from PR source (they own
# the bridges under test) and pulls the other service images from GHCR
@@ -368,3 +372,132 @@ jobs:
echo "| measured RTO | ${RTO_SECONDS}s |"
echo "| RPO | last completed backup (event-loss window = backup cadence) |"
} >> "$GITHUB_STEP_SUMMARY"
+
+ # ── Upgrade safety: install a prior release, then upgrade to HEAD ──────────
+ # "People upgrade. Nothing tested it." Apply the migration set as of a prior
+ # release, seed data on that released schema, then apply HEAD's migration set
+ # (the real self-host upgrade path) and assert the seeded rows survive and
+ # every HEAD migration applied. The signal this adds over the fresh-apply job
+ # above is destructive-migration detection: a migration that drops or rewrites
+ # existing data fails here even though it applies cleanly on an empty database.
+ #
+ # - v7.5.0 → HEAD: the real "last release → current" self-host upgrade path.
+ # - v7.3.1 → HEAD: an older baseline so several migrations land incrementally
+ # on a pre-existing, populated schema (not a fresh apply in disguise).
+ upgrade:
+ name: Upgrade (${{ matrix.from_ref }} → HEAD, data survives)
+ runs-on: ubuntu-latest
+ timeout-minutes: 15
+ strategy:
+ fail-fast: false
+ matrix:
+ from_ref: ['v7.3.1', 'v7.5.0']
+ services:
+ postgres:
+ image: postgres:16
+ env:
+ POSTGRES_USER: aisoc
+ POSTGRES_PASSWORD: aisoc_ci_secret
+ POSTGRES_DB: aisoc
+ ports:
+ - 5432:5432
+ options: >-
+ --health-cmd "pg_isready -U aisoc"
+ --health-interval 5s
+ --health-timeout 5s
+ --health-retries 10
+ env:
+ DATABASE_URL: postgresql+asyncpg://aisoc:aisoc_ci_secret@localhost:5432/aisoc
+ SECRET_KEY: ci-dummy-secret-key-at-least-32bytes!
+ ENVIRONMENT: development
+ AISOC_MIGRATIONS_STRICT: '1'
+ PGPASSWORD: aisoc_ci_secret
+ steps:
+ - uses: actions/checkout@9c091bb21b7c1c1d1991bb908d89e4e9dddfe3e0 # v7
+ with:
+ # Full history + tags so we can check out the released migration set.
+ fetch-depth: 0
+ fetch-tags: true
+
+ - uses: actions/setup-python@ece7cb06caefa5fff74198d8649806c4678c61a1 # v6
+ with:
+ python-version: '3.12'
+ cache: pip
+
+ # Same importable-subset recipe as the migrations job — run_migrations
+ # imports app.core.config, which needs the settings surface only.
+ - name: Install API dependencies
+ run: >-
+ pip install --quiet
+ fastapi "pydantic[email]" pydantic-settings "sqlalchemy[asyncio]" asyncpg
+ structlog "python-jose[cryptography]" "passlib[bcrypt]" tenacity pyyaml
+ prometheus-client "opentelemetry-sdk" "opentelemetry-api"
+ "opentelemetry-exporter-otlp-proto-grpc"
+ "opentelemetry-instrumentation-fastapi"
+ "strawberry-graphql[fastapi]"
+ neo4j redis celery httpx aiofiles email-validator PyJWT
+ "sqlglot>=23,<27" aiosqlite "croniter>=2.0,<7.0"
+
+ - name: Record HEAD sha
+ id: head
+ run: echo "sha=$(git rev-parse HEAD)" >> "$GITHUB_OUTPUT"
+
+ - name: Install the released schema (${{ matrix.from_ref }})
+ run: |
+ set -euo pipefail
+ # `git checkout <ref> -- <path>` only overwrites paths present in
+ # <ref>; it will NOT delete files added since. rm -rf first so the
+ # migrations dir exactly matches the released set (otherwise HEAD's
+ # newer files leak in and pass 1 silently becomes a fresh apply).
+ rm -rf services/api/migrations
+ git checkout "${{ matrix.from_ref }}" -- services/api/migrations
+ echo "migration files at ${{ matrix.from_ref }}: $(ls services/api/migrations/*.sql | wc -l)"
+ ( cd services/api && python -m app.scripts.run_migrations )
+
+ - name: Seed data on the released schema
+ run: |
+ set -euo pipefail
+ psql -h localhost -U aisoc -d aisoc <<'SQL'
+ CREATE TABLE upgrade_probe (id SERIAL PRIMARY KEY, payload TEXT NOT NULL);
+ INSERT INTO upgrade_probe (payload)
+ SELECT md5(g::text) FROM generate_series(1, 250) AS g;
+ SQL
+ COUNT=$(psql -h localhost -U aisoc -d aisoc -tAc "SELECT count(*) FROM upgrade_probe")
+ echo "seeded ${COUNT} probe rows on the ${{ matrix.from_ref }} schema"
+ [ "${COUNT}" = "250" ]
+
+ - name: Upgrade to HEAD and migrate
+ run: |
+ set -euo pipefail
+ rm -rf services/api/migrations
+ git checkout "${{ steps.head.outputs.sha }}" -- services/api/migrations
+ HEAD_COUNT=$(ls services/api/migrations/*.sql | wc -l)
+ echo "migration files at HEAD: ${HEAD_COUNT}"
+ ( cd services/api && python -m app.scripts.run_migrations )
+ echo "HEAD_MIGRATION_COUNT=${HEAD_COUNT}" >> "$GITHUB_ENV"
+
+ - name: Assert seeded data survived + every HEAD migration applied
+ run: |
+ set -euo pipefail
+ PROBE=$(psql -h localhost -U aisoc -d aisoc -tAc "SELECT count(*) FROM upgrade_probe")
+ TRACKED=$(psql -h localhost -U aisoc -d aisoc -tAc "SELECT count(*) FROM aisoc_schema_migrations")
+ echo "PROBE_ROWS=${PROBE}" >> "$GITHUB_ENV"
+ echo "probe rows after upgrade: ${PROBE} (expect 250)"
+ echo "tracked migrations: ${TRACKED} (expect ${HEAD_MIGRATION_COUNT})"
+ [ "${PROBE}" = "250" ] \
+ || { echo "::error::a migration destroyed data across the ${{ matrix.from_ref }} → HEAD upgrade (${PROBE}/250 rows survived)"; exit 1; }
+ [ "${TRACKED}" = "${HEAD_MIGRATION_COUNT}" ] \
+ || { echo "::error::not every HEAD migration applied on upgrade (${TRACKED}/${HEAD_MIGRATION_COUNT})"; exit 1; }
+
+ - name: Publish upgrade summary
+ if: always()
+ run: |
+ {
+ echo "## Upgrade gate — ${{ matrix.from_ref }} → HEAD"
+ echo ""
+ echo "| check | value |"
+ echo "|-------|-------|"
+ echo "| probe rows seeded on released schema | 250 |"
+ echo "| probe rows after upgrade | ${PROBE_ROWS:-?} |"
+ echo "| HEAD migrations applied | ${HEAD_MIGRATION_COUNT:-?} |"
+ } >> "$GITHUB_STEP_SUMMARY"
diff --git a/CHANGELOG.md b/CHANGELOG.md
index 7db8bb93b..192617a40 100644
--- a/CHANGELOG.md
+++ b/CHANGELOG.md
@@ -9,6 +9,7 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
### Added
+- **Phase 3.3 — upgrade-safety gate ("people upgrade; nothing tested it").** New `upgrade` job in `integration.yml` (matrix `v7.5.0 → HEAD` and `v7.3.1 → HEAD` against a real Postgres 16): install a prior release's migration set, seed 250 probe rows on that released schema, then apply HEAD's migration set — the actual self-host upgrade path — and assert the seeded rows survive **and** every HEAD migration applied. The signal this adds beyond the existing fresh-apply job is **destructive-migration detection**: a migration that drops or rewrites existing data fails here even though it applies cleanly on an empty database. The `v7.3.1 → HEAD` leg lands 8 incremental migrations on a pre-existing populated schema (verified locally: 250/250 rows survived, 55/55 migrations tracked). Uses `git checkout <ref> -- services/api/migrations` with an `rm -rf` first so the released set is exact (a `checkout … -- path` alone leaks HEAD-only files and silently degrades to a fresh apply).
- **Phase 3.2 — Postgres-outage chaos gate (fail-soft + self-heal).** Extends the `integration.yml` spine job with a second chaos scenario beyond the consumer-kill test: kill Postgres mid-stream, assert the fusion consumer keeps running (the `AlertSink` degrades to "fused alerts still stream over Kafka/WS, persistence dropped for the outage window" instead of crashing the worker), then restart Postgres and assert a *fresh* event persists — proving the sink's asyncpg pool and the API's SQLAlchemy pool both self-heal with no service restart. Distinct event titles side-step fusion's in-memory dedup so the post-restart event is a genuine fresh persist. Verified locally against a real Postgres 16 before gating (persist during outage → `None` with no raise; first persist after restart → row written). `scripts/integration/spine_test.py` polling now swallows transient HTTP errors during the recovery window so a 5xx blip while the API reconnects extends the poll rather than failing the run.
- **Phase 3.1 — the event spine is now continuous, and CI proves it with real containers.** The reality audit found two silent gaps in the product's central claim: ingest published normalized OCSF events to `aisoc.raw_events` that nothing consumed, and fusion published to `aisoc.alerts.fused` that nothing persisted — a raw event could never become an alert row without a human calling the API. Both bridges now exist in `services/fusion`: a deterministic promotion policy (`app/services/promoter.py` — OCSF Findings-category events and `severity_id >= 4` telemetry become `RawAlert`s; everything else is left to the detection engine by design) and a fail-soft, idempotent Postgres `AlertSink` (`app/services/alert_sink.py` — dedup-fingerprint-guarded insert, duplicates never persisted, DB outage degrades instead of crashing the consumer). New `.github/workflows/integration.yml` gates three claims against real containers, no mocks: (1) **spine** — `POST /v1/ingest/batch` → OCSF normalize → Kafka → fusion → WebSocket `alert.fused` frame + Postgres row via `GET /api/v1/alerts`, plus duplicate suppression and a chaos step (kill the fusion consumer mid-stream, produce, restart, assert zero event loss) — driver: `scripts/integration/spine_test.py`; (2) **migrations** — the full SQL chain applies on fresh Postgres 16, re-run is a no-op (new `AISOC_MIGRATIONS_STRICT=1` mode fails CI on any failed migration), and the UEBA alembic chain round-trips upgrade → downgrade → upgrade; (3) **backup-restore** — `scripts/backup.sh`/`restore.sh` against Postgres + MinIO: seed → backup → destroy → restore → assert full integrity, with measured RTO published to the job summary. 20 new fusion unit tests cover the mapping/skip logic (`test_promoter.py`, `test_alert_sink.py`); full fusion suite 71/71.