Self-hosted monitoring for homelabs: host and container metrics and uptime checks from all your machines in one dashboard, with an optional AI agent that diagnoses incidents and suggests fixes.
Python ai-agentsaiopsdashboarddockerfastapihomelabincident-managementllmmonitoringopenaireactroot-cause-analysisself-hosteduptime-monitoring
feat(db): Alembic migrations with a baseline that brings existing databases up to date
karloks2005bb8343692119
diff --git a/TESTS.md b/TESTS.mdindex 9c53b69..c187661 100644--- a/TESTS.md+++ b/TESTS.md@@ -164,6 +164,7 @@ The chart helper tests for the frontend are separate, in `src/frontend/tests/`.| [`test_health_checks.py`](#test_health_checkspy) | 4 | The assistant's fixed health checks and the critical-only webhook. || [`test_history.py`](#test_historypy) | 36 | The history API behind long-range charts: which level of detail answers each range, point limits, gaps, and agreement with the raw samples. || [`test_indexes.py`](#test_indexespy) | 9 | The database indexes on the metrics tables: only the ones queries use, none duplicated. |+| [`test_migrations.py`](#test_migrationspy) | 5 | Database migrations: new and pre-migration databases both end at the latest revision with the models' schema. || [`test_ingest_validation.py`](#test_ingest_validationpy) | 178 | What collectors report is checked and bounded: numbers, names, filesystems, spacing of check-ins. || [`test_input_limits.py`](#test_input_limitspy) | 38 | Size and shape limits on request bodies, the audit-log endpoint, and safe defaults. || [`test_network_checks.py`](#test_network_checkspy) | 94 | The host's real network totals, listening-port inventory, optional internet check, firewall fingerprints, and the validation of all of it. |@@ -682,6 +683,19 @@ The indexes on the metrics tables. The live migration is checked by hand against| `test_deleting_a_server_and_purging_old_rows_can_both_use_an_index` *(4 inputs)* | Every metrics table keeps an index starting with `server_id` (deleting a server cascades through it) and one on `time` alone (the hourly purge). || `test_a_primary_key_is_never_indexed_a_second_time` *(4 inputs)* | No table gets a redundant index on its own `id`. |+### test_migrations.py++The Alembic migrations in `src/backend/migrations`. Every test runs on SQLite, and also on PostgreSQL when+`MIGRATION_TEST_POSTGRES_URL` points at an empty database (its schema is wiped first).++| Test | What it checks |+|---|---|+| `test_the_migrations_live_next_to_the_backend` | The revisions are where the backend looks for them. |+| `test_a_new_database_ends_at_the_latest_revision_with_the_models_schema` | An empty database ends at the latest revision, and Alembic finds no difference between it and the models. |+| `test_upgrading_twice_changes_nothing` | Running the upgrade again at the latest revision changes no table. |+| `test_a_database_from_before_migrations_keeps_its_data_and_ends_at_the_latest_revision` | A database made the old way (`create_all`) keeps its rows and ends at the latest revision with the models' schema. |+| `test_an_old_database_missing_tables_and_columns_is_brought_up_to_date` | An install from an older version, without the availability tables and some `servers` columns, gets them. |+### test_ingest_validation.pyNothing a collector reports is trusted: numbers, names and filesystems are checked, check-ins are spaced, and one bad collector can't break pages for everyone.diff --git a/alembic.ini b/alembic.ininew file mode 100644index 0000000..50a6375--- /dev/null+++ b/alembic.ini@@ -0,0 +1,33 @@+# Database migrations (Alembic). The backend runs them itself at start (src/backend/migrate.py); this file is for+# writing new ones: `alembic revision --autogenerate -m "what changed"` against a database at the latest revision.+# The database comes from DATABASE_URL, as for the backend. See docs/development.md.+[alembic]+script_location = %(here)s/src/backend/migrations+file_template = %%(rev)s_%%(slug)s+prepend_sys_path = .++[loggers]+keys = root,alembic++[handlers]+keys = console++[formatters]+keys = generic++[logger_root]+level = WARNING+handlers = console++[logger_alembic]+level = INFO+handlers =+qualname = alembic++[handler_console]+class = StreamHandler+args = (sys.stderr,)+formatter = generic++[formatter_generic]+format = %(levelname)s [%(name)s] %(message)sdiff --git a/compose.yml b/compose.ymlindex 8f022b2..b230e3c 100644--- a/compose.yml+++ b/compose.yml@@ -11,7 +11,6 @@ services:POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}volumes:- postgres_data:/var/lib/postgresql/data- - ./docker/postgres/init:/docker-entrypoint-initdb.dhealthcheck:test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER:-infradeck} -d $${POSTGRES_DB:-infradeck}"]interval: 10sdiff --git a/docker-compose.server.yml b/docker-compose.server.ymlindex 50bef8f..082c90b 100644--- a/docker-compose.server.yml+++ b/docker-compose.server.yml@@ -16,7 +16,6 @@ services:POSTGRES_PASSWORD: ${POSTGRES_PASSWORD:?Set POSTGRES_PASSWORD in .env}volumes:- postgres_data:/var/lib/postgresql/data- - ./docker/postgres/init:/docker-entrypoint-initdb.dhealthcheck:test: ["CMD-SHELL", "pg_isready -U $${POSTGRES_USER:-infradeck} -d $${POSTGRES_DB:-infradeck}"]interval: 10sdiff --git a/docker/postgres/init/01_create_users.sql b/docker/postgres/init/01_create_users.sqldeleted file mode 100644index 9b1a67c..0000000--- a/docker/postgres/init/01_create_users.sql+++ /dev/null@@ -1,13 +0,0 @@-CREATE TABLE IF NOT EXISTS users (- id SERIAL PRIMARY KEY,- email VARCHAR(255) UNIQUE NOT NULL,- password_hash TEXT NOT NULL,- name VARCHAR(255) NOT NULL,- role VARCHAR(50) NOT NULL DEFAULT 'user',- is_active BOOLEAN NOT NULL DEFAULT TRUE,- created_at TIMESTAMPTZ NOT NULL DEFAULT NOW(),- updated_at TIMESTAMPTZ NOT NULL DEFAULT NOW()-);--CREATE INDEX IF NOT EXISTS idx_users_email ON users (email);-CREATE INDEX IF NOT EXISTS idx_users_role ON users (role);diff --git a/docs/containers.md b/docs/containers.mdindex bee8352..d06dec7 100644--- a/docs/containers.md+++ b/docs/containers.md@@ -140,8 +140,8 @@ while it was down are lost (the collector does not queue them), so the history hThe full list of tables is in [Database schema](database.md).**How it works.** The official `postgres:16-alpine` image with its data in the `postgres_data` volume.-On the very first start, [`docker/postgres/init/`](../docker/postgres/init) creates the users table; the-backend creates and upgrades everything else itself at startup. The healthcheck (`pg_isready`) holds the+The backend creates every table and upgrades the schema itself at startup (Alembic migrations in+[`src/backend/migrations/`](../src/backend/migrations)). The healthcheck (`pg_isready`) holds thebackend back until the database accepts connections. The container is on the internal `data` networkonly and publishes no port.diff --git a/docs/database.md b/docs/database.mdindex 152c676..d7c558c 100644--- a/docs/database.md+++ b/docs/database.md@@ -2,9 +2,8 @@# Database schema-Tables are created on backend startup (`Base.metadata.create_all`), and new-columns are added with `ALTER TABLE ... IF NOT EXISTS` in the startup hook in-`src/backend/main.py`. Each check-in stores one row in `host_metrics` (which also+The backend creates and upgrades the schema itself at startup with Alembic migrations+(see [Migrations](#migrations)). Each check-in stores one row in `host_metrics` (which alsocarries the boot ID, OOM counter, kernel flags, clock state and last shutdown), oneper filesystem in `mount_metrics` and one per container in `container_metrics` (whichalso records the exit code, out-of-memory flag, finish time and image ID), all with the@@ -12,8 +11,8 @@ same timestamp. `system_events` holds reboots, OOM kills, kernel messages, contachanges (appeared, removed, image changed) and network changes (ports opened or closed,firewall changes); `listener_states` and `firewall_states` hold what was last seen, `join_tokens` the hashed single-use tokens, `recovery_codes` the hashed two-factorrecovery codes (the encrypted two-factor secret lives on the `users` row), and per-mount disk-thresholds live in `app_settings`. `docker/postgres/init/01_create_users.sql` initialises-the `users` table on a fresh volume.+thresholds live in `app_settings`. Every table is created and changed by the migrations in+`src/backend/migrations/` (see Migrations below).**The agent's spending.** `agent_runs` has one row per call to the language model: what it was for (`chat`, later`diagnosis`), whether it worked, the model, input and output tokens, the estimated cost in USD, how long it took, and@@ -39,9 +38,29 @@ purge never deletes samples that have not been summarised yet. The Statistics panot purged yet; a retention setting for each level is planned.The metrics tables carry only the indexes their queries use: `time` (for the hourly purge) and-`(server_id, time)`, or `(server_id, container_id, time)` for containers. Older databases are-migrated at start: the new indexes are created and the redundant single-column ones are dropped,-which can be repeated safely. Creating an index briefly blocks writes to that table (a few-seconds for a table of millions of rows); on a very large install, create the indexes yourself-first with `CREATE INDEX CONCURRENTLY IF NOT EXISTS <name> ON <table> (<columns>)` using the names-in `src/backend/models.py`, and the start-up migration will skip them.+`(server_id, time)`, or `(server_id, container_id, time)` for containers. A database from before+migrations gets them from the baseline migration, which also drops the redundant single-column+ones. Creating an index briefly blocks writes to that table (a few seconds for a table of millions+of rows); on a very large install, create the indexes yourself first with+`CREATE INDEX CONCURRENTLY IF NOT EXISTS <name> ON <table> (<columns>)` using the names in+`src/backend/models.py`, and the migration will skip them.++## Migrations++The schema is versioned with [Alembic](https://alembic.sqlalchemy.org/). The revisions live in+`src/backend/migrations/versions/`, and the database records the one it is at in the `alembic_version`+table. At every start the backend runs the revisions the database doesn't have yet, in order, before it+serves anything (`src/backend/migrate.py`):++- On PostgreSQL an advisory lock makes a second process wait instead of migrating at the same time.+- Each revision runs in its own transaction. If one fails, the database stays at the revision before it and+ the backend refuses to start; the log says which revision failed.+- The log has one line per revision applied (`Database migration applied: 0002 (...)`).++The first revision, `0001_baseline`, is the schema as it was when migrations were introduced. A new database+gets every table from it. A database from before migrations (made by the backend's old start-up code) keeps+its tables and data: the baseline creates only the tables it lacks and brings the others up to date, so+upgrading needs no manual step.++How to add a revision is in [Development](development.md#database-migrations).+diff --git a/docs/development.md b/docs/development.mdindex d1a2762..84f0c12 100644--- a/docs/development.md+++ b/docs/development.md@@ -49,6 +49,22 @@ helpers have their own tests that need only Node 20 or newer: `cd src/frontend &[TESTS.md](../TESTS.md) for how the tests work, what they don't cover, and what everytest is for.+## Database migrations++Change the models in `src/backend/models.py`, then generate a revision against a database that is at the+latest revision (the development stack's PostgreSQL, or any empty database the backend has started on once):++```bash+DATABASE_URL=postgresql+psycopg://infradeck:<password>@127.0.0.1:5432/infradeck alembic revision --autogenerate -m "add servers.location"+```++Read the generated file in `src/backend/migrations/versions/` before committing it: Alembic can't tell a+renamed column from a dropped and added one, and data changes (backfills) have to be written by hand. Name+revisions with the next number (`--rev-id 0002`). The backend applies new revisions at its next start;+`alembic upgrade head` and `alembic downgrade -1` do it by hand. `tests/history/test_migrations.py` fails when+the models and the revisions describe different schemas; set `MIGRATION_TEST_POSTGRES_URL` to an empty+PostgreSQL database to run it there as well as on SQLite.+## Project layout```text@@ -61,6 +77,7 @@ SECURITY.md what each component can touch, and its limitsTESTS.md how to run the tests and what each one checksrequirements-dev.txt app requirements plus pytest, for running the testspytest.ini test settings (lets plain `pytest` find the `src` package)+alembic.ini settings for the `alembic` command (writing new migrations).github/workflows/images.yml builds, signs and publishes the collector images on a version tagdocker/@@ -73,12 +90,13 @@ docker/nginx.strict.conf server stack: public listener, collector endpoints refusednginx.collector.conf server stack: collector gate (VPN and internal listeners)Caddyfile optional HTTPS front door (dev compose, --profile https)- postgres/init/ SQL run on a fresh PostgreSQL volumesrc/backend/ API: auth, monitoring, enrollment, rate limits, modelsalerts.py alert engine: one incident per rule, auto-resolverules.py alert rules and default thresholdsrollups.py summarises check-ins into hourly/daily/weekly/monthly rollups+ migrate.py runs the database migrations at start+ migrations/ Alembic revisions (versions/); 0001 is the baselinehistory.py range-aware history behind the Statistics pagestats.py fleet statistics: incidents over time, availability, busy-hours heatmaptotp.py one-time codes and recovery codes (standard library only)diff --git a/requirements.txt b/requirements.txtindex a09676d..2bd2d00 100644--- a/requirements.txt+++ b/requirements.txt@@ -1,3 +1,4 @@+alembicbcryptcryptographyemail-validatordiff --git a/src/backend/main.py b/src/backend/main.pyindex 1762bfd..6c9f652 100644--- a/src/backend/main.py+++ b/src/backend/main.py@@ -22,7 +22,7 @@ from fastapi import Path as PathParamfrom fastapi.middleware.cors import CORSMiddlewarefrom fastapi.responses import JSONResponse, PlainTextResponsefrom pydantic import BaseModel, EmailStr, Field, field_validator-from sqlalchemy import case, func, inspect, text+from sqlalchemy import case, funcfrom .alerts import CONTAINER_RULE_PREFIX, CRITICAL, SERVER_OFFLINE_RULE, SEVERITIES, active_rule_keys, raise_alert, resolve_alerts, sync_alertsimport contextvars@@ -34,9 +34,10 @@ from .auth import (register_user, revoke_other_sessions, revoke_user_session, validate_password, verify_credentials, verify_password, verify_second_factor,)from .db import SessionLocal, check_database, get_db-from .models import ApiToken, Invite, NotificationChannel, AppSetting, AuditLog, Base, ChatMessage, Check, ContainerMetric, ContainerRollup, Conversation, EnrollmentKey, FirewallState, HostMetric, HostRollup, IncidentAlert, JoinToken, ListenerState, MountMetric, MountRollup, PendingCollector, RecoveryCode, Server, SystemEvent, User+from .models import ApiToken, Invite, NotificationChannel, AppSetting, AuditLog, ChatMessage, Check, ContainerMetric, ContainerRollup, Conversation, EnrollmentKey, FirewallState, HostMetric, HostRollup, IncidentAlert, JoinToken, ListenerState, MountMetric, MountRollup, PendingCollector, RecoveryCode, Server, SystemEvent, Userfrom .db import engine+from .migrate import upgrade_databasefrom .config import (AGENT_MODEL,APP_VERSION,@@ -330,83 +331,7 @@ def registration_warning(db) -> str | None:@app.on_event("startup")def startup_event():- Base.metadata.create_all(bind=engine)- with engine.begin() as connection:- if engine.dialect.name == "postgresql":- connection.execute(text("ALTER TABLE audit_logs ADD COLUMN IF NOT EXISTS task_title VARCHAR(255)"))- connection.execute(text("ALTER TABLE audit_logs ADD COLUMN IF NOT EXISTS steps TEXT"))- connection.execute(text("ALTER TABLE audit_logs ADD COLUMN IF NOT EXISTS conversation_id VARCHAR(255)"))- connection.execute(text("ALTER TABLE audit_logs ADD COLUMN IF NOT EXISTS event_type VARCHAR(50)"))- for column, column_type in (- ("memory_limit_mb", "DOUBLE PRECISION"), ("net_rx_mb", "DOUBLE PRECISION"), ("net_tx_mb", "DOUBLE PRECISION"),- ("block_read_mb", "DOUBLE PRECISION"), ("block_write_mb", "DOUBLE PRECISION"), ("pids", "INTEGER"),- ("started_at", "VARCHAR(64)"), ("ports", "VARCHAR(512)"),- ):- connection.execute(text(f"ALTER TABLE container_metrics ADD COLUMN IF NOT EXISTS {column} {column_type}"))- for column, column_type in (- ("boot_id", "VARCHAR(64)"), ("oom_kill_total", "INTEGER"), ("kernel_tainted", "INTEGER"),- ("ntp_synced", "BOOLEAN"), ("reboot_required", "BOOLEAN"), ("last_shutdown", "VARCHAR(16)"),- ("docker_status", "VARCHAR(16)"), ("docker_reason", "VARCHAR(16)"), ("docker_latency_ms", "INTEGER"),- ("internet_status", "VARCHAR(16)"), ("internet_latency_ms", "INTEGER"), ("collect_duration_ms", "INTEGER"),- ):- connection.execute(text(f"ALTER TABLE host_metrics ADD COLUMN IF NOT EXISTS {column} {column_type}"))- for column, column_type in (("exit_code", "INTEGER"), ("oom_killed", "BOOLEAN"), ("finished_at", "TIMESTAMP WITH TIME ZONE"), ("image_id", "VARCHAR(80)"), ("project", "VARCHAR(128)"), ("volumes", "VARCHAR(512)"), ("swap_mb", "DOUBLE PRECISION")):- connection.execute(text(f"ALTER TABLE container_metrics ADD COLUMN IF NOT EXISTS {column} {column_type}"))- connection.execute(text("ALTER TABLE system_events ADD COLUMN IF NOT EXISTS event_key VARCHAR(128)"))- connection.execute(text("ALTER TABLE pending_collectors ADD COLUMN IF NOT EXISTS join_token_id INTEGER REFERENCES join_tokens(id) ON DELETE SET NULL"))- connection.execute(text("ALTER TABLE pending_collectors ADD COLUMN IF NOT EXISTS details TEXT"))- for column, column_type in (("avg_swap_mb", "DOUBLE PRECISION"), ("n_swap_mb", "INTEGER NOT NULL DEFAULT 0"), ("max_swap_mb", "DOUBLE PRECISION")):- connection.execute(text(f"ALTER TABLE container_rollups ADD COLUMN IF NOT EXISTS {column} {column_type}"))- connection.execute(text("CREATE INDEX IF NOT EXISTS ix_system_events_event_key ON system_events (event_key)"))- # Composite indexes that match how the metrics tables are read, then the single-column ones they make- # redundant (and the pointless indexes on each table's own primary key). Safe to repeat. Creating an index- # briefly blocks writes to that table; on a very large table create them first by hand with CONCURRENTLY.- for table, name, columns in (- ("host_metrics", "ix_host_metrics_server_time", "server_id, time"),- ("mount_metrics", "ix_mount_metrics_server_time", "server_id, time"),- ("system_events", "ix_system_events_server_time", "server_id, time"),- ("container_metrics", "ix_container_metrics_server_container_time", "server_id, container_id, time"),- ):- connection.execute(text(f"CREATE INDEX IF NOT EXISTS {name} ON {table} ({columns})"))- for redundant in (- "ix_host_metrics_id", "ix_host_metrics_server_id", "ix_mount_metrics_id", "ix_mount_metrics_server_id",- "ix_system_events_id", "ix_system_events_server_id",- "ix_container_metrics_id", "ix_container_metrics_server_id", "ix_container_metrics_container_id",- ):- connection.execute(text(f"DROP INDEX IF EXISTS {redundant}"))- for column, column_type in (("totp_secret_enc", "TEXT"), ("totp_enabled", "BOOLEAN NOT NULL DEFAULT FALSE"), ("totp_last_step", "INTEGER"), ("password_changed_at", "TIMESTAMP WITH TIME ZONE")):- connection.execute(text(f"ALTER TABLE users ADD COLUMN IF NOT EXISTS {column} {column_type}"))- connection.execute(text("ALTER TABLE incident_alerts ADD COLUMN IF NOT EXISTS rule_key VARCHAR(255)"))- connection.execute(text("ALTER TABLE incident_alerts ADD COLUMN IF NOT EXISTS resolved_at TIMESTAMP WITH TIME ZONE"))- for column, column_type in (("resolution", "VARCHAR(255)"), ("acknowledged_at", "TIMESTAMP WITH TIME ZONE"), ("acknowledged_by", "INTEGER"), ("muted_until", "TIMESTAMP WITH TIME ZONE"),- ("notified_at", "TIMESTAMP WITH TIME ZONE"), ("resolve_notified_at", "TIMESTAMP WITH TIME ZONE"), ("reopen_after", "TIMESTAMP WITH TIME ZONE")):- connection.execute(text(f"ALTER TABLE incident_alerts ADD COLUMN IF NOT EXISTS {column} {column_type}"))- for column, column_type in (("username", "VARCHAR(64)"), ("avatar_color", "VARCHAR(16)"), ("preferences", "TEXT"), ("last_seen_at", "TIMESTAMP WITH TIME ZONE"), ("agent_chat_enabled", "BOOLEAN NOT NULL DEFAULT FALSE")):- connection.execute(text(f"ALTER TABLE users ADD COLUMN IF NOT EXISTS {column} {column_type}"))- connection.execute(text("CREATE INDEX IF NOT EXISTS ix_incident_alerts_rule_key ON incident_alerts (rule_key)"))- # Alerts from before the alert engine: offline alerts keep going under their rule, the rest were- # duplicated on every check-in with no way to resolve, so close them; the engine re-raises live ones.- connection.execute(text("UPDATE incident_alerts SET rule_key = :rule WHERE rule_key IS NULL AND description = :description"), {"rule": SERVER_OFFLINE_RULE, "description": OFFLINE_ALERT_DESCRIPTION})- connection.execute(text("UPDATE incident_alerts SET status = 'resolved', resolved_at = NOW() WHERE rule_key IS NULL AND status = 'active'"))- # Disk and inode alerts became per mount; the old keys were for the root filesystem.- connection.execute(text("UPDATE incident_alerts SET rule_key = rule_key || ':/' WHERE rule_key IN ('host:disk', 'host:inodes')"))- if "servers" in inspect(engine).get_table_names():- columns = {column["name"] for column in inspect(engine).get_columns("servers")}- if "collector_token_hash" not in columns:- with engine.begin() as connection:- connection.execute(text("ALTER TABLE servers ADD COLUMN collector_token_hash VARCHAR(64)"))- if "machine_id" not in columns:- with engine.begin() as connection:- connection.execute(text("ALTER TABLE servers ADD COLUMN machine_id VARCHAR(64)"))- if "is_dashboard_host" not in columns:- with engine.begin() as connection:- connection.execute(text("ALTER TABLE servers ADD COLUMN is_dashboard_host BOOLEAN NOT NULL DEFAULT FALSE"))- if "system_info" not in columns:- with engine.begin() as connection:- connection.execute(text("ALTER TABLE servers ADD COLUMN system_info TEXT"))- if "name_locked" not in columns:- with engine.begin() as connection:- connection.execute(text("ALTER TABLE servers ADD COLUMN name_locked BOOLEAN NOT NULL DEFAULT FALSE"))+ upgrade_database(engine) # src/backend/migrations; a failing revision stops the start heredb = SessionLocal()try:ensure_admin_exists(db)diff --git a/src/backend/migrate.py b/src/backend/migrate.pynew file mode 100644index 0000000..dc8c3af--- /dev/null+++ b/src/backend/migrate.py@@ -0,0 +1,40 @@+"""Bringing the database to the latest schema at start (Alembic revisions in src/backend/migrations/versions).++Runs before the backend serves anything. On PostgreSQL an advisory lock makes a second process (another backend, or+the agent worker) wait instead of migrating at the same time. Each revision runs in its own transaction, so a+revision that fails leaves the database at the one before it, and the backend refuses to start rather than run+on a half-changed schema.+"""+from __future__ import annotations++from pathlib import Path++from alembic import command+from alembic.config import Config+from sqlalchemy import text++# Any fixed number; it only has to be the same for every process that migrates this database.+ADVISORY_LOCK_ID = 7_120_351+MIGRATIONS_DIR = Path(__file__).resolve().parent / "migrations"+++def alembic_config(connection) -> Config:+ config = Config()+ config.set_main_option("script_location", str(MIGRATIONS_DIR))+ config.attributes["connection"] = connection+ return config+++def upgrade_database(engine) -> None:+ with engine.connect() as connection:+ postgres = connection.dialect.name == "postgresql"+ if postgres:+ connection.execute(text("SELECT pg_advisory_lock(:id)"), {"id": ADVISORY_LOCK_ID})+ connection.commit()+ try:+ command.upgrade(alembic_config(connection), "head")+ connection.commit()+ finally:+ if postgres:+ connection.execute(text("SELECT pg_advisory_unlock(:id)"), {"id": ADVISORY_LOCK_ID})+ connection.commit()diff --git a/src/backend/migrations/env.py b/src/backend/migrations/env.pynew file mode 100644index 0000000..0a8c940--- /dev/null+++ b/src/backend/migrations/env.py@@ -0,0 +1,28 @@+"""Alembic's entry point. The backend passes its own connection in (src/backend/migrate.py); the `alembic` command+line connects to DATABASE_URL instead."""+from alembic import context+from sqlalchemy import create_engine++from src.backend.config import DATABASE_URL+from src.backend.models import Base+++def _applied(ctx, step, heads, run_args) -> None:+ print(f"Database migration {'applied' if step.is_upgrade else 'reverted'}: {step.up_revision_id} ({step.up_revision.doc.splitlines()[0]})", flush=True)+++def _run(connection) -> None:+ # SQLite can't alter most of a table in place; batch mode copies the table instead. One transaction per+ # revision, so a failing one leaves the database at the revision before it.+ context.configure(connection=connection, target_metadata=Base.metadata, render_as_batch=connection.dialect.name == "sqlite", compare_type=True,+ transaction_per_migration=True, on_version_apply=_applied)+ with context.begin_transaction():+ context.run_migrations()+++connection = context.config.attributes.get("connection")+if connection is not None:+ _run(connection)+else:+ with create_engine(DATABASE_URL).connect() as own:+ _run(own)diff --git a/src/backend/migrations/script.py.mako b/src/backend/migrations/script.py.makonew file mode 100644index 0000000..efe908a--- /dev/null+++ b/src/backend/migrations/script.py.mako@@ -0,0 +1,20 @@+"""${message}++Revision ${up_revision}, after ${down_revision | comma,n}.+"""+from alembic import op+import sqlalchemy as sa+${imports if imports else ""}++revision = ${repr(up_revision)}+down_revision = ${repr(down_revision)}+branch_labels = ${repr(branch_labels)}+depends_on = ${repr(depends_on)}+++def upgrade() -> None:+ ${upgrades if upgrades else "pass"}+++def downgrade() -> None:+ ${downgrades if downgrades else "pass"}diff --git a/src/backend/migrations/versions/0001_baseline.py b/src/backend/migrations/versions/0001_baseline.pynew file mode 100644index 0000000..21c411d--- /dev/null+++ b/src/backend/migrations/versions/0001_baseline.py@@ -0,0 +1,696 @@+"""The schema as it was when migrations were introduced.++A new database gets every table here. A database from before Alembic (created by ``create_all`` and the hand-written+statements that used to run at start) keeps its tables: only the tables it lacks are created, and the old statements+below bring the rest up to this schema. Both end at the same schema, which tests/history/test_migrations.py checks.+"""+from alembic import op+import sqlalchemy as sa+++revision = '0001'+down_revision = None+branch_labels = None+depends_on = None++_existing: set[str] = set()+++def _table(name, *elements, **kwargs) -> None:+ if name not in _existing:+ op.create_table(name, *elements, **kwargs)+++def _index(name, table, columns, **kwargs) -> None:+ if table not in _existing:+ op.create_index(name, table, columns, **kwargs)+++def upgrade() -> None:+ bind = op.get_bind()+ _existing.clear()+ _existing.update(sa.inspect(bind).get_table_names())+ _create_tables()+ if _existing:+ _bring_up_to_date(bind)+++def _create_tables() -> None:+ _table('app_settings',+ sa.Column('key', sa.String(length=100), nullable=False),+ sa.Column('value', sa.Text(), nullable=False),+ sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),+ sa.PrimaryKeyConstraint('key')+ )+ _table('audit_logs',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('user_id', sa.Integer(), nullable=True),+ sa.Column('level', sa.String(length=50), nullable=False),+ sa.Column('message', sa.Text(), nullable=False),+ sa.Column('source', sa.String(length=100), nullable=False),+ sa.Column('task_title', sa.String(length=255), nullable=True),+ sa.Column('steps', sa.Text(), nullable=True),+ sa.Column('conversation_id', sa.String(length=255), nullable=True),+ sa.Column('event_type', sa.String(length=50), nullable=True),+ sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),+ sa.PrimaryKeyConstraint('id')+ )+ _index(op.f('ix_audit_logs_conversation_id'), 'audit_logs', ['conversation_id'], unique=False)+ _index(op.f('ix_audit_logs_id'), 'audit_logs', ['id'], unique=False)+ _index(op.f('ix_audit_logs_user_id'), 'audit_logs', ['user_id'], unique=False)+ _table('chat_messages',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('conversation_id', sa.String(length=64), nullable=False),+ sa.Column('role', sa.String(length=20), nullable=False),+ sa.Column('content', sa.Text(), nullable=False),+ sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),+ sa.PrimaryKeyConstraint('id')+ )+ _index(op.f('ix_chat_messages_conversation_id'), 'chat_messages', ['conversation_id'], unique=False)+ _index(op.f('ix_chat_messages_id'), 'chat_messages', ['id'], unique=False)+ _table('checks',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('name', sa.String(length=120), nullable=False),+ sa.Column('type', sa.String(length=16), nullable=False),+ sa.Column('target', sa.String(length=2048), nullable=False),+ sa.Column('options', sa.Text(), nullable=False),+ sa.Column('tags', sa.String(length=255), nullable=False),+ sa.Column('server_ids', sa.Text(), nullable=False),+ sa.Column('rule', sa.String(length=8), nullable=False),+ sa.Column('interval_seconds', sa.Integer(), nullable=False),+ sa.Column('retries', sa.Integer(), nullable=False),+ sa.Column('retry_interval_seconds', sa.Integer(), nullable=False),+ sa.Column('timeout_seconds', sa.Integer(), nullable=False),+ sa.Column('severity', sa.String(length=16), nullable=False),+ sa.Column('paused', sa.Boolean(), nullable=False),+ sa.Column('state', sa.String(length=16), nullable=False),+ sa.Column('state_since', sa.DateTime(timezone=True), nullable=True),+ sa.Column('last_result_at', sa.DateTime(timezone=True), nullable=True),+ sa.Column('rolled_until', sa.DateTime(timezone=True), nullable=True),+ sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),+ sa.PrimaryKeyConstraint('id')+ )+ _table('conversations',+ sa.Column('id', sa.String(length=64), nullable=False),+ sa.Column('user_id', sa.Integer(), nullable=False),+ sa.Column('title', sa.String(length=255), nullable=False),+ sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),+ sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),+ sa.PrimaryKeyConstraint('id')+ )+ _index(op.f('ix_conversations_user_id'), 'conversations', ['user_id'], unique=False)+ _table('users',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('email', sa.String(length=255), nullable=False),+ sa.Column('name', sa.String(length=255), nullable=False),+ sa.Column('password_hash', sa.Text(), nullable=False),+ sa.Column('role', sa.String(length=50), nullable=False),+ sa.Column('is_active', sa.Boolean(), nullable=False),+ sa.Column('totp_secret_enc', sa.Text(), nullable=True),+ sa.Column('totp_enabled', sa.Boolean(), nullable=False),+ sa.Column('totp_last_step', sa.Integer(), nullable=True),+ sa.Column('password_changed_at', sa.DateTime(timezone=True), nullable=True),+ sa.Column('username', sa.String(length=64), nullable=True),+ sa.Column('avatar_color', sa.String(length=16), nullable=True),+ sa.Column('preferences', sa.Text(), nullable=True),+ sa.Column('agent_chat_enabled', sa.Boolean(), nullable=False),+ sa.Column('last_seen_at', sa.DateTime(timezone=True), nullable=True),+ sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),+ sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),+ sa.PrimaryKeyConstraint('id')+ )+ _index(op.f('ix_users_email'), 'users', ['email'], unique=True)+ _index(op.f('ix_users_id'), 'users', ['id'], unique=False)+ _table('agent_runs',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('kind', sa.String(length=16), nullable=False),+ sa.Column('status', sa.String(length=16), nullable=False),+ sa.Column('model', sa.String(length=100), nullable=False),+ sa.Column('input_tokens', sa.Integer(), nullable=False),+ sa.Column('output_tokens', sa.Integer(), nullable=False),+ sa.Column('cost_usd', sa.Float(), nullable=True),+ sa.Column('duration_ms', sa.Integer(), nullable=False),+ sa.Column('user_id', sa.Integer(), nullable=True),+ sa.Column('conversation_id', sa.String(length=255), nullable=True),+ sa.Column('incident_id', sa.Integer(), nullable=True),+ sa.Column('error', sa.String(length=300), nullable=True),+ sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),+ sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='SET NULL'),+ sa.PrimaryKeyConstraint('id')+ )+ _index('ix_agent_runs_created', 'agent_runs', ['created_at'], unique=False)+ _index(op.f('ix_agent_runs_user_id'), 'agent_runs', ['user_id'], unique=False)+ _table('api_tokens',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('user_id', sa.Integer(), nullable=False),+ sa.Column('name', sa.String(length=64), nullable=False),+ sa.Column('scope', sa.String(length=16), nullable=False),+ sa.Column('token_hash', sa.String(length=64), nullable=False),+ sa.Column('hint', sa.String(length=8), nullable=False),+ sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),+ sa.Column('last_used_at', sa.DateTime(timezone=True), nullable=True),+ sa.Column('expires_at', sa.DateTime(timezone=True), nullable=True),+ sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id'),+ sa.UniqueConstraint('token_hash')+ )+ _index(op.f('ix_api_tokens_id'), 'api_tokens', ['id'], unique=False)+ _index(op.f('ix_api_tokens_user_id'), 'api_tokens', ['user_id'], unique=False)+ _table('check_results',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('check_id', sa.Integer(), nullable=False),+ sa.Column('server_id', sa.Integer(), nullable=False),+ sa.Column('time', sa.DateTime(timezone=True), nullable=False),+ sa.Column('ok', sa.Boolean(), nullable=False),+ sa.Column('response_ms', sa.Integer(), nullable=True),+ sa.Column('code', sa.Integer(), nullable=True),+ sa.Column('detail', sa.String(length=200), nullable=True),+ sa.Column('cert_days', sa.Integer(), nullable=True),+ sa.ForeignKeyConstraint(['check_id'], ['checks.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id')+ )+ _index('ix_check_results_check_time', 'check_results', ['check_id', 'time'], unique=False)+ _table('check_rollups',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('check_id', sa.Integer(), nullable=False),+ sa.Column('hour', sa.DateTime(timezone=True), nullable=False),+ sa.Column('slots', sa.Integer(), nullable=False),+ sa.Column('down_slots', sa.Integer(), nullable=False),+ sa.Column('avg_ms', sa.Float(), nullable=True),+ sa.Column('max_ms', sa.Integer(), nullable=True),+ sa.ForeignKeyConstraint(['check_id'], ['checks.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id'),+ sa.UniqueConstraint('check_id', 'hour')+ )+ _index(op.f('ix_check_rollups_check_id'), 'check_rollups', ['check_id'], unique=False)+ _table('enrollment_keys',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('owner_id', sa.Integer(), nullable=False),+ sa.Column('key_hash', sa.String(length=64), nullable=False),+ sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),+ sa.ForeignKeyConstraint(['owner_id'], ['users.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id'),+ sa.UniqueConstraint('key_hash'),+ sa.UniqueConstraint('owner_id')+ )+ _index(op.f('ix_enrollment_keys_id'), 'enrollment_keys', ['id'], unique=False)+ _table('invites',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('email', sa.String(length=255), nullable=False),+ sa.Column('role', sa.String(length=50), nullable=False),+ sa.Column('token_hash', sa.String(length=64), nullable=False),+ sa.Column('created_by', sa.Integer(), nullable=False),+ sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),+ sa.Column('expires_at', sa.DateTime(timezone=True), nullable=False),+ sa.Column('used_at', sa.DateTime(timezone=True), nullable=True),+ sa.ForeignKeyConstraint(['created_by'], ['users.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id'),+ sa.UniqueConstraint('token_hash')+ )+ _index(op.f('ix_invites_id'), 'invites', ['id'], unique=False)+ _table('join_tokens',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('token_hash', sa.String(length=64), nullable=False),+ sa.Column('created_by', sa.Integer(), nullable=False),+ sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),+ sa.Column('expires_at', sa.DateTime(timezone=True), nullable=False),+ sa.Column('used_at', sa.DateTime(timezone=True), nullable=True),+ sa.ForeignKeyConstraint(['created_by'], ['users.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id'),+ sa.UniqueConstraint('token_hash')+ )+ _index(op.f('ix_join_tokens_id'), 'join_tokens', ['id'], unique=False)+ _table('notification_channels',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('user_id', sa.Integer(), nullable=False),+ sa.Column('kind', sa.String(length=16), nullable=False),+ sa.Column('target_enc', sa.Text(), nullable=False),+ sa.Column('display', sa.String(length=120), nullable=False),+ sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),+ sa.Column('last_status', sa.String(length=64), nullable=True),+ sa.Column('last_ok', sa.Boolean(), nullable=True),+ sa.Column('last_checked_at', sa.DateTime(timezone=True), nullable=True),+ sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id')+ )+ _index(op.f('ix_notification_channels_id'), 'notification_channels', ['id'], unique=False)+ _index(op.f('ix_notification_channels_user_id'), 'notification_channels', ['user_id'], unique=False)+ _table('recovery_codes',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('user_id', sa.Integer(), nullable=False),+ sa.Column('code_hash', sa.String(length=64), nullable=False),+ sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),+ sa.Column('used_at', sa.DateTime(timezone=True), nullable=True),+ sa.ForeignKeyConstraint(['user_id'], ['users.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id'),+ sa.UniqueConstraint('code_hash')+ )+ _index(op.f('ix_recovery_codes_id'), 'recovery_codes', ['id'], unique=False)+ _index(op.f('ix_recovery_codes_user_id'), 'recovery_codes', ['user_id'], unique=False)+ _table('servers',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('owner_id', sa.Integer(), nullable=False),+ sa.Column('hostname', sa.String(length=255), nullable=False),+ sa.Column('ip_address', sa.String(length=64), nullable=True),+ sa.Column('os_info', sa.String(length=255), nullable=True),+ sa.Column('cpu_cores', sa.Integer(), nullable=True),+ sa.Column('total_ram_gb', sa.Float(), nullable=True),+ sa.Column('check_interval_seconds', sa.Integer(), nullable=False),+ sa.Column('collector_token_hash', sa.String(length=64), nullable=True),+ sa.Column('machine_id', sa.String(length=64), nullable=True),+ sa.Column('is_dashboard_host', sa.Boolean(), nullable=False),+ sa.Column('system_info', sa.Text(), nullable=True),+ sa.Column('name_locked', sa.Boolean(), nullable=False),+ sa.Column('last_seen_at', sa.DateTime(timezone=True), nullable=True),+ sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),+ sa.ForeignKeyConstraint(['owner_id'], ['users.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id'),+ sa.UniqueConstraint('collector_token_hash')+ )+ _index(op.f('ix_servers_id'), 'servers', ['id'], unique=False)+ _index(op.f('ix_servers_machine_id'), 'servers', ['machine_id'], unique=False)+ _index(op.f('ix_servers_owner_id'), 'servers', ['owner_id'], unique=False)+ _table('container_metrics',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('server_id', sa.Integer(), nullable=False),+ sa.Column('container_id', sa.String(length=128), nullable=False),+ sa.Column('name', sa.String(length=255), nullable=False),+ sa.Column('image', sa.String(length=255), nullable=True),+ sa.Column('status', sa.String(length=64), nullable=False),+ sa.Column('restart_count', sa.Integer(), nullable=False),+ sa.Column('cpu_pct', sa.Float(), nullable=True),+ sa.Column('memory_used_mb', sa.Float(), nullable=True),+ sa.Column('memory_pct', sa.Float(), nullable=True),+ sa.Column('memory_limit_mb', sa.Float(), nullable=True),+ sa.Column('net_rx_mb', sa.Float(), nullable=True),+ sa.Column('net_tx_mb', sa.Float(), nullable=True),+ sa.Column('block_read_mb', sa.Float(), nullable=True),+ sa.Column('block_write_mb', sa.Float(), nullable=True),+ sa.Column('pids', sa.Integer(), nullable=True),+ sa.Column('started_at', sa.String(length=64), nullable=True),+ sa.Column('ports', sa.String(length=512), nullable=True),+ sa.Column('project', sa.String(length=128), nullable=True),+ sa.Column('swap_mb', sa.Float(), nullable=True),+ sa.Column('volumes', sa.String(length=512), nullable=True),+ sa.Column('healthcheck_status', sa.String(length=64), nullable=True),+ sa.Column('exit_code', sa.Integer(), nullable=True),+ sa.Column('oom_killed', sa.Boolean(), nullable=True),+ sa.Column('finished_at', sa.DateTime(timezone=True), nullable=True),+ sa.Column('image_id', sa.String(length=80), nullable=True),+ sa.Column('time', sa.DateTime(timezone=True), nullable=False),+ sa.ForeignKeyConstraint(['server_id'], ['servers.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id')+ )+ _index('ix_container_metrics_server_container_time', 'container_metrics', ['server_id', 'container_id', 'time'], unique=False)+ _index(op.f('ix_container_metrics_time'), 'container_metrics', ['time'], unique=False)+ _table('container_rollups',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('server_id', sa.Integer(), nullable=False),+ sa.Column('container_id', sa.String(length=128), nullable=False),+ sa.Column('name', sa.String(length=255), nullable=False),+ sa.Column('tier', sa.String(length=8), nullable=False),+ sa.Column('bucket_start', sa.DateTime(timezone=True), nullable=False),+ sa.Column('samples', sa.Integer(), nullable=False),+ sa.Column('down_samples', sa.Integer(), nullable=False),+ sa.Column('avg_cpu', sa.Float(), nullable=True),+ sa.Column('n_cpu', sa.Integer(), nullable=False),+ sa.Column('max_cpu', sa.Float(), nullable=True),+ sa.Column('avg_memory_mb', sa.Float(), nullable=True),+ sa.Column('n_memory_mb', sa.Integer(), nullable=False),+ sa.Column('max_memory_mb', sa.Float(), nullable=True),+ sa.Column('avg_memory_pct', sa.Float(), nullable=True),+ sa.Column('n_memory_pct', sa.Integer(), nullable=False),+ sa.Column('max_memory_pct', sa.Float(), nullable=True),+ sa.Column('max_memory_limit_mb', sa.Float(), nullable=True),+ sa.Column('avg_swap_mb', sa.Float(), nullable=True),+ sa.Column('n_swap_mb', sa.Integer(), nullable=False),+ sa.Column('max_swap_mb', sa.Float(), nullable=True),+ sa.Column('max_restarts', sa.Integer(), nullable=True),+ sa.ForeignKeyConstraint(['server_id'], ['servers.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id'),+ sa.UniqueConstraint('server_id', 'container_id', 'tier', 'bucket_start')+ )+ _index('ix_container_rollups_tier_bucket', 'container_rollups', ['tier', 'bucket_start'], unique=False)+ _table('firewall_states',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('server_id', sa.Integer(), nullable=False),+ sa.Column('name', sa.String(length=16), nullable=False),+ sa.Column('sha', sa.String(length=16), nullable=False),+ sa.Column('rules', sa.Integer(), nullable=False),+ sa.Column('enabled', sa.Boolean(), nullable=True),+ sa.Column('updated_at', sa.DateTime(timezone=True), nullable=False),+ sa.ForeignKeyConstraint(['server_id'], ['servers.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id'),+ sa.UniqueConstraint('server_id', 'name')+ )+ _index(op.f('ix_firewall_states_id'), 'firewall_states', ['id'], unique=False)+ _index(op.f('ix_firewall_states_server_id'), 'firewall_states', ['server_id'], unique=False)+ _table('host_metrics',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('server_id', sa.Integer(), nullable=False),+ sa.Column('time', sa.DateTime(timezone=True), nullable=False),+ sa.Column('cpu_utilization_pct', sa.Float(), nullable=True),+ sa.Column('load_1m', sa.Float(), nullable=True),+ sa.Column('load_5m', sa.Float(), nullable=True),+ sa.Column('load_15m', sa.Float(), nullable=True),+ sa.Column('ram_used_gb', sa.Float(), nullable=True),+ sa.Column('ram_available_gb', sa.Float(), nullable=True),+ sa.Column('ram_pct', sa.Float(), nullable=True),+ sa.Column('swap_pct', sa.Float(), nullable=True),+ sa.Column('disk_usage_pct', sa.Float(), nullable=True),+ sa.Column('disk_inodes_pct', sa.Float(), nullable=True),+ sa.Column('net_bytes_sent_mb', sa.Float(), nullable=True),+ sa.Column('net_bytes_recv_mb', sa.Float(), nullable=True),+ sa.Column('system_uptime_seconds', sa.Float(), nullable=True),+ sa.Column('boot_id', sa.String(length=64), nullable=True),+ sa.Column('oom_kill_total', sa.Integer(), nullable=True),+ sa.Column('kernel_tainted', sa.Integer(), nullable=True),+ sa.Column('ntp_synced', sa.Boolean(), nullable=True),+ sa.Column('reboot_required', sa.Boolean(), nullable=True),+ sa.Column('last_shutdown', sa.String(length=16), nullable=True),+ sa.Column('docker_status', sa.String(length=16), nullable=True),+ sa.Column('docker_reason', sa.String(length=16), nullable=True),+ sa.Column('docker_latency_ms', sa.Integer(), nullable=True),+ sa.Column('internet_status', sa.String(length=16), nullable=True),+ sa.Column('internet_latency_ms', sa.Integer(), nullable=True),+ sa.Column('collect_duration_ms', sa.Integer(), nullable=True),+ sa.ForeignKeyConstraint(['server_id'], ['servers.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id')+ )+ _index('ix_host_metrics_server_time', 'host_metrics', ['server_id', 'time'], unique=False)+ _index(op.f('ix_host_metrics_time'), 'host_metrics', ['time'], unique=False)+ _table('host_rollups',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('server_id', sa.Integer(), nullable=False),+ sa.Column('tier', sa.String(length=8), nullable=False),+ sa.Column('bucket_start', sa.DateTime(timezone=True), nullable=False),+ sa.Column('samples', sa.Integer(), nullable=False),+ sa.Column('avg_cpu', sa.Float(), nullable=True),+ sa.Column('n_cpu', sa.Integer(), nullable=False),+ sa.Column('max_cpu', sa.Float(), nullable=True),+ sa.Column('avg_load', sa.Float(), nullable=True),+ sa.Column('n_load', sa.Integer(), nullable=False),+ sa.Column('max_load', sa.Float(), nullable=True),+ sa.Column('avg_ram', sa.Float(), nullable=True),+ sa.Column('n_ram', sa.Integer(), nullable=False),+ sa.Column('max_ram', sa.Float(), nullable=True),+ sa.Column('avg_swap', sa.Float(), nullable=True),+ sa.Column('n_swap', sa.Integer(), nullable=False),+ sa.Column('max_swap', sa.Float(), nullable=True),+ sa.Column('avg_disk', sa.Float(), nullable=True),+ sa.Column('n_disk', sa.Integer(), nullable=False),+ sa.Column('max_disk', sa.Float(), nullable=True),+ sa.Column('avg_inodes', sa.Float(), nullable=True),+ sa.Column('n_inodes', sa.Integer(), nullable=False),+ sa.Column('max_inodes', sa.Float(), nullable=True),+ sa.Column('avg_docker_ms', sa.Float(), nullable=True),+ sa.Column('n_docker_ms', sa.Integer(), nullable=False),+ sa.Column('max_docker_ms', sa.Float(), nullable=True),+ sa.Column('avg_collect_ms', sa.Float(), nullable=True),+ sa.Column('n_collect_ms', sa.Integer(), nullable=False),+ sa.Column('max_collect_ms', sa.Float(), nullable=True),+ sa.ForeignKeyConstraint(['server_id'], ['servers.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id'),+ sa.UniqueConstraint('server_id', 'tier', 'bucket_start')+ )+ _index('ix_host_rollups_tier_bucket', 'host_rollups', ['tier', 'bucket_start'], unique=False)+ _table('incident_alerts',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('server_id', sa.Integer(), nullable=False),+ sa.Column('severity', sa.String(length=16), nullable=False),+ sa.Column('title', sa.String(length=255), nullable=False),+ sa.Column('description', sa.Text(), nullable=False),+ sa.Column('status', sa.String(length=32), nullable=False),+ sa.Column('rule_key', sa.String(length=255), nullable=True),+ sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),+ sa.Column('resolved_at', sa.DateTime(timezone=True), nullable=True),+ sa.Column('resolution', sa.String(length=255), nullable=True),+ sa.Column('acknowledged_at', sa.DateTime(timezone=True), nullable=True),+ sa.Column('acknowledged_by', sa.Integer(), nullable=True),+ sa.Column('muted_until', sa.DateTime(timezone=True), nullable=True),+ sa.Column('notified_at', sa.DateTime(timezone=True), nullable=True),+ sa.Column('resolve_notified_at', sa.DateTime(timezone=True), nullable=True),+ sa.Column('reopen_after', sa.DateTime(timezone=True), nullable=True),+ sa.ForeignKeyConstraint(['server_id'], ['servers.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id')+ )+ _index(op.f('ix_incident_alerts_created_at'), 'incident_alerts', ['created_at'], unique=False)+ _index(op.f('ix_incident_alerts_id'), 'incident_alerts', ['id'], unique=False)+ _index(op.f('ix_incident_alerts_rule_key'), 'incident_alerts', ['rule_key'], unique=False)+ _index(op.f('ix_incident_alerts_server_id'), 'incident_alerts', ['server_id'], unique=False)+ _table('listener_states',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('server_id', sa.Integer(), nullable=False),+ sa.Column('proto', sa.String(length=8), nullable=False),+ sa.Column('port', sa.Integer(), nullable=False),+ sa.Column('scope', sa.String(length=16), nullable=False),+ sa.Column('first_seen', sa.DateTime(timezone=True), nullable=False),+ sa.Column('last_seen', sa.DateTime(timezone=True), nullable=False),+ sa.ForeignKeyConstraint(['server_id'], ['servers.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id'),+ sa.UniqueConstraint('server_id', 'proto', 'port', 'scope')+ )+ _index(op.f('ix_listener_states_id'), 'listener_states', ['id'], unique=False)+ _index(op.f('ix_listener_states_server_id'), 'listener_states', ['server_id'], unique=False)+ _table('mount_metrics',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('server_id', sa.Integer(), nullable=False),+ sa.Column('time', sa.DateTime(timezone=True), nullable=False),+ sa.Column('mount', sa.String(length=255), nullable=False),+ sa.Column('fs_type', sa.String(length=64), nullable=True),+ sa.Column('total_gb', sa.Float(), nullable=True),+ sa.Column('used_gb', sa.Float(), nullable=True),+ sa.Column('used_pct', sa.Float(), nullable=True),+ sa.Column('inodes_pct', sa.Float(), nullable=True),+ sa.ForeignKeyConstraint(['server_id'], ['servers.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id')+ )+ _index('ix_mount_metrics_server_time', 'mount_metrics', ['server_id', 'time'], unique=False)+ _index(op.f('ix_mount_metrics_time'), 'mount_metrics', ['time'], unique=False)+ _table('mount_rollups',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('server_id', sa.Integer(), nullable=False),+ sa.Column('mount', sa.String(length=255), nullable=False),+ sa.Column('tier', sa.String(length=8), nullable=False),+ sa.Column('bucket_start', sa.DateTime(timezone=True), nullable=False),+ sa.Column('samples', sa.Integer(), nullable=False),+ sa.Column('avg_used_pct', sa.Float(), nullable=True),+ sa.Column('n_used_pct', sa.Integer(), nullable=False),+ sa.Column('max_used_pct', sa.Float(), nullable=True),+ sa.Column('avg_used_gb', sa.Float(), nullable=True),+ sa.Column('n_used_gb', sa.Integer(), nullable=False),+ sa.Column('max_total_gb', sa.Float(), nullable=True),+ sa.Column('avg_inodes_pct', sa.Float(), nullable=True),+ sa.Column('n_inodes_pct', sa.Integer(), nullable=False),+ sa.Column('max_inodes_pct', sa.Float(), nullable=True),+ sa.ForeignKeyConstraint(['server_id'], ['servers.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id'),+ sa.UniqueConstraint('server_id', 'mount', 'tier', 'bucket_start')+ )+ _index('ix_mount_rollups_tier_bucket', 'mount_rollups', ['tier', 'bucket_start'], unique=False)+ _table('pending_collectors',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('poll_secret_hash', sa.String(length=64), nullable=False),+ sa.Column('pairing_code', sa.String(length=16), nullable=False),+ sa.Column('hostname', sa.String(length=255), nullable=False),+ sa.Column('machine_id', sa.String(length=64), nullable=True),+ sa.Column('ip_address', sa.String(length=64), nullable=True),+ sa.Column('os_info', sa.String(length=255), nullable=True),+ sa.Column('status', sa.String(length=16), nullable=False),+ sa.Column('details', sa.Text(), nullable=True),+ sa.Column('approved_server_id', sa.Integer(), nullable=True),+ sa.Column('join_token_id', sa.Integer(), nullable=True),+ sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),+ sa.Column('last_poll_at', sa.DateTime(timezone=True), nullable=False),+ sa.ForeignKeyConstraint(['approved_server_id'], ['servers.id'], ondelete='CASCADE'),+ sa.ForeignKeyConstraint(['join_token_id'], ['join_tokens.id'], ondelete='SET NULL'),+ sa.PrimaryKeyConstraint('id'),+ sa.UniqueConstraint('poll_secret_hash')+ )+ _index(op.f('ix_pending_collectors_id'), 'pending_collectors', ['id'], unique=False)+ _index(op.f('ix_pending_collectors_machine_id'), 'pending_collectors', ['machine_id'], unique=False)+ _table('system_events',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('server_id', sa.Integer(), nullable=False),+ sa.Column('time', sa.DateTime(timezone=True), nullable=False),+ sa.Column('kind', sa.String(length=32), nullable=False),+ sa.Column('event_key', sa.String(length=128), nullable=True),+ sa.Column('detail', sa.String(length=512), nullable=False),+ sa.ForeignKeyConstraint(['server_id'], ['servers.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id')+ )+ _index(op.f('ix_system_events_event_key'), 'system_events', ['event_key'], unique=False)+ _index('ix_system_events_server_time', 'system_events', ['server_id', 'time'], unique=False)+ _index(op.f('ix_system_events_time'), 'system_events', ['time'], unique=False)+ _table('agent_reports',+ sa.Column('id', sa.Integer(), nullable=False),+ sa.Column('incident_id', sa.Integer(), nullable=False),+ sa.Column('version', sa.Integer(), nullable=False),+ sa.Column('trigger', sa.String(length=32), nullable=False),+ sa.Column('status', sa.String(length=16), nullable=False),+ sa.Column('model', sa.String(length=100), nullable=False),+ sa.Column('confidence', sa.String(length=16), nullable=True),+ sa.Column('summary', sa.Text(), nullable=True),+ sa.Column('body', sa.Text(), nullable=True),+ sa.Column('evidence', sa.Text(), nullable=True),+ sa.Column('error', sa.String(length=300), nullable=True),+ sa.Column('created_at', sa.DateTime(timezone=True), nullable=False),+ sa.ForeignKeyConstraint(['incident_id'], ['incident_alerts.id'], ondelete='CASCADE'),+ sa.PrimaryKeyConstraint('id'),+ sa.UniqueConstraint('incident_id', 'version')+ )+ _index(op.f('ix_agent_reports_incident_id'), 'agent_reports', ['incident_id'], unique=False)+++def _bring_up_to_date(bind) -> None:+ """The statements that ran at every start before migrations, for databases created back then. Each is safe to+ repeat and does nothing on a database that already has the change."""+ if bind.dialect.name == "postgresql":+ execute = lambda statement, **values: bind.execute(sa.text(statement), values) # noqa: E731+ for column, column_type in (("task_title", "VARCHAR(255)"), ("steps", "TEXT"), ("conversation_id", "VARCHAR(255)"), ("event_type", "VARCHAR(50)")):+ execute(f"ALTER TABLE audit_logs ADD COLUMN IF NOT EXISTS {column} {column_type}")+ for column, column_type in (+ ("memory_limit_mb", "DOUBLE PRECISION"), ("net_rx_mb", "DOUBLE PRECISION"), ("net_tx_mb", "DOUBLE PRECISION"),+ ("block_read_mb", "DOUBLE PRECISION"), ("block_write_mb", "DOUBLE PRECISION"), ("pids", "INTEGER"),+ ("started_at", "VARCHAR(64)"), ("ports", "VARCHAR(512)"),+ ("exit_code", "INTEGER"), ("oom_killed", "BOOLEAN"), ("finished_at", "TIMESTAMP WITH TIME ZONE"), ("image_id", "VARCHAR(80)"),+ ("project", "VARCHAR(128)"), ("volumes", "VARCHAR(512)"), ("swap_mb", "DOUBLE PRECISION"),+ ):+ execute(f"ALTER TABLE container_metrics ADD COLUMN IF NOT EXISTS {column} {column_type}")+ for column, column_type in (+ ("boot_id", "VARCHAR(64)"), ("oom_kill_total", "INTEGER"), ("kernel_tainted", "INTEGER"),+ ("ntp_synced", "BOOLEAN"), ("reboot_required", "BOOLEAN"), ("last_shutdown", "VARCHAR(16)"),+ ("docker_status", "VARCHAR(16)"), ("docker_reason", "VARCHAR(16)"), ("docker_latency_ms", "INTEGER"),+ ("internet_status", "VARCHAR(16)"), ("internet_latency_ms", "INTEGER"), ("collect_duration_ms", "INTEGER"),+ ):+ execute(f"ALTER TABLE host_metrics ADD COLUMN IF NOT EXISTS {column} {column_type}")+ execute("ALTER TABLE system_events ADD COLUMN IF NOT EXISTS event_key VARCHAR(128)")+ execute("ALTER TABLE pending_collectors ADD COLUMN IF NOT EXISTS join_token_id INTEGER REFERENCES join_tokens(id) ON DELETE SET NULL")+ execute("ALTER TABLE pending_collectors ADD COLUMN IF NOT EXISTS details TEXT")+ for column, column_type in (("avg_swap_mb", "DOUBLE PRECISION"), ("n_swap_mb", "INTEGER NOT NULL DEFAULT 0"), ("max_swap_mb", "DOUBLE PRECISION")):+ execute(f"ALTER TABLE container_rollups ADD COLUMN IF NOT EXISTS {column} {column_type}")+ execute("CREATE INDEX IF NOT EXISTS ix_system_events_event_key ON system_events (event_key)")+ # Composite indexes that match how the metrics tables are read, then the single-column ones they made+ # redundant. Creating an index briefly blocks writes to that table.+ for table, name, columns in (+ ("host_metrics", "ix_host_metrics_server_time", "server_id, time"),+ ("mount_metrics", "ix_mount_metrics_server_time", "server_id, time"),+ ("system_events", "ix_system_events_server_time", "server_id, time"),+ ("container_metrics", "ix_container_metrics_server_container_time", "server_id, container_id, time"),+ ):+ execute(f"CREATE INDEX IF NOT EXISTS {name} ON {table} ({columns})")+ for redundant in (+ "ix_host_metrics_id", "ix_host_metrics_server_id", "ix_mount_metrics_id", "ix_mount_metrics_server_id",+ "ix_system_events_id", "ix_system_events_server_id",+ "ix_container_metrics_id", "ix_container_metrics_server_id", "ix_container_metrics_container_id",+ ):+ execute(f"DROP INDEX IF EXISTS {redundant}")+ for column, column_type in (+ ("totp_secret_enc", "TEXT"), ("totp_enabled", "BOOLEAN NOT NULL DEFAULT FALSE"), ("totp_last_step", "INTEGER"), ("password_changed_at", "TIMESTAMP WITH TIME ZONE"),+ ("username", "VARCHAR(64)"), ("avatar_color", "VARCHAR(16)"), ("preferences", "TEXT"), ("last_seen_at", "TIMESTAMP WITH TIME ZONE"), ("agent_chat_enabled", "BOOLEAN NOT NULL DEFAULT FALSE"),+ ):+ execute(f"ALTER TABLE users ADD COLUMN IF NOT EXISTS {column} {column_type}")+ for column, column_type in (+ ("rule_key", "VARCHAR(255)"), ("resolved_at", "TIMESTAMP WITH TIME ZONE"), ("resolution", "VARCHAR(255)"), ("acknowledged_at", "TIMESTAMP WITH TIME ZONE"),+ ("acknowledged_by", "INTEGER"), ("muted_until", "TIMESTAMP WITH TIME ZONE"), ("notified_at", "TIMESTAMP WITH TIME ZONE"),+ ("resolve_notified_at", "TIMESTAMP WITH TIME ZONE"), ("reopen_after", "TIMESTAMP WITH TIME ZONE"),+ ):+ execute(f"ALTER TABLE incident_alerts ADD COLUMN IF NOT EXISTS {column} {column_type}")+ execute("CREATE INDEX IF NOT EXISTS ix_incident_alerts_rule_key ON incident_alerts (rule_key)")+ # The users table used to come from docker/postgres/init with its own index names; give it the models'+ # ones. The unique index exists before the old unique constraint goes, so emails stay unique throughout.+ execute("CREATE UNIQUE INDEX IF NOT EXISTS ix_users_email ON users (email)")+ execute("CREATE INDEX IF NOT EXISTS ix_users_id ON users (id)")+ execute("ALTER TABLE users DROP CONSTRAINT IF EXISTS users_email_key")+ execute("DROP INDEX IF EXISTS idx_users_email")+ execute("DROP INDEX IF EXISTS idx_users_role")+ # Alerts from before the alert engine: offline alerts keep going under their rule, the rest were+ # duplicated on every check-in with no way to resolve, so close them; the engine re-raises live ones.+ execute("UPDATE incident_alerts SET rule_key = 'server_offline' WHERE rule_key IS NULL AND description = :description",+ description="The collector has stopped reporting. Check that the server is up and can reach the backend.")+ execute("UPDATE incident_alerts SET status = 'resolved', resolved_at = NOW() WHERE rule_key IS NULL AND status = 'active'")+ # Disk and inode alerts became per mount; the old keys were for the root filesystem.+ execute("UPDATE incident_alerts SET rule_key = rule_key || ':/' WHERE rule_key IN ('host:disk', 'host:inodes')")+ columns = {column["name"] for column in sa.inspect(bind).get_columns("servers")}+ for column, column_type in (("collector_token_hash", "VARCHAR(64)"), ("machine_id", "VARCHAR(64)"), ("is_dashboard_host", "BOOLEAN NOT NULL DEFAULT FALSE"),+ ("system_info", "TEXT"), ("name_locked", "BOOLEAN NOT NULL DEFAULT FALSE")):+ if column not in columns:+ bind.execute(sa.text(f"ALTER TABLE servers ADD COLUMN {column} {column_type}"))+++def downgrade() -> None:+ op.drop_index(op.f('ix_agent_reports_incident_id'), table_name='agent_reports')+ op.drop_table('agent_reports')+ op.drop_index(op.f('ix_system_events_time'), table_name='system_events')+ op.drop_index('ix_system_events_server_time', table_name='system_events')+ op.drop_index(op.f('ix_system_events_event_key'), table_name='system_events')+ op.drop_table('system_events')+ op.drop_index(op.f('ix_pending_collectors_machine_id'), table_name='pending_collectors')+ op.drop_index(op.f('ix_pending_collectors_id'), table_name='pending_collectors')+ op.drop_table('pending_collectors')+ op.drop_index('ix_mount_rollups_tier_bucket', table_name='mount_rollups')+ op.drop_table('mount_rollups')+ op.drop_index(op.f('ix_mount_metrics_time'), table_name='mount_metrics')+ op.drop_index('ix_mount_metrics_server_time', table_name='mount_metrics')+ op.drop_table('mount_metrics')+ op.drop_index(op.f('ix_listener_states_server_id'), table_name='listener_states')+ op.drop_index(op.f('ix_listener_states_id'), table_name='listener_states')+ op.drop_table('listener_states')+ op.drop_index(op.f('ix_incident_alerts_server_id'), table_name='incident_alerts')+ op.drop_index(op.f('ix_incident_alerts_rule_key'), table_name='incident_alerts')+ op.drop_index(op.f('ix_incident_alerts_id'), table_name='incident_alerts')+ op.drop_index(op.f('ix_incident_alerts_created_at'), table_name='incident_alerts')+ op.drop_table('incident_alerts')+ op.drop_index('ix_host_rollups_tier_bucket', table_name='host_rollups')+ op.drop_table('host_rollups')+ op.drop_index(op.f('ix_host_metrics_time'), table_name='host_metrics')+ op.drop_index('ix_host_metrics_server_time', table_name='host_metrics')+ op.drop_table('host_metrics')+ op.drop_index(op.f('ix_firewall_states_server_id'), table_name='firewall_states')+ op.drop_index(op.f('ix_firewall_states_id'), table_name='firewall_states')+ op.drop_table('firewall_states')+ op.drop_index('ix_container_rollups_tier_bucket', table_name='container_rollups')+ op.drop_table('container_rollups')+ op.drop_index(op.f('ix_container_metrics_time'), table_name='container_metrics')+ op.drop_index('ix_container_metrics_server_container_time', table_name='container_metrics')+ op.drop_table('container_metrics')+ op.drop_index(op.f('ix_servers_owner_id'), table_name='servers')+ op.drop_index(op.f('ix_servers_machine_id'), table_name='servers')+ op.drop_index(op.f('ix_servers_id'), table_name='servers')+ op.drop_table('servers')+ op.drop_index(op.f('ix_recovery_codes_user_id'), table_name='recovery_codes')+ op.drop_index(op.f('ix_recovery_codes_id'), table_name='recovery_codes')+ op.drop_table('recovery_codes')+ op.drop_index(op.f('ix_notification_channels_user_id'), table_name='notification_channels')+ op.drop_index(op.f('ix_notification_channels_id'), table_name='notification_channels')+ op.drop_table('notification_channels')+ op.drop_index(op.f('ix_join_tokens_id'), table_name='join_tokens')+ op.drop_table('join_tokens')+ op.drop_index(op.f('ix_invites_id'), table_name='invites')+ op.drop_table('invites')+ op.drop_index(op.f('ix_enrollment_keys_id'), table_name='enrollment_keys')+ op.drop_table('enrollment_keys')+ op.drop_index(op.f('ix_check_rollups_check_id'), table_name='check_rollups')+ op.drop_table('check_rollups')+ op.drop_index('ix_check_results_check_time', table_name='check_results')+ op.drop_table('check_results')+ op.drop_index(op.f('ix_api_tokens_user_id'), table_name='api_tokens')+ op.drop_index(op.f('ix_api_tokens_id'), table_name='api_tokens')+ op.drop_table('api_tokens')+ op.drop_index(op.f('ix_agent_runs_user_id'), table_name='agent_runs')+ op.drop_index('ix_agent_runs_created', table_name='agent_runs')+ op.drop_table('agent_runs')+ op.drop_index(op.f('ix_users_id'), table_name='users')+ op.drop_index(op.f('ix_users_email'), table_name='users')+ op.drop_table('users')+ op.drop_index(op.f('ix_conversations_user_id'), table_name='conversations')+ op.drop_table('conversations')+ op.drop_table('checks')+ op.drop_index(op.f('ix_chat_messages_id'), table_name='chat_messages')+ op.drop_index(op.f('ix_chat_messages_conversation_id'), table_name='chat_messages')+ op.drop_table('chat_messages')+ op.drop_index(op.f('ix_audit_logs_user_id'), table_name='audit_logs')+ op.drop_index(op.f('ix_audit_logs_id'), table_name='audit_logs')+ op.drop_index(op.f('ix_audit_logs_conversation_id'), table_name='audit_logs')+ op.drop_table('audit_logs')+ op.drop_table('app_settings')diff --git a/tests/history/test_migrations.py b/tests/history/test_migrations.pynew file mode 100644index 0000000..4a0df08--- /dev/null+++ b/tests/history/test_migrations.py@@ -0,0 +1,90 @@+"""Migrations (src/backend/migrations): a new database and one from before Alembic both end at the latest revision+with exactly the schema the models describe, and running the upgrade again changes nothing.++Set MIGRATION_TEST_POSTGRES_URL to an empty PostgreSQL database to run the same checks there too."""+import os++os.environ.setdefault("DATABASE_URL", "sqlite://")+os.environ.setdefault("REDIS_URL", "redis://localhost:6379/0")++import pytest+from alembic.autogenerate import compare_metadata+from alembic.migration import MigrationContext+from alembic.script import ScriptDirectory+from sqlalchemy import create_engine, inspect, text++from src.backend.migrate import MIGRATIONS_DIR, alembic_config, upgrade_database+from src.backend.models import Base++POSTGRES_URL = os.getenv("MIGRATION_TEST_POSTGRES_URL")++[email protected](params=["sqlite"] + (["postgresql"] if POSTGRES_URL else []))+def engine(request, tmp_path):+ if request.param == "sqlite":+ engine = create_engine(f"sqlite:///{tmp_path / 'test.db'}")+ else:+ engine = create_engine(POSTGRES_URL)+ with engine.begin() as connection:+ connection.execute(text("DROP SCHEMA public CASCADE"))+ connection.execute(text("CREATE SCHEMA public"))+ yield engine+ engine.dispose()+++def _head() -> str:+ with create_engine("sqlite://").connect() as connection:+ return ScriptDirectory.from_config(alembic_config(connection)).get_current_head()+++def _version(engine) -> str | None:+ with engine.connect() as connection:+ return MigrationContext.configure(connection).get_current_revision()+++def _differences(engine) -> list:+ with engine.connect() as connection:+ return compare_metadata(MigrationContext.configure(connection, opts={"compare_type": True}), Base.metadata)+++def test_the_migrations_live_next_to_the_backend():+ assert (MIGRATIONS_DIR / "versions").is_dir()+++def test_a_new_database_ends_at_the_latest_revision_with_the_models_schema(engine):+ upgrade_database(engine)+ assert _version(engine) == _head()+ assert _differences(engine) == []+++def test_upgrading_twice_changes_nothing(engine):+ upgrade_database(engine)+ before = {table: inspect(engine).get_columns(table) for table in inspect(engine).get_table_names()}+ upgrade_database(engine)+ after = {table: inspect(engine).get_columns(table) for table in inspect(engine).get_table_names()}+ assert _version(engine) == _head()+ assert [(table, [column["name"] for column in columns]) for table, columns in before.items()] == [(table, [column["name"] for column in columns]) for table, columns in after.items()]+++def test_a_database_from_before_migrations_keeps_its_data_and_ends_at_the_latest_revision(engine):+ Base.metadata.create_all(engine) # how the schema was made before Alembic+ with engine.begin() as connection:+ connection.execute(text("INSERT INTO app_settings (key, value, updated_at) VALUES ('instance_name', 'lab', CURRENT_TIMESTAMP)"))+ upgrade_database(engine)+ assert _version(engine) == _head()+ assert _differences(engine) == []+ with engine.connect() as connection:+ assert connection.execute(text("SELECT value FROM app_settings WHERE key = 'instance_name'")).scalar() == "lab"+++def test_an_old_database_missing_tables_and_columns_is_brought_up_to_date(engine):+ """An install that last ran an old version: no availability tables yet, and servers without the newer columns."""+ tables = [table for name, table in Base.metadata.tables.items() if not name.startswith("check")]+ Base.metadata.create_all(engine, tables=tables)+ with engine.begin() as connection:+ connection.execute(text("ALTER TABLE servers DROP COLUMN name_locked"))+ connection.execute(text("ALTER TABLE servers DROP COLUMN system_info"))+ upgrade_database(engine)+ assert {"checks", "check_results", "check_rollups"} <= set(inspect(engine).get_table_names())+ assert {"name_locked", "system_info"} <= {column["name"] for column in inspect(engine).get_columns("servers")}+ assert _differences(engine) == []