🐸 Rybbit - open-source and privacy-friendly alternative to Google Analytics that is 10x more intuitive.
Refactor Sidebar and Session Replay Page for Environment-Specific Rendering
Bill Yang5356038b9b95
diff --git a/ops/backups/README.md b/ops/backups/README.mdnew file mode 100644index 000000000..ed90ec835--- /dev/null+++ b/ops/backups/README.md@@ -0,0 +1,211 @@+# Database backups++These scripts create database-consistent backups for the three production+hosts. They intentionally do not archive live Docker volumes.++## Host map++| Host | PostgreSQL | ClickHouse |+| --- | --- | --- |+| FrogStats | Native `pg_dump` to B2 | Native `BACKUP` directly to B2 |+| Rybbit | Native `pg_dump` to B2 | Not backed up here; production uses the standalone host |+| ClickHouse | None | One native local backup copied to Hetzner and B2 |++PostgreSQL dumps include a custom-format database archive, cluster globals,+and SHA-256 checksums. ClickHouse backs up only the `analytics` database, not+the large and regenerable `system` log tables.++## Prerequisites++All hosts need Bash, Docker, `flock`, `curl`, and `rclone`. The `clickhouse`+container must provide `clickhouse-client`; the PostgreSQL container already+provides `pg_dump`, `pg_dumpall`, and `pg_restore`.++Create a private B2 bucket, enable server-side encryption, and optionally+enable a 7-14 day Object Lock. Give each host a separate application key+restricted to that host's prefix. Configure lifecycle rules in B2 rather than+giving the server credentials permission to delete old remote backups.++For PostgreSQL and locally staged ClickHouse backups, configure rclone as root:++```bash+sudo rclone config+sudo chmod 600 /root/.config/rclone/rclone.conf+```++The examples use an rclone `crypt` remote named `b2crypt` over a B2 remote.+This provides client-side encryption in addition to B2 server-side encryption.+The direct FrogStats ClickHouse backup uses B2 server-side encryption because+it is written directly through ClickHouse's S3-compatible client. Its B2+credentials live in a ClickHouse named collection rather than the backup SQL.++## Install on each host++From a checkout of this repository:++```bash+sudo install -d -m 0755 /opt/rybbit-backups/bin+sudo install -m 0755 ops/backups/bin/backup-postgres.sh /opt/rybbit-backups/bin/+sudo install -m 0755 ops/backups/bin/backup-clickhouse.sh /opt/rybbit-backups/bin/+sudo install -m 0644 ops/backups/bin/common.sh /opt/rybbit-backups/bin/+```++Install the appropriate configuration without committing secrets:++```bash+# Choose frogstats.env.example, rybbit.env.example, or clickhouse.env.example.+sudo install -m 0600 ops/backups/config/HOST.env.example /etc/rybbit-backup.env+sudoedit /etc/rybbit-backup.env+```++The configuration file is sourced by Bash. Quote values containing `$`, spaces,+single quotes, or other shell metacharacters.++Validate configuration without touching either database or remote storage:++```bash+sudo env BACKUP_CONFIG=/etc/rybbit-backup.env DRY_RUN=true \+ /opt/rybbit-backups/bin/backup-postgres.sh+sudo env BACKUP_CONFIG=/etc/rybbit-backup.env DRY_RUN=true \+ /opt/rybbit-backups/bin/backup-clickhouse.sh+```++Only run the script relevant to the host. The Rybbit application host does not+have ClickHouse backup settings.++## FrogStats ClickHouse named collection++The direct S3 backup requires a named collection containing its B2 endpoint+and restricted application key. Install the template outside the repository:++```bash+sudo install -d -m 0750 /etc/rybbit-backup+sudo install -m 0640 -o root -g 101 \+ ops/backups/clickhouse/s3-named-collection.xml.example \+ /etc/rybbit-backup/clickhouse-s3.xml+sudoedit /etc/rybbit-backup/clickhouse-s3.xml+```++Then mount it into the FrogStats ClickHouse container:++```yaml+services:+ clickhouse:+ volumes:+ - /etc/rybbit-backup/clickhouse-s3.xml:/etc/clickhouse-server/config.d/backup-s3.xml:ro+```++UID/GID 101 is the default ClickHouse user in the official image; confirm it+before installation if the image changes. Recreate ClickHouse and verify the+collection without printing its contents:++```bash+docker exec clickhouse clickhouse-client \+ --query "SELECT name FROM system.named_collections WHERE name = 'b2_backups'"+```++## Standalone ClickHouse staging disk++The standalone host creates one native backup and uploads the same completed+snapshot to both Hetzner and B2. Add these mounts to its ClickHouse service:++```yaml+services:+ clickhouse:+ volumes:+ - /var/backups/rybbit/clickhouse:/var/lib/clickhouse/backups+ - ./ops/backups/clickhouse/backup-disk.xml:/etc/clickhouse-server/config.d/backup-disk.xml:ro+```++Prepare the host directory before recreating the container. UID/GID 101 is the+default ClickHouse user in the official image; confirm it with `docker exec`+if the image changes:++```bash+sudo install -d -m 0750 -o 101 -g 101 /var/backups/rybbit/clickhouse+```++Restart ClickHouse after adding the disk configuration, then confirm it exists:++```bash+docker exec clickhouse clickhouse-client \+ --query "SELECT name, path FROM system.disks WHERE name = 'backups'"+```++FrogStats uses direct S3 mode and does not need this staging disk.++## Test manually, then enable timers++Run the relevant service manually before enabling its timer:++```bash+sudo install -m 0644 ops/backups/systemd/*.service ops/backups/systemd/*.timer /etc/systemd/system/+sudo systemctl daemon-reload++sudo systemctl start rybbit-postgres-backup.service+sudo journalctl -u rybbit-postgres-backup.service -n 100 --no-pager++# FrogStats and the standalone ClickHouse host only:+sudo systemctl start rybbit-clickhouse-backup.service+sudo journalctl -u rybbit-clickhouse-backup.service -n 100 --no-pager+```++After verifying objects at every configured destination:++```bash+sudo systemctl enable --now rybbit-postgres-backup.timer+# FrogStats and the standalone ClickHouse host only:+sudo systemctl enable --now rybbit-clickhouse-backup.timer+```++The PostgreSQL timer runs every six hours. The ClickHouse timer runs daily with+a randomized delay. Use systemd timer overrides if the two ClickHouse hosts+should run in different windows.++The standalone example prunes only the Hetzner destination after 14 days. B2+retention should be implemented with bucket lifecycle rules, especially when+Object Lock is enabled. Never add an Object-Locked B2 remote to+`CLICKHOUSE_PRUNE_REMOTES`.++## Restore drills++Always restore into a disposable container or a differently named database+first. Never test a restore against production.++For PostgreSQL, download one backup directory through the crypt remote, verify+it, restore globals, and then restore the custom archive:++```bash+cd /var/tmp/postgres-restore+sha256sum --check SHA256SUMS+docker exec -i postgres sh -ec \+ 'exec psql -U "$POSTGRES_USER" -d postgres' < globals.sql+docker exec postgres sh -ec \+ 'exec createdb -U "$POSTGRES_USER" analytics_restore'+docker exec -i postgres sh -ec \+ 'exec pg_restore -U "$POSTGRES_USER" --dbname analytics_restore --clean --if-exists' \+ < database.dump+```++Review the globals SQL before applying it when restoring into a shared+PostgreSQL cluster.++For a direct S3 ClickHouse backup, use the installed named collection and a+different destination database name:++```sql+SET log_queries = 0;+RESTORE DATABASE analytics AS analytics_restore+FROM S3(b2_backups, 'BACKUP_DIRECTORY');+```++For a locally staged ClickHouse backup, download the complete backup directory+to the configured host staging root and restore it from the backup disk:++```sql+RESTORE DATABASE analytics AS analytics_restore+FROM Disk('backups', 'BACKUP_DIRECTORY');+```++Perform and record a restore drill at least monthly. A successful upload is not+proof that the backup can be restored.diff --git a/ops/backups/bin/backup-clickhouse.sh b/ops/backups/bin/backup-clickhouse.shnew file mode 100755index 000000000..e3cfe8b7e--- /dev/null+++ b/ops/backups/bin/backup-clickhouse.sh@@ -0,0 +1,112 @@+#!/usr/bin/env bash++set -Eeuo pipefail++SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+# shellcheck source=common.sh+source "${SCRIPT_DIR}/common.sh"++finish() {+ local status=$?+ trap - EXIT++ if ((status == 0)); then+ log "ClickHouse backup completed successfully"+ notify_url "${CLICKHOUSE_SUCCESS_URL:-${BACKUP_SUCCESS_URL:-}}"+ else+ log "ClickHouse backup failed with exit status ${status}" >&2+ notify_url "${CLICKHOUSE_FAILURE_URL:-${BACKUP_FAILURE_URL:-}}"+ fi++ exit "$status"+}+trap finish EXIT++run_clickhouse_query() {+ local query="$1"++ # Feed SQL over stdin so S3 credentials do not appear in the host process+ # list. Query logging is disabled for this client session by the query.+ printf '%s\n' "$query" | docker exec -i "$CLICKHOUSE_CONTAINER" sh -ec \+ 'exec clickhouse-client --user "$CLICKHOUSE_USER" --password "$CLICKHOUSE_PASSWORD" --multiquery'+}++load_backup_config++CLICKHOUSE_CONTAINER="${CLICKHOUSE_CONTAINER:-clickhouse}"+CLICKHOUSE_DATABASE="${CLICKHOUSE_DATABASE:-analytics}"+CLICKHOUSE_BACKUP_MODE="${CLICKHOUSE_BACKUP_MODE:-s3}"+CLICKHOUSE_LOCK_FILE="${CLICKHOUSE_LOCK_FILE:-/run/lock/rybbit-clickhouse-backup.lock}"+BACKUP_HOST="${BACKUP_HOST:-$(hostname -s)}"+DRY_RUN="${DRY_RUN:-false}"++validate_simple_name BACKUP_HOST "$BACKUP_HOST"+validate_simple_name CLICKHOUSE_CONTAINER "$CLICKHOUSE_CONTAINER"+validate_identifier CLICKHOUSE_DATABASE "$CLICKHOUSE_DATABASE"+[[ "$CLICKHOUSE_BACKUP_MODE" == "s3" || "$CLICKHOUSE_BACKUP_MODE" == "local" ]] || \+ fatal "CLICKHOUSE_BACKUP_MODE must be 's3' or 'local'"++BACKUP_NAME="${BACKUP_HOST}-clickhouse-${CLICKHOUSE_DATABASE}-$(date -u '+%Y%m%dT%H%M%SZ')"++if [[ "$CLICKHOUSE_BACKUP_MODE" == "s3" ]]; then+ require_value CLICKHOUSE_S3_NAMED_COLLECTION+ validate_identifier CLICKHOUSE_S3_NAMED_COLLECTION "$CLICKHOUSE_S3_NAMED_COLLECTION"++ if [[ "$DRY_RUN" == "true" ]]; then+ log "DRY RUN: would create native ClickHouse backup ${BACKUP_NAME}"+ log "DRY RUN: destination named collection ${CLICKHOUSE_S3_NAMED_COLLECTION}"+ exit 0+ fi++ require_command docker+ require_command flock+ acquire_lock "$CLICKHOUSE_LOCK_FILE"++ log "Creating native ClickHouse backup of ${CLICKHOUSE_DATABASE} directly in object storage"+ QUERY="SET log_queries = 0; BACKUP DATABASE \`${CLICKHOUSE_DATABASE}\` TO S3(${CLICKHOUSE_S3_NAMED_COLLECTION}, $(sql_string "$BACKUP_NAME")) SETTINGS compression_method = 'zstd', compression_level = 1;"+ run_clickhouse_query "$QUERY"+else+ CLICKHOUSE_BACKUP_DISK="${CLICKHOUSE_BACKUP_DISK:-backups}"+ CLICKHOUSE_LOCAL_BACKUP_ROOT="${CLICKHOUSE_LOCAL_BACKUP_ROOT:-/var/backups/rybbit/clickhouse}"+ CLICKHOUSE_KEEP_LOCAL_BACKUP="${CLICKHOUSE_KEEP_LOCAL_BACKUP:-false}"++ require_value CLICKHOUSE_RCLONE_REMOTES+ validate_simple_name CLICKHOUSE_BACKUP_DISK "$CLICKHOUSE_BACKUP_DISK"++ if [[ "$DRY_RUN" == "true" ]]; then+ log "DRY RUN: would create native ClickHouse backup ${BACKUP_NAME} on disk ${CLICKHOUSE_BACKUP_DISK}"+ log "DRY RUN: would upload to ${CLICKHOUSE_RCLONE_REMOTES}"+ exit 0+ fi++ require_command docker+ require_command flock+ require_command rclone+ acquire_lock "$CLICKHOUSE_LOCK_FILE"++ mkdir -p "$CLICKHOUSE_LOCAL_BACKUP_ROOT"+ LOCAL_BACKUP_DIR="${CLICKHOUSE_LOCAL_BACKUP_ROOT%/}/${BACKUP_NAME}"+ [[ ! -e "$LOCAL_BACKUP_DIR" ]] || fatal "Local backup path already exists: $LOCAL_BACKUP_DIR"++ log "Creating native ClickHouse backup of ${CLICKHOUSE_DATABASE} on disk ${CLICKHOUSE_BACKUP_DISK}"+ QUERY="SET log_queries = 0; BACKUP DATABASE \`${CLICKHOUSE_DATABASE}\` TO Disk($(sql_string "$CLICKHOUSE_BACKUP_DISK"), $(sql_string "$BACKUP_NAME")) SETTINGS compression_method = 'zstd', compression_level = 1;"+ run_clickhouse_query "$QUERY"++ [[ -d "$LOCAL_BACKUP_DIR" ]] || fatal "ClickHouse reported success but local backup was not found: $LOCAL_BACKUP_DIR"+ [[ -n "$(find "$LOCAL_BACKUP_DIR" -mindepth 1 -print -quit)" ]] || fatal "ClickHouse local backup is empty"++ upload_directory "$LOCAL_BACKUP_DIR" "$BACKUP_NAME" "$CLICKHOUSE_RCLONE_REMOTES"++ if [[ -n "${CLICKHOUSE_PRUNE_REMOTES:-}" ]]; then+ CLICKHOUSE_REMOTE_RETENTION_DAYS="${CLICKHOUSE_REMOTE_RETENTION_DAYS:-14}"+ prune_remote_backup_directories \+ "$CLICKHOUSE_PRUNE_REMOTES" \+ "$CLICKHOUSE_REMOTE_RETENTION_DAYS" \+ "${BACKUP_HOST}-clickhouse-${CLICKHOUSE_DATABASE}-"+ fi++ if [[ "$CLICKHOUSE_KEEP_LOCAL_BACKUP" != "true" ]]; then+ log "Removing completed local staging backup"+ safe_delete_staging_dir "$CLICKHOUSE_LOCAL_BACKUP_ROOT" "$LOCAL_BACKUP_DIR"+ fi+fidiff --git a/ops/backups/bin/backup-postgres.sh b/ops/backups/bin/backup-postgres.shnew file mode 100755index 000000000..18b85481b--- /dev/null+++ b/ops/backups/bin/backup-postgres.sh@@ -0,0 +1,87 @@+#!/usr/bin/env bash++set -Eeuo pipefail++SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"+# shellcheck source=common.sh+source "${SCRIPT_DIR}/common.sh"++STAGING_DIR=""++finish() {+ local status=$?+ trap - EXIT++ if [[ -n "$STAGING_DIR" && -d "$STAGING_DIR" ]]; then+ safe_delete_staging_dir "$POSTGRES_STAGING_ROOT" "$STAGING_DIR" || true+ fi++ if ((status == 0)); then+ log "PostgreSQL backup completed successfully"+ notify_url "${POSTGRES_SUCCESS_URL:-${BACKUP_SUCCESS_URL:-}}"+ else+ log "PostgreSQL backup failed with exit status ${status}" >&2+ notify_url "${POSTGRES_FAILURE_URL:-${BACKUP_FAILURE_URL:-}}"+ fi++ exit "$status"+}+trap finish EXIT++load_backup_config++POSTGRES_CONTAINER="${POSTGRES_CONTAINER:-postgres}"+POSTGRES_DATABASE="${POSTGRES_DATABASE:-analytics}"+POSTGRES_STAGING_ROOT="${POSTGRES_STAGING_ROOT:-/var/backups/rybbit/postgres}"+POSTGRES_LOCK_FILE="${POSTGRES_LOCK_FILE:-/run/lock/rybbit-postgres-backup.lock}"+BACKUP_HOST="${BACKUP_HOST:-$(hostname -s)}"+DRY_RUN="${DRY_RUN:-false}"++require_value POSTGRES_RCLONE_REMOTES+validate_simple_name BACKUP_HOST "$BACKUP_HOST"+validate_simple_name POSTGRES_CONTAINER "$POSTGRES_CONTAINER"+validate_identifier POSTGRES_DATABASE "$POSTGRES_DATABASE"++BACKUP_NAME="${BACKUP_HOST}-postgres-${POSTGRES_DATABASE}-$(date -u '+%Y%m%dT%H%M%SZ')"++if [[ "$DRY_RUN" == "true" ]]; then+ log "DRY RUN: would create ${BACKUP_NAME} from container ${POSTGRES_CONTAINER}"+ log "DRY RUN: would upload to ${POSTGRES_RCLONE_REMOTES}"+ exit 0+fi++require_command docker+require_command flock+require_command rclone+require_command sha256sum+acquire_lock "$POSTGRES_LOCK_FILE"++mkdir -p "$POSTGRES_STAGING_ROOT"+STAGING_DIR="${POSTGRES_STAGING_ROOT%/}/${BACKUP_NAME}"+mkdir "$STAGING_DIR"++log "Creating consistent PostgreSQL dump for ${POSTGRES_DATABASE}"+docker exec "$POSTGRES_CONTAINER" sh -ec \+ 'export PGPASSWORD="$POSTGRES_PASSWORD"; exec pg_dump --username "$POSTGRES_USER" --dbname "$1" --format=custom --compress=zstd:3' \+ sh "$POSTGRES_DATABASE" >"${STAGING_DIR}/database.dump.partial"+mv "${STAGING_DIR}/database.dump.partial" "${STAGING_DIR}/database.dump"++log "Dumping PostgreSQL roles and cluster globals"+docker exec "$POSTGRES_CONTAINER" sh -ec \+ 'export PGPASSWORD="$POSTGRES_PASSWORD"; exec pg_dumpall --username "$POSTGRES_USER" --globals-only' \+ >"${STAGING_DIR}/globals.sql.partial"+mv "${STAGING_DIR}/globals.sql.partial" "${STAGING_DIR}/globals.sql"++[[ -s "${STAGING_DIR}/database.dump" ]] || fatal "PostgreSQL dump is empty"+[[ -s "${STAGING_DIR}/globals.sql" ]] || fatal "PostgreSQL globals dump is empty"++log "Validating PostgreSQL archive"+docker exec -i "$POSTGRES_CONTAINER" pg_restore --list <"${STAGING_DIR}/database.dump" >/dev/null++(+ cd "$STAGING_DIR"+ sha256sum database.dump globals.sql >SHA256SUMS+)++upload_directory "$STAGING_DIR" "$BACKUP_NAME" "$POSTGRES_RCLONE_REMOTES"+diff --git a/ops/backups/bin/common.sh b/ops/backups/bin/common.shnew file mode 100644index 000000000..61540b371--- /dev/null+++ b/ops/backups/bin/common.sh@@ -0,0 +1,152 @@+#!/usr/bin/env bash++# Shared functions for the database backup scripts. This file is sourced by+# the entrypoints and is not intended to be executed directly.++set -Eeuo pipefail+umask 077++log() {+ printf '[%s] %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$*"+}++fatal() {+ log "ERROR: $*" >&2+ exit 1+}++require_command() {+ command -v "$1" >/dev/null 2>&1 || fatal "Required command not found: $1"+}++require_value() {+ local name="$1"+ [[ -n "${!name:-}" ]] || fatal "Required configuration value is empty: ${name}"+}++load_backup_config() {+ local config_path="${BACKUP_CONFIG:-/etc/rybbit-backup.env}"+ [[ -r "$config_path" ]] || fatal "Cannot read backup configuration: $config_path"++ # shellcheck disable=SC1090+ source "$config_path"+}++validate_simple_name() {+ local label="$1"+ local value="$2"+ [[ "$value" =~ ^[A-Za-z0-9._-]+$ ]] || fatal "$label contains unsupported characters: $value"+}++validate_identifier() {+ local label="$1"+ local value="$2"+ [[ "$value" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] || fatal "$label is not a safe SQL identifier: $value"+}++sql_string() {+ local value=${1//\'/\'\'}+ printf "'%s'" "$value"+}++acquire_lock() {+ local lock_path="$1"+ mkdir -p "$(dirname "$lock_path")"+ exec {BACKUP_LOCK_FD}>"$lock_path"+ flock -n "$BACKUP_LOCK_FD" || fatal "Another backup is already running: $lock_path"+}++notify_url() {+ local url="${1:-}"+ [[ -n "$url" ]] || return 0++ if ! curl --fail --silent --show-error --retry 2 --max-time 15 "$url" >/dev/null; then+ log "WARNING: Could not send backup notification"+ fi+}++upload_directory() {+ local source_dir="$1"+ local backup_name="$2"+ local remotes_csv="$3"+ local -a remotes+ local remote_base+ local remote_path++ IFS=',' read -r -a remotes <<<"$remotes_csv"+ ((${#remotes[@]} > 0)) || fatal "No rclone destinations configured"++ for remote_base in "${remotes[@]}"; do+ remote_base="${remote_base#${remote_base%%[![:space:]]*}}"+ remote_base="${remote_base%${remote_base##*[![:space:]]}}"+ [[ -n "$remote_base" ]] || fatal "Empty destination in remote list"++ remote_path="${remote_base%/}/${backup_name}"+ log "Uploading ${backup_name} to ${remote_path}"+ rclone copy "${source_dir%/}/" "${remote_path}/" \+ --immutable \+ --transfers "${RCLONE_TRANSFERS:-4}" \+ --checkers "${RCLONE_CHECKERS:-8}" \+ --retries "${RCLONE_RETRIES:-5}" \+ --low-level-retries "${RCLONE_LOW_LEVEL_RETRIES:-10}" \+ --stats 30s++ log "Verifying ${remote_path}"+ rclone check "${source_dir%/}/" "${remote_path}/" --one-way+ done+}++prune_remote_backup_directories() {+ local remotes_csv="$1"+ local retention_days="$2"+ local backup_prefix="$3"+ local -a remotes+ local remote_base+ local listing+ local directory_name+ local timestamp+ local backup_epoch+ local cutoff_epoch++ [[ "$retention_days" =~ ^[1-9][0-9]*$ ]] || fatal "Retention days must be a positive integer"+ validate_simple_name "Backup prefix" "$backup_prefix"+ cutoff_epoch="$(date -u -d "${retention_days} days ago" '+%s')"+ IFS=',' read -r -a remotes <<<"$remotes_csv"++ for remote_base in "${remotes[@]}"; do+ remote_base="${remote_base#${remote_base%%[![:space:]]*}}"+ remote_base="${remote_base%${remote_base##*[![:space:]]}}"+ [[ -n "$remote_base" ]] || fatal "Empty destination in prune remote list"++ log "Looking for complete backups older than ${retention_days} days in ${remote_base}"+ listing="$(rclone lsf "${remote_base%/}/" --dirs-only --max-depth 1)"++ while IFS= read -r directory_name; do+ directory_name="${directory_name%/}"+ [[ "$directory_name" == "${backup_prefix}"* ]] || continue++ timestamp="${directory_name#"$backup_prefix"}"+ [[ "$timestamp" =~ ^[0-9]{8}T[0-9]{6}Z$ ]] || continue+ backup_epoch="$(date -u -d "${timestamp:0:4}-${timestamp:4:2}-${timestamp:6:2} ${timestamp:9:2}:${timestamp:11:2}:${timestamp:13:2} UTC" '+%s')" || \+ fatal "Could not parse backup timestamp: $directory_name"++ if ((backup_epoch < cutoff_epoch)); then+ log "Purging expired complete backup ${remote_base%/}/${directory_name}"+ rclone purge "${remote_base%/}/${directory_name}"+ fi+ done <<<"$listing"+ done+}++safe_delete_staging_dir() {+ local staging_root="$1"+ local target="$2"++ [[ -n "$staging_root" && -n "$target" ]] || fatal "Refusing to delete an unresolved staging path"+ [[ "$target" == "${staging_root%/}/"* ]] || fatal "Refusing to delete path outside staging root: $target"+ [[ "$target" != "$staging_root" && "$target" != "${staging_root%/}/" ]] || fatal "Refusing to delete staging root"++ if [[ -d "$target" ]]; then+ find "$target" -depth -delete+ fi+}diff --git a/ops/backups/clickhouse/backup-disk.xml b/ops/backups/clickhouse/backup-disk.xmlnew file mode 100644index 000000000..1375de50f--- /dev/null+++ b/ops/backups/clickhouse/backup-disk.xml@@ -0,0 +1,18 @@+<clickhouse>+ <!-- Required only for CLICKHOUSE_BACKUP_MODE=local. Bind-mount the host+ staging directory at this path inside the ClickHouse container. -->+ <storage_configuration>+ <disks>+ <backups>+ <type>local</type>+ <path>/var/lib/clickhouse/backups/</path>+ </backups>+ </disks>+ </storage_configuration>+ <backups>+ <allowed_disk>backups</allowed_disk>+ <allow_concurrent_backups>false</allow_concurrent_backups>+ <allow_concurrent_restores>false</allow_concurrent_restores>+ </backups>+</clickhouse>+diff --git a/ops/backups/clickhouse/s3-named-collection.xml.example b/ops/backups/clickhouse/s3-named-collection.xml.examplenew file mode 100644index 000000000..5d2fd9940--- /dev/null+++ b/ops/backups/clickhouse/s3-named-collection.xml.example@@ -0,0 +1,14 @@+<clickhouse>+ <!-- Install this file outside the repository, replace all placeholders,+ make it readable only by root and the ClickHouse group, and mount it+ read-only into /etc/clickhouse-server/config.d/. -->+ <allow_named_collection_override_by_default>false</allow_named_collection_override_by_default>+ <named_collections>+ <b2_backups>+ <url overridable="false">https://s3.YOUR-B2-REGION.backblazeb2.com/YOUR-BUCKET/frogstats/clickhouse/</url>+ <access_key_id overridable="false">REPLACE_WITH_B2_KEY_ID</access_key_id>+ <secret_access_key overridable="false">REPLACE_WITH_B2_APPLICATION_KEY</secret_access_key>+ </b2_backups>+ </named_collections>+</clickhouse>+diff --git a/ops/backups/config/clickhouse.env.example b/ops/backups/config/clickhouse.env.examplenew file mode 100644index 000000000..48c378f99--- /dev/null+++ b/ops/backups/config/clickhouse.env.example@@ -0,0 +1,23 @@+# Copy to /etc/rybbit-backup.env on the standalone ClickHouse host and chmod 600.++BACKUP_HOST=clickhouse++CLICKHOUSE_CONTAINER=clickhouse+CLICKHOUSE_DATABASE=analytics+CLICKHOUSE_BACKUP_MODE=local+CLICKHOUSE_BACKUP_DISK=backups+CLICKHOUSE_LOCAL_BACKUP_ROOT=/var/backups/rybbit/clickhouse+CLICKHOUSE_KEEP_LOCAL_BACKUP=false++# The same completed native snapshot is copied to both providers. Configure+# these names in root's /root/.config/rclone/rclone.conf. Use an rclone crypt+# wrapper for B2 if client-side encryption is required.+CLICKHOUSE_RCLONE_REMOTES=hetzner:clickhouse-backups,b2crypt:rybbit-db-backups/clickhouse/analytics++# Hetzner does not have B2 lifecycle rules, so prune it from the host. Do not+# include the Object-Locked B2 remote here; let B2 lifecycle rules expire it.+CLICKHOUSE_PRUNE_REMOTES=hetzner:clickhouse-backups+CLICKHOUSE_REMOTE_RETENTION_DAYS=14++# CLICKHOUSE_SUCCESS_URL=+# CLICKHOUSE_FAILURE_URL=diff --git a/ops/backups/config/frogstats.env.example b/ops/backups/config/frogstats.env.examplenew file mode 100644index 000000000..f90d25b92--- /dev/null+++ b/ops/backups/config/frogstats.env.example@@ -0,0 +1,21 @@+# Copy to /etc/rybbit-backup.env on the FrogStats host and chmod 600.++BACKUP_HOST=frogstats++# PostgreSQL: rclone should use a crypt remote backed by the private B2 bucket.+POSTGRES_CONTAINER=postgres+POSTGRES_DATABASE=analytics+POSTGRES_STAGING_ROOT=/var/backups/rybbit/postgres+POSTGRES_RCLONE_REMOTES=b2crypt:rybbit-db-backups/frogstats/postgres++# ClickHouse: direct native backup avoids staging ~58 GiB on this host.+CLICKHOUSE_CONTAINER=clickhouse+CLICKHOUSE_DATABASE=analytics+CLICKHOUSE_BACKUP_MODE=s3+CLICKHOUSE_S3_NAMED_COLLECTION=b2_backups++# Optional monitoring endpoints, such as Healthchecks.io ping URLs.+# POSTGRES_SUCCESS_URL=+# POSTGRES_FAILURE_URL=+# CLICKHOUSE_SUCCESS_URL=+# CLICKHOUSE_FAILURE_URL=diff --git a/ops/backups/config/rybbit.env.example b/ops/backups/config/rybbit.env.examplenew file mode 100644index 000000000..d92435239--- /dev/null+++ b/ops/backups/config/rybbit.env.example@@ -0,0 +1,15 @@+# Copy to /etc/rybbit-backup.env on the Rybbit application host and chmod 600.++BACKUP_HOST=rybbit++POSTGRES_CONTAINER=postgres+POSTGRES_DATABASE=analytics+POSTGRES_STAGING_ROOT=/var/backups/rybbit/postgres+POSTGRES_RCLONE_REMOTES=b2crypt:rybbit-db-backups/rybbit/postgres++# No ClickHouse backup is configured here. The backend uses the standalone+# ClickHouse host rather than this host's effectively empty local volume.++# POSTGRES_SUCCESS_URL=+# POSTGRES_FAILURE_URL=+diff --git a/ops/backups/systemd/rybbit-clickhouse-backup.service b/ops/backups/systemd/rybbit-clickhouse-backup.servicenew file mode 100644index 000000000..147755e55--- /dev/null+++ b/ops/backups/systemd/rybbit-clickhouse-backup.service@@ -0,0 +1,15 @@+[Unit]+Description=Rybbit native ClickHouse backup+After=docker.service network-online.target+Wants=network-online.target+Requires=docker.service++[Service]+Type=oneshot+Environment=BACKUP_CONFIG=/etc/rybbit-backup.env+ExecStart=/opt/rybbit-backups/bin/backup-clickhouse.sh+UMask=0077+Nice=10+IOSchedulingClass=best-effort+IOSchedulingPriority=7+diff --git a/ops/backups/systemd/rybbit-clickhouse-backup.timer b/ops/backups/systemd/rybbit-clickhouse-backup.timernew file mode 100644index 000000000..ad19890c3--- /dev/null+++ b/ops/backups/systemd/rybbit-clickhouse-backup.timer@@ -0,0 +1,12 @@+[Unit]+Description=Run Rybbit ClickHouse backup daily++[Timer]+OnCalendar=*-*-* 02:30:00+Persistent=true+RandomizedDelaySec=20m+Unit=rybbit-clickhouse-backup.service++[Install]+WantedBy=timers.target+diff --git a/ops/backups/systemd/rybbit-postgres-backup.service b/ops/backups/systemd/rybbit-postgres-backup.servicenew file mode 100644index 000000000..2153ec94a--- /dev/null+++ b/ops/backups/systemd/rybbit-postgres-backup.service@@ -0,0 +1,15 @@+[Unit]+Description=Rybbit native PostgreSQL backup+After=docker.service network-online.target+Wants=network-online.target+Requires=docker.service++[Service]+Type=oneshot+Environment=BACKUP_CONFIG=/etc/rybbit-backup.env+ExecStart=/opt/rybbit-backups/bin/backup-postgres.sh+UMask=0077+Nice=10+IOSchedulingClass=best-effort+IOSchedulingPriority=7+diff --git a/ops/backups/systemd/rybbit-postgres-backup.timer b/ops/backups/systemd/rybbit-postgres-backup.timernew file mode 100644index 000000000..a161fc7d2--- /dev/null+++ b/ops/backups/systemd/rybbit-postgres-backup.timer@@ -0,0 +1,12 @@+[Unit]+Description=Run Rybbit PostgreSQL backup every six hours++[Timer]+OnCalendar=*-*-* 00,06,12,18:15:00+Persistent=true+RandomizedDelaySec=10m+Unit=rybbit-postgres-backup.service++[Install]+WantedBy=timers.target+