Skip to content

🐸 Rybbit - open-source and privacy-friendly alternative to Google Analytics that is 10x more intuitive.

Refactor Sidebar and Session Replay Page for Environment-Specific Rendering

Bill Yang5356038b9b95
diff --git a/ops/backups/README.md b/ops/backups/README.md
new file mode 100644
index 000000000..ed90ec835
--- /dev/null
+++ b/ops/backups/README.md
@@ -0,0 +1,211 @@
+# Database backups
+
+These scripts create database-consistent backups for the three production
+hosts. They intentionally do not archive live Docker volumes.
+
+## Host map
+
+| Host | PostgreSQL | ClickHouse |
+| --- | --- | --- |
+| FrogStats | Native `pg_dump` to B2 | Native `BACKUP` directly to B2 |
+| Rybbit | Native `pg_dump` to B2 | Not backed up here; production uses the standalone host |
+| ClickHouse | None | One native local backup copied to Hetzner and B2 |
+
+PostgreSQL dumps include a custom-format database archive, cluster globals,
+and SHA-256 checksums. ClickHouse backs up only the `analytics` database, not
+the large and regenerable `system` log tables.
+
+## Prerequisites
+
+All hosts need Bash, Docker, `flock`, `curl`, and `rclone`. The `clickhouse`
+container must provide `clickhouse-client`; the PostgreSQL container already
+provides `pg_dump`, `pg_dumpall`, and `pg_restore`.
+
+Create a private B2 bucket, enable server-side encryption, and optionally
+enable a 7-14 day Object Lock. Give each host a separate application key
+restricted to that host's prefix. Configure lifecycle rules in B2 rather than
+giving the server credentials permission to delete old remote backups.
+
+For PostgreSQL and locally staged ClickHouse backups, configure rclone as root:
+
+```bash
+sudo rclone config
+sudo chmod 600 /root/.config/rclone/rclone.conf
+```
+
+The examples use an rclone `crypt` remote named `b2crypt` over a B2 remote.
+This provides client-side encryption in addition to B2 server-side encryption.
+The direct FrogStats ClickHouse backup uses B2 server-side encryption because
+it is written directly through ClickHouse's S3-compatible client. Its B2
+credentials live in a ClickHouse named collection rather than the backup SQL.
+
+## Install on each host
+
+From a checkout of this repository:
+
+```bash
+sudo install -d -m 0755 /opt/rybbit-backups/bin
+sudo install -m 0755 ops/backups/bin/backup-postgres.sh /opt/rybbit-backups/bin/
+sudo install -m 0755 ops/backups/bin/backup-clickhouse.sh /opt/rybbit-backups/bin/
+sudo install -m 0644 ops/backups/bin/common.sh /opt/rybbit-backups/bin/
+```
+
+Install the appropriate configuration without committing secrets:
+
+```bash
+# Choose frogstats.env.example, rybbit.env.example, or clickhouse.env.example.
+sudo install -m 0600 ops/backups/config/HOST.env.example /etc/rybbit-backup.env
+sudoedit /etc/rybbit-backup.env
+```
+
+The configuration file is sourced by Bash. Quote values containing `$`, spaces,
+single quotes, or other shell metacharacters.
+
+Validate configuration without touching either database or remote storage:
+
+```bash
+sudo env BACKUP_CONFIG=/etc/rybbit-backup.env DRY_RUN=true \
+ /opt/rybbit-backups/bin/backup-postgres.sh
+sudo env BACKUP_CONFIG=/etc/rybbit-backup.env DRY_RUN=true \
+ /opt/rybbit-backups/bin/backup-clickhouse.sh
+```
+
+Only run the script relevant to the host. The Rybbit application host does not
+have ClickHouse backup settings.
+
+## FrogStats ClickHouse named collection
+
+The direct S3 backup requires a named collection containing its B2 endpoint
+and restricted application key. Install the template outside the repository:
+
+```bash
+sudo install -d -m 0750 /etc/rybbit-backup
+sudo install -m 0640 -o root -g 101 \
+ ops/backups/clickhouse/s3-named-collection.xml.example \
+ /etc/rybbit-backup/clickhouse-s3.xml
+sudoedit /etc/rybbit-backup/clickhouse-s3.xml
+```
+
+Then mount it into the FrogStats ClickHouse container:
+
+```yaml
+services:
+ clickhouse:
+ volumes:
+ - /etc/rybbit-backup/clickhouse-s3.xml:/etc/clickhouse-server/config.d/backup-s3.xml:ro
+```
+
+UID/GID 101 is the default ClickHouse user in the official image; confirm it
+before installation if the image changes. Recreate ClickHouse and verify the
+collection without printing its contents:
+
+```bash
+docker exec clickhouse clickhouse-client \
+ --query "SELECT name FROM system.named_collections WHERE name = 'b2_backups'"
+```
+
+## Standalone ClickHouse staging disk
+
+The standalone host creates one native backup and uploads the same completed
+snapshot to both Hetzner and B2. Add these mounts to its ClickHouse service:
+
+```yaml
+services:
+ clickhouse:
+ volumes:
+ - /var/backups/rybbit/clickhouse:/var/lib/clickhouse/backups
+ - ./ops/backups/clickhouse/backup-disk.xml:/etc/clickhouse-server/config.d/backup-disk.xml:ro
+```
+
+Prepare the host directory before recreating the container. UID/GID 101 is the
+default ClickHouse user in the official image; confirm it with `docker exec`
+if the image changes:
+
+```bash
+sudo install -d -m 0750 -o 101 -g 101 /var/backups/rybbit/clickhouse
+```
+
+Restart ClickHouse after adding the disk configuration, then confirm it exists:
+
+```bash
+docker exec clickhouse clickhouse-client \
+ --query "SELECT name, path FROM system.disks WHERE name = 'backups'"
+```
+
+FrogStats uses direct S3 mode and does not need this staging disk.
+
+## Test manually, then enable timers
+
+Run the relevant service manually before enabling its timer:
+
+```bash
+sudo install -m 0644 ops/backups/systemd/*.service ops/backups/systemd/*.timer /etc/systemd/system/
+sudo systemctl daemon-reload
+
+sudo systemctl start rybbit-postgres-backup.service
+sudo journalctl -u rybbit-postgres-backup.service -n 100 --no-pager
+
+# FrogStats and the standalone ClickHouse host only:
+sudo systemctl start rybbit-clickhouse-backup.service
+sudo journalctl -u rybbit-clickhouse-backup.service -n 100 --no-pager
+```
+
+After verifying objects at every configured destination:
+
+```bash
+sudo systemctl enable --now rybbit-postgres-backup.timer
+# FrogStats and the standalone ClickHouse host only:
+sudo systemctl enable --now rybbit-clickhouse-backup.timer
+```
+
+The PostgreSQL timer runs every six hours. The ClickHouse timer runs daily with
+a randomized delay. Use systemd timer overrides if the two ClickHouse hosts
+should run in different windows.
+
+The standalone example prunes only the Hetzner destination after 14 days. B2
+retention should be implemented with bucket lifecycle rules, especially when
+Object Lock is enabled. Never add an Object-Locked B2 remote to
+`CLICKHOUSE_PRUNE_REMOTES`.
+
+## Restore drills
+
+Always restore into a disposable container or a differently named database
+first. Never test a restore against production.
+
+For PostgreSQL, download one backup directory through the crypt remote, verify
+it, restore globals, and then restore the custom archive:
+
+```bash
+cd /var/tmp/postgres-restore
+sha256sum --check SHA256SUMS
+docker exec -i postgres sh -ec \
+ 'exec psql -U "$POSTGRES_USER" -d postgres' < globals.sql
+docker exec postgres sh -ec \
+ 'exec createdb -U "$POSTGRES_USER" analytics_restore'
+docker exec -i postgres sh -ec \
+ 'exec pg_restore -U "$POSTGRES_USER" --dbname analytics_restore --clean --if-exists' \
+ < database.dump
+```
+
+Review the globals SQL before applying it when restoring into a shared
+PostgreSQL cluster.
+
+For a direct S3 ClickHouse backup, use the installed named collection and a
+different destination database name:
+
+```sql
+SET log_queries = 0;
+RESTORE DATABASE analytics AS analytics_restore
+FROM S3(b2_backups, 'BACKUP_DIRECTORY');
+```
+
+For a locally staged ClickHouse backup, download the complete backup directory
+to the configured host staging root and restore it from the backup disk:
+
+```sql
+RESTORE DATABASE analytics AS analytics_restore
+FROM Disk('backups', 'BACKUP_DIRECTORY');
+```
+
+Perform and record a restore drill at least monthly. A successful upload is not
+proof that the backup can be restored.
diff --git a/ops/backups/bin/backup-clickhouse.sh b/ops/backups/bin/backup-clickhouse.sh
new file mode 100755
index 000000000..e3cfe8b7e
--- /dev/null
+++ b/ops/backups/bin/backup-clickhouse.sh
@@ -0,0 +1,112 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=common.sh
+source "${SCRIPT_DIR}/common.sh"
+
+finish() {
+ local status=$?
+ trap - EXIT
+
+ if ((status == 0)); then
+ log "ClickHouse backup completed successfully"
+ notify_url "${CLICKHOUSE_SUCCESS_URL:-${BACKUP_SUCCESS_URL:-}}"
+ else
+ log "ClickHouse backup failed with exit status ${status}" >&2
+ notify_url "${CLICKHOUSE_FAILURE_URL:-${BACKUP_FAILURE_URL:-}}"
+ fi
+
+ exit "$status"
+}
+trap finish EXIT
+
+run_clickhouse_query() {
+ local query="$1"
+
+ # Feed SQL over stdin so S3 credentials do not appear in the host process
+ # list. Query logging is disabled for this client session by the query.
+ printf '%s\n' "$query" | docker exec -i "$CLICKHOUSE_CONTAINER" sh -ec \
+ 'exec clickhouse-client --user "$CLICKHOUSE_USER" --password "$CLICKHOUSE_PASSWORD" --multiquery'
+}
+
+load_backup_config
+
+CLICKHOUSE_CONTAINER="${CLICKHOUSE_CONTAINER:-clickhouse}"
+CLICKHOUSE_DATABASE="${CLICKHOUSE_DATABASE:-analytics}"
+CLICKHOUSE_BACKUP_MODE="${CLICKHOUSE_BACKUP_MODE:-s3}"
+CLICKHOUSE_LOCK_FILE="${CLICKHOUSE_LOCK_FILE:-/run/lock/rybbit-clickhouse-backup.lock}"
+BACKUP_HOST="${BACKUP_HOST:-$(hostname -s)}"
+DRY_RUN="${DRY_RUN:-false}"
+
+validate_simple_name BACKUP_HOST "$BACKUP_HOST"
+validate_simple_name CLICKHOUSE_CONTAINER "$CLICKHOUSE_CONTAINER"
+validate_identifier CLICKHOUSE_DATABASE "$CLICKHOUSE_DATABASE"
+[[ "$CLICKHOUSE_BACKUP_MODE" == "s3" || "$CLICKHOUSE_BACKUP_MODE" == "local" ]] || \
+ fatal "CLICKHOUSE_BACKUP_MODE must be 's3' or 'local'"
+
+BACKUP_NAME="${BACKUP_HOST}-clickhouse-${CLICKHOUSE_DATABASE}-$(date -u '+%Y%m%dT%H%M%SZ')"
+
+if [[ "$CLICKHOUSE_BACKUP_MODE" == "s3" ]]; then
+ require_value CLICKHOUSE_S3_NAMED_COLLECTION
+ validate_identifier CLICKHOUSE_S3_NAMED_COLLECTION "$CLICKHOUSE_S3_NAMED_COLLECTION"
+
+ if [[ "$DRY_RUN" == "true" ]]; then
+ log "DRY RUN: would create native ClickHouse backup ${BACKUP_NAME}"
+ log "DRY RUN: destination named collection ${CLICKHOUSE_S3_NAMED_COLLECTION}"
+ exit 0
+ fi
+
+ require_command docker
+ require_command flock
+ acquire_lock "$CLICKHOUSE_LOCK_FILE"
+
+ log "Creating native ClickHouse backup of ${CLICKHOUSE_DATABASE} directly in object storage"
+ QUERY="SET log_queries = 0; BACKUP DATABASE \`${CLICKHOUSE_DATABASE}\` TO S3(${CLICKHOUSE_S3_NAMED_COLLECTION}, $(sql_string "$BACKUP_NAME")) SETTINGS compression_method = 'zstd', compression_level = 1;"
+ run_clickhouse_query "$QUERY"
+else
+ CLICKHOUSE_BACKUP_DISK="${CLICKHOUSE_BACKUP_DISK:-backups}"
+ CLICKHOUSE_LOCAL_BACKUP_ROOT="${CLICKHOUSE_LOCAL_BACKUP_ROOT:-/var/backups/rybbit/clickhouse}"
+ CLICKHOUSE_KEEP_LOCAL_BACKUP="${CLICKHOUSE_KEEP_LOCAL_BACKUP:-false}"
+
+ require_value CLICKHOUSE_RCLONE_REMOTES
+ validate_simple_name CLICKHOUSE_BACKUP_DISK "$CLICKHOUSE_BACKUP_DISK"
+
+ if [[ "$DRY_RUN" == "true" ]]; then
+ log "DRY RUN: would create native ClickHouse backup ${BACKUP_NAME} on disk ${CLICKHOUSE_BACKUP_DISK}"
+ log "DRY RUN: would upload to ${CLICKHOUSE_RCLONE_REMOTES}"
+ exit 0
+ fi
+
+ require_command docker
+ require_command flock
+ require_command rclone
+ acquire_lock "$CLICKHOUSE_LOCK_FILE"
+
+ mkdir -p "$CLICKHOUSE_LOCAL_BACKUP_ROOT"
+ LOCAL_BACKUP_DIR="${CLICKHOUSE_LOCAL_BACKUP_ROOT%/}/${BACKUP_NAME}"
+ [[ ! -e "$LOCAL_BACKUP_DIR" ]] || fatal "Local backup path already exists: $LOCAL_BACKUP_DIR"
+
+ log "Creating native ClickHouse backup of ${CLICKHOUSE_DATABASE} on disk ${CLICKHOUSE_BACKUP_DISK}"
+ QUERY="SET log_queries = 0; BACKUP DATABASE \`${CLICKHOUSE_DATABASE}\` TO Disk($(sql_string "$CLICKHOUSE_BACKUP_DISK"), $(sql_string "$BACKUP_NAME")) SETTINGS compression_method = 'zstd', compression_level = 1;"
+ run_clickhouse_query "$QUERY"
+
+ [[ -d "$LOCAL_BACKUP_DIR" ]] || fatal "ClickHouse reported success but local backup was not found: $LOCAL_BACKUP_DIR"
+ [[ -n "$(find "$LOCAL_BACKUP_DIR" -mindepth 1 -print -quit)" ]] || fatal "ClickHouse local backup is empty"
+
+ upload_directory "$LOCAL_BACKUP_DIR" "$BACKUP_NAME" "$CLICKHOUSE_RCLONE_REMOTES"
+
+ if [[ -n "${CLICKHOUSE_PRUNE_REMOTES:-}" ]]; then
+ CLICKHOUSE_REMOTE_RETENTION_DAYS="${CLICKHOUSE_REMOTE_RETENTION_DAYS:-14}"
+ prune_remote_backup_directories \
+ "$CLICKHOUSE_PRUNE_REMOTES" \
+ "$CLICKHOUSE_REMOTE_RETENTION_DAYS" \
+ "${BACKUP_HOST}-clickhouse-${CLICKHOUSE_DATABASE}-"
+ fi
+
+ if [[ "$CLICKHOUSE_KEEP_LOCAL_BACKUP" != "true" ]]; then
+ log "Removing completed local staging backup"
+ safe_delete_staging_dir "$CLICKHOUSE_LOCAL_BACKUP_ROOT" "$LOCAL_BACKUP_DIR"
+ fi
+fi
diff --git a/ops/backups/bin/backup-postgres.sh b/ops/backups/bin/backup-postgres.sh
new file mode 100755
index 000000000..18b85481b
--- /dev/null
+++ b/ops/backups/bin/backup-postgres.sh
@@ -0,0 +1,87 @@
+#!/usr/bin/env bash
+
+set -Eeuo pipefail
+
+SCRIPT_DIR="$(cd "$(dirname "${BASH_SOURCE[0]}")" && pwd)"
+# shellcheck source=common.sh
+source "${SCRIPT_DIR}/common.sh"
+
+STAGING_DIR=""
+
+finish() {
+ local status=$?
+ trap - EXIT
+
+ if [[ -n "$STAGING_DIR" && -d "$STAGING_DIR" ]]; then
+ safe_delete_staging_dir "$POSTGRES_STAGING_ROOT" "$STAGING_DIR" || true
+ fi
+
+ if ((status == 0)); then
+ log "PostgreSQL backup completed successfully"
+ notify_url "${POSTGRES_SUCCESS_URL:-${BACKUP_SUCCESS_URL:-}}"
+ else
+ log "PostgreSQL backup failed with exit status ${status}" >&2
+ notify_url "${POSTGRES_FAILURE_URL:-${BACKUP_FAILURE_URL:-}}"
+ fi
+
+ exit "$status"
+}
+trap finish EXIT
+
+load_backup_config
+
+POSTGRES_CONTAINER="${POSTGRES_CONTAINER:-postgres}"
+POSTGRES_DATABASE="${POSTGRES_DATABASE:-analytics}"
+POSTGRES_STAGING_ROOT="${POSTGRES_STAGING_ROOT:-/var/backups/rybbit/postgres}"
+POSTGRES_LOCK_FILE="${POSTGRES_LOCK_FILE:-/run/lock/rybbit-postgres-backup.lock}"
+BACKUP_HOST="${BACKUP_HOST:-$(hostname -s)}"
+DRY_RUN="${DRY_RUN:-false}"
+
+require_value POSTGRES_RCLONE_REMOTES
+validate_simple_name BACKUP_HOST "$BACKUP_HOST"
+validate_simple_name POSTGRES_CONTAINER "$POSTGRES_CONTAINER"
+validate_identifier POSTGRES_DATABASE "$POSTGRES_DATABASE"
+
+BACKUP_NAME="${BACKUP_HOST}-postgres-${POSTGRES_DATABASE}-$(date -u '+%Y%m%dT%H%M%SZ')"
+
+if [[ "$DRY_RUN" == "true" ]]; then
+ log "DRY RUN: would create ${BACKUP_NAME} from container ${POSTGRES_CONTAINER}"
+ log "DRY RUN: would upload to ${POSTGRES_RCLONE_REMOTES}"
+ exit 0
+fi
+
+require_command docker
+require_command flock
+require_command rclone
+require_command sha256sum
+acquire_lock "$POSTGRES_LOCK_FILE"
+
+mkdir -p "$POSTGRES_STAGING_ROOT"
+STAGING_DIR="${POSTGRES_STAGING_ROOT%/}/${BACKUP_NAME}"
+mkdir "$STAGING_DIR"
+
+log "Creating consistent PostgreSQL dump for ${POSTGRES_DATABASE}"
+docker exec "$POSTGRES_CONTAINER" sh -ec \
+ 'export PGPASSWORD="$POSTGRES_PASSWORD"; exec pg_dump --username "$POSTGRES_USER" --dbname "$1" --format=custom --compress=zstd:3' \
+ sh "$POSTGRES_DATABASE" >"${STAGING_DIR}/database.dump.partial"
+mv "${STAGING_DIR}/database.dump.partial" "${STAGING_DIR}/database.dump"
+
+log "Dumping PostgreSQL roles and cluster globals"
+docker exec "$POSTGRES_CONTAINER" sh -ec \
+ 'export PGPASSWORD="$POSTGRES_PASSWORD"; exec pg_dumpall --username "$POSTGRES_USER" --globals-only' \
+ >"${STAGING_DIR}/globals.sql.partial"
+mv "${STAGING_DIR}/globals.sql.partial" "${STAGING_DIR}/globals.sql"
+
+[[ -s "${STAGING_DIR}/database.dump" ]] || fatal "PostgreSQL dump is empty"
+[[ -s "${STAGING_DIR}/globals.sql" ]] || fatal "PostgreSQL globals dump is empty"
+
+log "Validating PostgreSQL archive"
+docker exec -i "$POSTGRES_CONTAINER" pg_restore --list <"${STAGING_DIR}/database.dump" >/dev/null
+
+(
+ cd "$STAGING_DIR"
+ sha256sum database.dump globals.sql >SHA256SUMS
+)
+
+upload_directory "$STAGING_DIR" "$BACKUP_NAME" "$POSTGRES_RCLONE_REMOTES"
+
diff --git a/ops/backups/bin/common.sh b/ops/backups/bin/common.sh
new file mode 100644
index 000000000..61540b371
--- /dev/null
+++ b/ops/backups/bin/common.sh
@@ -0,0 +1,152 @@
+#!/usr/bin/env bash
+
+# Shared functions for the database backup scripts. This file is sourced by
+# the entrypoints and is not intended to be executed directly.
+
+set -Eeuo pipefail
+umask 077
+
+log() {
+ printf '[%s] %s\n' "$(date -u '+%Y-%m-%dT%H:%M:%SZ')" "$*"
+}
+
+fatal() {
+ log "ERROR: $*" >&2
+ exit 1
+}
+
+require_command() {
+ command -v "$1" >/dev/null 2>&1 || fatal "Required command not found: $1"
+}
+
+require_value() {
+ local name="$1"
+ [[ -n "${!name:-}" ]] || fatal "Required configuration value is empty: ${name}"
+}
+
+load_backup_config() {
+ local config_path="${BACKUP_CONFIG:-/etc/rybbit-backup.env}"
+ [[ -r "$config_path" ]] || fatal "Cannot read backup configuration: $config_path"
+
+ # shellcheck disable=SC1090
+ source "$config_path"
+}
+
+validate_simple_name() {
+ local label="$1"
+ local value="$2"
+ [[ "$value" =~ ^[A-Za-z0-9._-]+$ ]] || fatal "$label contains unsupported characters: $value"
+}
+
+validate_identifier() {
+ local label="$1"
+ local value="$2"
+ [[ "$value" =~ ^[A-Za-z_][A-Za-z0-9_]*$ ]] || fatal "$label is not a safe SQL identifier: $value"
+}
+
+sql_string() {
+ local value=${1//\'/\'\'}
+ printf "'%s'" "$value"
+}
+
+acquire_lock() {
+ local lock_path="$1"
+ mkdir -p "$(dirname "$lock_path")"
+ exec {BACKUP_LOCK_FD}>"$lock_path"
+ flock -n "$BACKUP_LOCK_FD" || fatal "Another backup is already running: $lock_path"
+}
+
+notify_url() {
+ local url="${1:-}"
+ [[ -n "$url" ]] || return 0
+
+ if ! curl --fail --silent --show-error --retry 2 --max-time 15 "$url" >/dev/null; then
+ log "WARNING: Could not send backup notification"
+ fi
+}
+
+upload_directory() {
+ local source_dir="$1"
+ local backup_name="$2"
+ local remotes_csv="$3"
+ local -a remotes
+ local remote_base
+ local remote_path
+
+ IFS=',' read -r -a remotes <<<"$remotes_csv"
+ ((${#remotes[@]} > 0)) || fatal "No rclone destinations configured"
+
+ for remote_base in "${remotes[@]}"; do
+ remote_base="${remote_base#${remote_base%%[![:space:]]*}}"
+ remote_base="${remote_base%${remote_base##*[![:space:]]}}"
+ [[ -n "$remote_base" ]] || fatal "Empty destination in remote list"
+
+ remote_path="${remote_base%/}/${backup_name}"
+ log "Uploading ${backup_name} to ${remote_path}"
+ rclone copy "${source_dir%/}/" "${remote_path}/" \
+ --immutable \
+ --transfers "${RCLONE_TRANSFERS:-4}" \
+ --checkers "${RCLONE_CHECKERS:-8}" \
+ --retries "${RCLONE_RETRIES:-5}" \
+ --low-level-retries "${RCLONE_LOW_LEVEL_RETRIES:-10}" \
+ --stats 30s
+
+ log "Verifying ${remote_path}"
+ rclone check "${source_dir%/}/" "${remote_path}/" --one-way
+ done
+}
+
+prune_remote_backup_directories() {
+ local remotes_csv="$1"
+ local retention_days="$2"
+ local backup_prefix="$3"
+ local -a remotes
+ local remote_base
+ local listing
+ local directory_name
+ local timestamp
+ local backup_epoch
+ local cutoff_epoch
+
+ [[ "$retention_days" =~ ^[1-9][0-9]*$ ]] || fatal "Retention days must be a positive integer"
+ validate_simple_name "Backup prefix" "$backup_prefix"
+ cutoff_epoch="$(date -u -d "${retention_days} days ago" '+%s')"
+ IFS=',' read -r -a remotes <<<"$remotes_csv"
+
+ for remote_base in "${remotes[@]}"; do
+ remote_base="${remote_base#${remote_base%%[![:space:]]*}}"
+ remote_base="${remote_base%${remote_base##*[![:space:]]}}"
+ [[ -n "$remote_base" ]] || fatal "Empty destination in prune remote list"
+
+ log "Looking for complete backups older than ${retention_days} days in ${remote_base}"
+ listing="$(rclone lsf "${remote_base%/}/" --dirs-only --max-depth 1)"
+
+ while IFS= read -r directory_name; do
+ directory_name="${directory_name%/}"
+ [[ "$directory_name" == "${backup_prefix}"* ]] || continue
+
+ timestamp="${directory_name#"$backup_prefix"}"
+ [[ "$timestamp" =~ ^[0-9]{8}T[0-9]{6}Z$ ]] || continue
+ backup_epoch="$(date -u -d "${timestamp:0:4}-${timestamp:4:2}-${timestamp:6:2} ${timestamp:9:2}:${timestamp:11:2}:${timestamp:13:2} UTC" '+%s')" || \
+ fatal "Could not parse backup timestamp: $directory_name"
+
+ if ((backup_epoch < cutoff_epoch)); then
+ log "Purging expired complete backup ${remote_base%/}/${directory_name}"
+ rclone purge "${remote_base%/}/${directory_name}"
+ fi
+ done <<<"$listing"
+ done
+}
+
+safe_delete_staging_dir() {
+ local staging_root="$1"
+ local target="$2"
+
+ [[ -n "$staging_root" && -n "$target" ]] || fatal "Refusing to delete an unresolved staging path"
+ [[ "$target" == "${staging_root%/}/"* ]] || fatal "Refusing to delete path outside staging root: $target"
+ [[ "$target" != "$staging_root" && "$target" != "${staging_root%/}/" ]] || fatal "Refusing to delete staging root"
+
+ if [[ -d "$target" ]]; then
+ find "$target" -depth -delete
+ fi
+}
diff --git a/ops/backups/clickhouse/backup-disk.xml b/ops/backups/clickhouse/backup-disk.xml
new file mode 100644
index 000000000..1375de50f
--- /dev/null
+++ b/ops/backups/clickhouse/backup-disk.xml
@@ -0,0 +1,18 @@
+<clickhouse>
+ <!-- Required only for CLICKHOUSE_BACKUP_MODE=local. Bind-mount the host
+ staging directory at this path inside the ClickHouse container. -->
+ <storage_configuration>
+ <disks>
+ <backups>
+ <type>local</type>
+ <path>/var/lib/clickhouse/backups/</path>
+ </backups>
+ </disks>
+ </storage_configuration>
+ <backups>
+ <allowed_disk>backups</allowed_disk>
+ <allow_concurrent_backups>false</allow_concurrent_backups>
+ <allow_concurrent_restores>false</allow_concurrent_restores>
+ </backups>
+</clickhouse>
+
diff --git a/ops/backups/clickhouse/s3-named-collection.xml.example b/ops/backups/clickhouse/s3-named-collection.xml.example
new file mode 100644
index 000000000..5d2fd9940
--- /dev/null
+++ b/ops/backups/clickhouse/s3-named-collection.xml.example
@@ -0,0 +1,14 @@
+<clickhouse>
+ <!-- Install this file outside the repository, replace all placeholders,
+ make it readable only by root and the ClickHouse group, and mount it
+ read-only into /etc/clickhouse-server/config.d/. -->
+ <allow_named_collection_override_by_default>false</allow_named_collection_override_by_default>
+ <named_collections>
+ <b2_backups>
+ <url overridable="false">https://s3.YOUR-B2-REGION.backblazeb2.com/YOUR-BUCKET/frogstats/clickhouse/</url>
+ <access_key_id overridable="false">REPLACE_WITH_B2_KEY_ID</access_key_id>
+ <secret_access_key overridable="false">REPLACE_WITH_B2_APPLICATION_KEY</secret_access_key>
+ </b2_backups>
+ </named_collections>
+</clickhouse>
+
diff --git a/ops/backups/config/clickhouse.env.example b/ops/backups/config/clickhouse.env.example
new file mode 100644
index 000000000..48c378f99
--- /dev/null
+++ b/ops/backups/config/clickhouse.env.example
@@ -0,0 +1,23 @@
+# Copy to /etc/rybbit-backup.env on the standalone ClickHouse host and chmod 600.
+
+BACKUP_HOST=clickhouse
+
+CLICKHOUSE_CONTAINER=clickhouse
+CLICKHOUSE_DATABASE=analytics
+CLICKHOUSE_BACKUP_MODE=local
+CLICKHOUSE_BACKUP_DISK=backups
+CLICKHOUSE_LOCAL_BACKUP_ROOT=/var/backups/rybbit/clickhouse
+CLICKHOUSE_KEEP_LOCAL_BACKUP=false
+
+# The same completed native snapshot is copied to both providers. Configure
+# these names in root's /root/.config/rclone/rclone.conf. Use an rclone crypt
+# wrapper for B2 if client-side encryption is required.
+CLICKHOUSE_RCLONE_REMOTES=hetzner:clickhouse-backups,b2crypt:rybbit-db-backups/clickhouse/analytics
+
+# Hetzner does not have B2 lifecycle rules, so prune it from the host. Do not
+# include the Object-Locked B2 remote here; let B2 lifecycle rules expire it.
+CLICKHOUSE_PRUNE_REMOTES=hetzner:clickhouse-backups
+CLICKHOUSE_REMOTE_RETENTION_DAYS=14
+
+# CLICKHOUSE_SUCCESS_URL=
+# CLICKHOUSE_FAILURE_URL=
diff --git a/ops/backups/config/frogstats.env.example b/ops/backups/config/frogstats.env.example
new file mode 100644
index 000000000..f90d25b92
--- /dev/null
+++ b/ops/backups/config/frogstats.env.example
@@ -0,0 +1,21 @@
+# Copy to /etc/rybbit-backup.env on the FrogStats host and chmod 600.
+
+BACKUP_HOST=frogstats
+
+# PostgreSQL: rclone should use a crypt remote backed by the private B2 bucket.
+POSTGRES_CONTAINER=postgres
+POSTGRES_DATABASE=analytics
+POSTGRES_STAGING_ROOT=/var/backups/rybbit/postgres
+POSTGRES_RCLONE_REMOTES=b2crypt:rybbit-db-backups/frogstats/postgres
+
+# ClickHouse: direct native backup avoids staging ~58 GiB on this host.
+CLICKHOUSE_CONTAINER=clickhouse
+CLICKHOUSE_DATABASE=analytics
+CLICKHOUSE_BACKUP_MODE=s3
+CLICKHOUSE_S3_NAMED_COLLECTION=b2_backups
+
+# Optional monitoring endpoints, such as Healthchecks.io ping URLs.
+# POSTGRES_SUCCESS_URL=
+# POSTGRES_FAILURE_URL=
+# CLICKHOUSE_SUCCESS_URL=
+# CLICKHOUSE_FAILURE_URL=
diff --git a/ops/backups/config/rybbit.env.example b/ops/backups/config/rybbit.env.example
new file mode 100644
index 000000000..d92435239
--- /dev/null
+++ b/ops/backups/config/rybbit.env.example
@@ -0,0 +1,15 @@
+# Copy to /etc/rybbit-backup.env on the Rybbit application host and chmod 600.
+
+BACKUP_HOST=rybbit
+
+POSTGRES_CONTAINER=postgres
+POSTGRES_DATABASE=analytics
+POSTGRES_STAGING_ROOT=/var/backups/rybbit/postgres
+POSTGRES_RCLONE_REMOTES=b2crypt:rybbit-db-backups/rybbit/postgres
+
+# No ClickHouse backup is configured here. The backend uses the standalone
+# ClickHouse host rather than this host's effectively empty local volume.
+
+# POSTGRES_SUCCESS_URL=
+# POSTGRES_FAILURE_URL=
+
diff --git a/ops/backups/systemd/rybbit-clickhouse-backup.service b/ops/backups/systemd/rybbit-clickhouse-backup.service
new file mode 100644
index 000000000..147755e55
--- /dev/null
+++ b/ops/backups/systemd/rybbit-clickhouse-backup.service
@@ -0,0 +1,15 @@
+[Unit]
+Description=Rybbit native ClickHouse backup
+After=docker.service network-online.target
+Wants=network-online.target
+Requires=docker.service
+
+[Service]
+Type=oneshot
+Environment=BACKUP_CONFIG=/etc/rybbit-backup.env
+ExecStart=/opt/rybbit-backups/bin/backup-clickhouse.sh
+UMask=0077
+Nice=10
+IOSchedulingClass=best-effort
+IOSchedulingPriority=7
+
diff --git a/ops/backups/systemd/rybbit-clickhouse-backup.timer b/ops/backups/systemd/rybbit-clickhouse-backup.timer
new file mode 100644
index 000000000..ad19890c3
--- /dev/null
+++ b/ops/backups/systemd/rybbit-clickhouse-backup.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Run Rybbit ClickHouse backup daily
+
+[Timer]
+OnCalendar=*-*-* 02:30:00
+Persistent=true
+RandomizedDelaySec=20m
+Unit=rybbit-clickhouse-backup.service
+
+[Install]
+WantedBy=timers.target
+
diff --git a/ops/backups/systemd/rybbit-postgres-backup.service b/ops/backups/systemd/rybbit-postgres-backup.service
new file mode 100644
index 000000000..2153ec94a
--- /dev/null
+++ b/ops/backups/systemd/rybbit-postgres-backup.service
@@ -0,0 +1,15 @@
+[Unit]
+Description=Rybbit native PostgreSQL backup
+After=docker.service network-online.target
+Wants=network-online.target
+Requires=docker.service
+
+[Service]
+Type=oneshot
+Environment=BACKUP_CONFIG=/etc/rybbit-backup.env
+ExecStart=/opt/rybbit-backups/bin/backup-postgres.sh
+UMask=0077
+Nice=10
+IOSchedulingClass=best-effort
+IOSchedulingPriority=7
+
diff --git a/ops/backups/systemd/rybbit-postgres-backup.timer b/ops/backups/systemd/rybbit-postgres-backup.timer
new file mode 100644
index 000000000..a161fc7d2
--- /dev/null
+++ b/ops/backups/systemd/rybbit-postgres-backup.timer
@@ -0,0 +1,12 @@
+[Unit]
+Description=Run Rybbit PostgreSQL backup every six hours
+
+[Timer]
+OnCalendar=*-*-* 00,06,12,18:15:00
+Persistent=true
+RandomizedDelaySec=10m
+Unit=rybbit-postgres-backup.service
+
+[Install]
+WantedBy=timers.target
+