Skip to content

PostgreSQL backup tool with Point-In-Time-Recovery and restore verification

7.0 KB
Raw Blame

1. Reproduce the failure

  • 1.1 Add e2e/docker-storage/no-unix-sockets.json, a seccomp profile that allows every system call except socket with address family AF_UNIX (value 1), which returns errno 97. Add a no-unix-sockets case to e2e/docker-storage/run.sh that starts the candidate with --security-opt seccomp=<profile> on a named volume and calls assert_running_container. Add the case to the all list. Verify that docker build --tag databasus-storage-test:local . and CASE=no-unix-sockets DATABASUS_IMAGE=databasus-storage-test:local bash e2e/docker-storage/run.sh fail, and that the container log contains could not create any Unix-domain sockets.

2. Bootstrap without a socket

  • 2.1 In docker/start.sh, drop /databasus-data/pgsocket from create_required_data_directories, normalize_data_permissions and verify_service_directories. Verify that grep -n pgsocket docker/start.sh lists only the lines that tasks 2.2 to 2.4 replace.
  • 2.2 Replace write_bootstrap_postgresql_authentication and activate_runtime_postgresql_authentication with one function. It writes the final pg_hba.conf, with SCRAM on 127.0.0.1/32 and ::1/128, reject for replication and no local rules, and an empty pg_ident.conf, both with mode 0600, before any PostgreSQL process starts. Delete the pg_ctl reload call. Verify that grep -nE 'peer|pg_ctl|databasus_bootstrap' docker/start.sh prints nothing.
  • 2.3 Add the single-user open step from design decision 2. It runs postgres --single with empty input under databasus, captures its output and prints that output only on failure. Move the pg_resetwal recovery from start_postgresql_with_recovery around this step, and remove a stale postmaster.pid before pg_resetwal runs. Verify after task 2.5 through the recovery case of task 3.4.
  • 2.4 Add the single-user password step from design decisions 2 and 3. It pipes ALTER ROLE postgres WITH PASSWORD '<generated>' from the printf builtin into postgres --single with -c exit_on_error=on, -c log_min_error_statement=panic and -c password_encryption=scram-sha-256, and stops startup on a non-zero exit. Verify that grep -nE 'env PGPASSWORD|-v internal_postgres_password' docker/start.sh prints nothing once task 2.5 has also converted the remaining psql calls.
  • 2.5 Start the regular server with -c unix_socket_directories='' and -c max_wal_senders=0 and poll pg_isready -h localhost -p 5437. Without a reset, a failed start stops the container. Create the metadata database through psql -h localhost with the existing \gexec statement. Pass PGPASSWORD as a shell assignment prefix in front of gosu in every psql call, including reject_legacy_wal_configuration. Delete the socket-login check from verify_postgresql_runtime_authentication and keep the loopback check. Verify that docker build --tag databasus-storage-test:local . succeeds and that CASE=no-unix-sockets DATABASUS_IMAGE=databasus-storage-test:local bash e2e/docker-storage/run.sh passes. Then verify that grep -n pgsocket docker/start.sh prints nothing.

3. Filesystem suite

  • 3.1 In assert_postgresql_runtime_authentication, delete the socket-login probe and the local all all reject check. Add a check that /proc/net/unix inside the container lists no .s.PGSQL entry. Add a check that neither the docker logs output nor any /proc/*/cmdline inside the container contains the password from /dev/shm/databasus-database-dsn. Add a probe that runs psql "host=localhost port=5437 user=postgres dbname=postgres replication=database" with the generated password from /dev/shm/databasus-database-dsn and asserts that PostgreSQL rejects it with number of requested standby connections exceeds. Keep the loopback probe without a password and the 0600 mode checks. Verify that CASE=bind-root DATABASUS_IMAGE=databasus-storage-test:local bash e2e/docker-storage/run.sh and the no-unix-sockets case pass.
  • 3.2 Stop creating pgsocket in prepare_bind_tree and in the NFS share setup. Delete both --tmpfs /databasus-data/pgsocket mounts from the NFS case. Verify that grep -n pgsocket e2e/docker-storage/run.sh prints nothing, and that the automatic-ids, split-mounts and nfs-root-squash cases pass one at a time through CASE=<case> DATABASUS_IMAGE=databasus-storage-test:local bash e2e/docker-storage/run.sh.
  • 3.3 Add v3.60.0 to run_upgrades. Add a check to run_upgrade that records whether /databasus-data/pgsocket exists before the update and, when it did, asserts that it still exists afterwards in the healthy container. Only v3.56.0 and later create that directory, so for those releases the check requires it to exist before the update. Verify that CASE=upgrades DATABASUS_IMAGE=databasus-storage-test:local bash e2e/docker-storage/run.sh passes.
  • 3.4 Add a postgresql-recovery case with five parts, and add it to the all list. In the first part, read the password from /dev/shm/databasus-database-dsn, write a marker row, stop the container with docker kill --signal KILL, start it again and assert that it is healthy, the marker exists and the log has no pg_resetwal line. Assert also that a loopback psql -h localhost login with the old password is refused and one with the newly published password succeeds. In the second part, stop the container cleanly, remove the segments under pgdata/pg_wal, start it again and assert that the log reports a write-ahead log reset and that the container is healthy. In the third part, write 16 into pgdata/PG_VERSION, which makes both the open step and pg_resetwal fail, start the container and assert that it exits with an error, the application never starts and the log reports the failed recovery. Run the remaining parts on fresh volumes. In the fourth part, append listen_addresses = '192.0.2.1' to postgresql.conf, start the container and assert that it exits with an error, that the log reports the password step without its failure, Starting PostgreSQL... and ERROR: PostgreSQL did not start., and that it has no pg_resetwal line. In the fifth part, append default_transaction_read_only = on to postgresql.conf, start the container and assert four things: it exits with a non-zero code, the log reports ERROR: PostgreSQL rejected the internal password change. and has no Starting PostgreSQL... line, the application never starts, and docker logs contains neither ALTER ROLE postgres WITH PASSWORD, nor PASSWORD ', nor any 64-character hexadecimal string. Verify that CASE=postgresql-recovery DATABASUS_IMAGE=databasus-storage-test:local bash e2e/docker-storage/run.sh passes.

4. Specs and review

  • 4.1 Verify that openspec validate remove-embedded-postgresql-socket --strict passes.
  • 4.2 Run the mandatory full implementation review from AGENTS.md and resolve its blocking findings. Then rerun the no-unix-sockets, bind-root, upgrades and postgresql-recovery cases through CASE=<case> DATABASUS_IMAGE=databasus-storage-test:local bash e2e/docker-storage/run.sh after rebuilding the image, and verify that they pass.