Skip to content

PostgreSQL backup tool with Point-In-Time-Recovery and restore verification

FIX (verification): Report the last connection error when a restore container never becomes ready and archive run-verification-on-podman (issue #833)

Rostislav Duginf67b48da41a6
diff --git a/agent/verification/internal/features/container/manager.go b/agent/verification/internal/features/container/manager.go
index 04d1be52..7c7eeb1a 100644
--- a/agent/verification/internal/features/container/manager.go
+++ b/agent/verification/internal/features/container/manager.go
@@ -236,30 +236,42 @@ func waitForReady(ctx context.Context, conn dbconn.Conn) error {
ticker := time.NewTicker(readyPollInterval)
defer ticker.Stop()
+ // The deadline alone hides why Postgres stayed unreachable: a refused port,
+ // a connection nobody answers and a rejected login need different fixes,
+ // and only the last ping error tells them apart.
+ var lastPingErr error
+
for {
- if pingOnce(readyCtx, conn) {
+ err := pingOnce(readyCtx, conn)
+ if err == nil {
return nil
}
+ // A ping the overall deadline cut short says nothing about Postgres, so
+ // it never replaces the error of a ping that finished on its own.
+ if readyCtx.Err() == nil || lastPingErr == nil {
+ lastPingErr = err
+ }
+
select {
case <-readyCtx.Done():
- return readyCtx.Err()
+ return fmt.Errorf("%w; last ping error: %w", readyCtx.Err(), lastPingErr)
case <-ticker.C:
}
}
}
-func pingOnce(ctx context.Context, conn dbconn.Conn) bool {
+func pingOnce(ctx context.Context, conn dbconn.Conn) error {
pingCtx, cancel := context.WithTimeout(ctx, readyPollInterval)
defer cancel()
pgConn, err := pgx.Connect(pingCtx, conn.DSN())
if err != nil {
- return false
+ return err
}
defer func() { _ = pgConn.Close(pingCtx) }()
- return pgConn.Ping(pingCtx) == nil
+ return pgConn.Ping(pingCtx)
}
func imageForJob(req SpawnRequest, engineImageRepo string) string {
diff --git a/agent/verification/internal/features/container/manager_test.go b/agent/verification/internal/features/container/manager_test.go
new file mode 100644
index 00000000..dd7e4834
--- /dev/null
+++ b/agent/verification/internal/features/container/manager_test.go
@@ -0,0 +1,36 @@
+package container
+
+import (
+ "context"
+ "net"
+ "testing"
+ "time"
+
+ "github.com/stretchr/testify/assert"
+ "github.com/stretchr/testify/require"
+
+ "databasus-verification-agent/internal/features/dbconn"
+)
+
+func Test_WaitForReady_WhenPortRefusesConnections_ReturnsDeadlineWithLastPingError(t *testing.T) {
+ listener, err := net.Listen("tcp", "127.0.0.1:0")
+ require.NoError(t, err)
+
+ closedPort := listener.Addr().(*net.TCPAddr).Port
+ require.NoError(t, listener.Close())
+
+ const password = "secret-restore-pw-123"
+
+ ctx, cancel := context.WithTimeout(t.Context(), 2*time.Second)
+ defer cancel()
+
+ err = waitForReady(ctx, dbconn.Conn{
+ Host: "127.0.0.1", Port: closedPort, User: restoreUser, Password: password, Database: restoreDB,
+ })
+
+ require.ErrorIs(t, err, context.DeadlineExceeded)
+ assert.Contains(t, err.Error(), "last ping error")
+ assert.Contains(t, err.Error(), "connection refused",
+ "the error must say why Postgres stayed unreachable, not only that time ran out")
+ assert.NotContains(t, err.Error(), password, "the report must not leak the database password")
+}
diff --git a/openspec/changes/run-verification-on-podman/.openspec.yaml b/openspec/changes/archive/2026-10-22-run-verification-on-podman/.openspec.yaml
similarity index 100%
rename from openspec/changes/run-verification-on-podman/.openspec.yaml
rename to openspec/changes/archive/2026-10-22-run-verification-on-podman/.openspec.yaml
diff --git a/openspec/changes/run-verification-on-podman/design.md b/openspec/changes/archive/2026-10-22-run-verification-on-podman/design.md
similarity index 74%
rename from openspec/changes/run-verification-on-podman/design.md
rename to openspec/changes/archive/2026-10-22-run-verification-on-podman/design.md
index 43039978..dd5d4b3d 100644
--- a/openspec/changes/run-verification-on-podman/design.md
+++ b/openspec/changes/archive/2026-10-22-run-verification-on-podman/design.md
@@ -59,17 +59,29 @@ The compose file takes the engine socket for `e2e-agent-runner` from `E2E_ENGINE
A new CI job, `e2e-verification-agent-podman`, runs on `ubuntu-24.04`, starts the rootful Podman socket with `sudo systemctl start podman.socket`, prints `podman version` and the OCI runtime for the log, and calls `just e2e-podman`. `determine-version` waits for it as it waits for the Docker job. The existing Docker job also runs `test-container-privileges.sh` through `run-all.sh`.
+Until the cause of the first failed run is fixed, the job also runs a temporary step, `Diagnose Podman port publishing`, before `just e2e-podman`. It runs `e2e/diagnose-podman-ports.sh` as root with `continue-on-error`, so it never fails the job. The script starts plain Postgres containers on a user-defined Podman network, one published on `127.0.0.1` as the agent publishes it and one on all interfaces. It checks them with `pg_isready` from the host network namespace through the loopback port, the host IP and the container IP. The first diagnostic run showed every one of these paths working while the agent got `connection refused`, so the script also creates containers through the Docker-compatible API with the agent's own request: the verification image, the tuned command, the hardening, the limits and `HostPort "0"`. Each further variant turns off one of the image, the capability drop and no-new-privileges, and the script prints each container's processes, capabilities and log. When the agent's path fails for plain containers, it prints the firewall rules, stops Docker, removes Docker's rules and repeats the checks. The diagnosis traced the hang to AppArmor denying signals between the container's processes under no-new-privileges. The change `start-restore-containers-unprivileged` fixes that and removes the step and the script.
+
Rejected alternatives:
- **The full suite on Podman.** The upgrade, start-stop and purge tests do not depend on the engine's reporting, and seven PostgreSQL majors would double the job time without exercising a different engine path. Major 18 matches the issue.
- **A matrix over the existing job.** The Podman run needs a different socket, a setup step and a different script list, so a matrix would need conditionals in every step.
- **Running the whole compose stack on Podman.** It tests `podman compose`, which the product does not use, and only the agent's engine calls matter here.
+### Report why a restore container never became ready
+
+On the first CI run on rootful Podman 4.9.3 (`ubuntu-24.04`), every restore container started but stayed unreachable through its published port, and the verification report said only `container never became ready: context deadline exceeded`. `waitForReady` polled once a second and returned only the deadline (`manager.go:232-249`), dropping every ping error. It now returns the deadline together with the error of the last ping that finished before the deadline. A ping the deadline itself cut short does not replace it, because its error only repeats the deadline. The report then tells a refused port from an unanswered connection and from a rejected login, on a user's host as well as in CI. pgx connection errors carry the user, database, host and port, not the password.
+
+Rejected alternatives:
+
+- **Log every failed ping.** Sixty lines per failed job, and the report the user sees would still say only "deadline exceeded".
+- **Keep the error of the very last ping.** The unit test showed that the last ping usually starts just before the deadline and fails with a dial timeout caused by the deadline, which hides the real cause.
+
## Risks / Trade-offs
- [The `docker` CLI in the runner may read Podman's compat API slightly differently, for example in `network ls` label filters] → The leak check fails loudly rather than passing silently. If a filter is unsupported, the script switches to an output the CLI and Podman agree on, still from the runner.
- [Podman may resolve the agent's short image name `databasus/verification-postgres:18` differently from Docker] → Podman's compat API resolves short names to Docker Hub. If the CI run shows otherwise, that is a real user-facing bug and is fixed in the agent, not worked around in the test.
-- [The `gosu` hang the reporter saw with `crun` may reproduce in CI] → The job prints the runtime, so the failure is attributable. It is then a separate issue, and this change does not merge with a red Podman job.
+- [The `gosu` hang the reporter saw with `crun` may reproduce in CI] → It did: under no-new-privileges, AppArmor on `ubuntu-24.04` denies signals between the container's processes. `start-restore-containers-unprivileged` fixes it, and this change does not merge with a red Podman job.
+- [The job does not run on `develop`, so its first run came after the merge to `main`, and it failed: Postgres in the restore containers never answered, because under no-new-privileges AppArmor denied signals between the container's processes] → `start-restore-containers-unprivileged` fixes the cause and tracks the green run on rootful Podman 4.9.3 in its tasks 3.1 and 4.2. That is the version `ubuntu-24.04` ships, which users on that LTS release also run. The runner is not switched to a newer Podman to make the job green, because that would hide a failure those users hit.
- [The 30-second delay lengthens each end-to-end run by about 30 seconds per engine] → Accepted; it buys a deterministic window instead of a race.
- [Without the read-back, an engine that silently ignores the requested hardening goes unnoticed at runtime] → No engine the agent supports does that, and the end-to-end test proves the applied state on both supported engines in CI.
diff --git a/openspec/changes/run-verification-on-podman/proposal.md b/openspec/changes/archive/2026-10-22-run-verification-on-podman/proposal.md
similarity index 100%
rename from openspec/changes/run-verification-on-podman/proposal.md
rename to openspec/changes/archive/2026-10-22-run-verification-on-podman/proposal.md
diff --git a/openspec/changes/run-verification-on-podman/specs/verification-container-resources/spec.md b/openspec/changes/archive/2026-10-22-run-verification-on-podman/specs/verification-container-resources/spec.md
similarity index 77%
rename from openspec/changes/run-verification-on-podman/specs/verification-container-resources/spec.md
rename to openspec/changes/archive/2026-10-22-run-verification-on-podman/specs/verification-container-resources/spec.md
index c11a5f8c..c84e19d4 100644
--- a/openspec/changes/run-verification-on-podman/specs/verification-container-resources/spec.md
+++ b/openspec/changes/archive/2026-10-22-run-verification-on-podman/specs/verification-container-resources/spec.md
@@ -37,3 +37,13 @@ The verification agent SHALL verify backups when its Docker endpoint is a rootfu
- **AND** the agent claims a verification job for a backup that cannot be restored
- **THEN** the agent reports the verification as failed with the restore error
- **AND** no container or network created for the job remains on the engine
+
+### Requirement: An unreachable restore container is reported with its cause
+
+When the agent cannot connect to the database in a restore container before the readiness timeout, the verification SHALL fail, and its failure message SHALL include the error of the agent's last connection attempt that ended before the timeout, such as a refused connection, a connection that got no answer or a rejected login. The failure message SHALL NOT contain the database password.
+
+#### Scenario: Published port refuses connections
+
+- **WHEN** the database in a restore container stays unreachable because its published port refuses every connection
+- **THEN** the agent reports the verification as failed
+- **AND** the failure message says that the readiness timeout passed and that the last connection attempt was refused
diff --git a/openspec/changes/run-verification-on-podman/tasks.md b/openspec/changes/archive/2026-10-22-run-verification-on-podman/tasks.md
similarity index 67%
rename from openspec/changes/run-verification-on-podman/tasks.md
rename to openspec/changes/archive/2026-10-22-run-verification-on-podman/tasks.md
index fe4f1a53..0318be80 100644
--- a/openspec/changes/run-verification-on-podman/tasks.md
+++ b/openspec/changes/archive/2026-10-22-run-verification-on-podman/tasks.md
@@ -21,6 +21,15 @@
- [x] 5.1 Run `just lint` in `agent/verification` and fix every finding until it passes.
- [x] 5.2 Search the repository for mentions of the deleted read-back and of the e2e entry points (`run-all.sh`, `just e2e`, `docker.sock` in the agent's e2e) and bring every affected comment and document in line with the change. Verify that `grep -rn "hardening regression" .` outside `openspec/` finds nothing.
+## 6. Make the restore container reachable on rootful Podman in CI
+
+The first run of `e2e-verification-agent-podman` failed all three tests: the restore container started, but the agent could not reach Postgres through its published port for 60 seconds. The tasks below diagnose it; the fix and the green CI run belong to `start-restore-containers-unprivileged`.
+
+- [x] 6.1 In `agent/verification/internal/features/container/manager.go`, make `waitForReady` return the readiness deadline together with the error of the last ping that finished before the deadline, so a refused port, an unanswered connection and a rejected login read differently in the verification report. Add `Test_WaitForReady_WhenPortRefusesConnections_ReturnsDeadlineWithLastPingError` to `manager_test.go`. Verify with `just test` in `agent/verification`.
+- [x] 6.2 Temporary: add `agent/verification/e2e/diagnose-podman-ports.sh` and the step `Diagnose Podman port publishing` with `continue-on-error: true` before `just e2e-podman` in `e2e-verification-agent-podman`. The script repeats the agent's connection path with plain Postgres containers and varies the bind address and the target address. It then creates containers through the Docker-compatible API with the agent's own request, turning off one of the image, the capability drop and no-new-privileges per variant, and prints each container's processes, capabilities and log. When the agent's path fails for plain containers, it repeats the checks with Docker's firewall rules removed. Verify with `bash -n agent/verification/e2e/diagnose-podman-ports.sh` and `actionlint .github/workflows/ci-release.yml`, which must report no new findings.
+
+The cause turned out to be AppArmor denying signals between the container's processes under no-new-privileges, not port publishing. The change `start-restore-containers-unprivileged` fixes it and removes the temporary diagnostics, replacing the former tasks 6.3 and 6.4.
+
## Change verification
```sh
diff --git a/openspec/specs/verification-container-resources/spec.md b/openspec/specs/verification-container-resources/spec.md
index 066cc9e6..78638031 100644
--- a/openspec/specs/verification-container-resources/spec.md
+++ b/openspec/specs/verification-container-resources/spec.md
@@ -49,3 +49,51 @@ A restore runs code taken from the backup with database superuser rights, so a r
- **WHEN** the agent creates a restore container without a CPU limit
- **THEN** the container still has the per-job memory limit, the process count limit, all capabilities dropped except the minimal set, and privilege escalation disabled
+
+### Requirement: Restore containers run with reduced privileges
+
+A restore runs code taken from the backup with database superuser rights. The verification agent SHALL create every restore container with all Linux capabilities dropped except `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `SETGID` and `SETUID`, with privilege escalation disabled, and with no directory of the host mounted into it. The agent SHALL request these settings from the container engine when it creates the container. It SHALL NOT fail a job because the engine describes the applied settings in a different form than the agent requested them.
+
+#### Scenario: Container is created with reduced privileges
+
+- **WHEN** the agent creates a restore container for a verification job
+- **THEN** the container engine receives a request that drops all capabilities, adds back only `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `SETGID` and `SETUID`, disables privilege escalation and mounts no host directory
+
+#### Scenario: Processes in the container hold only the minimal capabilities
+
+- **WHEN** a restore container created by the agent is running
+- **THEN** the capability bounding set of its processes contains only `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `SETGID` and `SETUID`
+- **AND** its processes cannot gain privileges through setuid binaries
+
+#### Scenario: Engine reports dropped capabilities as an explicit list
+
+- **WHEN** the container engine reports the dropped capabilities of a restore container as a list of capability names instead of `ALL`
+- **THEN** the agent proceeds with the restore and the verification reaches its normal terminal status
+
+### Requirement: Verification runs on Podman
+
+The verification agent SHALL verify backups when its Docker endpoint is a rootful Podman engine reached through Podman's Docker-compatible API. A restore of a valid backup SHALL complete, a restore of a broken backup SHALL fail with the restore error, and in both cases the agent SHALL remove the containers and networks it created for the job.
+
+#### Scenario: Valid backup on Podman
+
+- **WHEN** the agent's Docker endpoint is a rootful Podman engine
+- **AND** the agent claims a verification job for a valid PostgreSQL backup
+- **THEN** the agent reports the verification as completed with the restored database's statistics
+- **AND** no container or network created for the job remains on the engine
+
+#### Scenario: Broken backup on Podman
+
+- **WHEN** the agent's Docker endpoint is a rootful Podman engine
+- **AND** the agent claims a verification job for a backup that cannot be restored
+- **THEN** the agent reports the verification as failed with the restore error
+- **AND** no container or network created for the job remains on the engine
+
+### Requirement: An unreachable restore container is reported with its cause
+
+When the agent cannot connect to the database in a restore container before the readiness timeout, the verification SHALL fail, and its failure message SHALL include the error of the agent's last connection attempt that ended before the timeout, such as a refused connection, a connection that got no answer or a rejected login. The failure message SHALL NOT contain the database password.
+
+#### Scenario: Published port refuses connections
+
+- **WHEN** the database in a restore container stays unreachable because its published port refuses every connection
+- **THEN** the agent reports the verification as failed
+- **AND** the failure message says that the readiness timeout passed and that the last connection attempt was refused