PostgreSQL backup tool with Point-In-Time-Recovery and restore verification
TypeScript backupbackupsdatabasedatabase-backupdevopsdockergolangkubernetesmariadbmongodbmysqlpgpostgrespostgresqls3self-hostedsystem-administrationtoolsweb-ui
FIX (verification): Stop reading the restore container's hardening back after start, so verification works on Podman, and test the applied privileges end to end on Docker and Podman (issue #833)
Rostislav Dugin177ecf140cb6
diff --git a/.github/workflows/ci-release.yml b/.github/workflows/ci-release.ymlindex 12557889..92d44f9b 100644--- a/.github/workflows/ci-release.yml+++ b/.github/workflows/ci-release.yml@@ -239,6 +239,40 @@ jobs:docker compose down -v --rmi local || truerm -rf artifacts || true+ e2e-verification-agent-podman:+ if: github.ref != 'refs/heads/develop' && github.head_ref != 'develop'+ runs-on: ubuntu-24.04+ timeout-minutes: 30+ needs: [lint-verification-agent]+ steps:+ - name: Check out code+ uses: actions/checkout@34e114876b0b11c390a56381ad16ebd13914f8d5 # v4.3.1++ - name: Install tools from mise.toml+ uses: jdx/mise-action@2d8d4cafcbd33be2ea37d2b6f5ad595363d1f1ca # v5.1.1+ with:+ install_args: "just"++ - name: Start rootful Podman socket+ run: |+ sudo systemctl start podman.socket+ sudo podman version+ sudo podman info --format '{{.Host.OCIRuntime.Name}}'++ - name: Run e2e tests on Podman+ run: |+ cd agent/verification+ just e2e-podman++ - name: Cleanup+ if: always()+ run: |+ cd agent/verification/e2e+ docker compose down -v --rmi local || true+ rm -rf artifacts || true+ sudo podman ps -aq --filter label=databasus.verification.agent_id | xargs -r sudo podman rm -f || true+ sudo podman network ls -q --filter label=databasus.verification.agent_id | xargs -r sudo podman network rm || true+test-backend:if: github.ref != 'refs/heads/develop' && github.head_ref != 'develop'# 22.04 runs Linux 6.8. ubuntu-24.04 moved to Linux 7.0, where mongod 9.0 exits at startup:@@ -786,7 +820,7 @@ jobs:determine-version:runs-on: ubuntu-24.04timeout-minutes: 10- needs: [lint-backend, test-backend, test-frontend, test-verification-agent, e2e-verification-agent, test-filesystems]+ needs: [lint-backend, test-backend, test-frontend, test-verification-agent, e2e-verification-agent, e2e-verification-agent-podman, test-filesystems]if: ${{ github.ref == 'refs/heads/main' && !contains(github.event.head_commit.message, '[skip-release]') }}outputs:should_release: ${{ steps.version_bump.outputs.should_release }}diff --git a/agent/verification/AGENTS.md b/agent/verification/AGENTS.mdindex 8293e26a..ab3c8ccd 100644--- a/agent/verification/AGENTS.md+++ b/agent/verification/AGENTS.md@@ -150,7 +150,7 @@ Examples: `Test_DeriveCapacity_WhenConcurrentJobsExceedCPU_ReturnsError`, `Test_### Where tests live- **Unit / package tests**: alongside the code, named `*_test.go` (e.g. `internal/features/heartbeat/heartbeat_test.go`).-- **End-to-end tests**: under `agent/verification/e2e/`. Run via `just e2e` / `just e2e-clean`.+- **End-to-end tests**: under `agent/verification/e2e/`. Run via `just e2e` / `just e2e-clean`. `just e2e-podman` runs the e2e subset on Podman.The agent has no HTTP API of its own, so there are no controller tests — test exported functions of the packages, the CLI commands, and the HTTP-client/IO logic directly.diff --git a/agent/verification/Justfile b/agent/verification/Justfileindex 682fabc5..b6a3ab5b 100644--- a/agent/verification/Justfile+++ b/agent/verification/Justfile@@ -39,6 +39,20 @@ e2e:cd e2e && docker compose run --rm e2e-agent-runnercd e2e && docker compose down -v+# Runs the Podman subset with restore containers on Podman. Defaults to the+# rootful socket; override for rootless, e.g.+# E2E_ENGINE_SOCKET=$XDG_RUNTIME_DIR/podman/podman.sock just e2e-podman+e2e-podman:+ -cd e2e && docker compose down -v --remove-orphans 2>/dev/null+ rm -rf e2e/artifacts+ cd e2e && docker compose build --no-cache e2e-mock-server+ cd e2e && docker compose build+ cd e2e && docker compose run --rm e2e-agent-builder+ cd e2e && docker compose run --rm e2e-fixture-gen+ cd e2e && docker compose up -d e2e-mock-server+ cd e2e && E2E_ENGINE_SOCKET="${E2E_ENGINE_SOCKET:-/run/podman/podman.sock}" docker compose run --rm e2e-agent-runner bash /opt/agent/scripts/run-podman.sh+ cd e2e && docker compose down -v+e2e-clean:-cd e2e && docker compose down -v --rmi local 2>/dev/nullrm -rf e2e/artifactsdiff --git a/agent/verification/e2e/docker-compose.yml b/agent/verification/e2e/docker-compose.ymlindex fe9c1e83..972b781f 100644--- a/agent/verification/e2e/docker-compose.yml+++ b/agent/verification/e2e/docker-compose.yml@@ -46,9 +46,11 @@ services:- ./artifacts:/opt/agent/artifacts:ro- ./scripts:/opt/agent/scripts:ro# The agent's Docker SDK client spawns the ephemeral Postgres containers- # on the host daemon; pg_restore runs inside those containers (no host- # client, no assets bundle).- - /var/run/docker.sock:/var/run/docker.sock+ # on the host engine; pg_restore runs inside those containers (no host+ # client, no assets bundle). The scripts' docker CLI uses the same+ # socket. E2E_ENGINE_SOCKET swaps the engine for the agent only, e.g. a+ # Podman socket for `just e2e-podman`; the other services stay on Docker.+ - ${E2E_ENGINE_SOCKET:-/var/run/docker.sock}:/var/run/docker.sock# Mirrors production: the agent runs directly on the VM, so 127.0.0.1 IS# the host. Spawned Postgres containers publish 5432 to a random host port# bound to the host's 127.0.0.1 (dockerengine.go binds {HostIP:"127.0.0.1",diff --git a/agent/verification/e2e/mock-server/main.go b/agent/verification/e2e/mock-server/main.goindex 9e9c5fbe..bc73393c 100644--- a/agent/verification/e2e/mock-server/main.go+++ b/agent/verification/e2e/mock-server/main.go@@ -28,6 +28,7 @@ type server struct {streamFailRemain inttearStreamOnce boolreportGone bool+ streamDelay time.Duration}func main() {@@ -49,6 +50,7 @@ func main() {mux.HandleFunc("POST /mock/set-backup-fixture", s.handleSetBackupFixture)mux.HandleFunc("POST /mock/set-abort", s.handleSetAbort)mux.HandleFunc("POST /mock/set-stream-fail", s.handleSetStreamFail)+ mux.HandleFunc("POST /mock/set-stream-delay", s.handleSetStreamDelay)mux.HandleFunc("POST /mock/set-report-gone", s.handleSetReportGone)mux.HandleFunc("GET /mock/reports", s.handleGetReports)mux.HandleFunc("POST /mock/reset", s.handleReset)@@ -130,8 +132,21 @@ func (s *server) handleBackupStream(w http.ResponseWriter, r *http.Request) {tear := s.tearStreamOnces.tearStreamOnce = falsefixture := s.backupFixturePath+ delay := s.streamDelays.mu.Unlock()+ // The delay holds the response headers back, so the agent's restore+ // container stays up waiting for its backup while a test inspects it.+ if delay > 0 {+ log.Printf("GET backup-stream -> delaying %s", delay)++ select {+ case <-time.After(delay):+ case <-r.Context().Done():+ return+ }+ }+if failRemain > 0 {log.Printf("GET backup-stream -> injected 503 (remaining=%d)", failRemain-1)w.WriteHeader(http.StatusServiceUnavailable)@@ -301,6 +316,24 @@ func (s *server) handleSetStreamFail(w http.ResponseWriter, r *http.Request) {w.WriteHeader(http.StatusOK)}+func (s *server) handleSetStreamDelay(w http.ResponseWriter, r *http.Request) {+ var body struct {+ Seconds int `json:"seconds"`+ }++ if err := json.NewDecoder(r.Body).Decode(&body); err != nil {+ http.Error(w, err.Error(), http.StatusBadRequest)+ return+ }++ s.mu.Lock()+ s.streamDelay = time.Duration(body.Seconds) * time.Second+ s.mu.Unlock()++ log.Printf("POST /mock/set-stream-delay -> %ds", body.Seconds)+ w.WriteHeader(http.StatusOK)+}+func (s *server) handleSetReportGone(w http.ResponseWriter, r *http.Request) {var body struct {Gone bool `json:"gone"`@@ -342,7 +375,7 @@ func (s *server) handleHeartbeatStatus(w http.ResponseWriter, _ *http.Request) {}// handleReset wipes per-test state (reports, claim, abort/stream/report-gone-// flags, fixture path). Version + binary path are left alone — pin them via+// flags, stream delay, fixture path). Version + binary path are left alone — pin them via// /mock/set-version + /mock/set-binary-path. Each restore-style e2e test// should call this at the top to avoid seeing prior-test artifacts.func (s *server) handleReset(w http.ResponseWriter, _ *http.Request) {@@ -354,6 +387,7 @@ func (s *server) handleReset(w http.ResponseWriter, _ *http.Request) {s.streamFailRemain = 0s.tearStreamOnce = falses.reportGone = false+ s.streamDelay = 0s.heartbeats = 0s.lastBody = nils.mu.Unlock()diff --git a/agent/verification/e2e/scripts/run-all.sh b/agent/verification/e2e/scripts/run-all.shindex a92e8a3e..894945ac 100644--- a/agent/verification/e2e/scripts/run-all.sh+++ b/agent/verification/e2e/scripts/run-all.sh@@ -7,6 +7,7 @@ TESTS=("test-heartbeat-and-upgrade.sh""test-start-stop-status.sh""test-startup-purge.sh"+ "test-container-privileges.sh""test-restore-broken.sh""test-restore-disk-budget.sh""test-restore-timescale.sh"diff --git a/agent/verification/e2e/scripts/run-podman.sh b/agent/verification/e2e/scripts/run-podman.shnew file mode 100644index 00000000..78fb50f4--- /dev/null+++ b/agent/verification/e2e/scripts/run-podman.sh@@ -0,0 +1,42 @@+#!/bin/bash+# Podman subset of the e2e suite, run by `just e2e-podman` with the runner's+# engine socket pointed at Podman's Docker-compatible API. It covers the+# engine-dependent paths: the privileges the kernel applied, a successful+# restore and a failed one, each with its leak check. The other tests do not+# depend on the engine and run on Docker only, through run-all.sh.+set -uo pipefail++SCRIPT_DIR="$(dirname "$0")"++TESTS=(+ "test-container-privileges.sh"+ "test-restore-success.sh 18"+ "test-restore-broken.sh"+)++echo "========================================"+echo " Verification agent e2e (Podman)"+echo "========================================"++PASSED=0+FAILED=0++for test in "${TESTS[@]}"; do+ read -r -a args <<< "$test"+ echo ""+ echo "---- $test ----"+ if bash "$SCRIPT_DIR/${args[0]}" "${args[@]:1}"; then+ PASSED=$((PASSED + 1))+ else+ FAILED=$((FAILED + 1))+ fi+done++echo ""+echo "========================================"+echo " Results: $PASSED passed, $FAILED failed"+echo "========================================"++if [ "$FAILED" -ne 0 ]; then+ exit 1+fidiff --git a/agent/verification/e2e/scripts/test-container-privileges.sh b/agent/verification/e2e/scripts/test-container-privileges.shnew file mode 100644index 00000000..d34e0540--- /dev/null+++ b/agent/verification/e2e/scripts/test-container-privileges.sh@@ -0,0 +1,89 @@+#!/bin/bash+# Privileges e2e: checks what the kernel applied to the restore container, not+# what the engine reports back. The mock holds the backup stream for 30s, so+# the job's container is up and waiting for its backup while this script reads+# /proc/1/status through `docker exec`. The capability bounding set must hold+# exactly CHOWN, DAC_OVERRIDE, FOWNER, SETGID and SETUID (0xcb), and+# no-new-privileges must be set. CapEff is not checked: PID 1 is the Postgres+# server running as a non-root user, so it is zero under any configuration.+# The job must then finish COMPLETED and leave nothing behind.+set -euo pipefail++source "$(dirname "$0")/lib.sh"++WORK="/tmp/agent-work-container-privileges"+AGENT_ID="77777777-7777-7777-7777-777777777777"+VERIFICATION_ID="77777777-aaaa-aaaa-aaaa-777777777777"+BACKUP_ID="77777777-bbbb-bbbb-bbbb-777777777777"++EXPECTED_CAP_BND="00000000000000cb"+STREAM_DELAY_SECONDS=30++rm -rf "$WORK"+mkdir -p "$WORK"+cd "$WORK"++reset_mock_state+reset_mock_version++cp "$ARTIFACTS/agent-v1" ./databasus-verification-agent+chmod +x ./databasus-verification-agent++start_agent "$AGENT_ID"++curl -sf -X POST "$MOCK/mock/set-stream-delay" \+ -H 'Content-Type: application/json' \+ -d "{\"seconds\":${STREAM_DELAY_SECONDS}}"++curl -sf -X POST "$MOCK/mock/set-backup-fixture" \+ -H 'Content-Type: application/json' \+ -d '{"path":"/artifacts/good-pg18.dump"}'++curl -sf -X POST "$MOCK/mock/set-claim" \+ -H 'Content-Type: application/json' \+ -d "{\"verificationId\":\"$VERIFICATION_ID\",\"backupId\":\"$BACKUP_ID\",\"backupSizeMb\":1,\"maxContainerDiskMb\":2048,\"database\":{\"type\":\"POSTGRES_LOGICAL\",\"postgresqlLogical\":{\"version\":\"18\"}}}"++CONTAINER_ID=""+deadline=$((SECONDS + 120))+while [ $SECONDS -lt $deadline ]; do+ CONTAINER_ID="$(docker ps --filter "label=databasus.verification.agent_id=${AGENT_ID}" \+ --filter status=running --format '{{.ID}}' | head -n1)"+ if [ -n "$CONTAINER_ID" ]; then+ break+ fi+ sleep 1+done++if [ -z "$CONTAINER_ID" ]; then+ echo "FAIL: no running restore container labeled with agent ${AGENT_ID} within 120s"+ dump_diagnostics+ stop_agent+ exit 1+fi++PROC_STATUS="$(docker exec "$CONTAINER_ID" cat /proc/1/status)"+CAP_BND="$(echo "$PROC_STATUS" | awk '/^CapBnd:/ {print $2}')"+NO_NEW_PRIVS="$(echo "$PROC_STATUS" | awk '/^NoNewPrivs:/ {print $2}')"++if [ "$CAP_BND" != "$EXPECTED_CAP_BND" ] || [ "$NO_NEW_PRIVS" != "1" ]; then+ echo "FAIL: restore container privileges: CapBnd=${CAP_BND} (want ${EXPECTED_CAP_BND}), NoNewPrivs=${NO_NEW_PRIVS} (want 1)"+ echo "$PROC_STATUS"+ stop_agent+ exit 1+fi++echo "Restore container privileges OK: CapBnd=${CAP_BND} NoNewPrivs=${NO_NEW_PRIVS}"++wait_for_report '"status":"COMPLETED"' 240 '"status":"FAILED"'++assert_report "$VERIFICATION_ID" '.pgRestoreExitCode == 0'++stop_agent++if ! leak_check "$AGENT_ID"; then+ echo "---- agent.out ----"+ cat agent.out+ exit 1+fi++echo "Verification agent container-privileges e2e passed"diff --git a/agent/verification/internal/features/container/dockerengine.go b/agent/verification/internal/features/container/dockerengine.goindex f9424328..d634952c 100644--- a/agent/verification/internal/features/container/dockerengine.go+++ b/agent/verification/internal/features/container/dockerengine.go@@ -134,6 +134,23 @@ func (e *dockerEngine) RemoveNetwork(ctx context.Context, networkID string) erro}func (e *dockerEngine) CreateContainer(ctx context.Context, spec SpawnSpec) (string, error) {+ config, hostConfig := toContainerConfigs(spec)++ resp, err := e.cli.ContainerCreate(ctx, client.ContainerCreateOptions{+ Config: config,+ HostConfig: hostConfig,+ Name: spec.Name,+ })+ if err != nil {+ return "", err+ }++ return resp.ID, nil+}++// toContainerConfigs is the 1:1 translation of a SpawnSpec into the engine's+// create request; a unit test pins the hardening it carries.+func toContainerConfigs(spec SpawnSpec) (*container.Config, *container.HostConfig) {securityOpt := []string{}if spec.NoNewPrivileges {securityOpt = append(securityOpt, "no-new-privileges")@@ -168,22 +185,15 @@ func (e *dockerEngine) CreateContainer(ctx context.Context, spec SpawnSpec) (str},}- resp, err := e.cli.ContainerCreate(ctx, client.ContainerCreateOptions{- Config: &container.Config{- Image: spec.Image,- Cmd: spec.Cmd,- Env: spec.Env,- Labels: spec.Labels,- ExposedPorts: network.PortSet{pgPublishedPort: struct{}{}},- },- HostConfig: hostConfig,- Name: spec.Name,- })- if err != nil {- return "", err+ config := &container.Config{+ Image: spec.Image,+ Cmd: spec.Cmd,+ Env: spec.Env,+ Labels: spec.Labels,+ ExposedPorts: network.PortSet{pgPublishedPort: struct{}{}},}- return resp.ID, nil+ return config, hostConfig}func (e *dockerEngine) StartContainer(ctx context.Context, containerID string) error {@@ -212,38 +222,6 @@ func (e *dockerEngine) HostPort(return strconv.Atoi(bindings[0].HostPort)}-func (e *dockerEngine) InspectSecurity(- ctx context.Context, containerID string,-) (SecurityState, error) {- insp, err := e.cli.ContainerInspect(ctx, containerID, client.ContainerInspectOptions{})- if err != nil {- return SecurityState{}, err- }-- state := SecurityState{- ReadonlyRootfs: insp.Container.HostConfig.ReadonlyRootfs,- // Only flag mounts the AGENT requested. insp.Mounts (runtime) includes- // image-declared anonymous volumes (postgres:16 declares VOLUME- // /var/lib/postgresql/data for PGDATA) that we explicitly want; only- // host bind mounts from our spec are a hardening regression.- HasHostBinds: len(insp.Container.HostConfig.Binds) > 0 || len(insp.Container.HostConfig.Mounts) > 0,- }-- for _, opt := range insp.Container.HostConfig.SecurityOpt {- if strings.Contains(opt, "no-new-privileges") {- state.NoNewPrivileges = true- }- }-- for _, cap := range insp.Container.HostConfig.CapDrop {- if cap == "ALL" {- state.CapDropAll = true- }- }-- return state, nil-}-func (e *dockerEngine) Exec(ctx context.Context,containerID string,diff --git a/agent/verification/internal/features/container/dockerengine_test.go b/agent/verification/internal/features/container/dockerengine_test.goindex 7e2b5421..04788281 100644--- a/agent/verification/internal/features/container/dockerengine_test.go+++ b/agent/verification/internal/features/container/dockerengine_test.go@@ -1,10 +1,16 @@package containerimport (+ "net/netip""testing"+ "github.com/moby/moby/api/types/container"+ "github.com/moby/moby/api/types/network""github.com/moby/moby/api/types/system""github.com/stretchr/testify/assert"+ "github.com/stretchr/testify/require"++ "databasus-verification-agent/internal/testutil")func Test_ToHostCapabilities_ReadsCpuLimitSupportFromBothCfsFlags(t *testing.T) {@@ -63,3 +69,44 @@ func Test_ToHostCapabilities_ReadsUserNSRemapFromSecurityOptions(t *testing.T) {})}}++func Test_ToContainerConfigs_CarriesHardeningAndLimitsOfSpec(t *testing.T) {+ testCases := []struct {+ name string+ isCpuLimitSupported bool+ expectedNanoCPUs int64+ }{+ {name: "cpu limit supported", isCpuLimitSupported: true, expectedNanoCPUs: 2_000_000_000},+ {name: "cpu limit unsupported", isCpuLimitSupported: false, expectedNanoCPUs: 0},+ }++ for _, testCase := range testCases {+ t.Run(testCase.name, func(t *testing.T) {+ containerManager := NewManager(nil, "agent-1", "databasus/verification-postgres", testutil.DiscardLogger())+ containerManager.isCpuLimitSupported = testCase.isCpuLimitSupported+ spec := containerManager.buildSpec(newTestSpawnPlan())++ config, hostConfig := toContainerConfigs(spec)++ assert.Equal(t, []string{"ALL"}, hostConfig.CapDrop)+ assert.Equal(t, minimalCaps, hostConfig.CapAdd)+ assert.Contains(t, hostConfig.SecurityOpt, "no-new-privileges")+ assert.Empty(t, hostConfig.Binds)+ assert.Empty(t, hostConfig.Mounts)+ assert.Equal(t, int64(1024)*1024*1024, hostConfig.Memory)+ require.NotNil(t, hostConfig.PidsLimit)+ assert.EqualValues(t, containerPidsLimit, *hostConfig.PidsLimit)+ assert.Equal(t, testCase.expectedNanoCPUs, hostConfig.NanoCPUs)+ assert.Equal(t, container.NetworkMode("net-id"), hostConfig.NetworkMode)++ assert.Equal(t, network.PortMap{+ pgPublishedPort: []network.PortBinding{+ {HostIP: netip.MustParseAddr("127.0.0.1"), HostPort: "0"},+ },+ }, hostConfig.PortBindings)+ assert.Equal(t, network.PortSet{pgPublishedPort: struct{}{}}, config.ExposedPorts)+ assert.Equal(t, spec.Image, config.Image)+ assert.Equal(t, spec.Labels, config.Labels)+ })+ }+}diff --git a/agent/verification/internal/features/container/manager.go b/agent/verification/internal/features/container/manager.goindex 6cab3498..04d1be52 100644--- a/agent/verification/internal/features/container/manager.go+++ b/agent/verification/internal/features/container/manager.go@@ -191,11 +191,6 @@ func (m *Manager) Spawn(jobCtx context.Context, req SpawnRequest) (*PostgresContreturn nil, fmt.Errorf("container never became ready: %w", err)}- if err := m.assertSecurity(jobCtx, containerID); err != nil {- _ = c.Terminate(jobCtx)- return nil, err- }-return c, nil}@@ -234,24 +229,6 @@ func (m *Manager) getNanoCPUs(cpuPerJob int) int64 {return int64(cpuPerJob) * 1_000_000_000}-func (m *Manager) assertSecurity(ctx context.Context, containerID string) error {- state, err := m.engine.InspectSecurity(ctx, containerID)- if err != nil {- return fmt.Errorf("inspect security state: %w", err)- }-- switch {- case !state.NoNewPrivileges:- return fmt.Errorf("hardening regression: no-new-privileges not applied")- case !state.CapDropAll:- return fmt.Errorf("hardening regression: CapDrop ALL not applied")- case state.HasHostBinds:- return fmt.Errorf("hardening regression: container has host bind mounts")- }-- return nil-}-func waitForReady(ctx context.Context, conn dbconn.Conn) error {readyCtx, cancel := context.WithTimeout(ctx, containerReadyTimeout)defer cancel()diff --git a/agent/verification/internal/features/container/securitystate.go b/agent/verification/internal/features/container/securitystate.godeleted file mode 100644index e56bba02..00000000--- a/agent/verification/internal/features/container/securitystate.go+++ /dev/null@@ -1,10 +0,0 @@-package container--// SecurityState is read back from the live container so each job re-asserts its-// own hardening landed (a daemon glitch fails that job, never the process).-type SecurityState struct {- NoNewPrivileges bool- CapDropAll bool- ReadonlyRootfs bool- HasHostBinds bool-}diff --git a/openspec/changes/run-verification-on-podman/.openspec.yaml b/openspec/changes/run-verification-on-podman/.openspec.yamlnew file mode 100644index 00000000..7cc41bb5--- /dev/null+++ b/openspec/changes/run-verification-on-podman/.openspec.yaml@@ -0,0 +1,2 @@+schema: spec-driven+created: 2026-10-22diff --git a/openspec/changes/run-verification-on-podman/design.md b/openspec/changes/run-verification-on-podman/design.mdnew file mode 100644index 00000000..43039978--- /dev/null+++ b/openspec/changes/run-verification-on-podman/design.md@@ -0,0 +1,78 @@+## Context++See proposal.md for why. The relevant current state:++- `Manager.buildSpec` sets `CapDropAll: true`, `CapAdd: minimalCaps` and `NoNewPrivileges: true` on every restore container (`agent/verification/internal/features/container/manager.go:203-226`, `minimalCaps` at `:35`). `postgres_test.go:19-20` and `:38-39` assert these fields on the spec.+- `dockerEngine.CreateContainer` translates the spec into the engine's host configuration inline (`dockerengine.go:136-176`): `CapDropAll` becomes `CapDrop: ["ALL"]`, `NoNewPrivileges` becomes `SecurityOpt: ["no-new-privileges"]`. No test covers this translation.+- After the container is ready, `Manager.Spawn` calls `assertSecurity` (`manager.go:194-197`), which reads the container back through `dockerEngine.InspectSecurity` (`dockerengine.go:215-245`) and fails the job unless `HostConfig.CapDrop` contains the literal `ALL`, `SecurityOpt` contains `no-new-privileges`, and `Binds` and `Mounts` are empty (`manager.go:237-253`). `SecurityState` (`securitystate.go`) exists only for this check; its `ReadonlyRootfs` field is read by nobody.+- Docker's inspect returns the host configuration the client sent, so on Docker the check compares the agent's request with itself. Podman's compat API returns `CapDrop` as the capabilities it removed from its own default set and an empty `CapAdd`. Reproduced locally on Podman 5.8.7: `CapDrop` was `["FSETID","KILL","NET_BIND_SERVICE","SETFCAP","SETPCAP","SYS_CHROOT"]`, while `/proc/1/status` in the same container showed `CapBnd: 00000000000000cb` (exactly the five minimal capabilities) and `NoNewPrivs: 1`.+- The runner spawns the container before it downloads the backup (`agent/verification/internal/features/runner/runner.go:166`, download at `:454`). The stream HTTP client has no timeout (`internal/features/api/api.go:86`), and the idle timeout of 60 seconds (`runner.go:28`) starts only once the body is being read.+- The end-to-end suite runs the agent binary in `e2e-agent-runner` with `network_mode: host` and the host's `/var/run/docker.sock` mounted (`agent/verification/e2e/docker-compose.yml`). The agent and the scripts' `docker` CLI both reach the engine through that socket. `just e2e` runs it (`agent/verification/Justfile`), and the CI job `e2e-verification-agent` calls `just e2e` (`.github/workflows/ci-release.yml:216-240`).+- The mock server serves the backup stream in `handleBackupStream` (`agent/verification/e2e/mock-server/main.go`). It has knobs for 503s and a torn stream, not for a delay.++The approach follows `agent/verification/AGENTS.md`: pure translation functions get table-driven unit tests, as `toHostCapabilities` already does in `dockerengine_test.go`, and behavior that needs a real engine is covered in `e2e/` rather than in `internal/` tests.++## Goals / Non-Goals++**Goals:**++- A Podman engine with a correctly hardened container no longer fails the job.+- Each hardening control has a test that can fail: the request the agent sends is covered by a unit test, and what the kernel applied is covered by an end-to-end test on Docker and on Podman.++**Non-Goals:**++- Detecting at runtime an engine that silently ignores the requested hardening.+- Changing how the agent talks to the engine beyond deleting the read-back.++## Decisions++### Delete the runtime read-back instead of fixing it++The agent stops reading the container's security settings back at all. `assertSecurity`, `InspectSecurity` and `SecurityState` are deleted, and `Spawn` returns the container once it is ready.++Rejected alternatives:++- **Accept either `ALL` or an explicit list in `CapDrop`.** A list would have to be compared with the engine's default capability set, which the agent cannot know. Podman reads it from `containers.conf`, which an administrator can change, and Podman also reports `CapAdd` as empty, so the agent cannot rebuild the effective set from inspect.+- **Read `CapBnd` and `NoNewPrivs` from `/proc/1/status` through an exec on every job.** This is the honest form of the check, and it works on any engine. It would only catch an engine that accepts the request and silently does not apply it, which none of the engines the agent runs on does. It costs an exec per job and a parser, and a false positive fails every verification on that host, as #833 shows. The end-to-end test below makes the same kernel-level check once per engine in CI instead.+- **Keep the no-new-privileges and bind-mount branches and drop only the capability branch.** Both remaining branches compare the agent's request with itself in the same way, and Podman's reporting of either can change in a future version with the same effect as #833.++### Test the request with a pure translation function++The body of `CreateContainer` that builds `container.HostConfig` and `container.Config` from a `SpawnSpec` moves into a pure function, and `CreateContainer` calls it. A table-driven unit test feeds it specs and asserts the dropped and added capabilities, the security options, the absence of binds and mounts, the resource limits and the loopback-only port binding. This mirrors `toHostCapabilities` and needs no engine.++Rejected alternative: a fake Docker API server over `httptest` that records the create request. It tests the same mapping through the SDK's JSON encoding, which adds a moving part without adding coverage of anything the agent decides.++### Check the applied privileges end to end, on both engines++A new end-to-end script, `test-container-privileges.sh`, makes the mock delay the backup stream by 30 seconds. While the job's container is running and waiting for its backup, the script finds the container by the agent's label and reads `/proc/1/status` through `docker exec`. It asserts `CapBnd` is `00000000000000cb` and `NoNewPrivs` is `1`, then lets the job finish and runs the usual leak check. At that point PID 1 is the Postgres server, which runs as a non-root user, so `CapEff` is zero for any configuration; the bounding set is what limits a root process in the container, such as the agent's own `pg_restore` exec.++The mock gets a `POST /mock/set-stream-delay` knob, cleared by `/mock/reset`. The delay applies before the response headers, so no agent timeout fires: the stream client has none, and the idle timeout starts with the body.++Rejected alternatives:++- **A Go test in `internal/` against a real engine.** `just test` and the CI job `test-verification-agent` run without engine access by design, and the module keeps engine-dependent tests in `e2e/`.+- **Polling for the container without a delay.** On a small fixture the container lives for a few seconds after it becomes ready, so the check would race the restore.++### Run a subset of the end-to-end suite on Podman in a separate CI job++The compose file takes the engine socket for `e2e-agent-runner` from `E2E_ENGINE_SOCKET`, defaulting to `/var/run/docker.sock`. The builder, the fixture generator and the mock server stay on Docker; only the agent and the scripts' `docker` CLI talk to Podman. A new script, `run-podman.sh`, runs `test-container-privileges.sh`, `test-restore-success.sh 18` and `test-restore-broken.sh`. A new Justfile recipe, `e2e-podman`, prepares artifacts as `e2e` does and runs the runner with the socket and script swapped. It defaults the socket to `/run/podman/podman.sock` and lets the developer override it, for example with a rootless socket.++A new CI job, `e2e-verification-agent-podman`, runs on `ubuntu-24.04`, starts the rootful Podman socket with `sudo systemctl start podman.socket`, prints `podman version` and the OCI runtime for the log, and calls `just e2e-podman`. `determine-version` waits for it as it waits for the Docker job. The existing Docker job also runs `test-container-privileges.sh` through `run-all.sh`.++Rejected alternatives:++- **The full suite on Podman.** The upgrade, start-stop and purge tests do not depend on the engine's reporting, and seven PostgreSQL majors would double the job time without exercising a different engine path. Major 18 matches the issue.+- **A matrix over the existing job.** The Podman run needs a different socket, a setup step and a different script list, so a matrix would need conditionals in every step.+- **Running the whole compose stack on Podman.** It tests `podman compose`, which the product does not use, and only the agent's engine calls matter here.++## Risks / Trade-offs++- [The `docker` CLI in the runner may read Podman's compat API slightly differently, for example in `network ls` label filters] → The leak check fails loudly rather than passing silently. If a filter is unsupported, the script switches to an output the CLI and Podman agree on, still from the runner.+- [Podman may resolve the agent's short image name `databasus/verification-postgres:18` differently from Docker] → Podman's compat API resolves short names to Docker Hub. If the CI run shows otherwise, that is a real user-facing bug and is fixed in the agent, not worked around in the test.+- [The `gosu` hang the reporter saw with `crun` may reproduce in CI] → The job prints the runtime, so the failure is attributable. It is then a separate issue, and this change does not merge with a red Podman job.+- [The 30-second delay lengthens each end-to-end run by about 30 seconds per engine] → Accepted; it buys a deterministic window instead of a race.+- [Without the read-back, an engine that silently ignores the requested hardening goes unnoticed at runtime] → No engine the agent supports does that, and the end-to-end test proves the applied state on both supported engines in CI.++## Migration Plan++None. The change ships with the next agent release. Agents on Docker behave as before, and agents on Podman start completing verifications. Rollback is reverting the commit; the agent stores no state this change touches.diff --git a/openspec/changes/run-verification-on-podman/proposal.md b/openspec/changes/run-verification-on-podman/proposal.mdnew file mode 100644index 00000000..10030864--- /dev/null+++ b/openspec/changes/run-verification-on-podman/proposal.md@@ -0,0 +1,40 @@+## Why++Backup verification fails on every job when the agent talks to Podman through its Docker-compatible API (issue #833). After the restore container starts, the agent reads the container's configuration back and requires the literal `ALL` in the list of dropped capabilities. Podman does drop all capabilities except the minimal set, but it reports the dropped list differently: as the capabilities it removed from its own default set. The check therefore rejects a correctly hardened container, and the job ends with "hardening regression: CapDrop ALL not applied".++The check also protects nothing on Docker. Docker reports back the configuration the agent sent, so the check only confirms that the agent's own request contained `ALL`. It never looks at what the kernel applied. A unit test covers that request better, and a test against Podman catches this class of engine difference before a user does.++## What Changes++- The agent stops reading the container's security settings back after start. The read-back covered dropped capabilities, no-new-privileges and host bind mounts. All three branches only repeated the agent's own request, so the whole check goes: `Manager.assertSecurity`, `dockerEngine.InspectSecurity` and the `SecurityState` type are deleted, with no replacement at runtime.+- Restore containers keep the same hardening: every capability dropped except `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `SETGID` and `SETUID`, privilege escalation disabled, no host bind mounts, and the existing memory, CPU and process limits.+- A unit test checks that the container configuration the agent sends to the engine carries that hardening. The test covers the translation from the agent's container spec to the engine's host configuration, which nothing tests today.+- A new CI job runs a subset of the agent's end-to-end tests with restore containers on a rootful Podman engine, so the agent is tested against both engines it is used with.++Nothing breaks. No flag, config field, API or report field changes. The error "hardening regression: ..." disappears from verification failures.++## Capabilities++### New Capabilities++None.++### Modified Capabilities++- `verification-container-resources`: adds the hardening every restore container carries as its own requirement (today it appears only inside a CPU-limit scenario), and adds the requirement that verification runs on Podman through its Docker-compatible API.++## Impact++- Code: `agent/verification` only. The Docker engine wrapper and the container manager in `internal/features/container`, the end-to-end compose file, scripts and Justfile in `agent/verification/e2e` and `agent/verification/Justfile`.+- CI: one new job in `.github/workflows/ci-release.yml`, which `determine-version` also waits for.+- Rules this change answers to: `AGENTS.md` and `agent/verification/AGENTS.md`.+- Backend, frontend, the agent-to-backend API and the heartbeat payload are untouched.++## Out of scope++- Dropping or loosening any container hardening control. They stay as they are.+- Blocking outbound network access from restore containers.+- The hang at `gosu` that the issue reporter saw on Podman with the `crun` runtime. It did not reproduce locally on rootless Podman 5.8.7 with `crun`; if the new Podman CI job shows it, it becomes its own issue.+- Rootless Podman in CI, Podman on macOS or Windows, and other container engines.+- Running the full end-to-end suite, every PostgreSQL major included, on Podman.+- Documenting Podman as a supported engine on the website.diff --git a/openspec/changes/run-verification-on-podman/specs/verification-container-resources/spec.md b/openspec/changes/run-verification-on-podman/specs/verification-container-resources/spec.mdnew file mode 100644index 00000000..c11a5f8c--- /dev/null+++ b/openspec/changes/run-verification-on-podman/specs/verification-container-resources/spec.md@@ -0,0 +1,39 @@+## ADDED Requirements++### Requirement: Restore containers run with reduced privileges++A restore runs code taken from the backup with database superuser rights. The verification agent SHALL create every restore container with all Linux capabilities dropped except `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `SETGID` and `SETUID`, with privilege escalation disabled, and with no directory of the host mounted into it. The agent SHALL request these settings from the container engine when it creates the container. It SHALL NOT fail a job because the engine describes the applied settings in a different form than the agent requested them.++#### Scenario: Container is created with reduced privileges++- **WHEN** the agent creates a restore container for a verification job+- **THEN** the container engine receives a request that drops all capabilities, adds back only `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `SETGID` and `SETUID`, disables privilege escalation and mounts no host directory++#### Scenario: Processes in the container hold only the minimal capabilities++- **WHEN** a restore container created by the agent is running+- **THEN** the capability bounding set of its processes contains only `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `SETGID` and `SETUID`+- **AND** its processes cannot gain privileges through setuid binaries++#### Scenario: Engine reports dropped capabilities as an explicit list++- **WHEN** the container engine reports the dropped capabilities of a restore container as a list of capability names instead of `ALL`+- **THEN** the agent proceeds with the restore and the verification reaches its normal terminal status++### Requirement: Verification runs on Podman++The verification agent SHALL verify backups when its Docker endpoint is a rootful Podman engine reached through Podman's Docker-compatible API. A restore of a valid backup SHALL complete, a restore of a broken backup SHALL fail with the restore error, and in both cases the agent SHALL remove the containers and networks it created for the job.++#### Scenario: Valid backup on Podman++- **WHEN** the agent's Docker endpoint is a rootful Podman engine+- **AND** the agent claims a verification job for a valid PostgreSQL backup+- **THEN** the agent reports the verification as completed with the restored database's statistics+- **AND** no container or network created for the job remains on the engine++#### Scenario: Broken backup on Podman++- **WHEN** the agent's Docker endpoint is a rootful Podman engine+- **AND** the agent claims a verification job for a backup that cannot be restored+- **THEN** the agent reports the verification as failed with the restore error+- **AND** no container or network created for the job remains on the enginediff --git a/openspec/changes/run-verification-on-podman/tasks.md b/openspec/changes/run-verification-on-podman/tasks.mdnew file mode 100644index 00000000..fe4f1a53--- /dev/null+++ b/openspec/changes/run-verification-on-podman/tasks.md@@ -0,0 +1,28 @@+## 1. Pin the container request with a unit test++- [x] 1.1 In `agent/verification/internal/features/container/dockerengine.go`, move the construction of `container.Config` and `container.HostConfig` out of `CreateContainer` into a pure function that takes a `SpawnSpec`, and make `CreateContainer` call it. Add a table-driven test in `dockerengine_test.go` that asserts, for a hardened spec: `CapDrop` is exactly `["ALL"]`, `CapAdd` equals `minimalCaps`, `SecurityOpt` contains `no-new-privileges`, `Binds` and `Mounts` are empty, the memory, process and CPU limits are copied, a zero `NanoCPUs` stays zero, and port 5432 is published only on `127.0.0.1`. Verify with `just test` in `agent/verification`.++## 2. Delete the runtime read-back++- [x] 2.1 Delete `Manager.assertSecurity` and its call in `Manager.Spawn`, `dockerEngine.InspectSecurity` and `securitystate.go`. Leave `buildSpec`, `minimalCaps` and the hardening fields of `SpawnSpec` unchanged. Update any comment that names the read-back. Verify that `grep -rn "assertSecurity\|InspectSecurity\|SecurityState\|hardening regression" agent/verification` finds nothing, and that `just build` and `just test` in `agent/verification` pass.++## 3. Check the applied privileges end to end++- [x] 3.1 Add `POST /mock/set-stream-delay` with a body `{"seconds": N}` to `agent/verification/e2e/mock-server/main.go`. `handleBackupStream` waits that long before writing the response, and `/mock/reset` clears the delay. Verify with `just lint` in `agent/verification`, which covers `./e2e/...`.+- [x] 3.2 Add `agent/verification/e2e/scripts/test-container-privileges.sh`. It sets a 30-second stream delay, claims a job for `good-pg18.dump`, waits until a running container labeled with the test's agent ID exists, and reads `/proc/1/status` through `docker exec`. It fails unless `CapBnd` is `00000000000000cb` and `NoNewPrivs` is `1`. It then waits for the `COMPLETED` report and runs `leak_check`. Add the script to `TESTS` in `run-all.sh`. Verify with `just e2e` in `agent/verification`, which must report every test passed.++## 4. Run the end-to-end subset on Podman++- [x] 4.1 In `agent/verification/e2e/docker-compose.yml`, mount the runner's engine socket from `${E2E_ENGINE_SOCKET:-/var/run/docker.sock}` and update the runner's comments. Add `agent/verification/e2e/scripts/run-podman.sh`, which runs `test-container-privileges.sh`, `test-restore-success.sh 18` and `test-restore-broken.sh` and reports results the way `run-all.sh` does. Add an `e2e-podman` recipe to `agent/verification/Justfile` that prepares artifacts as `e2e` does and runs the runner with `E2E_ENGINE_SOCKET` defaulting to `/run/podman/podman.sock` and the command `bash /opt/agent/scripts/run-podman.sh`. Verify with `E2E_ENGINE_SOCKET=$XDG_RUNTIME_DIR/podman/podman.sock just e2e-podman` in `agent/verification`, which must report every test passed, and with `just e2e` still passing.+- [x] 4.2 Add the job `e2e-verification-agent-podman` to `.github/workflows/ci-release.yml`, with the same trigger condition, runner image, timeout, `needs` and tool setup as `e2e-verification-agent`. Before `just e2e-podman` it runs `sudo systemctl start podman.socket` and prints `podman version` and `podman info --format '{{.Host.OCIRuntime.Name}}'`. Its cleanup step also removes leftover Podman containers labeled by the agent. Add the job to the `needs` of `determine-version`. Verify with `actionlint .github/workflows/ci-release.yml`, which must report no new findings compared with the file before the change.++## 5. Final checks++- [x] 5.1 Run `just lint` in `agent/verification` and fix every finding until it passes.+- [x] 5.2 Search the repository for mentions of the deleted read-back and of the e2e entry points (`run-all.sh`, `just e2e`, `docker.sock` in the agent's e2e) and bring every affected comment and document in line with the change. Verify that `grep -rn "hardening regression" .` outside `openspec/` finds nothing.++## Change verification++```sh+just --justfile agent/verification/Justfile test+```