PostgreSQL backup tool with Point-In-Time-Recovery and restore verification
TypeScript backupbackupsdatabasedatabase-backupdevopsdockergolangkubernetesmariadbmongodbmysqlpgpostgrespostgresqls3self-hostedsystem-administrationtoolsweb-ui
FIX (verification): Start restore containers as postgres with no capabilities instead of no-new-privileges, which hangs Postgres under AppArmor on rootful Podman, and archive start-restore-containers-unprivileged (issue #833)
Rostislav Dugindff7d41fe4fb
diff --git a/agent/verification/e2e/scripts/test-container-privileges.sh b/agent/verification/e2e/scripts/test-container-privileges.shindex d34e0540..7f828078 100644--- a/agent/verification/e2e/scripts/test-container-privileges.sh+++ b/agent/verification/e2e/scripts/test-container-privileges.sh@@ -2,10 +2,10 @@# Privileges e2e: checks what the kernel applied to the restore container, not# what the engine reports back. The mock holds the backup stream for 30s, so# the job's container is up and waiting for its backup while this script reads-# /proc/1/status through `docker exec`. The capability bounding set must hold-# exactly CHOWN, DAC_OVERRIDE, FOWNER, SETGID and SETUID (0xcb), and-# no-new-privileges must be set. CapEff is not checked: PID 1 is the Postgres-# server running as a non-root user, so it is zero under any configuration.+# /proc/1/status through `docker exec`. PID 1 must run as a non-root user, and+# the capability bounding set must be empty, so even a process that reaches+# user ID 0 inside the container holds no capabilities. CapEff is not checked:+# for a non-root process it is zero under any configuration.# The job must then finish COMPLETED and leave nothing behind.set -euo pipefail@@ -16,7 +16,7 @@ AGENT_ID="77777777-7777-7777-7777-777777777777"VERIFICATION_ID="77777777-aaaa-aaaa-aaaa-777777777777"BACKUP_ID="77777777-bbbb-bbbb-bbbb-777777777777"-EXPECTED_CAP_BND="00000000000000cb"+EXPECTED_CAP_BND="0000000000000000"STREAM_DELAY_SECONDS=30rm -rf "$WORK"@@ -61,18 +61,23 @@ if [ -z "$CONTAINER_ID" ]; thenexit 1fi-PROC_STATUS="$(docker exec "$CONTAINER_ID" cat /proc/1/status)"+if ! PROC_STATUS="$(docker exec "$CONTAINER_ID" cat /proc/1/status)"; then+ echo "FAIL: could not read /proc/1/status of restore container ${CONTAINER_ID}"+ dump_diagnostics+ stop_agent+ exit 1+fiCAP_BND="$(echo "$PROC_STATUS" | awk '/^CapBnd:/ {print $2}')"-NO_NEW_PRIVS="$(echo "$PROC_STATUS" | awk '/^NoNewPrivs:/ {print $2}')"+UID_REAL="$(echo "$PROC_STATUS" | awk '/^Uid:/ {print $2}')"-if [ "$CAP_BND" != "$EXPECTED_CAP_BND" ] || [ "$NO_NEW_PRIVS" != "1" ]; then- echo "FAIL: restore container privileges: CapBnd=${CAP_BND} (want ${EXPECTED_CAP_BND}), NoNewPrivs=${NO_NEW_PRIVS} (want 1)"+if [ -z "$UID_REAL" ] || [ "$UID_REAL" = "0" ] || [ "$CAP_BND" != "$EXPECTED_CAP_BND" ]; then+ echo "FAIL: restore container privileges: Uid=${UID_REAL} (want non-zero), CapBnd=${CAP_BND} (want ${EXPECTED_CAP_BND})"echo "$PROC_STATUS"stop_agentexit 1fi-echo "Restore container privileges OK: CapBnd=${CAP_BND} NoNewPrivs=${NO_NEW_PRIVS}"+echo "Restore container privileges OK: Uid=${UID_REAL} CapBnd=${CAP_BND}"wait_for_report '"status":"COMPLETED"' 240 '"status":"FAILED"'diff --git a/agent/verification/images/postgres/Dockerfile b/agent/verification/images/postgres/Dockerfileindex 3fd8bc3c..975dadc2 100644--- a/agent/verification/images/postgres/Dockerfile+++ b/agent/verification/images/postgres/Dockerfile@@ -62,8 +62,8 @@ postgresql-pltcl-${PG_MAJOR}"# pass -c shared_preload_libraries, so this value stands at runtime.ARG PRELOAD_LIBRARIES="pg_stat_statements,pg_cron,pg_stat_kcache,pg_qualstats,pg_wait_sampling,pgaudit,set_user"-# Root is the base image's user; the official entrypoint still gosu-drops to the-# postgres user before exec, so the agent's hardened runtime is unchanged.+# The build installs packages as root, the base image's user. At runtime the agent+# starts the container as the postgres user, so nothing in it runs as root.RUN set -eux; \version_specific_extension_packages=""; \version_specific_preload_libraries=""; \diff --git a/agent/verification/internal/features/container/dockerengine.go b/agent/verification/internal/features/container/dockerengine.goindex d634952c..8b7b7aab 100644--- a/agent/verification/internal/features/container/dockerengine.go+++ b/agent/verification/internal/features/container/dockerengine.go@@ -116,7 +116,8 @@ func (e *dockerEngine) CreateNetwork(// only reach the restored DB via the published host port (verifier package// doc). Outbound from PG to the host network is therefore not blocked at// the docker layer; rely on the host firewall + the other hardening- // controls (cap-drop, no-new-privs, pids/memory, ephemeral, userns).+ // controls (unprivileged user, no capabilities, pids/memory, ephemeral,+ // userns).resp, err := e.cli.NetworkCreate(ctx, name, client.NetworkCreateOptions{Driver: "bridge",Labels: labels,@@ -151,11 +152,6 @@ func (e *dockerEngine) CreateContainer(ctx context.Context, spec SpawnSpec) (str// toContainerConfigs is the 1:1 translation of a SpawnSpec into the engine's// create request; a unit test pins the hardening it carries.func toContainerConfigs(spec SpawnSpec) (*container.Config, *container.HostConfig) {- securityOpt := []string{}- if spec.NoNewPrivileges {- securityOpt = append(securityOpt, "no-new-privileges")- }-capDrop := []string{}if spec.CapDropAll {capDrop = append(capDrop, "ALL")@@ -168,9 +164,7 @@ func toContainerConfigs(spec SpawnSpec) (*container.Config, *container.HostConfi}hostConfig := &container.HostConfig{- SecurityOpt: securityOpt,CapDrop: capDrop,- CapAdd: spec.CapAdd,ReadonlyRootfs: spec.ReadonlyRootfs,NetworkMode: container.NetworkMode(spec.NetworkID),PortBindings: network.PortMap{@@ -187,6 +181,7 @@ func toContainerConfigs(spec SpawnSpec) (*container.Config, *container.HostConficonfig := &container.Config{Image: spec.Image,+ User: spec.User,Cmd: spec.Cmd,Env: spec.Env,Labels: spec.Labels,@@ -229,13 +224,7 @@ func (e *dockerEngine) Exec(stdin io.Reader,env []string,) (restore.ExecResult, error) {- created, err := e.cli.ExecCreate(ctx, containerID, client.ExecCreateOptions{- Cmd: cmd,- Env: env,- AttachStdin: stdin != nil,- AttachStdout: true,- AttachStderr: true,- })+ created, err := e.cli.ExecCreate(ctx, containerID, toExecOptions(cmd, env, stdin != nil))if err != nil {return restore.ExecResult{}, fmt.Errorf("exec create: %w", err)}@@ -284,6 +273,19 @@ func (e *dockerEngine) Exec(}, nil}+// toExecOptions builds the engine's exec request. It names no user, so the+// command runs as the container's user, the same unprivileged user the+// container starts as; a unit test pins that.+func toExecOptions(cmd, env []string, isStdinAttached bool) client.ExecCreateOptions {+ return client.ExecCreateOptions{+ Cmd: cmd,+ Env: env,+ AttachStdin: isStdinAttached,+ AttachStdout: true,+ AttachStderr: true,+ }+}+func (e *dockerEngine) RemoveContainer(ctx context.Context, containerID string) error {_, err := e.cli.ContainerRemove(ctx, containerID, client.ContainerRemoveOptions{Force: true,diff --git a/agent/verification/internal/features/container/dockerengine_test.go b/agent/verification/internal/features/container/dockerengine_test.goindex 04788281..c907b466 100644--- a/agent/verification/internal/features/container/dockerengine_test.go+++ b/agent/verification/internal/features/container/dockerengine_test.go@@ -89,8 +89,8 @@ func Test_ToContainerConfigs_CarriesHardeningAndLimitsOfSpec(t *testing.T) {config, hostConfig := toContainerConfigs(spec)assert.Equal(t, []string{"ALL"}, hostConfig.CapDrop)- assert.Equal(t, minimalCaps, hostConfig.CapAdd)- assert.Contains(t, hostConfig.SecurityOpt, "no-new-privileges")+ assert.Empty(t, hostConfig.CapAdd)+ assert.Empty(t, hostConfig.SecurityOpt)assert.Empty(t, hostConfig.Binds)assert.Empty(t, hostConfig.Mounts)assert.Equal(t, int64(1024)*1024*1024, hostConfig.Memory)@@ -107,6 +107,33 @@ func Test_ToContainerConfigs_CarriesHardeningAndLimitsOfSpec(t *testing.T) {assert.Equal(t, network.PortSet{pgPublishedPort: struct{}{}}, config.ExposedPorts)assert.Equal(t, spec.Image, config.Image)assert.Equal(t, spec.Labels, config.Labels)+ assert.Equal(t, "postgres", config.User)+ })+ }+}++func Test_ToExecOptions_CarriesCommandAndNamesNoUser(t *testing.T) {+ testCases := []struct {+ name string+ isStdinAttached bool+ }{+ {name: "with stdin", isStdinAttached: true},+ {name: "without stdin", isStdinAttached: false},+ }++ for _, testCase := range testCases {+ t.Run(testCase.name, func(t *testing.T) {+ cmd := []string{"pg_restore", "--dbname", "restore"}+ env := []string{"PGPASSWORD=pw"}++ options := toExecOptions(cmd, env, testCase.isStdinAttached)++ assert.Equal(t, cmd, options.Cmd)+ assert.Equal(t, env, options.Env)+ assert.Equal(t, testCase.isStdinAttached, options.AttachStdin)+ assert.True(t, options.AttachStdout)+ assert.True(t, options.AttachStderr)+ assert.Empty(t, options.User, "the command runs as the container's unprivileged user")})}}diff --git a/agent/verification/internal/features/container/manager.go b/agent/verification/internal/features/container/manager.goindex 7c7eeb1a..f9054617 100644--- a/agent/verification/internal/features/container/manager.go+++ b/agent/verification/internal/features/container/manager.go@@ -29,10 +29,16 @@ const (timescaleImageRepo = "timescale/timescaledb")-// minimalCaps are the only capabilities added back after CapDrop ALL — exactly-// what the official entrypoint needs for initdb + the gosu privilege drop, not-// a blanket keep.-var minimalCaps = []string{"CHOWN", "DAC_OVERRIDE", "FOWNER", "SETGID", "SETUID"}+// containerUser is the unprivileged user every Postgres image the agent runs+// ships. Starting as this user makes the official entrypoint skip its switch+// from root, so no step in the container needs a capability and all of them+// are dropped. It replaces the kernel's no_new_privs flag: on rootful Podman+// with crun and AppArmor that flag stacks the container's AppArmor profile on+// crun's, which denies every signal between the container's processes and+// hangs Postgres.+// With an empty bounding set, a setuid binary still yields no capability.+// The agent's execs (dd, pg_restore, du) run as this user too.+const containerUser = "postgres"// restoreTunedPostgresCmd runs the restore container's Postgres with crash// safety traded for a smaller on-disk footprint. The container is destroyed@@ -206,13 +212,12 @@ func (m *Manager) buildSpec(plan spawnPlan) SpawnSpec {PidsLimit: containerPidsLimit,NetworkID: plan.networkID,- NoNewPrivileges: true,- CapDropAll: true,- CapAdd: minimalCaps,+ CapDropAll: true,+ User: containerUser,// rootfs is writable because the official image needs a writable PGDATA;// the per-job disk budget is enforced by the agent's disk watcher, not a- // Docker cap. The dominant controls (cap-drop, no-new-privs, pids,- // memory, userns, ephemeral lifecycle) remain in force. The job network+ // Docker cap. The dominant controls (unprivileged user, no capabilities,+ // pids, memory, userns, ephemeral lifecycle) remain in force. The job network// is a per-job user-defined bridge (job isolation) but NOT --internal:// see CreateNetwork in dockerengine.go for why.ReadonlyRootfs: false,diff --git a/agent/verification/internal/features/container/postgres_test.go b/agent/verification/internal/features/container/postgres_test.goindex 64b6075f..bafa8baf 100644--- a/agent/verification/internal/features/container/postgres_test.go+++ b/agent/verification/internal/features/container/postgres_test.go@@ -15,9 +15,8 @@ func Test_BuildSpec_AppliesHardeningControls(t *testing.T) {spec := containerManager.buildSpec(newTestSpawnPlan())- assert.True(t, spec.NoNewPrivileges)assert.True(t, spec.CapDropAll)- assert.Equal(t, minimalCaps, spec.CapAdd)+ assert.Equal(t, "postgres", spec.User, "the container starts unprivileged, so the entrypoint never switches users")assert.EqualValues(t, containerPidsLimit, spec.PidsLimit)assert.Equal(t, "net-id", spec.NetworkID)assert.Equal(t, int64(2)*1_000_000_000, spec.NanoCPUs)@@ -34,9 +33,8 @@ func Test_BuildSpec_WhenCpuLimitUnsupported_OmitsCpuLimitAndKeepsOtherControls(tassert.Zero(t, spec.NanoCPUs, "Docker treats 0 as no CPU limit")assert.Equal(t, int64(1024)*1024*1024, spec.MemoryBytes)assert.EqualValues(t, containerPidsLimit, spec.PidsLimit)- assert.True(t, spec.NoNewPrivileges)assert.True(t, spec.CapDropAll)- assert.Equal(t, minimalCaps, spec.CapAdd)+ assert.Equal(t, "postgres", spec.User)}func Test_BuildSpec_SetsRestoreTunedPostgresCmd(t *testing.T) {diff --git a/agent/verification/internal/features/container/spawnspec.go b/agent/verification/internal/features/container/spawnspec.goindex f5ba02f7..445464c6 100644--- a/agent/verification/internal/features/container/spawnspec.go+++ b/agent/verification/internal/features/container/spawnspec.go@@ -37,8 +37,7 @@ type SpawnSpec struct {// The hardening invariants, explicit so a unit test asserts them and the// engine translation cannot silently drop one.- NoNewPrivileges bool- CapDropAll bool- CapAdd []string- ReadonlyRootfs bool+ CapDropAll bool+ ReadonlyRootfs bool+ User string}diff --git a/openspec/changes/archive/2026-10-22-start-restore-containers-unprivileged/.openspec.yaml b/openspec/changes/archive/2026-10-22-start-restore-containers-unprivileged/.openspec.yamlnew file mode 100644index 00000000..7cc41bb5--- /dev/null+++ b/openspec/changes/archive/2026-10-22-start-restore-containers-unprivileged/.openspec.yaml@@ -0,0 +1,2 @@+schema: spec-driven+created: 2026-10-22diff --git a/openspec/changes/archive/2026-10-22-start-restore-containers-unprivileged/design.md b/openspec/changes/archive/2026-10-22-start-restore-containers-unprivileged/design.mdnew file mode 100644index 00000000..be23333e--- /dev/null+++ b/openspec/changes/archive/2026-10-22-start-restore-containers-unprivileged/design.md@@ -0,0 +1,88 @@+## Context++See proposal.md for why. The relevant current state, on the branch `fix/podman-port-diagnostics`:++- `Manager.buildSpec` requests every restore container with `NoNewPrivileges: true`, `CapDropAll: true`, `CapAdd: minimalCaps` and `User: containerUser` (`agent/verification/internal/features/container/manager.go:215-218`). `minimalCaps` holds `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `SETGID` and `SETUID` (`manager.go:34`), the set the official entrypoint needs to prepare the data directory and switch from root with `gosu`. `containerUser` is `postgres` (`manager.go:41`); it was added on this branch (commit a8fe37a6) as a first attempt and did not fix CI by itself.+- `toContainerConfigs` turns `NoNewPrivileges` into `SecurityOpt: ["no-new-privileges"]` and `CapDropAll` into `CapDrop: ["ALL"]`, and copies `CapAdd` and `User` (`dockerengine.go:155-190`). `SpawnSpec` carries the fields (`spawnspec.go:40-43`). The unit tests pin all of it (`postgres_test.go:18-21`, `:38-40`, `dockerengine_test.go:92-93`).+- `dockerEngine.Exec` sets no user (`dockerengine.go`, `Exec`), so the agent's commands in the container (`dd` writing the dump to `/tmp`, `pg_restore`, `du`) run as the container's user.+- `test-container-privileges.sh` reads `/proc/1/status` of a running restore container and requires `CapBnd` `00000000000000cb` and `NoNewPrivs` `1` (`agent/verification/e2e/scripts/test-container-privileges.sh:19`, `:66-69`).+- The diagnostics behind the root cause are temporary: `agent/verification/e2e/diagnose-podman-ports.sh`, the step `Diagnose Podman port publishing` in `e2e-verification-agent-podman` (`.github/workflows/ci-release.yml:265-267`), the workflow `.github/workflows/debug-podman.yml`, which runs only that job on the debugging branch, and the exclusion of that branch from the push trigger of `ci-release.yml` (`ci-release.yml:6`).+- `waitForReady` reports the last ping error on a readiness timeout (`manager.go:239`). That belongs to `run-verification-on-podman` (its task 6.1) and stays.+- `run-verification-on-podman` is not archived. Its section 6 tracks the Podman CI failure with two open tasks, 6.3 (fix the cause) and 6.4 (remove the diagnostics), and its delta spec adds the two requirements this change modifies.++Diagnostic runs on `ubuntu-24.04` with Podman 4.9.3, `crun` 1.14.1 and AppArmor enabled (runs 39831270927 and 39835622435) showed:++| Container request | Postgres came up |+|---|---|+| root start, no-new-privileges (the agent before this branch) | no: `gosu` spins in `sched_yield` |+| `postgres` start, no-new-privileges (the branch now) | no: a backend sleeps in `authentication` forever |+| root start, no-new-privileges, AppArmor unconfined | yes |+| root start, no-new-privileges, seccomp unconfined | no |+| root start, no no-new-privileges | yes |+| `postgres` start, no no-new-privileges | yes |++The kernel log held AppArmor `DENIED` signal lines for `gosu` (`rtmin+1`, glibc's thread signal for `setuid`) and for `postgres` (`urg`, `usr1`, `term`), each with the peer label `containers-default-0.57.4-apparmor1//&crun`.++Constraints come from `agent/verification/AGENTS.md`: pure translation functions get unit tests, engine-dependent behavior is covered in `e2e/`, and tests mirror the source split.++## Goals / Non-Goals++**Goals:**++- Verification completes on rootful Podman with `crun` on an AppArmor host, and stays green on Docker and on rootless Podman.+- The restore container starts as an unprivileged user with an empty capability bounding set, so even a process that reaches user ID 0 inside it holds no capability.++**Non-Goals:**++- Changing anything on the host: AppArmor profiles, Podman's configuration or `crun`.+- A runtime check of the applied privileges; the end-to-end test keeps that role, as `run-verification-on-podman` decided.++## Decisions++### Start as `postgres` with no capabilities, and drop no-new-privileges++The container request sets `User: postgres`, `CapDrop: ["ALL"]`, no `CapAdd` and no `no-new-privileges`. The official entrypoint skips `gosu` and the ownership fixes when it does not start as root, so `initdb` and the server run as `postgres` from the first instruction. No step in the container needs a capability, so the bounding set can be empty.++no-new-privileges exists to stop a process from gaining privileges through a setuid or file-capability binary. With an empty bounding set, such a binary can still give a process user ID 0, but the kernel grants it no capability, so it cannot pass permission checks, load modules, change namespaces or do anything else root's capabilities allow. What it keeps is ownership of root-owned files inside a container that is deleted after the job. AppArmor, seccomp, the namespaces and the limits stay in force, and they are what stands between the container and the host.++`SpawnSpec` loses `NoNewPrivileges` and `CapAdd`, and `minimalCaps` is deleted, since nothing sets them any more. `CapDropAll` and `User` stay.++Rejected alternatives:++- **Keep no-new-privileges and start as `postgres`.** Tested: Postgres's own processes signal each other, and AppArmor denies those signals under the stacked label too.+- **Request `apparmor=unconfined`.** It fixes the hang by removing the stronger control, and the agent would ask for less confinement than the engine applies by default.+- **Drop no-new-privileges only on Podman hosts with AppArmor.** The agent would have to detect the host's security modules and runtime through the engine API and carry two container configurations. A Docker host configured with `crun` would hit the same stacking without matching the detection.+- **Drop no-new-privileges and keep the five capabilities.** A process that gains user ID 0 through a setuid binary would then hold `DAC_OVERRIDE`, `FOWNER`, `SETUID` and the others inside the container, for no benefit, because nothing needs them once the container starts unprivileged.+- **Ship an AppArmor profile, or tell users to change theirs.** The agent does not own the host's AppArmor configuration, and a manual step per host is what this change avoids.++### The agent's commands run as `postgres` too++`Exec` keeps not setting a user, so `dd`, `pg_restore` and `du` run as the container's user. `dd` writes the dump to `/tmp`, which any user may write. `pg_restore` connects over the local socket as the database superuser, which does not depend on the operating-system user. `du` reads the data directory and `/tmp`, both owned by `postgres`. The full Docker suite passed locally with this on the branch, TimescaleDB and the disk-budget test included.++The exec request moves into a pure function, as the container request did in `run-verification-on-podman`, and a unit test pins that it names no user.++Rejected alternative: run execs as root through the exec request. Root without capabilities gains nothing for these commands, and it would put a root process back into the container.++### Check the new privileges end to end++`test-container-privileges.sh` keeps its approach: it holds the backup stream, reads `/proc/1/status` of the running container and then lets the job finish. It now requires the first `Uid` field to be non-zero and `CapBnd` to be `0000000000000000`. It no longer checks `NoNewPrivs`. The unit tests assert that the request sets the user, drops all capabilities, adds none and sets no security options.++### Verify on CI as part of the tasks, then restore the CI files++The failure needs rootful Podman with `crun` on an AppArmor host, which no local check reproduces: the development host runs Fedora with SELinux and rootless Podman. The user asked for the CI run to be part of the tasks, which overrides the project's default of keeping CI runs out of tasks. The apply phase therefore pushes to `fix/podman-port-diagnostics`, where `debug-podman.yml` runs only the Podman job, and reads its result with `gh`. Once that job is green, it removes the diagnostics, restores `ci-release.yml` to its state before the debugging, deletes `debug-podman.yml`, pushes again and checks that the full `CI and Release` workflow passes on the branch.++### Hand over section 6 of `run-verification-on-podman`++Task 6.3 of that change is the fix this change makes, and task 6.4 is its cleanup. Both are replaced by a line that points here, and its risk entry about the red Podman job is updated the same way. `run-verification-on-podman` keeps its delta as it was proposed, with five capabilities and no-new-privileges, and this change's MODIFIED delta replaces that wording. So `run-verification-on-podman` is archived first and this change right after it, in the same step, so the main spec never ships the superseded wording.++## Risks / Trade-offs++- [An image set through `verification-pg-image-repo` without a `postgres` user, or with an entrypoint that needs root] → Container creation or startup fails, and the job is reported as a setup failure with the engine's error. Every image built on the official Postgres image works, and the proposal states the expectation.+- [The data directory in the anonymous volume is not writable by `postgres` on some engine] → The engines copy the image directory's ownership into a new anonymous volume, and the official image creates it owned by `postgres`. The end-to-end tests cover Docker and rootful Podman, and rootless Podman passed locally.+- [The TimescaleDB image is Alpine-based, with a `postgres` user of a different ID] → The user is resolved by name inside the image; the TimescaleDB end-to-end test already passed on Docker with the user switch.+- [Without no-new-privileges, a flaw in a setuid binary of the image gives user ID 0 inside the container] → It comes with no capabilities, inside an ephemeral container under AppArmor, seccomp and namespaces.+- [The diagnosis rests on `ubuntu-24.04`; other AppArmor hosts may stack labels differently] → The change removes the dependency on signals crossing a stacked label altogether, rather than matching one host's labels.++## Migration Plan++None. The change ships with the next agent release; agents on Docker keep working, and agents on rootful Podman with AppArmor start completing verifications. Rollback is reverting the commit; the agent stores no state this change touches.diff --git a/openspec/changes/archive/2026-10-22-start-restore-containers-unprivileged/proposal.md b/openspec/changes/archive/2026-10-22-start-restore-containers-unprivileged/proposal.mdnew file mode 100644index 00000000..94552cad--- /dev/null+++ b/openspec/changes/archive/2026-10-22-start-restore-containers-unprivileged/proposal.md@@ -0,0 +1,42 @@+## Why++On Ubuntu hosts where rootful Podman runs containers through `crun`, no verification ever completes: the restore container starts, but Postgres in it never answers. The CI job `e2e-verification-agent-podman` (Podman 4.9.3, `crun` 1.14.1, `ubuntu-24.04`) fails all its tests for this reason, and it is most likely the hang the reporter of issue #833 saw.++The cause is the interaction of no-new-privileges with AppArmor. Under no-new-privileges, AppArmor cannot switch a process from `crun`'s profile to the container profile on exec, so it stacks the two, and the container's processes run under the label `containers-default-0.57.4-apparmor1//&crun`. The container profile allows signals only to its exact own label, so the kernel denies every signal between the container's processes. The official entrypoint's `gosu` then spins forever waiting for its threads to acknowledge a user switch, and when the container starts unprivileged, Postgres backends never wake from their latches. CI showed it directly: AppArmor `DENIED` lines for `gosu` and `postgres` signals, and every variant with no-new-privileges hung while the same container without the flag, or with AppArmor off, came up.++## What Changes++- Restore containers start as the image's unprivileged `postgres` user instead of root. The official entrypoint then never switches users, and the server, `initdb` and the agent's own commands inside the container (writing the dump, `pg_restore`, disk usage) all run as that user.+- Restore containers get no Linux capabilities at all. Today all are dropped and five are added back (`CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `SETGID`, `SETUID`) because the entrypoint needed them to switch from root; starting as `postgres` makes them unnecessary.+- The agent stops requesting no-new-privileges for restore containers. With no capabilities left in the bounding set, a process that exploits a setuid binary in the image gets user ID 0 with no capabilities, so the flag no longer guards a privilege the container could gain. AppArmor, seccomp, namespaces and the memory, process and CPU limits stay as they are.+- The end-to-end privileges test checks the new state: the first process runs as a non-root user and its capability bounding set is empty.+- The temporary Podman diagnostics are removed once CI is green: the diagnostic script, its CI step, the debug-only workflow and the debugging branch's exclusion from the main CI workflow. The improved readiness-timeout message stays.+- The still-open change `run-verification-on-podman` hands its unfinished section on the Podman CI failure over to this change.++No flag, configuration field, API or report field changes. An image configured through `verification-pg-image-repo` now has to ship a `postgres` user whose entrypoint works without root, which every image built on the official Postgres image does, the bundled verification image included.++## Capabilities++### New Capabilities++None.++### Modified Capabilities++- `verification-container-resources`: the reduced-privileges requirement changes from "five capabilities and no-new-privileges" to "starts as an unprivileged user with no capabilities"; the Podman requirement extends to hosts that confine containers with AppArmor; the scenario that lists the controls kept without a CPU limit names the new controls.++## Impact++- Code: `agent/verification/internal/features/container` (the container request and its unit tests), `agent/verification/e2e/scripts/test-container-privileges.sh`, the verification image's Dockerfile comment.+- CI: `.github/workflows/ci-release.yml` returns to its state before the debugging, and `.github/workflows/debug-podman.yml` and `agent/verification/e2e/diagnose-podman-ports.sh` are deleted.+- OpenSpec: `openspec/changes/run-verification-on-podman` closes its section 6 by pointing here.+- Rules this change answers to: `AGENTS.md` and `agent/verification/AGENTS.md`.+- Backend, frontend, the agent-to-backend API and the verification image's contents are untouched.++## Out of scope++- Fixing or working around Ubuntu's AppArmor profiles or Podman's container profile on the host.+- Turning off AppArmor or seccomp for restore containers.+- Rootless Podman in CI, Podman on macOS or Windows, and other container engines.+- Blocking outbound network access from restore containers.+- Running the full end-to-end suite on Podman.diff --git a/openspec/changes/archive/2026-10-22-start-restore-containers-unprivileged/specs/verification-container-resources/spec.md b/openspec/changes/archive/2026-10-22-start-restore-containers-unprivileged/specs/verification-container-resources/spec.mdnew file mode 100644index 00000000..33c5a04a--- /dev/null+++ b/openspec/changes/archive/2026-10-22-start-restore-containers-unprivileged/specs/verification-container-resources/spec.md@@ -0,0 +1,65 @@+## MODIFIED Requirements++### Requirement: Restore containers run with reduced privileges++A restore runs code taken from the backup with database superuser rights. The verification agent SHALL start every restore container as the image's unprivileged database user, SHALL run its own commands inside the container as that user, SHALL leave the container's capability bounding set empty, so that even a process that reaches user ID 0 inside the container holds no capabilities, and SHALL mount no directory of the host into it. The agent SHALL request these settings from the container engine when it creates the container. It SHALL NOT fail a job because the engine describes the applied settings in a different form than the agent requested them.++#### Scenario: Container is created with reduced privileges++- **WHEN** the agent creates a restore container for a verification job+- **THEN** the container engine receives a request that starts the container as the image's unprivileged database user, drops all capabilities, adds none back and mounts no host directory++#### Scenario: Agent's commands run as the unprivileged user++- **WHEN** the agent runs a command inside a restore container, such as writing the backup into it, restoring it or measuring its disk usage+- **THEN** the agent's request names no user, so the command runs as the container's unprivileged database user++#### Scenario: Processes in the container hold no privileges++- **WHEN** a restore container created by the agent is running+- **THEN** its first process runs as a non-root user+- **AND** the capability bounding set of its processes is empty, so a process that gains user ID 0 inside the container still holds no capabilities++#### Scenario: Engine reports dropped capabilities as an explicit list++- **WHEN** the container engine reports the dropped capabilities of a restore container as a list of capability names instead of `ALL`+- **THEN** the agent proceeds with the restore and the verification reaches its normal terminal status++### Requirement: Verification runs on Podman++The verification agent SHALL verify backups when its Docker endpoint is a rootful Podman engine reached through Podman's Docker-compatible API, including on a host that confines containers with AppArmor. A restore of a valid backup SHALL complete, a restore of a broken backup SHALL fail with the restore error, and in both cases the agent SHALL remove the containers and networks it created for the job.++#### Scenario: Valid backup on Podman++- **WHEN** the agent's Docker endpoint is a rootful Podman engine+- **AND** the agent claims a verification job for a valid PostgreSQL backup+- **THEN** the agent reports the verification as completed with the restored database's statistics+- **AND** no container or network created for the job remains on the engine++#### Scenario: Broken backup on Podman++- **WHEN** the agent's Docker endpoint is a rootful Podman engine+- **AND** the agent claims a verification job for a backup that cannot be restored+- **THEN** the agent reports the verification as failed with the restore error+- **AND** no container or network created for the job remains on the engine++#### Scenario: Podman on a host with AppArmor++- **WHEN** the agent's Docker endpoint is a rootful Podman engine on a host where AppArmor confines containers and the container runtime has its own AppArmor profile+- **AND** the agent claims a verification job for a valid PostgreSQL backup+- **THEN** the database in the restore container starts and accepts the agent's connection+- **AND** the agent reports the verification as completed++### Requirement: A missing CPU limit is announced and leaves other isolation intact++A restore runs code taken from the backup with database superuser rights, so a restore container without a CPU limit can use every CPU core on the host. The verification agent SHALL log a warning at startup when it will run restore containers without a CPU limit, and SHALL keep every other container isolation control in force.++#### Scenario: Startup warning on an unsupported host++- **WHEN** the Docker host reports at agent startup that it cannot enforce a CPU limit+- **THEN** the agent logs a warning that restore containers will run without a CPU limit and can use every CPU core on the host++#### Scenario: Other limits remain applied++- **WHEN** the agent creates a restore container without a CPU limit+- **THEN** the container still has the per-job memory limit, the process count limit, no capabilities, and runs as the image's unprivileged database userdiff --git a/openspec/changes/archive/2026-10-22-start-restore-containers-unprivileged/tasks.md b/openspec/changes/archive/2026-10-22-start-restore-containers-unprivileged/tasks.mdnew file mode 100644index 00000000..56d1fe6f--- /dev/null+++ b/openspec/changes/archive/2026-10-22-start-restore-containers-unprivileged/tasks.md@@ -0,0 +1,32 @@+## 1. Request an unprivileged container++- [x] 1.1 In `agent/verification/internal/features/container/postgres_test.go` and `dockerengine_test.go`, change the hardening assertions to the new request: the spec and the engine config start as `postgres`, drop all capabilities, add none (`CapAdd` empty) and carry no security options (`SecurityOpt` empty). Run `just test` in `agent/verification` and see these tests fail on the current code for those reasons.+- [x] 1.2 In `agent/verification/internal/features/container/dockerengine.go`, move the construction of the exec request in `dockerEngine.Exec` into a pure function, as `toContainerConfigs` does for the container request, and add a test in `dockerengine_test.go` that asserts the request carries the command, the environment and the stdin attachment and names no user, so the agent's commands run as the container's user. Verify with `just test` in `agent/verification`.+- [x] 1.3 Delete `NoNewPrivileges` and `CapAdd` from `SpawnSpec`, `minimalCaps` from `manager.go`, and their use in `Manager.buildSpec` and `toContainerConfigs`, keeping `CapDropAll` and `User`. Rewrite the comments on `containerUser`, on `buildSpec`'s hardening and on `CreateNetwork` so they name the controls that remain and say why the container starts as `postgres` (the AppArmor signal denial under no-new-privileges, per design.md). Verify that `just test` and `just lint` in `agent/verification` pass and that `grep -rn "minimalCaps\|NoNewPrivileges\|no-new-priv" agent/verification --include=*.go` finds nothing.++## 2. Check the applied privileges end to end++- [x] 2.1 In `agent/verification/e2e/scripts/test-container-privileges.sh`, require the first `Uid` field of `/proc/1/status` to be non-zero and `CapBnd` to be `0000000000000000`, drop the `NoNewPrivs` check, and update the header comment and the failure and success messages, which no longer mention no-new-privileges. Verify that `just e2e` in `agent/verification` reports every test passed, and that `E2E_ENGINE_SOCKET=$XDG_RUNTIME_DIR/podman/podman.sock just e2e-podman` does too.++## 3. Confirm the fix on rootful Podman in CI++- [x] 3.1 Commit the work so far and push the branch `fix/podman-port-diagnostics`, which runs only `Debug Podman e2e`. The user approved this commit and push for this task: the failure reproduces only on rootful Podman with AppArmor, and the user asked for the CI check inside the tasks (design.md, "Verify on CI as part of the tasks"). Then wait for the run with `gh run watch <run-id> --exit-status`. The step `Run e2e tests on Podman` must report `Results: 3 passed, 0 failed`. If it fails, read the run's log with `gh run view <run-id> --log`, fix the cause in groups 1 and 2, and repeat this task.++## 4. Remove the temporary diagnostics and restore CI++- [x] 4.1 Delete `agent/verification/e2e/diagnose-podman-ports.sh` and `.github/workflows/debug-podman.yml`, remove the step `Diagnose Podman port publishing` from `e2e-verification-agent-podman`, and restore the push trigger of `.github/workflows/ci-release.yml` to `branches: ["**"]` without the temporary comment. Verify that `git diff origin/develop -- .github/workflows/ci-release.yml` prints nothing, that `actionlint .github/workflows/ci-release.yml` reports no findings, and that `grep -rn "diagnose-podman-ports\|debug-podman\|podman-port-diagnostics" . --exclude-dir=.git --exclude-dir=openspec` finds nothing.+- [x] 4.2 Commit the cleanup and push the branch, which now runs the full `CI and Release` workflow; the user approved this commit and push for the same reason as in 3.1. Wait for the run with `gh run watch <run-id> --exit-status`. Every job that runs on a branch push must pass, `e2e-verification-agent` and `e2e-verification-agent-podman` included.++## 5. Hand over the open section of `run-verification-on-podman`++- [x] 5.1 In `openspec/changes/run-verification-on-podman/tasks.md`, replace the open tasks 6.3 and 6.4 with one line saying that `start-restore-containers-unprivileged` fixes the cause and removes the diagnostics, so that change has no open task. In its `design.md`, update the risk entry about the red Podman job and the paragraph on the temporary diagnostic step the same way. Verify that `openspec validate run-verification-on-podman --strict` passes and that `openspec list` shows all of its tasks complete.++## 6. Final checks++- [x] 6.1 Run `just lint` at the repository root and fix every finding. Verify that `grep -rn 'NoNewPrivileges\|NoNewPrivs\|"no-new-privileges"' agent/verification --exclude-dir=artifacts` finds nothing, so neither the code, the unit tests nor the end-to-end test still sets or checks the flag, and that `openspec validate start-restore-containers-unprivileged --strict` passes.++## Change verification++```sh+just --justfile agent/verification/Justfile test+```diff --git a/openspec/specs/verification-container-resources/spec.md b/openspec/specs/verification-container-resources/spec.mdindex 78638031..01e2f7e0 100644--- a/openspec/specs/verification-container-resources/spec.md+++ b/openspec/specs/verification-container-resources/spec.md@@ -48,22 +48,27 @@ A restore runs code taken from the backup with database superuser rights, so a r#### Scenario: Other limits remain applied- **WHEN** the agent creates a restore container without a CPU limit-- **THEN** the container still has the per-job memory limit, the process count limit, all capabilities dropped except the minimal set, and privilege escalation disabled+- **THEN** the container still has the per-job memory limit, the process count limit, no capabilities, and runs as the image's unprivileged database user### Requirement: Restore containers run with reduced privileges-A restore runs code taken from the backup with database superuser rights. The verification agent SHALL create every restore container with all Linux capabilities dropped except `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `SETGID` and `SETUID`, with privilege escalation disabled, and with no directory of the host mounted into it. The agent SHALL request these settings from the container engine when it creates the container. It SHALL NOT fail a job because the engine describes the applied settings in a different form than the agent requested them.+A restore runs code taken from the backup with database superuser rights. The verification agent SHALL start every restore container as the image's unprivileged database user, SHALL run its own commands inside the container as that user, SHALL leave the container's capability bounding set empty, so that even a process that reaches user ID 0 inside the container holds no capabilities, and SHALL mount no directory of the host into it. The agent SHALL request these settings from the container engine when it creates the container. It SHALL NOT fail a job because the engine describes the applied settings in a different form than the agent requested them.#### Scenario: Container is created with reduced privileges- **WHEN** the agent creates a restore container for a verification job-- **THEN** the container engine receives a request that drops all capabilities, adds back only `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `SETGID` and `SETUID`, disables privilege escalation and mounts no host directory+- **THEN** the container engine receives a request that starts the container as the image's unprivileged database user, drops all capabilities, adds none back and mounts no host directory-#### Scenario: Processes in the container hold only the minimal capabilities+#### Scenario: Agent's commands run as the unprivileged user++- **WHEN** the agent runs a command inside a restore container, such as writing the backup into it, restoring it or measuring its disk usage+- **THEN** the agent's request names no user, so the command runs as the container's unprivileged database user++#### Scenario: Processes in the container hold no privileges- **WHEN** a restore container created by the agent is running-- **THEN** the capability bounding set of its processes contains only `CHOWN`, `DAC_OVERRIDE`, `FOWNER`, `SETGID` and `SETUID`-- **AND** its processes cannot gain privileges through setuid binaries+- **THEN** its first process runs as a non-root user+- **AND** the capability bounding set of its processes is empty, so a process that gains user ID 0 inside the container still holds no capabilities#### Scenario: Engine reports dropped capabilities as an explicit list@@ -72,7 +77,7 @@ A restore runs code taken from the backup with database superuser rights. The ve### Requirement: Verification runs on Podman-The verification agent SHALL verify backups when its Docker endpoint is a rootful Podman engine reached through Podman's Docker-compatible API. A restore of a valid backup SHALL complete, a restore of a broken backup SHALL fail with the restore error, and in both cases the agent SHALL remove the containers and networks it created for the job.+The verification agent SHALL verify backups when its Docker endpoint is a rootful Podman engine reached through Podman's Docker-compatible API, including on a host that confines containers with AppArmor. A restore of a valid backup SHALL complete, a restore of a broken backup SHALL fail with the restore error, and in both cases the agent SHALL remove the containers and networks it created for the job.#### Scenario: Valid backup on Podman@@ -88,6 +93,13 @@ The verification agent SHALL verify backups when its Docker endpoint is a rootfu- **THEN** the agent reports the verification as failed with the restore error- **AND** no container or network created for the job remains on the engine+#### Scenario: Podman on a host with AppArmor++- **WHEN** the agent's Docker endpoint is a rootful Podman engine on a host where AppArmor confines containers and the container runtime has its own AppArmor profile+- **AND** the agent claims a verification job for a valid PostgreSQL backup+- **THEN** the database in the restore container starts and accepts the agent's connection+- **AND** the agent reports the verification as completed+### Requirement: An unreachable restore container is reported with its causeWhen the agent cannot connect to the database in a restore container before the readiness timeout, the verification SHALL fail, and its failure message SHALL include the error of the agent's last connection attempt that ended before the timeout, such as a refused connection, a connection that got no answer or a rejected login. The failure message SHALL NOT contain the database password.