Skip to content

PostgreSQL backup tool with Point-In-Time-Recovery and restore verification

FEATURE (ci): Run backend tests alongside the lint as three parallel groups, physical, logical and the rest, and gate the release on the lint directly, so a pull request gets its backend test result without waiting for the lint and a single 4-vCPU runner

Rostislav Dugin120848fc6e8e
diff --git a/.github/workflows/ci-release.yml b/.github/workflows/ci-release.yml
index 32058da1..594c0359 100644
--- a/.github/workflows/ci-release.yml
+++ b/.github/workflows/ci-release.yml
@@ -245,7 +245,10 @@ jobs:
# (SERVER-121912, SERVER-136848).
runs-on: ubuntu-22.04
timeout-minutes: 90
- needs: [lint-backend]
+ strategy:
+ fail-fast: false
+ matrix:
+ group: [physical, logical, other]
steps:
- name: Free disk space for database images
run: |
@@ -345,11 +348,15 @@ jobs:
# Each package runs in its own metadata DB, so packages run in parallel.
# -p must equal the provisioned worker count.
TEST_PARALLEL_WORKERS: "8"
+ TEST_GROUP: ${{ matrix.group }}
run: |
cd backend
- # 8 workers oversubscribe the runner's 4 vCPU, so the suite needs more wall-clock than
- # the 15m scripts/test.sh allows on a developer machine.
- go test -p=8 -count=1 -failfast -timeout 40m ./internal/...
+ # A separate assignment lets a failing recipe stop the step instead of running no packages.
+ packages=$(just test-group-packages "$TEST_GROUP")
+ # Each leg runs one group on its own runner. 8 workers oversubscribe its 4 vCPU, so a
+ # group needs more wall-clock than the 15m scripts/test.sh allows on a developer machine.
+ # shellcheck disable=SC2086 # one import path per word
+ go test -p=8 -count=1 -failfast -timeout 40m $packages
test-filesystems:
runs-on: ubuntu-24.04
@@ -777,7 +784,7 @@ jobs:
determine-version:
runs-on: ubuntu-24.04
timeout-minutes: 10
- needs: [test-backend, test-frontend, test-verification-agent, e2e-verification-agent, test-filesystems]
+ needs: [lint-backend, test-backend, test-frontend, test-verification-agent, e2e-verification-agent, test-filesystems]
if: ${{ github.ref == 'refs/heads/main' && !contains(github.event.head_commit.message, '[skip-release]') }}
outputs:
should_release: ${{ steps.version_bump.outputs.should_release }}
diff --git a/backend/Justfile b/backend/Justfile
index 24a1bf45..ec695bcc 100644
--- a/backend/Justfile
+++ b/backend/Justfile
@@ -17,6 +17,18 @@ test-package +packages:
apply_libedit_shim
go test -count=1 {{ packages }}
+# CI runs one group per runner, so the three groups must cover every package exactly once. The
+# other group drops only the physical and logical subtrees, so a new package lands in it.
+test-group-packages group:
+ #!/usr/bin/env bash
+ set -euo pipefail
+ case "{{ group }}" in
+ physical) go list ./internal/features/tests/physical/... ;;
+ logical) go list ./internal/features/tests/logical/... ;;
+ other) go list ./internal/... | grep -Ev '/internal/features/tests/(physical|logical)(/|$)' ;;
+ *) echo "unknown test group: {{ group }} (expected physical, logical or other)" >&2; exit 1 ;;
+ esac
+
clean-testcontainers:
#!/usr/bin/env bash
set -euo pipefail
diff --git a/openspec/changes/archive/2026-10-21-parallelize-backend-ci-tests/.openspec.yaml b/openspec/changes/archive/2026-10-21-parallelize-backend-ci-tests/.openspec.yaml
new file mode 100644
index 00000000..dc36fb1b
--- /dev/null
+++ b/openspec/changes/archive/2026-10-21-parallelize-backend-ci-tests/.openspec.yaml
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-10-21
diff --git a/openspec/changes/archive/2026-10-21-parallelize-backend-ci-tests/design.md b/openspec/changes/archive/2026-10-21-parallelize-backend-ci-tests/design.md
new file mode 100644
index 00000000..9852a917
--- /dev/null
+++ b/openspec/changes/archive/2026-10-21-parallelize-backend-ci-tests/design.md
@@ -0,0 +1,81 @@
+## Context
+
+See `proposal.md` for the motivation and `specs/ci-pipeline/spec.md` for the required behavior.
+
+`test-backend` waits for `lint-backend` (`.github/workflows/ci-release.yml:248`) and runs the whole backend suite in one step: `go test -p=8 -count=1 -failfast -timeout 40m ./internal/...` (`.github/workflows/ci-release.yml:352`). The comment above that command (`.github/workflows/ci-release.yml:350-351`) notes that 8 workers oversubscribe the runner's 4 vCPU. Before the tests, the job frees disk space, starts the compose fixtures, provisions and migrates 8 slot databases and pulls every test image (`.github/workflows/ci-release.yml:250-339`).
+
+In the last green run of the full suite (run `38079240506`), `lint-backend` took 4:09 and `test-backend` took 13:45. Of those 13:45, setup took about 4:25 and `go test` took 9:16. The 88 packages with tests reported a combined 2635 seconds:
+
+| Group | Packages with tests | Combined time | Longest package |
+|---|---|---|---|
+| `physical` (`internal/features/tests/physical/...`) | 2 | 514 s | `pg17`, 318 s |
+| `logical` (`internal/features/tests/logical/...`) | 4 | 511 s | `postgresql`, 181 s |
+| `other` | 82 | 1610 s | `backups/backups/usecases/physical/postgresql`, 366 s |
+
+These times were measured under the CPU contention described above, so each package should run faster on a less loaded runner. That speed-up is an expectation, not a measurement.
+
+The publishing jobs gate on `determine-version` (`.github/workflows/ci-release.yml:780`), which needs every test job but not `lint-backend`. The main image reaches the lint through `build-image` (`.github/workflows/ci-release.yml:620`). The verification image reaches it only through `test-backend`: `push-verification-image` needs `determine-version` and `build-verification-image` (`.github/workflows/ci-release.yml:982`), and `build-verification-image` needs only the agent lint and the Dockerfile scan (`.github/workflows/ci-release.yml:754`).
+
+Constraints: the root `AGENTS.md` requires every new action to be pinned to a full commit SHA and keeps `permissions: contents: read` at the top level. This change adds no action and no permission. `backend/AGENTS.md` governs the new `backend/Justfile` recipe. ADR-0013 boots one shared container per database version inside a test process. Splitting the suite across runners keeps that model, and each runner holds fewer containers at once.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- Remove the lint wait from the path to the backend test result.
+- Spread the backend suite over three runners without a hand-maintained list of packages.
+- Keep every publishing path gated on the backend lint and on all three groups.
+
+**Non-Goals:**
+
+- An even balance between the groups. The `other` group stays the longest; see Risks.
+- Faster setup inside a group.
+
+## Decisions
+
+### 1. Gate the release on the lint directly instead of through the tests
+
+`test-backend` drops `needs: [lint-backend]`, and `determine-version` adds `lint-backend` to its `needs`. Every publishing job already needs `determine-version`, and its `if:` conditions use the implicit `success()`, so a failed lint skips all of them. The lint always finishes before the tests, so the extra dependency adds no wall-clock time.
+
+Alternatives rejected:
+
+- Drop the dependency and change nothing else. The verification image would then be published for a commit whose backend lint failed.
+- Add `lint-backend` to `push-verification-image` only. It closes today's gap, but the next publishing job that hangs off `determine-version` would reopen it.
+
+### 2. Split by package path, with `other` as the catch-all
+
+The `physical` group is `./internal/features/tests/physical/...`, and the `logical` group is `./internal/features/tests/logical/...`. The `other` group is `go list ./internal/...` without packages whose import path matches `/internal/features/tests/(physical|logical)(/|$)`. The filter excludes only these two subtrees, not all of `internal/features/tests/`. A new sibling directory under `tests/` therefore lands in `other` instead of being skipped.
+
+Alternatives rejected:
+
+- Split by database engine, with each group pulling only its own images. It saves pull time and disk space, but every new package would need a deliberate assignment, and the proposal leaves per-group images out of scope.
+- Shard automatically, by hash or round-robin over `go list`. Nothing would need maintenance, but the balance would be accidental and would shift with every new package.
+- Exclude all of `internal/features/tests/` from `other`. A new directory there would run in no group.
+
+### 3. One recipe prints a group's packages
+
+A recipe `test-group-packages <group>` in `backend/Justfile` prints the import paths of `physical`, `logical` or `other` and fails for any other name. The CI step passes its output to `go test`. A local check combines the three outputs and compares them with `go list ./internal/...`, which proves that the groups cover every package exactly once.
+
+Alternative rejected: inline the `go list` and `grep` calls in the workflow step. The partition could then be checked only by a CI run, and a typo in a pattern would silently drop packages from the suite.
+
+### 4. A matrix job with `fail-fast: false`
+
+`test-backend` gets `strategy.matrix.group: [physical, logical, other]` and `strategy.fail-fast: false`. The group reaches the shell through an `env` entry rather than a `${{ }}` expression inside the script. Every leg keeps the current setup steps unchanged: 8 slot databases, `-p=8`, `-failfast` and the 40-minute `go test` timeout. GitHub names the checks `test-backend (physical)`, `test-backend (logical)` and `test-backend (other)`, and `needs: test-backend` waits for every leg.
+
+Alternatives rejected:
+
+- Three copies of the job. Each copy would repeat about 100 lines of setup that must then be kept in sync by hand.
+- Keep the default `fail-fast: true`. A failing leg would cancel the others, so a rerun of the failed jobs would repeat the cancelled legs too.
+- A reusable workflow or a composite action for the setup. The matrix already removes the duplication, and both options add a file and an indirection for a single caller.
+
+## Risks / Trade-offs
+
+- [The `other` group stays the long pole, with 1610 of the 2635 seconds and the longest single package] → The gain on the test step is smaller than an even split would give. By rough estimate, the step drops from 9:16 to about 6–7 minutes. The path to a test result drops from about 18 minutes to about 11–12, with most of the gain coming from the removed lint wait. A follow-up can move the heaviest packages of `other` into `physical` once real timings are in.
+- [Each leg repeats the setup of about 4.5 minutes, including the "Free disk space" step, which took 5:40 in one run] → The legs run in parallel, so the duplicated setup costs runner minutes but no wall-clock time. Public repositories do not pay for hosted runners.
+- [Three legs pull about 22 GB of images each, three times the registry traffic of today] → The pull already retries three times per image. A per-group image set is a follow-up.
+- [A lint failure no longer saves the test runners] → Publishing still stops at the lint, through decision 1.
+- [The check name changes from `test-backend` to three names] → No branch protection and no active ruleset requires the old name, so nothing has to be updated.
+
+## Migration Plan
+
+The change lands in one commit. To roll back, revert that commit.
diff --git a/openspec/changes/archive/2026-10-21-parallelize-backend-ci-tests/proposal.md b/openspec/changes/archive/2026-10-21-parallelize-backend-ci-tests/proposal.md
new file mode 100644
index 00000000..8cf6ffa7
--- /dev/null
+++ b/openspec/changes/archive/2026-10-21-parallelize-backend-ci-tests/proposal.md
@@ -0,0 +1,43 @@
+## Why
+
+A pull request into `main` waits about 18 minutes for its backend test result, and nearly all of that time sits on one sequential path. The backend tests start only after the backend lint finishes, which adds about 4 minutes, and then all 138 backend packages run on a single 4-vCPU runner for about 9 minutes. Lint almost never fails in CI, because the pre-commit hooks already run it locally. The suite is CPU-bound on one runner, while the repository is public and extra hosted runners cost nothing.
+
+## Governing docs
+
+This change answers to [`AGENTS.md`](../../../AGENTS.md), including its Security rules for workflows, and to [`backend/AGENTS.md`](../../../backend/AGENTS.md) for the backend test recipe it adds. It changes `.github/workflows/ci-release.yml` and `backend/Justfile`. It changes no Go code, migrations or tests.
+
+## What Changes
+
+- The backend test job no longer waits for the backend lint job. Both start together.
+- The backend test job runs as three parallel groups on separate runners. The `physical` group holds the end-to-end physical backup tests, and the `logical` group holds the end-to-end logical backup tests. The `other` group holds every other backend package, including any new package that does not belong to the first two groups.
+- One `backend/Justfile` recipe prints the package list of a group. CI uses it, and a local check uses it to prove that the three groups cover every package exactly once.
+- A failure in one group no longer cancels the other two, so the developer can rerun only the failed group.
+- Image publishing and the release still require a green backend lint and all three test groups. Today the verification image reaches the lint only through the backend tests, so the release gate gains the lint as a direct dependency.
+- The backend test check appears in GitHub as three checks named `test-backend (physical)`, `test-backend (logical)` and `test-backend (other)` instead of one `test-backend`. No branch protection or active ruleset requires the old name.
+
+## Capabilities
+
+### New Capabilities
+
+- `ci-pipeline`: the order of the backend checks in CI, how the backend suite is split across runners, and which checks gate image publishing and the release.
+
+### Modified Capabilities
+
+None.
+
+## Impact
+
+- `.github/workflows/ci-release.yml`: `test-backend` drops `needs: [lint-backend]` and becomes a matrix job with `fail-fast: false`. Its test step runs the package list of its group. `determine-version` keeps `needs: test-backend`, which now waits for every matrix leg, and adds `lint-backend` to its `needs`.
+- `backend/Justfile`: a new recipe that prints the packages of one test group.
+- Each pull request into `main` and each push to `main` uses two more hosted runners. Every group repeats the full setup of about 4.5 minutes and pulls the full set of test images.
+- The real wall-clock gain can only be measured by a CI run after merge.
+
+## Out of scope
+
+- Flaky backend tests. They are being fixed separately.
+- Skipping lint and tests on pushes to `develop`. That is deliberate and stays.
+- Pulling a smaller image set per group and dropping or shortening the "Free disk space" step.
+- Speeding up the lint job itself, for example by caching `golangci-lint`.
+- Rebalancing the groups beyond the three agreed paths, for example by moving large packages from `other` into `physical`.
+- Adding `actionlint` or any other tool to `mise.toml` or to CI.
+- Changing `scripts/test.sh` or `just test`. The local full suite still runs every package in one process.
diff --git a/openspec/changes/archive/2026-10-21-parallelize-backend-ci-tests/specs/ci-pipeline/spec.md b/openspec/changes/archive/2026-10-21-parallelize-backend-ci-tests/specs/ci-pipeline/spec.md
new file mode 100644
index 00000000..29ac1c85
--- /dev/null
+++ b/openspec/changes/archive/2026-10-21-parallelize-backend-ci-tests/specs/ci-pipeline/spec.md
@@ -0,0 +1,66 @@
+## Purpose
+
+Defines when the backend checks run in CI, how the backend test suite is split across runners, and which checks must pass before an image is published or a release is made.
+
+## ADDED Requirements
+
+### Requirement: Backend tests start without waiting for the backend lint
+
+When CI runs the backend lint and the backend tests for a commit, the backend tests SHALL start without waiting for the lint result. A failed lint SHALL NOT cancel or skip the backend tests.
+
+#### Scenario: Lint and tests start together
+
+- **WHEN** CI starts the backend checks for a pull request
+- **THEN** the backend lint check and every backend test check start without one waiting for the other
+
+#### Scenario: Lint fails
+
+- **WHEN** the backend lint check fails while the backend tests run
+- **THEN** the backend test checks run to completion and report their own results
+
+### Requirement: The backend suite runs as three groups that cover every package once
+
+CI SHALL run the backend test suite as three checks on separate runners, and the checks SHALL run at the same time. The `physical` group SHALL run the end-to-end physical backup tests, and the `logical` group SHALL run the end-to-end logical backup tests. The `other` group SHALL run every remaining backend package. Each backend package SHALL run in exactly one group. A backend package added outside the end-to-end physical and logical backup tests SHALL run in the `other` group without any change to the CI configuration.
+
+#### Scenario: The groups cover the suite
+
+- **WHEN** the package lists of the three groups are combined
+- **THEN** they contain every backend package exactly once
+
+#### Scenario: A new package is added
+
+- **WHEN** a developer adds a backend package outside the end-to-end physical and logical backup tests
+- **THEN** CI runs its tests in the `other` group
+
+#### Scenario: Checks are reported per group
+
+- **WHEN** CI runs the backend tests
+- **THEN** each group reports its own check, named `test-backend (physical)`, `test-backend (logical)` and `test-backend (other)`
+
+### Requirement: A failed group neither cancels nor hides the others
+
+When one backend test group fails, the other groups SHALL run to completion and report their own results. The developer SHALL be able to rerun only the failed group.
+
+#### Scenario: One group fails
+
+- **WHEN** a test in the `other` group fails while the `physical` and `logical` groups still run
+- **THEN** the `physical` and `logical` groups keep running and report their own results
+
+#### Scenario: Rerun of the failed group
+
+- **WHEN** the developer reruns the failed jobs of that workflow run
+- **THEN** only the failed group runs again, and the groups that passed keep their results
+
+### Requirement: Publishing and release require the lint and every test group
+
+CI SHALL publish an image or create a release only when the backend lint and all three backend test groups have passed for that commit. Running the lint and the tests in parallel SHALL NOT let a commit with a failed lint or a failed group reach publishing or release.
+
+#### Scenario: Lint fails on main
+
+- **WHEN** the backend lint fails on a push to `main` while all three test groups pass
+- **THEN** CI publishes no image and creates no release for that commit
+
+#### Scenario: One group fails on main
+
+- **WHEN** the lint and two test groups pass on a push to `main` but the third group fails
+- **THEN** CI publishes no image and creates no release for that commit
diff --git a/openspec/changes/archive/2026-10-21-parallelize-backend-ci-tests/tasks.md b/openspec/changes/archive/2026-10-21-parallelize-backend-ci-tests/tasks.md
new file mode 100644
index 00000000..ed14cf57
--- /dev/null
+++ b/openspec/changes/archive/2026-10-21-parallelize-backend-ci-tests/tasks.md
@@ -0,0 +1,13 @@
+## 1. Package groups
+
+- [x] 1.1 Add a `test-group-packages group` recipe to `backend/Justfile`, following design decisions 2 and 3. For `physical` it prints `go list ./internal/features/tests/physical/...`, and for `logical` it prints `go list ./internal/features/tests/logical/...`. For `other` it prints `go list ./internal/...` without import paths matching `/internal/features/tests/(physical|logical)(/|$)`. Any other name prints an error to stderr and exits non-zero. A comment above the recipe says that CI runs one group per runner. Verify in `backend/` that `diff <(go list ./internal/... | sort) <(for group in physical logical other; do just test-group-packages "$group"; done | sort)` prints nothing and exits `0`, that `just test-group-packages physical` prints only packages under `internal/features/tests/physical/`, that `just test-group-packages unknown` exits non-zero, and that `just --justfile backend/Justfile --fmt --check --unstable` passes at the repository root.
+
+## 2. CI workflow
+
+- [x] 2.1 In `.github/workflows/ci-release.yml`, follow design decision 4. Remove `needs: [lint-backend]` from `test-backend` and add `strategy` with `fail-fast: false` and `matrix.group: [physical, logical, other]`. In the "Run Go tests" step, pass the group through a `TEST_GROUP` `env` entry. After the step's existing `cd backend`, run `go test` with the same flags as today on the output of `just test-group-packages "$TEST_GROUP"`. Update the comment above the command so it still describes the worker count correctly. Verify that `go run github.com/rhysd/actionlint/cmd/[email protected] -shellcheck= .github/workflows/ci-release.yml` passes at the repository root. Then verify the shape of the job with repository searches. `grep -n -A12 '^ test-backend:' .github/workflows/ci-release.yml` shows `fail-fast: false` and `group: [physical, logical, other]` and no `needs` line with `lint-backend`. `grep -nF 'test-group-packages "$TEST_GROUP"' .github/workflows/ci-release.yml` finds the call in the "Run Go tests" step. `grep -n 'go test -p=8.*\./internal/\.\.\.' .github/workflows/ci-release.yml` finds nothing. The pattern keeps the `-p=8` flag so it skips the verification agent's own `go test ./internal/...` call.
+- [x] 2.2 Add `lint-backend` to the `needs` of `determine-version`, following design decision 1. Verify that the actionlint command from 2.1 passes. Also verify that `grep -n -A3 '^ determine-version:' .github/workflows/ci-release.yml` shows a `needs` list that contains both `lint-backend` and `test-backend`.
+
+## 3. Specs and review
+
+- [x] 3.1 Verify that `openspec validate parallelize-backend-ci-tests --strict` passes.
+- [x] 3.2 Run the mandatory full implementation review from `AGENTS.md` and resolve its blocking findings. Then rerun the checks from 1.1 and 2.1.
diff --git a/openspec/specs/ci-pipeline/spec.md b/openspec/specs/ci-pipeline/spec.md
new file mode 100644
index 00000000..9c71a046
--- /dev/null
+++ b/openspec/specs/ci-pipeline/spec.md
@@ -0,0 +1,68 @@
+# CI Pipeline Specification
+
+## Purpose
+
+Defines when the backend checks run in CI, how the backend test suite is split across runners, and which checks must pass before an image is published or a release is made.
+
+## Requirements
+
+### Requirement: Backend tests start without waiting for the backend lint
+
+When CI runs the backend lint and the backend tests for a commit, the backend tests SHALL start without waiting for the lint result. A failed lint SHALL NOT cancel or skip the backend tests.
+
+#### Scenario: Lint and tests start together
+
+- **WHEN** CI starts the backend checks for a pull request
+- **THEN** the backend lint check and every backend test check start without one waiting for the other
+
+#### Scenario: Lint fails
+
+- **WHEN** the backend lint check fails while the backend tests run
+- **THEN** the backend test checks run to completion and report their own results
+
+### Requirement: The backend suite runs as three groups that cover every package once
+
+CI SHALL run the backend test suite as three checks on separate runners, and the checks SHALL run at the same time. The `physical` group SHALL run the end-to-end physical backup tests, and the `logical` group SHALL run the end-to-end logical backup tests. The `other` group SHALL run every remaining backend package. Each backend package SHALL run in exactly one group. A backend package added outside the end-to-end physical and logical backup tests SHALL run in the `other` group without any change to the CI configuration.
+
+#### Scenario: The groups cover the suite
+
+- **WHEN** the package lists of the three groups are combined
+- **THEN** they contain every backend package exactly once
+
+#### Scenario: A new package is added
+
+- **WHEN** a developer adds a backend package outside the end-to-end physical and logical backup tests
+- **THEN** CI runs its tests in the `other` group
+
+#### Scenario: Checks are reported per group
+
+- **WHEN** CI runs the backend tests
+- **THEN** each group reports its own check, named `test-backend (physical)`, `test-backend (logical)` and `test-backend (other)`
+
+### Requirement: A failed group neither cancels nor hides the others
+
+When one backend test group fails, the other groups SHALL run to completion and report their own results. The developer SHALL be able to rerun only the failed group.
+
+#### Scenario: One group fails
+
+- **WHEN** a test in the `other` group fails while the `physical` and `logical` groups still run
+- **THEN** the `physical` and `logical` groups keep running and report their own results
+
+#### Scenario: Rerun of the failed group
+
+- **WHEN** the developer reruns the failed jobs of that workflow run
+- **THEN** only the failed group runs again, and the groups that passed keep their results
+
+### Requirement: Publishing and release require the lint and every test group
+
+CI SHALL publish an image or create a release only when the backend lint and all three backend test groups have passed for that commit. Running the lint and the tests in parallel SHALL NOT let a commit with a failed lint or a failed group reach publishing or release.
+
+#### Scenario: Lint fails on main
+
+- **WHEN** the backend lint fails on a push to `main` while all three test groups pass
+- **THEN** CI publishes no image and creates no release for that commit
+
+#### Scenario: One group fails on main
+
+- **WHEN** the lint and two test groups pass on a push to `main` but the third group fails
+- **THEN** CI publishes no image and creates no release for that commit