PostgreSQL backup tool with Point-In-Time-Recovery and restore verification
TypeScript backupbackupsdatabasedatabase-backupdevopsdockergolangkubernetesmariadbmongodbmysqlpgpostgrespostgresqls3self-hostedsystem-administrationtoolsweb-ui
FEATURE (docs): Add OpenSpec specifications
Rostislav Dugin45af63278d93
diff --git a/openspec/changes/archive/2026-09-04-automatic-physical-reanchor-after-failover/.openspec.yaml b/openspec/changes/archive/2026-09-04-automatic-physical-reanchor-after-failover/.openspec.yamlnew file mode 100644index 00000000..9696e00f--- /dev/null+++ b/openspec/changes/archive/2026-09-04-automatic-physical-reanchor-after-failover/.openspec.yaml@@ -0,0 +1,2 @@+schema: spec-driven+created: 2026-09-03diff --git a/openspec/changes/archive/2026-09-04-automatic-physical-reanchor-after-failover/design.md b/openspec/changes/archive/2026-09-04-automatic-physical-reanchor-after-failover/design.mdnew file mode 100644index 00000000..e1308321--- /dev/null+++ b/openspec/changes/archive/2026-09-04-automatic-physical-reanchor-after-failover/design.md@@ -0,0 +1,101 @@+## Context++See `proposal.md` for motivation and `specs/postgresql-physical-backup-failover/spec.md` for observable behavior.++`CheckTimelineCompatibility` currently compares the live timeline with the maximum timeline found in completed FULL and `.history` rows (`backend/internal/features/backups/backups/usecases/physical/postgresql/timeline.go:55-110,293-322`). Once the WAL streamer catalogs a new `.history` row, this maximum can match the promoted server even though an INCR still points to a root FULL on the previous timeline.++The scheduler checks incremental cadence before it reacts to a missing extendable chain (`backend/internal/features/backups/backups/backuping/physical/scheduler.go:161-242`). A newly broken chain can therefore wait for a full incremental interval. Existing terminal backup rows, the single in-flight claim, and the three-attempt FULL brake already provide enough durable coordination to avoid another state table.++Orphan WAL cleanup discovers and deletes segments without serializing against FULL creation (`backend/internal/features/backups/backups/backuping/physical/cleaner.go:508-546`). Restore selection also uses the retention-owned chain span directly, so an older chain cannot see same-timeline WAL beyond the start of a newer FULL (`backend/internal/features/backups/backups/core/physical/chain_view/restore_set.go:41-82`).++The implementation is constrained by `AGENTS.md` and `backend/AGENTS.md`.++## Goals / Non-Goals++**Goals:**++- Use the exact root FULL as the timeline authority for every INCR.+- Detect promotion both before and during `pg_basebackup`.+- Start one prompt replacement FULL without creating a scheduler retry loop.+- Preserve WAL and same-timeline PITR while that replacement is running.++**Non-Goals:**++- Restore through more than one PostgreSQL timeline.+- Persist the current timeline on the database model.+- Add a reanchor table, new backup configuration, or a new retry policy.++## Decisions++### Use root FULL timeline for INCR compatibility++Create separate FULL and INCR identity decisions. The INCR decision receives the root FULL timeline and compares it directly with the live value. The FULL decision keeps the current comparison with catalog history because a FULL may establish a new timeline and must still reject a regression or another cluster.++The INCR executor will load its root FULL through the existing repository before opening the backup slot. Missing or incomplete root metadata remains a broken-parent condition.++Rejected alternative: store a current timeline on the database row. That duplicates data already attached to backup artifacts and still does not identify the timeline that a particular INCR must extend.++Rejected alternative: add a reanchor state table. The broken INCR or failed FULL row already survives restarts and records why a replacement is needed.++### Recheck identity only after unexpected stream failure++The preflight decision returns the live timeline observed immediately before streaming. When a stream returns a non-completed result and its execution context is still active, the executor opens a fresh inspection connection and evaluates identity again.++For INCR, a newer timeline replaces a transient stream result with `CHAIN_BROKEN/TIMELINE_SWITCH_DETECTED`. For FULL, a timeline newer than its preflight value becomes `ERROR/FAILOVER_DURING_BACKUP`. A lower timeline or changed system identifier uses the existing safety refusal. A failed recheck leaves the original result unchanged and logs the failure.++Rejected alternative: classify `signal: interrupt` text. The same text can result from cancellation or an internal stream watchdog, so stderr alone cannot prove promotion.++Rejected alternative: recheck every successful backup. Successful stream metadata already supplies the completed artifact timeline, and the existing history validation handles that path.++### Derive one immediate replacement from terminal rows++Scheduler precedence will be:++1. Explicit forced FULL.+2. Timeline-related terminal backup that has not yet received a replacement attempt.+3. Existing forced INCR and cadence decisions.++A timeline-broken INCR is pending replacement only while no FULL row has a `CreatedAt` later than the INCR's `CompletedAt`. A latest FULL with `FAILOVER_DURING_BACKUP` is itself the pending signal. Inserting the replacement row changes the latest catalog state before its goroutine starts, and the existing in-flight claim prevents concurrent inserts.++An immediate decision carries the triggering backup kind and ID. After acquiring the same per-database advisory lock used for claim creation, `claimAndInsert` reloads the latest FULL and INCR rows and repeats the trigger predicate. It inserts the replacement only if that exact terminal row is still current and no later FULL attempt exists. This closes the stale-decision window where another scheduler could finish the first replacement and release its claim before a peer acts on an older decision.++If the replacement fails for another reason or is canceled, its terminal row no longer matches either immediate condition. Existing cadence and `recentFullAttemptsWindow` logic then controls later attempts. If another promotion interrupts it, the new `FAILOVER_DURING_BACKUP` result creates one new immediate attempt.++`SYSTEM_IDENTIFIER_MISMATCH` is not repairable by a FULL against the wrong cluster. The scheduler finds the newest mismatch completion time across FULL and INCR rows, then compares it with the newest completed FULL completion time. Automatic cadence and reanchor decisions remain suppressed while no completed FULL is newer than that mismatch. An explicit forced FULL remains higher priority so an operator can retry after correcting the connection, but normal preflight still refuses a different cluster. A forced FULL that fails generically or is canceled does not prove cluster identity and cannot clear suppression. Only a later completed forced FULL supersedes the mismatch, creates the new extendable chain, and restores normal scheduling.++Rejected alternative: set `ForceFullRequestedAt` from the executor. Persisting the terminal result and setting the request would span two writes with a crash window, while scheduler derivation needs no second state transition.++Rejected alternative: move every broken chain ahead of cadence. That would change `SUMMARIZER_OFF` behavior and could run repeated FULL backups for a source-side setting that a FULL cannot repair.++Rejected alternative: rely only on the in-flight claim to deduplicate stale decisions. The claim disappears when the first replacement finishes, so a peer can otherwise insert a second replacement from a decision made before that first attempt.++### Serialize FULL creation and orphan deletion++Add one shared repository-level operation that takes a transaction-scoped PostgreSQL advisory lock using `hashtextextended('physical-backup:' || database_id::text, 0)`. Both paths acquire it before any row lock:++- Scheduler transaction: advisory lock, in-flight claim, typed backup row insert.+- Orphan deletion transaction: advisory lock, read in-flight claim, skip if it is a FULL, otherwise delete the selected storage objects and catalog rows before commit.++Rejected alternative: check the in-flight claim without a shared lock. A FULL can claim the database immediately after the cleaner's check and lose WAL during its run.++Rejected alternative: lock a backup configuration row. The physical catalog service would then depend directly on another feature's private persistence model.++### Separate restore reachability from retention ownership++Retention keeps the existing chain span ending at the next completed FULL. `ResolveRestoreSet` will instead query committed WAL from the selected root FULL start through the unbounded end of the same timeline, then apply the existing contiguous-run and target-time checks. This preserves the current behavior of shipping the full contiguous replay window while allowing an older FULL to reach a target during a newer FULL.++Rejected alternative: widen every chain span. That would give the same WAL to multiple retention chains and change deletion accounting.++Rejected alternative: follow `.history` ancestry into a new timeline. The current manifest and restore path describe one WAL range, so that requires a separate cross-timeline design.++## Risks / Trade-offs++- [Identity recheck cannot reach the promoted server immediately] -> Preserve the original failure; the next normal attempt will run preflight again.+- [A corrected connection remains suppressed until the operator acts] -> Permit an explicit forced FULL; only its successful completion after the newest mismatch restores normal scheduling.+- [Advisory lock collision] -> Prefix the UUID before hashing so this lock namespace does not overlap other application advisory locks; the remaining 64-bit hash collision risk is negligible.+- [Object storage deletion holds the advisory lock] -> Keep the existing WAL byte and row budgets so the transaction remains bounded.+- [No cross-timeline PITR before replacement FULL completes] -> Return the existing unreachable-target error and document this as an explicit capability boundary.++## Migration Plan++No database migration is required. Deploy backend code and run the PostgreSQL 17 and 18 promotion tests before release. Rollback consists of reverting the backend change; existing rows and API payloads remain readable because the added error reasons are text values.diff --git a/openspec/changes/archive/2026-09-04-automatic-physical-reanchor-after-failover/proposal.md b/openspec/changes/archive/2026-09-04-automatic-physical-reanchor-after-failover/proposal.mdnew file mode 100644index 00000000..94f4d3a2--- /dev/null+++ b/openspec/changes/archive/2026-09-04-automatic-physical-reanchor-after-failover/proposal.md@@ -0,0 +1,34 @@+## Why++PostgreSQL promotion changes the live timeline and invalidates an incremental chain rooted on the previous timeline. Databasus can currently miss that mismatch after a `.history` file is cataloged, and a failed backup does not cause a prompt replacement FULL, so operators must trigger one manually.++## What Changes++- Validate each INCR against the timeline of its own root FULL instead of the newest timeline found across FULL and `.history` rows.+- Recheck cluster identity after an unexpected stream failure so promotion during `pg_basebackup` is classified correctly.+- Schedule one immediate replacement FULL after a timeline-related INCR or FULL failure, then retain the existing cadence and failed-FULL brake for later failures.+- Prevent orphan cleanup from deleting WAL while a FULL is active.+- Allow same-timeline PITR from an older FULL to use WAL received while a newer FULL was still running.+- Add distinct `TIMELINE_SWITCH_DETECTED` and `FAILOVER_DURING_BACKUP` error reasons. This extends the existing API enum without removing or renaming existing values.++Out of scope:++- PITR across multiple PostgreSQL timelines.+- Timeline ancestry resolution and manifests with multiple `WAL-Range` entries.+- Changing retention policy or backup cadence configuration.++This change answers to `AGENTS.md` and `backend/AGENTS.md`.++## Capabilities++### New Capabilities++- `postgresql-physical-backup-failover`: Detect timeline changes around physical backups, re-anchor automatically, and preserve the WAL needed during re-anchoring.++### Modified Capabilities++None.++## Impact++The change affects PostgreSQL physical backup timeline validation, backup result classification, scheduler decisions, WAL cleanup coordination, and same-timeline restore selection. It adds no table, configuration field, endpoint, or external dependency.diff --git a/openspec/changes/archive/2026-09-04-automatic-physical-reanchor-after-failover/specs/postgresql-physical-backup-failover/spec.md b/openspec/changes/archive/2026-09-04-automatic-physical-reanchor-after-failover/specs/postgresql-physical-backup-failover/spec.mdnew file mode 100644index 00000000..430dc55f--- /dev/null+++ b/openspec/changes/archive/2026-09-04-automatic-physical-reanchor-after-failover/specs/postgresql-physical-backup-failover/spec.md@@ -0,0 +1,108 @@+## Purpose++Keep PostgreSQL physical backups recoverable after promotion by rejecting stale incremental chains, creating a replacement FULL, and retaining the WAL needed during that transition.++## ADDED Requirements++### Requirement: Incremental backups use their root FULL timeline+The system SHALL run an incremental backup only when the live PostgreSQL timeline matches the timeline of that incremental chain's root FULL.++#### Scenario: Timeline matches the root FULL+- **WHEN** an incremental backup starts and the live timeline equals its root FULL timeline+- **THEN** the system proceeds with the incremental backup++#### Scenario: Promotion happened before the incremental started+- **WHEN** the live timeline is newer than the root FULL timeline+- **THEN** the system refuses the incremental backup as `CHAIN_BROKEN` with reason `TIMELINE_SWITCH_DETECTED`++#### Scenario: Source timeline regressed+- **WHEN** the live timeline is older than the root FULL timeline+- **THEN** the system refuses the incremental backup with reason `TIMELINE_REGRESSION`++#### Scenario: Cataloged history is newer than the root FULL+- **WHEN** a history file for the live timeline is already cataloged but the root FULL belongs to an older timeline+- **THEN** the system still refuses the incremental backup as `TIMELINE_SWITCH_DETECTED`++### Requirement: Backup failures are rechecked for promotion+The system SHALL recheck PostgreSQL cluster identity after an unexpected physical backup stream failure while the backup execution remains active.++#### Scenario: Promotion interrupts an incremental stream+- **WHEN** the timeline advances after incremental preflight and the stream fails+- **THEN** the system records the incremental as `CHAIN_BROKEN` with reason `TIMELINE_SWITCH_DETECTED`++#### Scenario: Promotion interrupts a FULL stream+- **WHEN** the timeline advances after FULL preflight and the stream fails+- **THEN** the system records the FULL as `ERROR` with reason `FAILOVER_DURING_BACKUP`++#### Scenario: Backup was canceled+- **WHEN** the backup execution context was canceled by an operator or configuration change+- **THEN** the system preserves the cancellation result and does not classify it as a promotion++#### Scenario: Recheck cannot connect+- **WHEN** the identity recheck fails+- **THEN** the system preserves the original backup result++#### Scenario: Cluster identity changed+- **WHEN** the identity recheck reaches a cluster with a different system identifier+- **THEN** the system rejects that cluster and suppresses automatic backup scheduling until an operator explicitly requests a FULL that completes against the expected cluster++#### Scenario: Operator requests a FULL after cluster identity changed+- **WHEN** automatic scheduling is suppressed by a system identifier mismatch and an operator explicitly requests a FULL+- **THEN** the system attempts preflight again and still refuses to write a backup unless the live system identifier matches the configured database++#### Scenario: Forced FULL still reaches a different cluster+- **WHEN** the operator's forced FULL is refused with `SYSTEM_IDENTIFIER_MISMATCH`+- **THEN** automatic backup scheduling remains suppressed++#### Scenario: Forced FULL fails without proving cluster identity+- **WHEN** a forced FULL after a system identifier mismatch fails for another reason or is canceled+- **THEN** automatic backup scheduling remains suppressed because no later FULL has completed against the expected cluster++#### Scenario: Forced FULL reaches the expected cluster+- **WHEN** the operator's forced FULL completes against the configured system identifier+- **THEN** that newer completed FULL supersedes earlier mismatch results and normal automatic scheduling resumes++### Requirement: Timeline failures trigger one immediate replacement FULL+The system SHALL create one immediate replacement FULL after a timeline-related terminal backup result.++#### Scenario: Incremental chain broke after promotion+- **WHEN** an incremental ends as `CHAIN_BROKEN` with reason `TIMELINE_SWITCH_DETECTED` and no later FULL attempt exists+- **THEN** the system schedules a FULL without waiting for the configured incremental or FULL interval++#### Scenario: FULL was interrupted by promotion+- **WHEN** the latest FULL ends as `ERROR` with reason `FAILOVER_DURING_BACKUP`+- **THEN** the system schedules one replacement FULL without waiting for the configured interval++#### Scenario: Replacement FULL fails normally+- **WHEN** the immediate replacement FULL ends with a non-timeline error or is canceled+- **THEN** the system resumes its existing cadence and failed-FULL retry limits instead of scheduling a FULL on every scheduler tick++#### Scenario: Another promotion interrupts the replacement+- **WHEN** the timeline advances again during the replacement FULL+- **THEN** the system schedules one new immediate replacement FULL for the later promotion++#### Scenario: Two schedulers hold the same stale replacement decision+- **WHEN** two scheduler instances decide that the same terminal backup needs an immediate replacement+- **THEN** only the instance that revalidates the still-current trigger creates a FULL row++### Requirement: Active FULL backups retain archived WAL+The system SHALL prevent orphan cleanup from deleting WAL for a database while a FULL backup for that database is active.++#### Scenario: Cleaner runs during the first FULL+- **WHEN** WAL is archived while the database's first FULL is still in progress and orphan cleanup runs+- **THEN** the WAL remains cataloged and stored for use after the FULL completes++#### Scenario: Cleaner finishes before a FULL starts+- **WHEN** orphan cleanup has begun before the scheduler starts a FULL for the same database+- **THEN** the FULL starts only after that cleanup operation has finished++### Requirement: Same-timeline PITR remains available during a newer FULL+The system SHALL allow a point-in-time restore from an older FULL when the target precedes completion of a newer FULL on the same timeline.++#### Scenario: Target falls during the newer FULL+- **WHEN** the requested target is later than the older FULL completion and earlier than the newer FULL completion+- **THEN** the restore set uses the older FULL and contiguous WAL through the target even when that WAL crosses the newer FULL start position++#### Scenario: Target crosses a promotion+- **WHEN** reaching the target would require WAL from more than one timeline+- **THEN** the system reports that the target is not restorable until a FULL on the promoted timeline is availablediff --git a/openspec/changes/archive/2026-09-04-automatic-physical-reanchor-after-failover/tasks.md b/openspec/changes/archive/2026-09-04-automatic-physical-reanchor-after-failover/tasks.mdnew file mode 100644index 00000000..4b78a5d6--- /dev/null+++ b/openspec/changes/archive/2026-09-04-automatic-physical-reanchor-after-failover/tasks.md@@ -0,0 +1,19 @@+## 1. Timeline classification++- [x] 1.1 Add `TIMELINE_SWITCH_DETECTED` and `FAILOVER_DURING_BACKUP` error reasons, and verify enum serialization with `cd backend && TEST_PARALLEL_WORKERS=1 go test -p=1 -count=1 ./internal/features/backups/backups/core/physical/enums`+- [x] 1.2 Split FULL and INCR timeline decisions so INCR uses its root FULL timeline, and verify equal, newer, older, and different-cluster cases with `cd backend && TEST_PARALLEL_WORKERS=1 go test -p=1 -count=1 ./internal/features/backups/backups/usecases/physical/postgresql`+- [x] 1.3 Recheck cluster identity after non-completed FULL and INCR streams while execution remains active, preserving cancellation and failed-probe results, and verify the executor tests with `cd backend && TEST_PARALLEL_WORKERS=1 go test -p=1 -count=1 ./internal/features/backups/backups/usecases/physical/postgresql`++## 2. Automatic re-anchoring++- [x] 2.1 Add scheduler decisions for one immediate FULL after `TIMELINE_SWITCH_DETECTED` or `FAILOVER_DURING_BACKUP`, carry the triggering row into transaction-time revalidation, suppress automatic scheduling while no completed FULL is newer than the latest `SYSTEM_IDENTIFIER_MISMATCH`, and verify priority, two-scheduler stale decisions, generic replacement failure, second promotion, mismatch failure, generic failure, cancellation and successful forced FULL attempts after mismatch, plus `SUMMARIZER_OFF`, with `cd backend && TEST_PARALLEL_WORKERS=1 go test -p=1 -count=1 ./internal/features/backups/backups/backuping/physical`++## 3. WAL and restore safety++- [x] 3.1 Serialize FULL claim creation and orphan WAL deletion with the shared per-database transaction advisory lock, and verify both lock orderings plus `Test_CleanOrphanWalForDatabase_WhenFirstFullInProgress_KeepsWalArchivedDuringBackup` with `cd backend && TEST_PARALLEL_WORKERS=1 go test -p=1 -count=1 ./internal/features/backups/backups/backuping/physical`+- [x] 3.2 Let PITR resolution read unbounded contiguous WAL on the selected root FULL's timeline without changing retention spans, and verify `Test_ResolveRestoreSet_WhenTargetDuringNewerFull_UsesOlderChainWalThroughTarget` plus existing gap tests with `cd backend && TEST_PARALLEL_WORKERS=1 go test -p=1 -count=1 ./internal/features/backups/backups/core/physical/chain_view`++## 4. Promotion coverage and final validation++- [x] 4.1 Add PostgreSQL 17 and 18 promotion coverage showing that retained timeline history does not let a stale INCR cross its root FULL timeline, followed by automatic FULL on the promoted timeline, cleanup, and restore from the new chain, and verify with `cd backend && make test`+- [x] 4.2 Run the complete backend checks with `cd backend && make test && make lint`diff --git a/openspec/changes/archive/2026-09-04-fix-boundary-wal-orphan-deletion/.openspec.yaml b/openspec/changes/archive/2026-09-04-fix-boundary-wal-orphan-deletion/.openspec.yamlnew file mode 100644index 00000000..b4b3ece7--- /dev/null+++ b/openspec/changes/archive/2026-09-04-fix-boundary-wal-orphan-deletion/.openspec.yaml@@ -0,0 +1,2 @@+schema: spec-driven+created: 2026-09-01diff --git a/openspec/changes/archive/2026-09-04-fix-boundary-wal-orphan-deletion/design.md b/openspec/changes/archive/2026-09-04-fix-boundary-wal-orphan-deletion/design.mdnew file mode 100644index 00000000..f90ead04--- /dev/null+++ b/openspec/changes/archive/2026-09-04-fix-boundary-wal-orphan-deletion/design.md@@ -0,0 +1,102 @@+## Context++See the Why section in `proposal.md` for the motivation and the field report.++The orphan sweep first finds candidates with an anti-join. Before deleting a candidate, `DeleteOrphanWalSegmentsInSpan` acquires the database advisory lock and rechecks active FULL claims and completed anchors in the deletion transaction. Cascade deletion acquires the same lock before calculating its WAL span. The scheduler acquires that lock before inserting a FULL claim, so no segment can be deleted across the transition from an unanchored database to an active FULL.++Two facts about LSN alignment decide the whole design:++- A segment's `start_lsn` and `end_lsn` come from its filename, not from the stream. `segmentBounds` (`backend/internal/features/backups/backups/usecases/physical/postgresql/wal_upload.go:428`) derives them, so both are always exact multiples of the segment size.+- A FULL's `start_lsn` is a real recovery position inside a segment. In the field report it is `0/AB000028` in every reproduction, offset `0x28` past the segment start, which is where the first record lands after a long page header.++So `f.start_lsn <= w.start_lsn` is false for the boundary segment in every realistic case, and the segment holding both `start_lsn` and (for a short backup) `stop_lsn` is classified as an orphan.++The correct predicate already exists a few lines above in the same file. `FindByChainSpan` (same file, line 60) selects on `end_lsn > startLSN AND start_lsn < endLSN`, and its comment states the invariant outright: a FULL's `start_lsn` is usually mid-segment, so excluding the segment whose file-boundary start sits below it "would drop the very segment that bridges the FULL's stop_lsn into the next segment, manufacturing a false WAL gap". `FindOrphans` never got the same treatment. The fix is to make the two agree.++Constrained by `backend/AGENTS.md`: comments carry *why* only, tests follow `Test_What_WhenCondition_Expected`, and coverage is preferred at the level real callers see rather than at the repository.++## Goals / Non-Goals++**Goals:**++- One predicate governs whether a segment is expendable, and it agrees with `FindByChainSpan` on what a chain covers.+- Regression coverage fails against today's code, at both the sweep level and the full backup-to-restore level.+- Restore visibility is independent of the non-overlapping ownership spans used by retention.+- Deleting a FULL never removes WAL that a surviving predecessor needs after it becomes the selected restore base.+- Every cascade path that removes WAL participates in the active-FULL claim handoff.++**Non-Goals:**++- Introducing a retention margin (extra segments kept "just in case"). The boundary segment is genuinely referenced; keeping it is correctness, not slack.+- Schema or index changes.++## Decisions++### Compare against the segment's end, not its start++The predicate becomes: a segment is referenced when some COMPLETED FULL on the same timeline has `f.start_lsn < w.end_lsn`. Read plainly: keep the segment if any retained backup starts before the segment runs out, which is exactly "some retained backup needs bytes from this segment or from a later one".++Alternatives rejected:++- **Round the FULL's `start_lsn` down to its segment boundary** before comparing (`f.start_lsn - (f.start_lsn % segment_size) <= w.start_lsn`). Arithmetically equivalent for aligned segments, but it puts the 16 MB segment size into SQL. The size is runtime-configurable (`walmath.SetWalSize`), and the value would have to be threaded into the query or hardcoded. Comparing against the stored `end_lsn` needs no size at all.+- **Keep the predicate and special-case the boundary segment in the cleaner** by skipping any segment whose range contains a FULL's `start_lsn`. Two places would then encode chain coverage, and the query would still return rows that the caller must remember to filter. The next caller of `FindOrphans` inherits the bug.+- **Retain the newest N segments unconditionally** as a safety margin. Papers over the wrong predicate, keeps deleting referenced WAL once the margin scrolls past, and makes the retained set depend on arrival order rather than on what a restore needs.++### Keep the sweep anchored on FULLs only, not on incrementals++An incremental's span sits inside its chain, and the chain is anchored by its root FULL, whose `start_lsn` is lower. `GetChainSpan` already treats the root FULL's `start_lsn` as the chain's lower bound.++Alternatives rejected:++- **Add COMPLETED incrementals to the anti-join** so the sweep anchors on any backup rather than on FULLs. It would not retain a single extra segment, because every incremental's chain starts at a FULL whose `start_lsn` is lower, while it doubles the anti-join's cost on a query the comment at `wal_segment_repository.go:80` already flags as a sequential scan.++### Separate WAL ownership from restore visibility++`GetChainSpan` keeps the non-overlapping retention range `[full.start_lsn, successor.start_lsn)`. `ResolveRestoreSet` uses `CompletedAt` to select a safe base backup, then queries WAL from that chain through `LSNMax` and ships the full contiguous run up to the first LSN gap. PostgreSQL applies `recovery_target_time`; `ReceivedAt` is only the existing catalog estimate for rejecting targets beyond available WAL.++Moving the ownership boundary to `successor.stop_lsn` is rejected because it gives two chains overlapping deletion ranges. Trimming the restore set at the first segment received after the target is also rejected: receipt time does not prove that a segment contains a transaction timestamp at or after the requested target.++### Preserve shared WAL when deleting a successor++The WAL attached to a newer FULL is also the continuation of every older restore base on the same timeline. If an older completed FULL survives, cascade deletion removes no WAL with the newer FULL. If no predecessor survives, deletion uses the target FULL's ownership span and removes only fully contained segments.++`GetDependentsSummary` and cascade deletion share this selection rule, including the active FULL claim guard. The preview takes the same advisory lock and database snapshot as cascade deletion, so it cannot report WAL or timeline history that an observed active claim makes non-deletable. The preview cannot use the overlap query because the destructive predicate deliberately excludes a segment crossing the upper ownership boundary.++`DeleteChainDependentsKeepFull` remains different by design. The `FULL_BACKUPS` policy keeps a standalone FULL while explicitly shedding its incrementals and owned WAL, so predecessor preservation applies only when the FULL row is removed.++### Serialize first-FULL scheduling with orphan deletion++Scheduling a FULL claim and deleting orphan WAL acquire the same transaction-scoped advisory lock keyed by database ID. The deletion transaction checks for an active FULL claim first, then checks whether a completed FULL covers the candidate. An `IN_PROGRESS` row without a live claim does not protect WAL because the claim is the source of truth for active work.++Cascade deletion acquires this lock before locking its target FULL. After the lock is acquired, both full removal and `DeleteChainDependentsKeepFull` recheck the active FULL claim. If a claim exists, they preserve WAL. Full removal also preserves timeline history until the active FULL has published its completed anchor. This lock ordering prevents a cascade from calculating an unbounded ownership span while a replacement FULL claim is being committed.++### Coverage at two levels++`chain_view/service_test.go:343` seeds a FULL with a segment-aligned `start_lsn` (`4*segmentBytes`), and `cleaner_test.go:249` does the same through `seedChainFull`. That is the one alignment where the buggy predicate is accidentally correct. Both files get a case with a mid-segment `start_lsn`, so the offset that matters lives in the fixture rather than being assumed away. (`cleaner_test.go:236` seeds no FULL at all, so it covers the "no COMPLETED FULL on this timeline" branch and never the boundary case under either predicate.)++That alone would not have caught the field failure, because it does not exercise the upload-then-sweep sequence against a real cluster. The `features/tests/physical/postgresql` suite already has the pieces, all unexported inside its `shared` package: `seedChainAndStreamPastTarget` (`shared/helpers.go:786`), `waitForReplayableThroughLSN` (`:760`) and `requestRestoreTokenViaAPI` (`:606`), plus a working PITR assertion in `RunFullTwoIncrementalsPlusWalRecoversToTarget`. The new scenario therefore lands in `shared` as an exported `Run...` function and is wired into both `pg17/backup_restore_test.go` and `pg18/backup_restore_test.go`, matching every existing case: the per-version files are one-line dispatchers.++Alternative rejected: asserting on log output (`deleted orphan wal segment`). It tests the message rather than the outcome, and it would pass on any refactor that deletes the segment silently.++### The cleaner needs a single-pass test entry point++`shared` cannot drive the sweep today. `cleanOrphans` and `cleanOrphanWalForDatabase` are unexported (`cleaner.go:492` and `:508`), and the only exported entry is `Run` (`cleaner.go:40`), an infinite ticker that panics when called twice. `shared/setup.go` already follows a house pattern for this with `StartPhysicalSchedulerForTest` and `StartPhysicalWalStreamSupervisorForTest`; the cleaner gets its sibling, running one pass rather than starting a loop.++Alternatives rejected:++- **Call `Run` and wait out a tick.** The 3 second `cleanerTickInterval` becomes a sleep in an already slow suite, the panic guard makes a second pass impossible in the same process, and the test would depend on timing rather than on sequence.+- **Move the e2e case into the `backuping/physical` package** to reach the unexported methods. That package has no real cluster fixture, and `features/tests/` exists precisely for backup-to-restore cycles per `backend/AGENTS.md`.++## Risks / Trade-offs++- **The new predicate keeps a segment that is genuinely expendable.** The window is one segment per database and timeline, since the anti-join matches on `timeline_id`: a segment ending after the earliest retained FULL's start but beginning before it. That segment holds the bytes leading into the backup, so keeping 16 MB is the intended cost of a restorable chain. No mitigation needed.+- **Installations already damaged stay damaged.** WAL deleted before the fix is gone from storage, and PostgreSQL will have recycled it from `pg_wal` once the slot advanced. Users must take a fresh FULL after upgrading to get a restorable chain. This belongs in the release note, not in code.+- **The e2e case adds container time to an already slow suite.** Extend an existing scenario rather than standing up a fresh cluster if the runtime grows past what `make test` tolerates.+- **The unit fixtures could drift back to aligned LSNs** in a later refactor, quietly disarming the regression. The test name states the condition (`WhenFullStartsMidSegment`), so a fixture change that flattens it is visible in review.+- **Deleting a newer FULL retains more WAL than its preview used to report.** This is required while an older FULL survives. Later deletion of the earliest surviving FULL or the orphan sweep reclaims the WAL when no restore base needs it.++## Migration Plan++No schema change and no data migration. Deployment is the ordinary upgrade; the sweep simply stops deleting the boundary segment on its next run. Rollback is reverting the predicate, which restores the old, incorrect deletions and needs no data step.++Operationally, an installation that already lost boundary segments needs a new FULL backup before PITR works again. Nothing in the code can recover the deleted WAL.diff --git a/openspec/changes/archive/2026-09-04-fix-boundary-wal-orphan-deletion/proposal.md b/openspec/changes/archive/2026-09-04-fix-boundary-wal-orphan-deletion/proposal.mdnew file mode 100644index 00000000..3364c629--- /dev/null+++ b/openspec/changes/archive/2026-09-04-fix-boundary-wal-orphan-deletion/proposal.md@@ -0,0 +1,54 @@+## Why++The WAL orphan sweep deletes the very WAL segment that carries a FULL backup's `start_lsn` and `stop_lsn`, so point-in-time restore breaks a few seconds after a FULL backup reports success. `FindOrphans` in `backend/internal/features/backups/backups/core/physical/repositories/wal_segment_repository.go:83` keeps a segment only when some COMPLETED FULL satisfies `f.start_lsn <= w.start_lsn`. A segment's `start_lsn` is derived from its filename and is therefore file-aligned (`0/AB000000`), while a FULL's `start_lsn` sits just past the segment's long page header (`0/AB000028`). The comparison fails on the boundary segment every time.++This is not a rare race. A user report against `v3.55.3` reproduces it five times in a row, always with the same `0x28` offset (`0/8F000028`, `0/92000028`, `0/96000028`, `0/98000028`, `0/AB000028`), and the cleanup log follows two seconds behind the upload log. After the deletion `ResolveRestoreSet` cannot advance past the FULL's `stop_lsn` and reports `wal gap before target; latest restorable point is <stop_lsn>`, while the UI still shows the FULL and the WAL archive as successful. A user believes PITR works when the recovery chain is already broken.++The blast radius grows over time. The predicate is satisfied by *any* older COMPLETED FULL, so with several FULLs on a database only the oldest chain loses its boundary segment. When retention prunes that oldest chain, the next chain's boundary segment becomes an orphan and is deleted in turn, so a chain that restored correctly yesterday stops restoring today.++Two longer windows expose the same failure through different paths. While a new FULL runs, point-in-time restore must still use the preceding completed FULL, but restore-set assembly used that chain's retention boundary and hid WAL archived after the new FULL started. When every backup has been deleted, WAL archived during the replacement FULL had no completed anchor yet, so the orphan sweep could delete it before the FULL published its recovery positions.++## What Changes++- The orphan predicate anchors on the segment's end rather than its start: a WAL segment is retained when a COMPLETED FULL on the same timeline starts anywhere before the segment's `end_lsn`. The boundary segment that straddles a FULL's `start_lsn` is no longer classified as an orphan.+- Regression coverage pins the real-world shape the current tests miss. `chain_view/service_test.go:343` places the FULL's `start_lsn` exactly on a segment boundary, which is the one alignment where the buggy predicate happens to be correct; new coverage uses a mid-segment `start_lsn`.+- End-to-end coverage asserts that a FULL followed by WAL streaming leaves a restorable chain after the retention cleanup job has run, so the upload-then-delete sequence cannot regress silently.+- Restore-set assembly selects the base backup by completion time, then reads the full contiguous WAL run independently of the chain's retention boundary. Targets during a newer FULL continue to restore from the preceding completed FULL.+- FULL scheduling, orphan deletion, and cascade deletion use the same database advisory lock. Each destructive transaction rechecks the active FULL claim before removing WAL, so a newly claimed FULL cannot lose WAL before it publishes a completed anchor.+- Deleting a newer FULL preserves WAL while an older completed FULL remains on the same timeline, because that older FULL becomes the restore base after deletion. The deletion preview uses the same predicate as the destructive path.+- Not BREAKING. No schema, API, config or DTO changes. Existing rows are unaffected; the sweep simply stops deleting a segment it should never have deleted. WAL already destroyed on affected installations cannot be recovered by this change, and the proposal does not claim otherwise.++## Capabilities++### New Capabilities++- `physical-backups/wal-retention`: which archived WAL segments a database keeps and which the retention sweep may delete, expressed in terms of the recovery chains a user can still restore from.++### Modified Capabilities++None. `openspec/specs/` holds no physical-backup capability yet, so the retention behavior is stated for the first time here.++## Impact++Code:++- `backend/internal/features/backups/backups/core/physical/repositories/wal_segment_repository.go`: the `FindOrphans` predicate.+- `backend/internal/features/backups/backups/core/physical/chain_view/service_test.go`: orphan coverage with a mid-segment FULL `start_lsn`.+- `backend/internal/features/backups/backups/backuping/physical/cleaner_test.go`: the sweep leaves the boundary segment in place.+- `backend/internal/features/backups/backups/backuping/physical/`: a single-pass cleaner entry point for tests, since the only exported entry today is an infinite ticker.+- `backend/internal/features/tests/physical/postgresql/shared/` plus its `pg17` and `pg18` dispatchers: the backup-to-restore case that runs the sweep in between.++Behavior downstream: `ChainViewService.ResolveRestoreSet` reads WAL beyond the selected chain's retention boundary. `FindWalGapsInChain` continues to describe the chain-owned span used by retention.++Operations: one extra 16 MB segment is retained per database and timeline whose oldest COMPLETED FULL has no earlier FULL to anchor it. The anti-join matches on `timeline_id`, so the bound is per timeline, not per database. That is the correct cost of a restorable chain.++Rules this change answers to: `AGENTS.md` (root) and `backend/AGENTS.md`.++## Out of scope++- The missing `pg_switch_wal` grant on the replication-only user Databasus provisions. Separate change: `grant-wal-switch-to-replication-user`.+- Surfacing chain health in the UI, so that a broken chain stops reading as a successful backup. Worth doing and deliberately not done here.+- Any repair or re-fetch of WAL already deleted on a running installation.+- The `physical_backup_retention_cleanup` job's other duties: chain retention policies, GFS tiers, the grace period, and abandoned-claim reaping.+- Adding a switch to disable orphan cleanup.+- WAL archived on a new timeline after a promotion when no FULL claim or completed FULL anchors that timeline. Automatic re-anchoring after failover belongs to its own change.diff --git a/openspec/changes/archive/2026-09-04-fix-boundary-wal-orphan-deletion/specs/physical-backups/wal-retention/spec.md b/openspec/changes/archive/2026-09-04-fix-boundary-wal-orphan-deletion/specs/physical-backups/wal-retention/spec.mdnew file mode 100644index 00000000..c76d7f42--- /dev/null+++ b/openspec/changes/archive/2026-09-04-fix-boundary-wal-orphan-deletion/specs/physical-backups/wal-retention/spec.md@@ -0,0 +1,90 @@+## Purpose++Defines which archived WAL segments a database keeps and which the retention sweep may delete, stated in terms of the recovery chains a user can still restore from. A segment is expendable only when no retained backup needs it.++## ADDED Requirements++### Requirement: A WAL segment overlapping a retained backup's recovery window is never deleted as unreferenced++The retention sweep SHALL treat a WAL segment as unreferenced only when no retained, successfully completed physical full backup on the same timeline begins before the end of that segment.++Recovery positions inside a full backup do not fall on segment boundaries: a backup begins a short distance into the segment that holds it. The segment holding that position therefore begins earlier than the backup does, and it still carries every write recorded after the backup finished. That segment SHALL be retained.++#### Scenario: The full backup starts and ends inside one segment++- **WHEN** a full backup's start and end positions both fall inside a single WAL segment, and that segment has been archived+- **THEN** the retention sweep leaves the segment in place+- **AND** a restore to any point in time after the backup's end position and covered by the archived WAL succeeds++#### Scenario: A segment entirely older than every retained backup++- **WHEN** an archived WAL segment ends at or before the start of the earliest retained full backup on its timeline+- **THEN** the retention sweep deletes it as unreferenced++#### Scenario: The anchoring backup is pruned++- **WHEN** retention removes the oldest full backup, and a later full backup remains whose start position falls inside a WAL segment that the removed backup used to anchor+- **THEN** that segment is still retained, because the remaining backup starts before the segment ends+- **AND** point-in-time restore from the remaining backup stays available++### Requirement: A completed full backup leaves an unbroken recovery chain++Once a physical full backup reports success and WAL streaming stays healthy, the system SHALL keep the archived WAL contiguous from that backup's end position onward, for as long as the backup is retained. No background maintenance SHALL introduce a gap immediately after a successful backup.++#### Scenario: Retention runs seconds after a full backup completes++- **WHEN** a full backup completes, its boundary WAL segment is archived, and the retention sweep runs before any further backup exists+- **THEN** the archived WAL is unbroken from the backup's end position to the newest archived segment+- **AND** the reported latest restorable point is later than the backup's end position rather than equal to it++#### Scenario: Restoring to a point shortly after the backup++- **WHEN** a user requests a restore to a point in time a few seconds after a successful full backup, and WAL covering that point has been archived+- **THEN** the restore proceeds instead of being refused for a WAL gap++#### Scenario: Restoring while a newer full backup is running++- **WHEN** a newer full backup has started but has not completed, and the archived WAL remains contiguous from the preceding completed full backup+- **THEN** a point-in-time restore during that interval uses the preceding completed full backup+- **AND** the restore set contains the full available contiguous WAL run, including WAL beyond the newer backup's start and stop positions++### Requirement: WAL archived during an active full backup is protected++No orphan or cascade deletion SHALL remove WAL for a database while that database has an active FULL backup claim. Scheduling the claim and every WAL-deleting maintenance path SHALL be serialized for each database, and each deletion transaction SHALL recheck the claim before removing a segment.++#### Scenario: All backups were deleted and a replacement full is running++- **WHEN** no completed full backup remains, a replacement FULL has an active claim, and WAL is archived during that backup+- **THEN** the orphan sweep leaves the WAL in place+- **AND** the WAL becomes anchored when the FULL completes++#### Scenario: An in-progress row has no active claim++- **WHEN** an `IN_PROGRESS` full backup row remains without an active FULL claim and no completed full backup anchors the WAL+- **THEN** the orphan sweep may reclaim the WAL++#### Scenario: Cascade deletion overlaps replacement full claim creation++- **WHEN** full removal or `FULL_BACKUPS` dependent cleanup starts while a replacement FULL claim is being committed+- **THEN** the cascade waits for claim creation to finish+- **AND** it leaves WAL in place after observing the active claim+- **AND** full removal leaves timeline history in place until the active FULL publishes a completed anchor+- **AND** the deletion preview reports zero deletable WAL and timeline history while that claim remains active++### Requirement: Deleting a full backup preserves WAL required by a surviving predecessor++Deleting a completed full backup SHALL retain its WAL when an older completed full backup remains on the same database and timeline. The older backup becomes the restore base after deletion and SHALL remain capable of replaying the shared WAL. The deletion preview SHALL report the same WAL count and size that the destructive operation can remove.++This rule applies when the FULL itself is removed. A `FULL_BACKUPS` retention policy may deliberately keep a FULL as a standalone restore point while removing its incrementals and owned WAL.++#### Scenario: A newer full backup is deleted++- **WHEN** a completed full backup is deleted while an older completed full backup remains on the same timeline+- **THEN** no WAL is deleted with the newer full backup+- **AND** point-in-time restore during the deleted backup's former interval succeeds from the older backup++#### Scenario: The earliest surviving full backup is deleted++- **WHEN** a completed full backup has no older completed predecessor on its timeline+- **THEN** deletion removes only WAL segments fully contained in its ownership span+- **AND** a segment crossing the next full backup's start position remains availablediff --git a/openspec/changes/archive/2026-09-04-fix-boundary-wal-orphan-deletion/tasks.md b/openspec/changes/archive/2026-09-04-fix-boundary-wal-orphan-deletion/tasks.mdnew file mode 100644index 00000000..accd3b92--- /dev/null+++ b/openspec/changes/archive/2026-09-04-fix-boundary-wal-orphan-deletion/tasks.md@@ -0,0 +1,58 @@+## 1. Pin the bug with a failing test++- [x] 1.1 Add `Test_CleanOrphanWalForDatabase_WhenFullStartsMidSegment_KeepsBoundarySegment` to `backend/internal/features/backups/backups/backuping/physical/cleaner_test.go`: seed a COMPLETED FULL whose `start_lsn` and `stop_lsn` both fall inside segment 1 (offset `0x28` past the boundary, matching the field report), seed the archived segment 1 with file-aligned bounds, run `cleanOrphanWalForDatabase`, assert the segment still exists. Verify with `make test` in `backend/` that the test fails against the current predicate and names the deleted segment in its failure message.+- [x] 1.2 Add `Test_FindWalOrphansByDatabase_WhenFullStartsMidSegment_BoundarySegmentIsNotOrphan` to `backend/internal/features/backups/backups/core/physical/chain_view/service_test.go`, next to the aligned-LSN case at line 343 so the two alignments sit side by side. Verify it fails for the same reason via `make test`.++## 2. Fix the predicate++- [x] 2.1 Change the anti-join in `PhysicalWalSegmentRepository.FindOrphans` from `f.start_lsn <= w.start_lsn` (`backend/internal/features/backups/backups/core/physical/repositories/wal_segment_repository.go:100`) to `f.start_lsn < w.end_lsn`. Verify both tests from group 1 pass and the existing `Test_CleanOrphanWalForDatabase_WhenWalOutsideAllChains_DeletesOrphan`, `Test_CleanOrphanWalForDatabase_WhenWalCoveredByChain_KeepsIt` and `Test_FindWalOrphansByDatabase_WhenWalOutsideAllChainSpans_ReturnsOrphan` still pass.+- [x] 2.2 Add a second why line to the comment above `FindOrphans` (`wal_segment_repository.go:80`) stating that the comparison anchors on `end_lsn` because a FULL's `start_lsn` sits mid-segment while a segment's bounds are file-aligned. Keep the existing sequential-scan note, which is unrelated and still accurate. Verify with `make lint` in `backend/` and by the group 1 tests still passing.++- [x] 2.3 Add `Test_DeleteFull_WhenSuccessorStartsMidSegment_KeepsSuccessorBoundarySegment` to `backend/internal/features/backups/backups/core/physical/service/service_test.go`: prune a chain whose successor FULL starts mid-segment and assert the successor's boundary segment and its stored object survive. Not in the original plan, which assumed `FindOrphans` was the only place the alignment mistake lives. It covers the delta spec's "The anchoring backup is pruned" scenario, which the predicate fix alone does not satisfy.+- [x] 2.4 Bound `deleteWalInSpanBudgeted` on `end_lsn <= span.End` instead of `start_lsn < span.End` (`backend/internal/features/backups/backups/core/physical/service/service.go:604`), so a chain's deletion span stops at segment boundaries rather than cutting into the successor's boundary segment. Reached from retention pruning (`cascadeDelete` via `GetChainSpan`) and from the user-initiated `DeleteWalSegment`, both of which end their span on a mid-segment anchor LSN.++## 3. Make the sweep drivable from the end-to-end suite++- [x] 3.1 Add a single-pass cleaner entry point for tests to `backend/internal/features/backups/backups/backuping/physical/`, alongside `StartPhysicalSchedulerForTest` and `StartPhysicalWalStreamSupervisorForTest` in `backend/internal/features/tests/physical/postgresql/shared/setup.go`. `Run` (`cleaner.go:40`) cannot serve: it is an infinite ticker that panics on a second call. Verify with `make test` that a test can run one sweep and observe its effect without waiting out `cleanerTickInterval`.++## 4. Cover the sequence end to end++- [x] 4.1 Add an exported `Run...` scenario to `backend/internal/features/tests/physical/postgresql/shared/` that runs the sweep between the backup and the restore: take a FULL, stream WAL past a target time via `seedChainAndStreamPastTarget` (`shared/helpers.go:786`), run one cleaner pass from 3.1, then request a restore token for that target and assert it is issued rather than refused with 422. Wire it into both `pg17/backup_restore_test.go` and `pg18/backup_restore_test.go` as one-line dispatchers, matching every existing case. Verify with `make test` (or `make test-fedora` on a Fedora host) that both versions pass and that they fail when the predicate from 2.1 is reverted.+- [x] 4.2 Add a case asserting the sweep still reclaims WAL that sits entirely below the earliest COMPLETED FULL's `start_lsn`. The existing `Test_CleanOrphanWalForDatabase_WhenWalOutsideAllChains_DeletesOrphan` (`cleaner_test.go:236`) seeds no FULL at all, so it covers the "no COMPLETED FULL on this timeline" branch and never this one. Verify with `make test`.++## 5. Close out++- [x] 5.1 Run `make lint` and `make test` in `backend/`; fix every finding before finishing.+- [x] 5.2 Run the `reviewer` subagent over the working-tree diff and resolve every CHANGES REQUIRED finding.+- [ ] 5.3 Put the upgrade note in the pull request description, since the repository keeps no changelog file: installations affected before the upgrade must take a fresh FULL backup, because the deleted WAL cannot be recovered, and the symptom is a `latest restorable point` equal to the FULL's `stop_lsn`. Verify the PR description contains it before requesting review.++## 6. Cover FULL execution windows++- [x] 6.1 Make `ResolveRestoreSet` query WAL through `LSNMax` after selecting the completed base backup, instead of limiting PITR to the selected chain's retention span.+- [x] 6.2 Serialize FULL claim creation and orphan deletion with the per-database advisory lock, then recheck the live FULL claim and completed anchor in the deletion transaction.+- [x] 6.3 Add regressions for PITR during a newer FULL and WAL archived while the first FULL has an active claim.++## 7. Preserve shared WAL during cascade deletion++- [x] 7.1 Add a regression that deletes a successor FULL while its predecessor remains, asserts that no WAL is removed, and resolves a target from the predecessor afterward.+- [x] 7.2 Make cascade deletion skip WAL when an older COMPLETED FULL remains on the same database and timeline. Keep the existing ownership span for a FULL with no surviving predecessor.+- [x] 7.3 Make `GetDependentsSummary` use the same WAL selection rule and fully-contained predicate as cascade deletion, including count and size.++## 8. Complete regression coverage++- [x] 8.1 Strengthen the restore-set regression so the target lies after the successor's stop position but before its completion time, and assert that all available contiguous WAL is returned.+- [x] 8.2 Add a restore-token controller regression for a target during a newer FULL.+- [x] 8.3 Add a cleaner regression proving that an `IN_PROGRESS` FULL row without a live claim does not retain orphan WAL.++## 9. Validate the completed change++- [x] 9.1 Run focused chain-view, physical-service, cleaner, scheduler, and controller tests. Confirm the new deletion and preview regressions fail when their production change is reverted.+- [x] 9.2 Run `make lint` and the full backend test suite. On Fedora, record the expected host-native `make test` loader failure for `libedit.so.2`, then run the supported `make test-fedora` shim to completion.+- [x] 9.3 Run the mandatory implementation reviewer and resolve every `CHANGES REQUIRED` finding.++## 10. Serialize cascade deletion with active FULL creation++- [x] 10.1 Acquire the database backup-and-cleanup advisory lock before cascade deletion locks its target FULL, then recheck the active FULL claim inside the transaction.+- [x] 10.2 Preserve WAL in both full-removal and `DeleteChainDependentsKeepFull` paths while a FULL claim is active. Preserve timeline history in the full-removal path until the active FULL publishes a completed anchor.+- [x] 10.3 Add a concurrency regression for both cascade modes. Hold the advisory lock, start deletion, commit a FULL row and claim, then prove deletion waited and retained the WAL row and object.+- [x] 10.4 Make `GetDependentsSummary` use the shared active-claim decision under the advisory lock, and cover a preview/delete pair that both report zero WAL and history while a FULL claim is active.diff --git a/openspec/changes/archive/2026-09-04-grant-wal-switch-to-replication-user/.openspec.yaml b/openspec/changes/archive/2026-09-04-grant-wal-switch-to-replication-user/.openspec.yamlnew file mode 100644index 00000000..b4b3ece7--- /dev/null+++ b/openspec/changes/archive/2026-09-04-grant-wal-switch-to-replication-user/.openspec.yaml@@ -0,0 +1,2 @@+schema: spec-driven+created: 2026-09-01diff --git a/openspec/changes/archive/2026-09-04-grant-wal-switch-to-replication-user/design.md b/openspec/changes/archive/2026-09-04-grant-wal-switch-to-replication-user/design.mdnew file mode 100644index 00000000..ffeaa54c--- /dev/null+++ b/openspec/changes/archive/2026-09-04-grant-wal-switch-to-replication-user/design.md@@ -0,0 +1,134 @@+## Context++See `proposal.md`: Why for the motivation and the field report.++Four pieces of the current implementation shape the approach.++The connection test is binary and already carries machine-readable refusals. `checkReplicationReadiness` (`backend/internal/features/databases/databases/postgresql/physical/model.go:805`) opens a replication connection, then an ordinary one for inspection queries (`:824`), and returns a `ConnectionTestError` with a code for each unmet precondition: `wal_level`, `max_wal_senders`, `max_replication_slots`, `summarize_wal`, custom tablespaces, system-identifier mismatch (`:841-865`). The controller turns that into `400 {"code": ...}` and nothing else (`controller.go:280`). There is no shape in this API for "passed, with a remark".++One of those checks is already conditional on the backup type. `summarize_wal` is demanded only when `p.BackupType.IsRequireWalSummary()` (`model.go:852`), and its sibling predicate `IsWalStreaming()` already exists (`enums.go:15`) with no caller in this file.++Provisioning is one short transaction. `CreateReplicationOnlyUser` (`model.go:723`) runs `CREATE USER ... LOGIN`, calls `grantReplication` for the detected platform (`model.go:983`), verifies the role exists and commits. `grantReplication` already carries per-platform branches and already converts a permission failure into an actionable message for Azure and GCP, so the shape for a second, best-effort grant is established there.++The frontend already owns every word of connection-test copy. `physicalConnectionErrorContent.ts` maps each code to a title, a summary, optional shell steps and a `managedNote` for RDS, Azure and GCP where the commands do not apply. The map is an exhaustive `Record<ConnectionErrorCode, ...>`, so adding an enum member makes the compiler demand the copy.++Constrained by `backend/AGENTS.md` (Swagger comments on every endpoint, controller-level tests over unit tests, comments explain *why*) and `frontend/AGENTS.md`.++## Goals / Non-Goals++**Goals:**++- A configuration that promises recovery to a point in time is refused unless the source can actually deliver it.+- Backup types that do not read archived WAL keep working with credentials that cannot force a switch.+- Provisioning improves the outcome where it can, so Databasus does not hand a user credentials it will then reject.++**Non-Goals:**++- Detecting *why* a platform refuses the grant. RDS, Azure and GCP each refuse differently; the product needs the answer, not the taxonomy.+- Guaranteeing the grant survives. An operator can revoke it later, which is why the streamer's runtime detection stays the source of truth.+- A privilege model for replication users beyond this one function.+- Reworking notifications, or re-testing saved databases in the background.++## Decisions++### Refuse in the connection test rather than report a finding beside it++A source that can neither force a switch nor close segments on a timer fails `checkReplicationReadiness` with a new `ConnectionErrorCode`, alongside the preconditions already checked there.++Alternatives rejected:++- **Pass the test and return a finding.** The endpoint returns `200 {"message": "connection successful"}` or a `400` with a code (`controller.go:280`); a third outcome means inventing a response shape and teaching every caller to read it, to describe a configuration that cannot do its job. The recent verification of minimum dump privileges for logical engines took the refusal route, not the remark route.+- **Store the answer on the database and show it as a state in the database view.** A nullable column, writes from provisioning, from testing and from the streamer, and a rule for clearing it, all to display a state the user could not have created if the test had refused it.+- **Leave it to the `WAL_ROTATION_DENIED` notification.** That is today's behavior. It arrives after the first quiet interval and only when the database has `CHAIN_BROKEN` enabled (`wal_supervisor.go:453`).++### Demand it only for `FULL_INCREMENTAL_WAL_STREAM`++The gate is `p.BackupType.IsWalStreaming()`, mirroring the `summarize_wal` line two checks above it.++`FULL` and `FULL_INCREMENTAL` never read an archived segment on restore: `pg_basebackup --wal-method=fetch` inlines the WAL a base backup needs (`pg_basebackup.go:69`) and `ResolveRestoreSetForBackup` ships no WAL and replays nothing (`restore_set.go:83`). Only `ResolveRestoreSet`, which walks the contiguous archived run from the last backup's `stop_lsn` (`restore_set.go:61`), depends on segments being finalized.++Alternatives rejected:++- **Demand it for every physical backup type.** Refuses configurations that work, and refuses them on managed platforms where nothing the operator does can produce the privilege.+- **Demand it for the incremental type too.** Incrementals are built by `pg_basebackup --incremental` against a manifest and restored by `pg_combinebackup` without replay. The archived WAL is not in that path.++### The privilege is the only accepted answer++The check passes on `has_function_privilege` alone. No source setting substitutes for it.++`archive_timeout` is the obvious candidate and `buildChainRiskAlert` names it as a remedy (`wal_supervisor.go:432`). It would work mechanically: the server switches the segment on its own timer, our `pg_receivewal` reaches the boundary and closes the file, the uploader ships it. Nothing here waits on the source's archiver. The problem is the precondition. `archive_timeout` has no effect while `archive_mode` is off, and turning `archive_mode` on commits the cluster to archiving every segment: until `archive_command` reports success, PostgreSQL keeps the segment in `pg_wal` and will not recycle it. On a source with no real archive command that is a full disk, and the failure lands on the user's production cluster rather than on a backup. A remedy Databasus prints in an error card has to be safe to follow literally.++Alternatives rejected:++- **Accept `archive_mode` on with a non-zero `archive_timeout`.** Keeps WAL streaming reachable on managed platforms, at the price of telling users to enable archiving without an archiver. The disk-fill risk is on their primary.+- **Accept it, and also tell the user to set `archive_command` to a no-op.** Databasus would be dictating an unrelated cluster-wide setting whose failure mode it cannot see, to work around a missing privilege.+- **Read `archive_timeout` alone.** Accepts a source where the setting has no effect at all.++### Probe with `has_function_privilege`, not by calling the function++Availability is read with `SELECT has_function_privilege(<role>, 'pg_switch_wal()', 'EXECUTE')` over the ordinary connection `checkReplicationReadiness` already opens.++Alternatives rejected:++- **Call `pg_switch_wal()` and catch `42501`.** A successful probe rotates a segment as a side effect, so testing a connection would archive a padded segment. Unacceptable in a path a user may run repeatedly, and worse in `CreateDatabase` and `UpdateDatabase`, which run the test on every save.+- **Infer from `rolsuper`.** Wrong in both directions: a non-superuser with an explicit grant can rotate, and a managed platform's "superuser-like" role often cannot.++### The grant is best effort inside the existing provisioning transaction++`grantReplication` gains a sibling that issues `GRANT EXECUTE ON FUNCTION pg_switch_wal() TO "<role>"` and treats a permission failure as a recorded outcome rather than an error. The result travels back with the created credentials so the caller can report it.++Failing provisioning on a refused grant is not an option: the replication user is fully functional for `FULL` and `FULL_INCREMENTAL` without it, and on managed platforms the refusal is the normal case.++The statement runs inside a savepoint. A refused statement puts the enclosing transaction in an aborted state, so without one the refusal would take down the `CREATE USER` it was meant to survive, and provisioning would fail on exactly the platforms this decision exists to serve.++Alternatives rejected:++- **A separate transaction after provisioning.** A failure would leave a user with no record of whether the grant was attempted, and the caller would have to reconcile two outcomes.+- **Skip the grant and let the connection test refuse.** Databasus would offer to create a user and then reject the user it created, on the platform where it could have simply granted the privilege.+- **Grant to `PUBLIC`.** Widens the privilege far beyond the role Databasus created.++### The refusal carries a sentence as well as a code++`ConnectionTestError` has an optional `Message` "for paths that surface only `Error()` (database create / update), so they don't show a bare code". Those are exactly the paths this change breaks: `CreateDatabase` and `UpdateDatabase` run the same test but answer with `{"error": ...}` rather than `{"code": ...}` (`controller.go`), so a user editing a saved streaming database would otherwise be shown the literal string `no_wal_switch_privilege`. The code therefore carries a sentence naming the role, the `GRANT`, and the option of a backup type without WAL replay.++Alternatives rejected:++- **Leave `Message` empty, like `wal_level_invalid` and its neighbours do.** Those codes are reached mostly from the test-connection endpoints, where the frontend renders a full card. This one is reached from the edit form by every user whose existing configuration this change refuses.+- **Teach create and update to answer with the code instead.** A wider API change than this needs, and it would alter the shape of every other failure those endpoints return.++### The backend sends a code, the frontend owns the words++The new code gets its entry in `physicalConnectionErrorContent.ts`: the `GRANT EXECUTE` line as a command step, and a `managedNote` stating that RDS, Azure and GCP do not let a customer role confer this privilege, so continuous WAL streaming is unavailable there and the backup type stays at `FULL_INCREMENTAL`. `buildChainRiskAlert` keeps its own wording unchanged, including the `archive_timeout` hint it gives an operator who is already streaming.++Alternatives rejected:++- **Extract one shared remedy sentence for both surfaces.** The alert's `Message` interpolates `database_id=%s reason=%s` around the remedies (`wal_supervisor.go:427-433`), it is log-shaped by design, and the card is a titled panel with commands and a managed-platform note. Sharing one sentence between them would fit neither.+- **Send a human-readable message from the backend.** Every other physical connection error sends a bare code and lets the frontend render it; `ConnectionErrorCode.ts` says so in its header comment.++### Names++Named here because `AGENTS.md` puts the naming check in the post-planning review, while names are still cheap to change:++The capability is called *forced WAL rotation* throughout, following the streamer's existing `runForcedWalRotation` and `DefaultForcedRotationInterval`. The exception is the wire code and the grant helper, which name the SQL function instead, because what is missing is a privilege on `pg_switch_wal()` and the remedy is a `GRANT` naming that function.++- `canForceWalRotation(ctx, q, roleName)` for the probe. A predicate, so no `Get` prefix. Unexported: no caller outside the package.+- `grantWalSwitchIfPermitted(ctx, tx, username) (isGranted bool, err error)` as `grantReplication`'s sibling. The suffix carries the best-effort contract, so a caller does not have to read the body to learn that a refusal is not an error. No platform argument: the statement is identical everywhere and only the outcome differs.+- `ConnErrNoWalSwitchPrivilege = "no_wal_switch_privilege"` for the code, matching the existing `no_replication_privilege`, and `ConnectionErrorCode.NoWalSwitchPrivilege` for its frontend mirror.+- `ReplicationOnlyUser{Username, Password, IsForcedWalRotationAvailable}` as `CreateReplicationOnlyUser`'s result, so the call sites do not carry two adjacent strings plus a bare flag.+- `CreateReplicationOnlyUserResponse` for the endpoint's own response, since the shared `CreateReadOnlyUserResponse` also serves the logical read-only user, which has no such flag.++The new probe and grant live in `forced_wal_rotation.go` with a mirrored `_test.go`, rather than growing `model.go`, which already holds the model, platform detection, readiness and provisioning.++## Risks / Trade-offs++- **Continuous WAL streaming becomes unavailable on RDS, Azure and GCP.** This is the deliberate cost of the decision above. Those users keep `FULL` and `FULL_INCREMENTAL`, and learn it at the moment they configure rather than from a recovery point that silently stops advancing. If a platform later exposes the grant, nothing in this change blocks it: the probe simply starts returning true.+- **An existing `FULL_INCREMENTAL_WAL_STREAM` database becomes uneditable** until its source is fixed, because `UpdateDatabase` runs the same test (`service.go:209`). Its backups keep running; nothing re-tests a saved database. The error card names both ways out, one of which is lowering the backup type. Called out in the proposal as the breaking part of this change and belongs in the upgrade note.+- **The probe runs as the admin, not the new role.** `has_function_privilege` takes a role argument, so the provisioning path probes the created role by name rather than `current_user`. Getting this wrong reports a false positive on every platform, so it is worth a dedicated test.+- **A managed platform may report the grant as succeeding while still refusing at call time.** The streamer's runtime detection remains authoritative and still stops the loop and notifies.+- **A privilege revoked after setup is not caught until a rotation is attempted.** Unchanged from today, and the existing notification covers it.++## Migration Plan++None. No schema change, no new column, no data to backfill. Rollback is reverting the change.++Existing databases are re-evaluated only when someone saves them. Replication users provisioned by earlier versions are not re-provisioned; their next connection test names the limitation and the remedies, and acting on it is the operator's call.diff --git a/openspec/changes/archive/2026-09-04-grant-wal-switch-to-replication-user/proposal.md b/openspec/changes/archive/2026-09-04-grant-wal-switch-to-replication-user/proposal.mdnew file mode 100644index 00000000..b8c8d904--- /dev/null+++ b/openspec/changes/archive/2026-09-04-grant-wal-switch-to-replication-user/proposal.md@@ -0,0 +1,58 @@+## Why++Databasus offers to provision a replication-only PostgreSQL user, then relies on a privilege it never grants that user. `CreateReplicationOnlyUser` (`backend/internal/features/databases/databases/postgresql/physical/model.go:723`) issues `CREATE USER ... LOGIN` plus a replication grant and stops there. Meanwhile `runForcedWalRotation` (`backend/internal/features/backups/backups/usecases/physical/postgresql/wal_rotation.go:70`) calls `pg_switch_wal()` every five minutes, and PostgreSQL restricts that function to superusers by default. The provisioned user gets `ERROR: permission denied for function pg_switch_wal (SQLSTATE 42501)`, the rotation loop logs one warning and returns for the lifetime of the streamer.++The degradation is real and silent, and it falls on exactly one feature. A FULL backup does not need the privilege: `pg_basebackup` runs with `--wal-method=fetch` (`pg_basebackup.go:69`), so the WAL between `start_lsn` and `stop_lsn` travels inside the same tar, and a restore to a specific backup ships no archived WAL at all (`chain_view/restore_set.go:83`). Incrementals inherit that. What breaks is recovery to a point in time. `ResolveRestoreSet` replays the contiguous run of archived segments starting at the last backup's `stop_lsn` (`restore_set.go:61`) and refuses any target beyond it with `WalGapBeforeTargetError`. A segment reaches storage only once finalized, so on a source that writes a few bytes an hour the newest WAL sits in `pg_wal` until the 16 MB segment fills. The user can restore the FULL and nothing after it, for hours or days. Bounding exactly that is why `DefaultForcedRotationInterval` exists (`wal_rotation.go:22`).++So the product currently accepts a configuration it knows cannot deliver what that configuration promises. A user who picks `FULL_INCREMENTAL_WAL_STREAM` and accepts Databasus's own offer to create a user ends up with a worse recovery point than a user who pastes in superuser credentials, and nothing in the interface says so. The only signal arrives later, as a notification, and only when `CHAIN_BROKEN` is among the database's enabled notification kinds (`wal_supervisor.go:453`), so a user who declined that notification learns nothing at all.++The streaming side itself behaves as designed and stays as it is. `runForcedWalRotation` recognizes SQLSTATE `42501`, stops the loop and reports through `buildChainRiskAlert` (`wal_supervisor.go:425`), which already names both remedies. Two gaps remain: provisioning never issues the grant it could issue, and connection testing accepts credentials that cannot support the backup type the user chose.++## What Changes++- Connection testing refuses credentials that cannot force a WAL segment switch, for the WAL-streaming backup type only. The test fails with a new machine-readable code, the way `wal_level`, `max_wal_senders` and `summarize_wal` already fail. The privilege is the only accepted answer: `archive_timeout` would also close segments on a timer, but it does nothing unless `archive_mode` is on, and an enabled `archive_mode` with no working `archive_command` makes PostgreSQL keep every segment in `pg_wal` until archiving succeeds. Databasus is not going to hand a user a remedy that fills their disk.+- `FULL` and `FULL_INCREMENTAL` are unaffected. Those backup types do not read archived WAL on restore, so demanding the privilege for them would refuse working configurations. The gate reuses the shape of the existing `summarize_wal` check, which is already conditional on the backup type (`model.go:852`).+- User provisioning grants `EXECUTE` on `pg_switch_wal()` to the role it creates, on platforms where the connected admin can. The grant is best effort: on managed PostgreSQL the admin frequently cannot grant execute on a restricted system function, and that must not fail the provisioning that otherwise succeeded. This matters more now that the test refuses: without the grant, Databasus's own offer to create a user would produce credentials Databasus then rejects.+- Provisioning reports whether forced WAL rotation will be available with the credentials it just created, so the answer is visible at the moment the user makes the choice rather than buried in a background log line.+- The frontend gets one more entry in its connection-error content map (`frontend/src/entity/databases/model/postgresql/physical/physicalConnectionErrorContent.ts`), with the `GRANT` command and a managed-platform note saying that where the platform will not confer the privilege, continuous WAL streaming is not available and the backup type has to stay at `FULL_INCREMENTAL`.++BREAKING for one existing configuration. A database already saved as `FULL_INCREMENTAL_WAL_STREAM` whose source refuses the switch will fail its connection test, which both `CreateDatabase` and `UpdateDatabase` run (`service.go:117`, `:209`). Backups of that database keep running: nothing re-tests a saved database on its own. But editing it fails until the operator grants the privilege or drops the backup type down to `FULL_INCREMENTAL`. That is the intended consequence of refusing a configuration that cannot do what it claims, and the error names both ways out.++## Capabilities++### New Capabilities++- `physical-backups/replication-credentials`: what a Databasus-provisioned replication user can do on the source cluster, and which source capabilities the product requires before it accepts a configuration that promises recovery to a point in time.++### Modified Capabilities++None. `openspec/specs/` holds no physical-backup capability yet, so this behavior is stated for the first time here.++## Impact++Code:++- `backend/internal/features/databases/databases/postgresql/shared/connection_error.go`: one new `ConnectionErrorCode`.+- `backend/internal/features/databases/databases/postgresql/physical/model.go`: the readiness check gains the privilege probe; the provisioning transaction gains the best-effort grant.+- `backend/internal/features/databases/service.go`, `controller.go` and `dto.go`: the provisioning endpoint gets its own `CreateReplicationOnlyUserResponse` carrying the rotation-capability result, instead of borrowing the logical `CreateReadOnlyUserResponse`, plus the Swagger annotations the endpoints already require.+- `frontend/src/entity/databases/model/postgresql/physical/`: the new code in `ConnectionErrorCode.ts` and its content in `physicalConnectionErrorContent.ts`. That map is an exhaustive `Record` over the enum, so the compiler demands the copy.+- `frontend/src/features/databases/ui/edit/CreateReadOnlyComponent.tsx`: the provisioning wizard step reports a refused grant, with the response type in `entity/databases/model/`.++External behavior: on self-managed PostgreSQL the grant normally succeeds during provisioning and WAL streaming works as intended. On RDS, Azure and GCP the grant is expected to fail, because the function's owner there is the platform's own superuser. Continuous WAL streaming becomes unavailable on those platforms until the platform allows the grant, and the error card says so instead of leaving the user to discover it from a recovery point that never advances. Backups themselves stay available there at `FULL` and `FULL_INCREMENTAL`.++No schema change, no migration, no new persisted state. Rotation availability is a property of the credentials and the source settings, read when they are tested rather than stored.++Dependency: none blocking. `physicalConnectionErrorContent.ts` holds English literals today, which the root `AGENTS.md` language rule allows; if the unmerged `translate-frontend-ui` change has converted that file to dictionary lookups by the time this lands, the new entry follows whatever shape the file then has.++Rules this change answers to: `AGENTS.md` (root), `backend/AGENTS.md`, `frontend/AGENTS.md`.++## Out of scope++- Persisting rotation availability on the database record and showing it as a state in the database view. Refusing the configuration at the door removes the state that surface existed to show. The one case it does not cover, a privilege revoked after setup, is already covered by the `WAL_ROTATION_DENIED` notification.+- Accepting a source-side archive timer in place of the privilege, or teaching Databasus to configure `archive_mode` and `archive_command` on the source.+- Changing anything about `FULL` or `FULL_INCREMENTAL` readiness.+- The WAL orphan sweep deleting a FULL backup's boundary segment. Separate change: `fix-boundary-wal-orphan-deletion`. The two were reported together and are unrelated in cause.+- Changing the rotation interval, the rotation trigger, or the decision to stop the loop permanently after a refusal.+- Re-testing saved databases in the background so an existing bad configuration surfaces without an edit.+- Retrofitting the grant onto replication users provisioned by earlier versions. The connection test names the remedy; the product does not silently re-provision.+- Granting anything beyond `pg_switch_wal` execute. Widening what a replication-only user can do needs its own justification.diff --git a/openspec/changes/archive/2026-09-04-grant-wal-switch-to-replication-user/specs/physical-backups/replication-credentials/spec.md b/openspec/changes/archive/2026-09-04-grant-wal-switch-to-replication-user/specs/physical-backups/replication-credentials/spec.mdnew file mode 100644index 00000000..9c3e79cb--- /dev/null+++ b/openspec/changes/archive/2026-09-04-grant-wal-switch-to-replication-user/specs/physical-backups/replication-credentials/spec.md@@ -0,0 +1,92 @@+## Purpose++Covers what a Databasus-provisioned replication user can do on a PostgreSQL source, and which source capabilities Databasus requires before it accepts a physical-backup configuration that promises recovery to a point in time.++## ADDED Requirements++### Requirement: A provisioned replication user can force WAL rotation where the platform allows it++When Databasus provisions a replication user on a source cluster, it SHALL also give that user the ability to force a WAL segment switch, so the recovery point of a rarely-written database is bounded by the rotation interval rather than by how long the current segment takes to fill.++Where the connected administrator cannot confer that ability, provisioning SHALL still succeed and deliver a working replication user, and SHALL report that forced WAL rotation is unavailable with those credentials.++#### Scenario: Self-managed source with an administrator that can confer the privilege++- **WHEN** a user asks Databasus to create a replication user on a self-managed PostgreSQL cluster, connected as an administrator able to confer it+- **THEN** the created user can force a WAL segment switch+- **AND** the credentials are accepted for a configuration that streams WAL++#### Scenario: Managed platform that withholds the privilege++- **WHEN** the same request is made against a managed platform whose administrator cannot confer the privilege+- **THEN** the replication user is still created and usable for physical backups+- **AND** the result reports that forced WAL rotation is unavailable++### Requirement: Provisioning widens the role's privileges by exactly one ability++Provisioning SHALL confer the ability to force a WAL segment switch and nothing else. The provisioned role SHALL remain limited to replication: no data access, no ability to create roles or databases, and no membership in an administrative role.++The ability SHALL be conferred on the provisioned role alone, never on a group that would extend it to other roles on the cluster. Where the source refuses, the cluster's privileges SHALL be left exactly as they were, with no partial or substitute grant.++#### Scenario: The provisioned role's reach after a successful grant++- **WHEN** provisioning succeeds including the ability to force a segment switch+- **THEN** the role can stream WAL and force a segment switch+- **AND** it still cannot read table data, create roles or databases, or act as an administrator++#### Scenario: No other role gains the ability++- **WHEN** provisioning confers the ability to force a segment switch+- **THEN** no role other than the one Databasus just created gains it++#### Scenario: The source refuses++- **WHEN** the connected administrator cannot confer the ability+- **THEN** the source's privileges are unchanged apart from the replication user's own creation+- **AND** no weaker or broader substitute privilege is granted in its place++### Requirement: A configuration that promises recovery to a point in time is refused unless the source can bound it++Recovery to an arbitrary point in time replays WAL segments that reached storage, and a segment reaches storage only once the source closes it. When a user configures continuous WAL streaming, Databasus SHALL refuse the configuration unless the supplied credentials can force a WAL segment switch.++The refusal SHALL name both ways out: confer the ability on the source, or choose a backup type that does not replay archived WAL. It SHALL NOT direct the user to enable archiving on the source, which would make the cluster retain every segment until an archiver confirms it. The refusal SHALL be reported the same way as the other unmet preconditions of a physical source, so a user meets it at the same moment they meet the rest.++This SHALL apply to credentials a user enters by hand as well as to credentials Databasus provisioned, and to a database being created as well as one being changed.++#### Scenario: Streaming configuration with credentials that cannot force a switch++- **WHEN** a user configures continuous WAL streaming with credentials that can stream WAL but cannot force a segment switch+- **THEN** the configuration is refused+- **AND** the refusal names conferring the ability and choosing a backup type without WAL replay++#### Scenario: A managed platform that will not confer the ability++- **WHEN** the source is a managed platform whose administrator cannot confer the ability on any role+- **THEN** the streaming configuration is refused there as well+- **AND** the refusal says continuous WAL streaming is unavailable on that source, rather than offering a source setting that would make the cluster retain WAL indefinitely++#### Scenario: Streaming configuration with credentials that can force a switch++- **WHEN** the credentials can force a segment switch+- **THEN** the configuration is accepted++#### Scenario: An existing streaming database that no longer qualifies++- **WHEN** a user changes a saved streaming database whose credentials cannot force a segment switch+- **THEN** the change is refused with the same explanation+- **AND** the backups already running for that database are not stopped by the refusal++### Requirement: Backup types that never replay archived WAL are not held to this requirement++A backup type that restores only to the point a backup itself finished does not read an archived segment, so the ability to force a segment switch SHALL NOT be required of it. Databasus SHALL accept such a configuration with credentials that can only stream.++#### Scenario: Full backups with credentials that cannot force a switch++- **WHEN** a user configures full backups, with or without incrementals, using credentials that cannot force a segment switch+- **THEN** the configuration is accepted+- **AND** backups and restores to those backups work with no privilege error++#### Scenario: Raising the backup type later++- **WHEN** a user changes such a database to stream WAL continuously+- **THEN** the requirement above applies to the change, and it is refused if the source cannot bound the recovery pointdiff --git a/openspec/changes/archive/2026-09-04-grant-wal-switch-to-replication-user/tasks.md b/openspec/changes/archive/2026-09-04-grant-wal-switch-to-replication-user/tasks.mdnew file mode 100644index 00000000..cbcb878e--- /dev/null+++ b/openspec/changes/archive/2026-09-04-grant-wal-switch-to-replication-user/tasks.md@@ -0,0 +1,28 @@+## 1. Refuse a streaming configuration the source cannot support++- [x] 1.1 Add `canForceWalRotation` in a new `backend/internal/features/databases/databases/postgresql/physical/forced_wal_rotation.go`: a read-only check answering whether a named role can execute `pg_switch_wal()`, via `has_function_privilege` over the ordinary connection `checkReplicationReadiness` already opens (`model.go:824`). Verify against a real PostgreSQL container with `Test_CanForceWalRotation_ForReplicationOnlyRole_ReturnsFalse` and `Test_CanForceWalRotation_AfterExecuteIsGranted_ReturnsTrue`, both dropping their roles through the existing `createTempUser` cleanup.+- [x] 1.2 Confirm the probe answers for the role passed in rather than for the connected administrator: `Test_CanForceWalRotation_WhenAnAdminProbesAnotherRole_AnswersForThatRole` connects as the superuser fixture, asserts the administrator can rotate, and asserts a replication-only role cannot. This is the false-positive trap named in `design.md`; without it the probe reports success on every platform. It stays a container-backed model test because no endpoint reaches the case of probing one role while connected as another.+- [x] 1.3 Add `ConnErrNoWalSwitchPrivilege` to `backend/internal/features/databases/databases/postgresql/shared/connection_error.go` and the gate to `checkReplicationReadiness`, next to the `summarize_wal` check it mirrors (`model.go:852`): when `BackupType.IsWalStreaming()` and the probe says the role cannot execute `pg_switch_wal()`, return that code. Verify with `Test_TestDatabaseConnectionDirect_ForWalStreamWhenRoleCannotSwitchWal_ReturnsNoWalSwitchPrivilegeCode` on `POST /databases/test-connection-direct`.+- [x] 1.4 Verify the accepting path with `Test_TestDatabaseConnectionDirect_ForWalStreamWhenRoleCanSwitchWal_ReturnsSuccess` on the same endpoint.+- [x] 1.5 Verify the backup-type gate with `Test_TestReplicationConnection_WhenRoleCannotSwitchWal_SucceedsForBackupTypesWithoutWalReplay`: the same refusing credentials pass for `FULL` and for `FULL_INCREMENTAL`. Without this the change quietly blocks configurations that work.+- [x] 1.6 Verify the saved-database paths: `Test_TestDatabaseConnection_WhenTheSourceRevokedTheWalSwitchPrivilege_ReturnsNoWalSwitchPrivilegeCode` on `POST /databases/{id}/test-connection` (`controller.go:225`), and `Test_UpdateDatabase_RaisingToWalStreamWhenRoleCannotSwitchWal_ReturnsNoWalSwitchPrivilegeCode` proving a saved database cannot be raised to `FULL_INCREMENTAL_WAL_STREAM` against such a source, since `UpdateDatabase` runs the same test (`service.go:209`). Create and update surface only `Error()`, so the code carries a remedy sentence as its `Message` and the update test asserts that sentence.+- [x] 1.7 Update the Swagger annotations on both test-connection endpoints (`controller.go:216-224`, `:247-255`) for the added failure, per the mandatory-Swagger rule in `backend/AGENTS.md`. Verify with `make swagger` in `backend/` that generation succeeds and both endpoints document the code.++## 2. Grant the ability during provisioning++- [x] 2.1 Add `grantWalSwitchIfPermitted` in `forced_wal_rotation.go` issuing `GRANT EXECUTE ON FUNCTION pg_switch_wal()` for the created role inside a savepoint, since a refused statement would abort the transaction that created the role, and call it from the provisioning transaction in `CreateReplicationOnlyUser` (`model.go:723`). Verify with `Test_CreateReplicationOnlyUser_OnSelfManagedSource_GrantsForcedWalRotation`, which asserts the created user passes a streaming connection test it could not pass without the grant, and drops the role in `t.Cleanup`.+- [x] 2.2 Verify a refused grant does not fail provisioning: `Test_CreateReplicationOnlyUser_WhenTheSourceRefusesTheGrant_StillReturnsWorkingCredentials` provisions as an administrator that may create roles and confer REPLICATION but owns no `pg_catalog` function, and asserts the returned credentials pass a `FULL_INCREMENTAL` connection test.+- [x] 2.3 Verify the grant reaches only the created role: `Test_CreateReplicationOnlyUser_ConfersForcedWalRotationOnTheCreatedRoleAlone` asserts a second replication-only role on the same cluster still cannot execute `pg_switch_wal()`, and that the created role gained no other attribute.+- [x] 2.4 Return `IsForcedWalRotationAvailable` in a `CreateReplicationOnlyUserResponse` of its own, rather than on the shared `CreateReadOnlyUserResponse` the logical read-only user also uses, and update the Swagger annotations on that endpoint (`controller.go:470-483`). Verify with `make swagger` in `backend/` that generation succeeds and the field appears in the generated spec.++## 3. Give the refusal its words in the interface++- [x] 3.1 Add `ConnectionErrorCode.NoWalSwitchPrivilege = 'no_wal_switch_privilege'` to `frontend/src/entity/databases/model/postgresql/physical/ConnectionErrorCode.ts` and its content to `physicalConnectionErrorContent.ts`: the `GRANT EXECUTE ON FUNCTION pg_switch_wal()` command step, a note offering full plus incremental as the other way out, and a `managedNote` saying RDS, Azure and GCP do not let a customer role confer it. The map is an exhaustive `Record` over the enum, so the compiler demands the entry. Verify with `pnpm lint` and `pnpm build` in `frontend/`.+- [x] 3.2 Show the provisioning result's `isForcedWalRotationAvailable` in `frontend/src/features/databases/ui/edit/CreateReadOnlyComponent.tsx`, the wizard step that provisions the user, so a managed-platform user learns at that moment that continuous WAL streaming will not be available with these credentials. Verify with `pnpm lint` and `pnpm build`.+- [x] 3.3 Keep the new copy English. `frontend/src/shared/i18n/` does not exist on this branch, so the copy stays as literals, which the root `AGENTS.md` language rule allows; the unmerged `translate-frontend-ui` change converts it along with the rest of the file. Verify with `pnpm lint`.++## 4. Close out++- [x] 4.1 Run `make lint` and `make test` in `backend/`, and `pnpm lint`, `pnpm test` and `pnpm build` in `frontend/`; fix every finding. On a Fedora host the suite is run with `make test-fedora`, which supplies the `libedit.so.2` shim the bundled MariaDB client needs; without it the MariaDB restore test fails for that reason alone.+- [x] 4.2 Run the `reviewer` subagent over the working-tree diff and resolve every CHANGES REQUIRED finding.+- [ ] 4.3 Put the upgrade note in the pull request description, since the repository keeps no changelog file: a saved `FULL_INCREMENTAL_WAL_STREAM` database whose credentials cannot force a segment switch keeps backing up but can no longer be saved from the edit form until the operator grants the privilege or lowers the backup type, and on RDS, Azure and GCP the privilege cannot be conferred at all. The symptom before this change was `permission denied for function pg_switch_wal` in the logs and a recovery point that only advanced when a segment filled. Verify the PR description contains it before requesting review.diff --git a/openspec/changes/translate-frontend-ui/.openspec.yaml b/openspec/changes/translate-frontend-ui/.openspec.yamlnew file mode 100644index 00000000..b4b3ece7--- /dev/null+++ b/openspec/changes/translate-frontend-ui/.openspec.yaml@@ -0,0 +1,2 @@+schema: spec-driven+created: 2026-09-01diff --git a/openspec/changes/translate-frontend-ui/README.md b/openspec/changes/translate-frontend-ui/README.mdnew file mode 100644index 00000000..b593b3ca--- /dev/null+++ b/openspec/changes/translate-frontend-ui/README.md@@ -0,0 +1,3 @@+# translate-frontend-ui++Translate the whole frontend UI into ru, es, pt, zh, fr with compiler-enforced dictionary completenessdiff --git a/openspec/changes/translate-frontend-ui/design.md b/openspec/changes/translate-frontend-ui/design.mdnew file mode 100644index 00000000..8f466027--- /dev/null+++ b/openspec/changes/translate-frontend-ui/design.md@@ -0,0 +1,221 @@+## Context++See proposal.md - Why.++The state that shapes the approach:++- `frontend/src` is 32,700 lines. Strings are inlined at their point of use, with no copy layer. A crude grep finds 299 text attributes, 411 JSX text nodes and 136 literals in `.ts` files; the real count is closer to 1,500 once multi-line JSX, template literals, option arrays and `alert()` calls are included.+- Copy is not confined to `ui/` segments. `entity/databases/model/postgresql/ConnectionStringParser.ts` returns error prose, `entity/notifiers/models/getNotifierNameFromType.ts` returns display names. The dictionary has to be reachable from `entity/` and `shared/`, which puts it in `shared/`.+- Two rules in `frontend/AGENTS.md` bear on where the dictionaries go. FSD MUST rule 4 says shared holds only infrastructure, and the dictionaries are domain-scoped (`backups`, `databases`, `storages`). The reading this design adopts: the provider, `t()` and the locale wiring are infrastructure, and copy strings carry no domain calculation, so `shared/i18n` is the right layer. The alternative reading, that copy is domain content belonging in `entity/`, would scatter one dictionary across seven slices and break the single-source-of-truth typing that the whole approach rests on. Recorded here so it is a decision rather than an oversight.+- `ThemeProvider` (`shared/theme/`) already establishes the pattern for a user preference: React context, `localStorage`, a toggle in the header. Locale follows it rather than inventing a second shape.+- `ThemeToggleComponent` already renders at three call sites: `widgets/main/MainScreenComponent.tsx:254`, `widgets/main/SidebarComponent.tsx:132` and `features/users/ui/AuthNavbarComponent.tsx:49`. Merging the language control into it covers the navbar, the narrow-viewport sidebar and the pre-login screen in one move.+- The website's `website/app/components/LanguageSelectorComponent.tsx` is working prior art, including an `isSplitEnd` variant that glues the selector to a neighbouring button as one split control.+- ESLint uses flat config (`tseslint.config(...)`). There is no `overrides` key; per-directory rules are additional config objects.+- `shared/api/apiHelper.ts` already parses `json.code` into `ApiError.code`, and `EditPostgreSqlPhysicalSpecificDataComponent.tsx:204` already branches on it. The error-code contract exists in one place and needs generalizing, not inventing.++The website solved the same problem differently, with five full page copies per page and no dictionaries. That works there because each URL needs its own static HTML for search engines and the pages are prose. Applying it here would mean 525 component copies held structurally in sync. The application diverges deliberately.++## Goals / Non-Goals++**Goals:**++- A forgotten translation is a compile error, not a runtime fallback.+- The migration is a burn-down with a machine-checkable finish line, not an open-ended sweep.+- Each language reads as if written in it, so the key is the identity and the English text is just the English value.++**Non-Goals:**++- Right-to-left languages. None of the six need it, and no layout work assumes it.+- Lazy-loading dictionaries per feature slice. One dictionary per language, loaded whole. Revisit if a dictionary passes a few hundred KB.+- Machine translation in the build. Translations are written and reviewed, like the website's.++## Decisions++### Dictionaries, not per-language component copies++See Context. Component reuse is the deciding factor: the same `EditStorageComponent` serves six languages, so language must be a runtime lookup rather than a file variant.++### `i18next` + `react-i18next`++Alternatives considered:++- **LinguiJS.** Rejected on the key strategy. Lingui uses the English source text as the key, so editing an English phrase silently invalidates all five translations. Since each language is written independently rather than rendered literally, keys must be stable and text-independent.+- **A hand-rolled `t()`, roughly 60 lines.** Attractive given the project's tight dependency budget (AntD only, one icon set). Rejected because Russian needs three plural forms, and interpolation, plural selection, and namespacing grow into a quarter of a library within a year. If usage stays limited to `{name}` substitution and plurals, this decision is cheap to revisit.+- **`react-intl` / FormatJS.** Comparable, but a heavier compile step for ICU messages than this codebase needs.++Required configuration, because the defaults are wrong here:++- `returnNull: false`, otherwise `t()` is typed `string | null` and every call site needs a guard.+- `nsSeparator: false`, otherwise a key containing a colon is misparsed as a namespace reference.+- `interpolation.escapeValue: false`, since React escapes already.+- `fallbackLng: 'en'` as a last-resort runtime guard only. Completeness is enforced by types (below); the fallback exists for a dictionary that failed to load, not for a missing key.++### Type safety comes from the locale files, not the library++This is the load-bearing decision, and no library provides it out of the box: i18next falls back at runtime, Lingui reports gaps in a CLI run. Both are checks a person can ignore.++Two independent layers:++```+shared/i18n/locales/en.ts <- source of truth for the key set+ export const en = { backups: { logical: { title: 'Logical backups' } } }+ | |+ layer 1: valid keys layer 2: complete dictionaries+ declare module 'i18next' { locales/ru.ts:+ interface CustomTypeOptions { import type { en } from './en'+ resources: { translation: typeof en }+ } export const ru: typeof en = { ... }+ }+ t('backups.logical.titel') fails missing key OR stray key fails tsc -b+```++`const ru: typeof en` is an exact structural check including nesting: it rejects a missing key and a leftover one alike. `en` is declared without `as const`, so values widen to `string` and translations are not forced to match the English text.++The `import type` link from each translation to `en` is erased at build time, so the English dictionary does not end up in the Russian bundle.++Consequence worth naming: a language cannot be added partially. Either its dictionary is complete or the build is red. That is why the four remaining languages come last, after the key set has settled.++### Keys are domain-scoped, not path-scoped++`t('backups.logical.emptyState.title')`, not `t('features.backups.logical.ui.EditLogicalBackupConfigComponent.emptyState.title')`.++Top-level namespaces follow the domain, not the file tree: `backups`, `databases`, `storages`, `notifiers`, `verification`, `workspaces`, `users`, `settings`, `status`, `errors`, `common`. A path-shaped key would look tidy today and rot at the first component move, and FSD already permits a component to migrate between `pages/`, `features/` and `entity/` as consumers appear.++### Status labels become typed lookup tables++Today the same five words are copy-pasted with different markup:++```+features/verification/runs/ui/VerificationDetailDrawer.tsx:51 renderStatusTag() -> <Tag color="green">Successful</Tag>+features/verification/runs/ui/VerificationsComponent.tsx:58 renderStatus() -> <span>Successful</span>+```++Seven enums are affected: `VerificationStatus`, `RestoreStatus`, `LogicalBackupStatus`, `PhysicalBackupStatus`, `RestoreVerificationStatus`, `HealthStatus`, `AddMemberStatusEnum`.++The label lookup lives in `entity/*/model/` in its own file next to the enum (`VerificationStatusLabelKeys.ts` beside `VerificationStatus.ts`), re-exported from the slice's `index.ts` so consumers reach it through the slice's public API. FSD MUST rule 5 governs types rather than constants, so co-locating the `Record` inside the enum's own file would also be legal; naming the choice here keeps the seven slices from diverging.++It is an explicit `Record`, not a template key:++```ts+// no: a dynamic key defeats the type check entirely+t(`status.verification.${status}`)++// yes: adding an enum member leaves the Record incomplete and fails the build+const verificationStatusLabelKeys: Record<VerificationStatus, TranslationKey> = { ... }+```++Components keep their own markup (a tag here, an icon and span there) and take the words from one place. This deduplicates as a side effect; that is a bonus, not the goal.++### The lint rule is what makes the migration finite++`eslint-plugin-i18next`, rule `i18next/no-literal-string`. The already-installed `react/jsx-no-literals` is not enough: it sees JSX text nodes and misses `placeholder="..."`, `okText="..."` and `alert('...')`, which together are more than half the strings.++Enabled per directory, appended to the flat config as each slice lands:++```js+{+ files: ['src/features/settings/**/*.tsx'],+ plugins: { i18next },+ rules: { 'i18next/no-literal-string': 'error' },+},+```++Exclusions are needed for 51 sites containing `pg_dump`, `pg_basebackup`, `mongodump`, `localhost` or `https://`, 27 sites with code and command blocks, and AntD props that look like prose but are not (`color`, `type`, `size`, `variant`). Configured exclusions cover the recurring ones; one-off cases take an inline suppression with a stated reason, which keeps the config from turning into a dumping ground. The exact option names are version-specific and get pinned when the plugin is installed.++Slices are migrated smallest first, so the infrastructure is debugged on `features/settings` (4 strings) rather than `features/databases` (159).++### The language control is merged with the theme control++The website's selector is the reference: a compact button showing the current language, a dropdown listing all six by native name, opening on `onMouseEnter` and closing on `onMouseLeave`, with the panel using top padding rather than a margin so the hover area between button and list stays contiguous. That detail is not cosmetic; a gap makes the dropdown close while the pointer crosses it.++Rather than adding a second standalone button, the language selector and the theme selector become one paired control, split down the middle, the way the website glues its selector to the GitHub button. Two reasons: the navbar has no room for a third control at narrow widths, and the two settings are the same kind of thing (a per-browser display preference), so grouping them is honest rather than decorative.++The paired control replaces `ThemeToggleComponent` at its three existing call sites, which is what satisfies the spec's placement requirement without a separate placement task.++Alternatives considered:++- **A separate language button beside the theme button.** Simpler to build, but adds a third control to a navbar that already competes for width, and the website has already established the paired shape.+- **Language inside the theme dropdown as a submenu.** Fewer pixels, but buries a setting a first-time visitor needs to find immediately, and nested dropdowns on hover are fragile.++Trigger consistency is an open question below.++### Three surfaces that are not React state++Three things follow the language but sit outside the component tree, and each needs wiring the dictionary alone does not give:++- `index.html` hardcodes `<html lang="en">` and an English `<title>`. Both are set from the provider once the locale resolves. The `lang` attribute is what screen readers switch voices on and what stops the browser offering to translate a page already in the reader's language; the title is what a bookmark keeps.+- Eight `toLocaleString()` calls carry no locale argument, so they follow the browser rather than the selection. On an English browser with Russian selected, dates would read Russian and numbers English on the same screen. They are in `LogicalBackupsComponent.tsx:513,515`, `PhysicalBackupsComponent.tsx:49,52`, `VerificationDetailDrawer.tsx:45,48,157` and `StarButtonComponent.tsx:56`. The formatter moves to `shared/i18n` and takes the selected locale.+- The unit words stay English (`MB`, `GB`), only the digits regroup.++Two decisions this raises:++`index.html` is a static Vite entry, so its `lang` and `<title>` can only follow the language at runtime. They are set from `LocaleProvider` rather than from a separate `useDocumentLanguage` hook: the provider already owns the resolved locale and runs before first paint, and a second hook would need a consumer in the tree to fire at all, which is exactly the coupling the provider exists to avoid. The markup keeps English values as the pre-hydration fallback.++Chinese does not follow the website's number rule, and the spec deliberately says so. `website/AGENTS.md` fixes `zh 180 万` for millions, qualified as the form for prose. `(1800000).toLocaleString('zh')` returns `1,800,000`; the 万 form needs `Intl.NumberFormat` with `notation: 'compact'`. A row count in a table cell is not prose, and a compact form loses precision a size or a count needs, so the interface shows full grouped numerals in every language and reserves 万 for the website's marketing copy. Verified: `en 1,800,000`, `ru 1 800 000`, `fr 1 800 000`, `es/pt 1.800.000`, `zh 1,800,000`.++### Two axes: words follow the language, clocks follow the region++The product ships one control, and it selects a language. Formatting conventions are a second, independent axis, and conflating them silently is how a plan like this goes wrong. The split adopted here:++- Words follow the selected language. Labels, month and weekday names, component-library text, status labels.+- Digit grouping follows the selected language too. A user who picks French and reads `1,234.5` will file it as a bug, and the product gives them no other control to fix it. This is a deliberate simplification, not a claim that grouping is linguistic.+- Clock format and date order keep following the browser. `shared/time/getUserTimeFormat.ts:3,43` and `shared/time/utils.ts:5` derive 12-hour versus 24-hour and the day/month order from `navigator.language`, and they stay that way. Language does not determine these: English speakers use 12-hour in the United States and 24-hour in the United Kingdom, so deriving a clock from a language selection would be guessing. The browser reports the user's actual region, which is the better signal.++The residual oddity is that a French selection can show French month names above 12-hour AM/PM times. That is correct rather than inconsistent: the words are in the reader's language and the clock is in their region's convention.++Durations (`formatDuration`, `2h 15m 3s`) need no work under this split. They are technical readouts whose digits never reach a grouping threshold, and their unit letters stay English along with the other unit words.++### Locale persistence mirrors the theme++`localStorage` under `databasus-locale`. No user profile field: the backend has no notion of a user language, and adding one only becomes necessary when notifier emails need translating, which is outside this change.++Reading the stored value during provider initialization (as `ThemeProvider` does with `getStoredTheme`) is what satisfies the "no flash of English" scenario in the spec.++Initial detection reads the browser, not the operating system: a page cannot observe the system setting. What it has is `navigator.languages`, the browser's ordered preference list, which usually but not always follows the system.++The rule: walk `navigator.languages` in order, take the first entry whose primary subtag matches a supported language, fall back to English. Walking the list matters. A user whose preferences are `["de", "ru"]` genuinely prefers Russian over English, and checking only `navigator.language` would hand them English.++Region subtags are stripped, which forces two mappings that should be stated rather than discovered:++- `zh-TW` and `zh-HK` resolve to the Simplified Chinese dictionary, though their users read Traditional. Simplified is what the website ships and what this change ships.+- `pt-PT` resolves to the Brazilian Portuguese dictionary, matching `website/AGENTS.md`, which fixes Portuguese as pt-BR.++Both are the least-bad option while only one variant exists per language, and both are a reason to add a variant later rather than a defect to fix now. `website/app/i18n.ts` already carries the same kind of mapping in `HTML_LANG_CODES` (`zh: "zh-CN"`).++### Backend errors: fix the frontend half of the contract now++The backend returns roughly 416 free-text English messages, surfaced by 20+ `alert(e.message)` calls. Translating them is out of scope, but the frontend's handling is not: it standardizes on translating by `ApiError.code` and falling back to `message`. The backend can then acquire codes incrementally without any further frontend rework. `ConnectionErrorCode` in `backend/internal/features/databases/databases/postgresql/shared/connection_error.go` is the existing precedent on the backend side.++## Risks / Trade-offs++- **The migration stalls half-finished.** → The per-directory lint rule makes progress visible and irreversible, and each slice is an independently shippable commit. The interface is bilingual in the meantime, which is worse than English but better than a long-lived branch.+- **Text expansion breaks layouts.** Russian and French run 20-30% longer than English, and AntD forms with fixed `labelCol` will wrap or clip. → Visual checks are done in Russian, not English. Russian is migrated alongside English from the first slice precisely so this surfaces immediately rather than at step 4.+- **Chinese is shorter and denser**, which hides different problems (line height, mixed CJK and Latin spacing). → Spot-check Chinese on the table-heavy screens once its dictionary lands.+- **Existing tests assert English text.** Twelve test files exist; some assert on user-facing copy. → Assert on keys or on stable identifiers rather than rendered prose, changed per slice as it is migrated.+- **The exclusion list for technical strings grows into a loophole.** → Prefer inline suppression with a reason for one-offs; only recurring patterns go in the config.+- **Six dictionaries in one bundle.** At ~1,500 strings each the total is meaningful but not alarming. → Ship all six initially and measure; dynamic import per language is a contained follow-up if the number justifies it.+- **The type-level completeness check makes adding a language all-or-nothing.** → Accepted deliberately. A partially translated language is worse than an absent one, and this is the mechanism that guarantees it cannot ship.++## Migration Plan++No data migration and no deployment coupling: this is a frontend build. Rollback for any slice is reverting its commit, which also reverts its lint config entry.++Sequencing is by slice, smallest first, English and Russian together:++```+settings(4) -> databases(159) -> restores(13) -> healthcheck(19) -> workspaces(34)+ -> users(48) -> verification(60) -> notifiers(102) -> backups(119) -> storages(136)+ -> entity/, widgets/, pages/, shared/ui/ (~136 strings)+```++Counts are the crude-grep numbers and understate the real ones by roughly half; the ordering holds regardless.++`databases` is second rather than last on purpose. Strict smallest-first validates the key namespace only on trivial slices, so a design flaw surfacing in the largest one would mean reworking every slice already migrated. Doing it right after the infrastructure is proven costs fluency and buys the ability to revise the key layout while only one other slice depends on it.++Spanish, Portuguese, Chinese and French are written only after the key set has stopped moving. Translating earlier means redoing the work each time a key is renamed.++## Open Questions++- Which AntD locale bundles to import statically versus dynamically. Affects bundle size only, decidable once real numbers exist.+- Whether the paired control should open both halves on hover, or keep the theme half on click as it is today. The website opens the language list on hover; `ThemeToggleComponent` uses an AntD `Dropdown` with `trigger={['click']}`. Two different triggers inside one visually joined button is odd, but changing the theme control's behaviour is scope the user has not asked for. Default: the language half opens on hover, the theme half keeps click, revisit after seeing it built.+- Whether `common.*` should hold shared verbs (`Save`, `Cancel`, `Delete`) or whether each domain repeats them. Sharing risks a verb that reads correctly in one context and wrong in another, in languages with grammatical gender or aspect. Resolvable during the first two slices, with the default being: share only where every language confirms the wording carries over.diff --git a/openspec/changes/translate-frontend-ui/proposal.md b/openspec/changes/translate-frontend-ui/proposal.mdnew file mode 100644index 00000000..13e8e970--- /dev/null+++ b/openspec/changes/translate-frontend-ui/proposal.md@@ -0,0 +1,77 @@+## Why++The marketing website ships in six languages, but the product behind it is English only. A user who arrives through the Russian or Chinese landing page logs in and hits an entirely English interface. The website translation set the audience expectation; the application has to meet it.++There is also a window that closes. Today's 32,700 lines of frontend code carry roughly 1,500 hardcoded strings, and that number grows with every feature. Extraction gets more expensive the longer it waits, and nothing currently stops a new hardcoded string from landing.++## Governing docs++This change answers to [`AGENTS.md`](../../../AGENTS.md) at the repo root and to [`frontend/AGENTS.md`](../../../frontend/AGENTS.md). It also amends the root doc's language rule, which [`assets/readme/AGENTS.md`](../../../assets/readme/AGENTS.md) restates, so that doc is in scope for consistency even though no README translation changes. It touches no backend, agent or website code, so those module docs do not apply. `website/AGENTS.md` is relevant only as prior art: its translation-quality rules are the reference for how each language is written.++### **BREAKING** rule change, approved and applied++The root doc's "Language in code" section permitted translated end-user content in three places and nowhere else: the website page copies, the README translations, and the six language-switcher labels in the root `README.md`. It forbade any language other than English inside `frontend/src/`, fallback copy and error messages included.++This change puts Russian, Spanish, Portuguese, Chinese and French text into `frontend/src/shared/i18n/locales/*.ts`, which that wording forbade. The rule now names a fourth permitted location, and nothing about the change is implementable without it.++The amendment adds exactly one location, `frontend/src/shared/i18n/locales/<locale>.ts`, and changes nothing else: identifiers, file names, dictionary keys, comments, log messages, API strings, test assertions and commit messages stay English, and every other file under `frontend/src/` stays English including fallback copy and error messages. It is recorded in the root `AGENTS.md` under "Language in code" (task 2.2).++## What Changes++- Add an i18n layer under `frontend/src/shared/i18n`: `i18next` + `react-i18next`, a `LocaleProvider` mirroring the existing `ThemeProvider`, and a language control in the navbar.+- Ship the same six languages as the website: `en` (source of truth) plus `ru`, `es`, `pt`, `zh`, `fr`.+- Author locale dictionaries as TypeScript modules typed against the English one, so a missing or stray key fails `tsc -b` instead of silently falling back at runtime.+- Replace every user-facing hardcoded string in `frontend/src` with a dictionary lookup, slice by slice.+- Add `eslint-plugin-i18next` and enable `i18next/no-literal-string` per directory as each slice is migrated, so migrated code cannot regress.+- Route AntD's own strings through `ConfigProvider locale` and date formatting through `dayjs.locale`, both following the selected language.+- Group numbers by the selected language instead of the browser's, so digits and dates on the same screen stop disagreeing.+- Set the document's language attribute and title from the selected language, replacing the hardcoded `lang="en"` and English `<title>` in `frontend/index.html`.+- Translate status labels via lookup tables keyed by the existing status enums, replacing the copy-pasted `renderStatusTag` / `renderStatus` variants. Enum values stay English: they are wire protocol.+- **BREAKING** for contributors, not users: after a slice is migrated, a literal string in its JSX fails the lint job.++### The language control++The website's `LanguageSelectorComponent` is the reference: a compact button in the navbar showing the current language, a dropdown that opens on hover and lists the six languages by their native names.++Here it is merged with the existing theme control into one paired button, language on one side and Dark/Light/System on the other, in the style the website already uses to glue the language selector to the GitHub button (`isSplitEnd`).++That merge also settles placement. `ThemeToggleComponent` already renders in three places, and the paired control replaces it at all three:++- `widgets/main/MainScreenComponent.tsx:254` - the main navbar+- `widgets/main/SidebarComponent.tsx:132` - the sidebar, for narrow viewports+- `features/users/ui/AuthNavbarComponent.tsx:49` - the authentication screen++So the language is selectable before login, not only after it. The choice is stored in `localStorage`, and on a first visit with nothing stored the interface starts in the highest-ranked language the browser prefers that the product actually ships.++### Out of scope++- The backend. Its ~416 free-text error responses keep reaching the user in English through `alert(e.message)`. This change only fixes the frontend's side of the contract: translate by `ApiError.code` when one is present, fall back to `message` when it is not, so the backend work later is additive.+- Replacing `alert()` with AntD notifications (20+ sites). A real improvement, a separate change.+- Unit words stay English: `h`/`m`/`s` in `formatDuration`, `MB` and `GB` in sizes. `formatDuration` needs no change at all, its digits being too small to group. The digits in sizes and counts are in scope and follow the selected language.+- Clock format and date order stay derived from the browser, not from the selected language. See design.md - Two axes.+- Technical strings: engine names, `pg_dump` and friends, connection strings, command blocks, cron expressions.+- Persisting the language choice server side. It lives in `localStorage`, like the theme.++## Capabilities++### New Capabilities++- `frontend-localization`: the frontend renders its interface in a user-selected language, keeps dictionaries provably complete, and prevents untranslated strings from being added.++### Modified Capabilities++None. This is the repository's first change; `openspec/specs/` is empty.++## Impact++- `frontend/package.json`: adds `i18next`, `react-i18next`, `eslint-plugin-i18next`. Runtime cost is roughly 40 KB plus one dictionary per session.+- `frontend/eslint.config.js`: gains one config object per migrated directory.+- `frontend/src/App.tsx`: `LocaleProvider` wraps the tree, `ConfigProvider` receives an AntD locale.+- `frontend/index.html`: the hardcoded `lang="en"` and English `<title>` are set from the selected locale instead.+- Eight `toLocaleString()` call sites: number grouping follows the selected language rather than the browser's.+- `frontend/src/shared/ui/ThemeToggleComponent.tsx`: becomes half of a paired control; its three call sites change with it.+- `frontend/src/features/**` (105 files, 23,800 lines), `frontend/src/entity/**` (~136 strings), `frontend/src/widgets/**`, `frontend/src/pages/**`: every user-facing literal moves to a dictionary key.+- `AGENTS.md` (root): the "Language in code" rule gains a fourth permitted location. See the rule change above.+- `frontend/AGENTS.md`: the copy rules gain the i18n convention.+- Existing frontend tests that assert on English text.+- No backend, agent, or API changes.diff --git a/openspec/changes/translate-frontend-ui/specs/frontend-localization/spec.md b/openspec/changes/translate-frontend-ui/specs/frontend-localization/spec.mdnew file mode 100644index 00000000..05e43260--- /dev/null+++ b/openspec/changes/translate-frontend-ui/specs/frontend-localization/spec.md@@ -0,0 +1,193 @@+## Purpose++Lets a user read the Databasus interface in their own language, and gives the project a way to keep every language complete as the product changes.++## ADDED Requirements++### Requirement: The interface renders in the selected language++The frontend SHALL render all user-facing copy in one of six languages: English, Russian, Spanish, Portuguese, Chinese and French. Copy includes labels, placeholders, button text, headings, table columns, empty states, confirmation dialogs, tooltips and validation messages.++Technical strings are excluded and stay identical in every language: command lines and code blocks, connection strings, cron expressions, database engine and tool names, file paths, and duration readouts such as `2h 15m 3s` in full, and the unit words in size readouts such as the `GB` in `1.4 GB`. The digits in a size or a count are not excluded; see the number formatting requirement below.++#### Scenario: A user switches language++- **WHEN** a user selects Russian+- **THEN** every visible label, button and message re-renders in Russian without a page reload+- **AND** command blocks, connection strings and engine names remain byte-identical to what English showed++### Requirement: The language control sits beside the theme control++The language control SHALL be presented together with the light/dark theme control as a single paired element, and SHALL appear everywhere that element appears: the main navigation bar, the narrow-viewport sidebar, and the authentication screen. A user SHALL be able to choose a language before signing in.++The control SHALL show which language is currently active, and SHALL list all six by their native names rather than by language code or flag.++#### Scenario: Choosing a language before signing in++- **WHEN** an unauthenticated user opens the authentication screen+- **THEN** the language control is visible there+- **AND** selecting a language translates the authentication screen immediately and keeps that language after signing in++#### Scenario: Opening the language list++- **WHEN** a user moves the pointer onto the language control+- **THEN** the list of six languages opens without requiring a click+- **AND** moving the pointer from the control to the list does not close it++#### Scenario: Narrow viewport++- **WHEN** the viewport is too narrow for the main navigation bar+- **THEN** the paired language and theme control remains reachable in the sidebar++### Requirement: The language choice defaults to the browser's preference and persists++On a first visit with no stored choice, the frontend SHALL take the browser's ordered list of preferred languages and use the first entry that matches a supported language, ignoring any region subtag, and English when none matches. Once a user makes a choice, that choice SHALL be stored in the browser and applied on subsequent visits.++#### Scenario: First visit with Spanish preferred++- **WHEN** a user whose browser reports Spanish as its preferred language opens the application for the first time+- **THEN** the interface renders in Spanish++#### Scenario: The first preference is unsupported++- **WHEN** a user whose browser reports German first and Russian second opens the application for the first time+- **THEN** the interface renders in Russian rather than English, because Russian is the highest-ranked supported preference++#### Scenario: No preference is supported++- **WHEN** a user whose browser reports no supported language opens the application for the first time+- **THEN** the interface renders in English++#### Scenario: A regional variant with no dictionary of its own++- **WHEN** a user whose browser reports Taiwanese Chinese or European Portuguese opens the application for the first time+- **THEN** the interface renders in the single Chinese or Portuguese variant the product ships, rather than falling through to English++#### Scenario: An explicit choice outranks the browser preference++- **WHEN** a user whose browser prefers Spanish selects English+- **THEN** the interface renders in English on this and every later visit from the same browser++#### Scenario: The choice survives a reload++- **WHEN** a user has selected a language and then reloads the page or returns in a new session on the same browser+- **THEN** the interface renders in the previously selected language+- **AND** the first paint is already in that language, with no visible flash of English++### Requirement: Library and framework text follows the selected language++Text produced by the component library and by date formatting SHALL follow the selected language, not the browser default and not English.++#### Scenario: Component library strings++- **WHEN** the interface renders in a non-English language+- **THEN** table pagination, empty-table placeholders, date pickers, select dropdowns and confirmation popovers display their built-in text in that language++#### Scenario: Dates and times++- **WHEN** the interface renders in a non-English language+- **THEN** month names, weekday names and relative times render in that language++### Requirement: Numbers are grouped by the selected language++Numeric values displayed to the user SHALL be grouped and separated according to the selected language rather than the browser's own locale, using each language's standard digit grouping. Unit words attached to a number stay English. Abbreviated forms that read as prose, such as the Chinese 万 form the website uses for millions in running text, are not used in the interface: counts and sizes appear in full.++#### Scenario: A browser locale that differs from the selected language++- **WHEN** a user whose browser is English selects French, and a screen shows a row count of one million eight hundred thousand+- **THEN** the number is grouped in the French convention rather than the English one+- **AND** it does not disagree with the dates on the same screen, which already follow the selected language++#### Scenario: Clock and calendar conventions are regional, not linguistic++- **WHEN** a user selects a language whose speakers commonly use a 24-hour clock, while their browser reports a region that uses a 12-hour clock+- **THEN** times continue to display in the 12-hour form their browser reports+- **AND** the words around those times, including month and weekday names, are still in the selected language++#### Scenario: A size readout++- **WHEN** a backup size is displayed in a language that groups digits differently from English+- **THEN** the digits are grouped for that language+- **AND** the unit remains `MB` or `GB`++### Requirement: The document reports the selected language++The page SHALL declare the selected language to the browser, and its title SHALL be translated.++#### Scenario: Assistive technology and browser translation++- **WHEN** the interface renders in a given language+- **THEN** the document's language attribute reports that language+- **AND** the browser does not offer to translate a page that is already in the reader's language++#### Scenario: The browser tab++- **WHEN** the interface renders in a non-English language+- **THEN** the browser tab and any bookmark made from it show a title in that language++### Requirement: Status labels are translated while wire values stay stable++Status values exchanged with the backend SHALL remain unchanged English identifiers. Their displayed labels SHALL be translated.++#### Scenario: A backup status is displayed++- **WHEN** the backend reports a backup status of `COMPLETED` and the interface renders in Spanish+- **THEN** the user sees the Spanish label for a completed backup+- **AND** the value sent back to the backend in any subsequent request is still `COMPLETED`++#### Scenario: A new status value is introduced++- **WHEN** a new value is added to a status enum and no label is supplied for it+- **THEN** the type check fails++### Requirement: Dictionary completeness is enforced by the type checker++Each non-English dictionary SHALL be checked against the English dictionary at build time. English is the source of truth for the set of keys; the wording of each language is written independently and is not a literal rendering of the English text.++#### Scenario: A key is missing from a translation++- **WHEN** a key exists in the English dictionary and is absent from any other language+- **THEN** the type check fails and names the missing key++#### Scenario: A key is left over in a translation++- **WHEN** a key is removed from the English dictionary but remains in another language+- **THEN** the type check fails and names the stray key++#### Scenario: A lookup uses an unknown key++- **WHEN** code requests a key that does not exist in the English dictionary+- **THEN** the type check fails++### Requirement: Migrated code cannot regress to hardcoded strings++Once a directory has been migrated, the lint job SHALL reject a user-facing string literal added to it. Directories not yet migrated are exempt until they are.++#### Scenario: A hardcoded string is added to migrated code++- **WHEN** a contributor adds a literal label, placeholder or message to a migrated directory+- **THEN** the lint job fails and points at the literal++#### Scenario: A technical string is added to migrated code++- **WHEN** a contributor adds a command block, connection string or engine name to a migrated directory+- **THEN** the lint job accepts it, either through the configured exclusions or through an inline suppression carrying a reason++### Requirement: Backend errors are translated by code, with the message as fallback++The frontend SHALL translate an error response by its stable error code when the backend supplies one, and SHALL display the backend's message text only when no code is present. This change does not add codes to the backend.++#### Scenario: The backend supplies a known error code++- **WHEN** a request fails with an error code the frontend has a translation for+- **THEN** the user sees the translated message in the selected language++#### Scenario: The backend supplies no code++- **WHEN** a request fails with only a free-text message+- **THEN** the user sees that message as-is, in a container whose surrounding copy is translated++#### Scenario: The backend supplies an unrecognized code++- **WHEN** a request fails with an error code the frontend has no translation for+- **THEN** the user sees the backend's message text rather than a raw code or a blank messagediff --git a/openspec/changes/translate-frontend-ui/tasks.md b/openspec/changes/translate-frontend-ui/tasks.mdnew file mode 100644index 00000000..382ba736--- /dev/null+++ b/openspec/changes/translate-frontend-ui/tasks.md@@ -0,0 +1,66 @@+## 1. i18n infrastructure++- [ ] 1.1 Add `i18next`, `react-i18next` and `eslint-plugin-i18next` to `frontend/package.json` and verify `pnpm install` succeeds and `pnpm build` still passes+- [ ] 1.2 Create `shared/i18n/locales/en.ts` with the top-level namespaces (`common`, `status`, `errors`, plus one per domain) and verify `tsc -b` passes+- [ ] 1.3 Add the `i18next` module augmentation typing `resources` as `typeof en`, and verify that `t('does.not.exist')` is a type error while a real key is not+- [ ] 1.4 Create `shared/i18n/locales/ru.ts` typed `const ru: typeof en`, and verify that deleting one key from it makes `tsc -b` fail with that key named+- [ ] 1.5 Configure the i18next instance with `returnNull: false`, `nsSeparator: false`, `interpolation.escapeValue: false` and `fallbackLng: 'en'`, and verify `t()` is typed `string` at call sites+- [ ] 1.6 Add `LocaleProvider` and `useLocale` in `shared/i18n`, mirroring `shared/theme/ThemeProvider.tsx`, reading `databasus-locale` from `localStorage` during initialization and falling back to `navigator.language` matched against the six supported languages, then English, and verify a stored locale is applied on first paint with no flash of English+- [ ] 1.7 Wrap the tree in `LocaleProvider` in `App.tsx` and pass the matching AntD locale to `ConfigProvider`, and verify table pagination and an empty table render in Russian when Russian is selected+- [ ] 1.8 Wire `dayjs.locale()` to follow the selected locale and verify month and weekday names in a `DatePicker` change with the language+- [ ] 1.8a Set `document.documentElement.lang` and the document title from the selected locale, replacing the hardcoded `lang="en"` and English `<title>` in `frontend/index.html`, and verify the tab title and the reported language both change with the selection+- [ ] 1.8b Add a locale-aware number formatter in `shared/i18n` and verify with a unit test that an English browser with French selected groups digits the French way, that Chinese groups Western-style rather than in the 万 form, and that unit words are not its concern+- [ ] 1.8c Confirm that `shared/time/getUserTimeFormat.ts:3,43` and `shared/time/utils.ts:5` keep deriving clock format and date order from `navigator.language`, and record in `frontend/AGENTS.md` why those three reads are correct where a bare `toLocaleString()` is not, so a later reader does not "fix" them+- [ ] 1.9 Build the paired language and theme control in `shared/ui`, with the language half showing the current language and opening its native-name list on hover, using contiguous hover padding rather than a margin so crossing from button to list does not close it, and verify the list stays open while the pointer moves onto it+- [ ] 1.10 Replace `ThemeToggleComponent` with the paired control at all three call sites (`widgets/main/MainScreenComponent.tsx:254`, `widgets/main/SidebarComponent.tsx:132`, `features/users/ui/AuthNavbarComponent.tsx:49`) and verify the control renders in the navbar, in the narrow-viewport sidebar and on the authentication screen+- [ ] 1.11 Verify end to end that a language chosen before signing in survives the sign-in, that a first visit on a Spanish system starts in Spanish, that an unsupported system language starts in English, and that an explicit choice outranks the system language on later visits++## 2. Lint enforcement++- [ ] 2.1 Add an `i18next/no-literal-string` config object to `frontend/eslint.config.js` scoped to `src/features/settings/**` with exclusions for technical strings and non-prose AntD props, and verify `pnpm lint` passes on the migrated slice and fails when a literal label is reintroduced+- [x] 2.2 Amend the "Language in code" section of the root `AGENTS.md` to permit target-language text in `frontend/src/shared/i18n/locales/<locale>.ts` and nowhere else under `frontend/src/`, and verify the amended wording still forbids translated fallback copy and error messages outside that directory. Do not start section 5 until this task is done and approved+- [ ] 2.3 Document the i18n convention in `frontend/AGENTS.md`: dictionary keys are domain-scoped, English is the key source of truth, technical strings stay literal, and each migrated directory gets a lint config entry++## 3. Error handling contract++- [ ] 3.1 Add an `errors` namespace keyed by backend error code and a `translateApiError` function that resolves an `ApiError` by `code`, falling back to `message` when the code is absent or unknown, and verify all three branches with a unit test+- [ ] 3.2 Route the existing `physicalConnectionErrorContent` lookup in `EditPostgreSqlPhysicalSpecificDataComponent.tsx` through that helper and verify the physical connection error paths still show their specific copy++## 4. Status labels++- [ ] 4.1 Add a `status` namespace and a typed `Record<Enum, TranslationKey>` beside each of the seven status enums in `entity/*/model/`, and verify that adding an enum member without a label fails `tsc -b`+- [ ] 4.2 Replace the duplicated `renderStatusTag` in `VerificationDetailDrawer.tsx` and `renderStatus` in `VerificationsComponent.tsx` with lookups against the shared table, keeping each component's own markup, and verify both screens render the same labels++## 5. Slice migration (English and Russian together)++Ordering is deliberate and not by size alone. `features/settings` comes first to prove the infrastructure on four strings. `features/databases` comes second, out of size order, because it is the largest and most varied slice: if the key namespace design breaks anywhere it breaks there, and finding that out after nine migrated slices would mean reworking all of them. The cost is doing the hardest slice before the pattern is fluent. The rest then run ascending.++- [ ] 5.1 Migrate `features/settings` and enable the lint rule for it, and verify `pnpm lint`, `pnpm test` and `tsc -b` pass+- [ ] 5.2 Migrate `features/databases`, the largest slice, and enable the lint rule for it, then review whether the namespace design survived it and revise the key layout before continuing if it did not, and verify the same three commands pass+- [ ] 5.3 Migrate `features/restores` and enable the lint rule for it, and verify the same three commands pass+- [ ] 5.4 Migrate `features/healthcheck` and enable the lint rule for it, and verify the same three commands pass+- [ ] 5.5 Migrate `features/workspaces` and enable the lint rule for it, and verify the same three commands pass+- [ ] 5.6 Migrate `features/users` and enable the lint rule for it, and verify the same three commands pass+- [ ] 5.7 Migrate `features/verification` and enable the lint rule for it, and verify the same three commands pass+- [ ] 5.8 Migrate `features/notifiers` and enable the lint rule for it, and verify the same three commands pass+- [ ] 5.9 Migrate `features/backups` and enable the lint rule for it, and verify the same three commands pass+- [ ] 5.10 Migrate `features/storages` and enable the lint rule for it, and verify the same three commands pass+- [ ] 5.10a Replace the bare `toLocaleString()` calls with the formatter from 1.8b as each owning slice is migrated (`LogicalBackupsComponent.tsx:513,515` and `PhysicalBackupsComponent.tsx:49,52` with `features/backups`, `VerificationDetailDrawer.tsx:45,48,157` with `features/verification`, `StarButtonComponent.tsx:56` with `shared/ui/`), rather than in a separate pass over the same files, and verify no bare `toLocaleString()` survives+- [ ] 5.11 Migrate the remaining copy in `entity/`, `widgets/`, `pages/` and `shared/ui/`, including `ConnectionStringParser` error text and `getNotifierNameFromType`, and enable the lint rule for those directories, and verify the same three commands pass+- [ ] 5.12 Update frontend tests that assert on rendered English prose to assert on keys or stable identifiers instead, and verify `pnpm test` passes++## 6. Remaining languages++- [ ] 6.1 Confirm the key set has stopped changing after section 5, then write `es.ts`, and verify `tsc -b` passes with no missing or stray keys+- [ ] 6.2 Write `pt.ts` (Brazilian Portuguese, matching the website's pt-BR conventions) and verify `tsc -b` passes+- [ ] 6.3 Write `fr.ts` and verify `tsc -b` passes+- [ ] 6.4 Write `zh.ts` (Simplified, matching the website's conventions) and verify `tsc -b` passes+- [ ] 6.5 Add each new locale to the switcher, the AntD locale map and the dayjs locale map, and verify each language renders end to end++## 7. Verification++- [ ] 7.1 Walk every screen in Russian and French looking for clipped or wrapped labels in AntD forms with fixed `labelCol`, and fix the layouts that break+- [ ] 7.2 Spot-check Chinese on the table-heavy screens (backups, databases, verification runs) for line height and CJK-Latin spacing+- [ ] 7.3 Confirm in every language that command blocks, connection strings, cron expressions, engine names and the unit words in duration and size readouts are byte-identical to English, while the digits inside those readouts follow the selected language+- [ ] 7.4 Confirm in every language that the browser tab title and the document language attribute match the selection, and that no bare `toLocaleString()` remains+- [ ] 7.5 Run `pnpm lint`, `pnpm format`, `pnpm test` and `pnpm build` from `frontend/` and confirm all passdiff --git a/openspec/specs/physical-backups/replication-credentials/spec.md b/openspec/specs/physical-backups/replication-credentials/spec.mdnew file mode 100644index 00000000..83e1946b--- /dev/null+++ b/openspec/specs/physical-backups/replication-credentials/spec.md@@ -0,0 +1,94 @@+# Replication Credentials Specification++## Purpose++Covers what a Databasus-provisioned replication user can do on a PostgreSQL source, and which source capabilities Databasus requires before it accepts a physical-backup configuration that promises recovery to a point in time.++## Requirements++### Requirement: A provisioned replication user can force WAL rotation where the platform allows it++When Databasus provisions a replication user on a source cluster, it SHALL also give that user the ability to force a WAL segment switch, so the recovery point of a rarely-written database is bounded by the rotation interval rather than by how long the current segment takes to fill.++Where the connected administrator cannot confer that ability, provisioning SHALL still succeed and deliver a working replication user, and SHALL report that forced WAL rotation is unavailable with those credentials.++#### Scenario: Self-managed source with an administrator that can confer the privilege++- **WHEN** a user asks Databasus to create a replication user on a self-managed PostgreSQL cluster, connected as an administrator able to confer it+- **THEN** the created user can force a WAL segment switch+- **AND** the credentials are accepted for a configuration that streams WAL++#### Scenario: Managed platform that withholds the privilege++- **WHEN** the same request is made against a managed platform whose administrator cannot confer the privilege+- **THEN** the replication user is still created and usable for physical backups+- **AND** the result reports that forced WAL rotation is unavailable++### Requirement: Provisioning widens the role's privileges by exactly one ability++Provisioning SHALL confer the ability to force a WAL segment switch and nothing else. The provisioned role SHALL remain limited to replication: no data access, no ability to create roles or databases, and no membership in an administrative role.++The ability SHALL be conferred on the provisioned role alone, never on a group that would extend it to other roles on the cluster. Where the source refuses, the cluster's privileges SHALL be left exactly as they were, with no partial or substitute grant.++#### Scenario: The provisioned role's reach after a successful grant++- **WHEN** provisioning succeeds including the ability to force a segment switch+- **THEN** the role can stream WAL and force a segment switch+- **AND** it still cannot read table data, create roles or databases, or act as an administrator++#### Scenario: No other role gains the ability++- **WHEN** provisioning confers the ability to force a segment switch+- **THEN** no role other than the one Databasus just created gains it++#### Scenario: The source refuses++- **WHEN** the connected administrator cannot confer the ability+- **THEN** the source's privileges are unchanged apart from the replication user's own creation+- **AND** no weaker or broader substitute privilege is granted in its place++### Requirement: A configuration that promises recovery to a point in time is refused unless the source can bound it++Recovery to an arbitrary point in time replays WAL segments that reached storage, and a segment reaches storage only once the source closes it. When a user configures continuous WAL streaming, Databasus SHALL refuse the configuration unless the supplied credentials can force a WAL segment switch.++The refusal SHALL name both ways out: confer the ability on the source, or choose a backup type that does not replay archived WAL. It SHALL NOT direct the user to enable archiving on the source, which would make the cluster retain every segment until an archiver confirms it. The refusal SHALL be reported the same way as the other unmet preconditions of a physical source, so a user meets it at the same moment they meet the rest.++This SHALL apply to credentials a user enters by hand as well as to credentials Databasus provisioned, and to a database being created as well as one being changed.++#### Scenario: Streaming configuration with credentials that cannot force a switch++- **WHEN** a user configures continuous WAL streaming with credentials that can stream WAL but cannot force a segment switch+- **THEN** the configuration is refused+- **AND** the refusal names conferring the ability and choosing a backup type without WAL replay++#### Scenario: A managed platform that will not confer the ability++- **WHEN** the source is a managed platform whose administrator cannot confer the ability on any role+- **THEN** the streaming configuration is refused there as well+- **AND** the refusal says continuous WAL streaming is unavailable on that source, rather than offering a source setting that would make the cluster retain WAL indefinitely++#### Scenario: Streaming configuration with credentials that can force a switch++- **WHEN** the credentials can force a segment switch+- **THEN** the configuration is accepted++#### Scenario: An existing streaming database that no longer qualifies++- **WHEN** a user changes a saved streaming database whose credentials cannot force a segment switch+- **THEN** the change is refused with the same explanation+- **AND** the backups already running for that database are not stopped by the refusal++### Requirement: Backup types that never replay archived WAL are not held to this requirement++A backup type that restores only to the point a backup itself finished does not read an archived segment, so the ability to force a segment switch SHALL NOT be required of it. Databasus SHALL accept such a configuration with credentials that can only stream.++#### Scenario: Full backups with credentials that cannot force a switch++- **WHEN** a user configures full backups, with or without incrementals, using credentials that cannot force a segment switch+- **THEN** the configuration is accepted+- **AND** backups and restores to those backups work with no privilege error++#### Scenario: Raising the backup type later++- **WHEN** a user changes such a database to stream WAL continuously+- **THEN** the requirement above applies to the change, and it is refused if the source cannot bound the recovery pointdiff --git a/openspec/specs/physical-backups/wal-retention/spec.md b/openspec/specs/physical-backups/wal-retention/spec.mdnew file mode 100644index 00000000..5b195421--- /dev/null+++ b/openspec/specs/physical-backups/wal-retention/spec.md@@ -0,0 +1,92 @@+# WAL Retention Specification++## Purpose++Defines which archived WAL segments a database keeps and which the retention sweep may delete, stated in terms of the recovery chains a user can still restore from. A segment is expendable only when no retained backup needs it.++## Requirements++### Requirement: A WAL segment overlapping a retained backup's recovery window is never deleted as unreferenced++The retention sweep SHALL treat a WAL segment as unreferenced only when no retained, successfully completed physical full backup on the same timeline begins before the end of that segment.++Recovery positions inside a full backup do not fall on segment boundaries: a backup begins a short distance into the segment that holds it. The segment holding that position therefore begins earlier than the backup does, and it still carries every write recorded after the backup finished. That segment SHALL be retained.++#### Scenario: The full backup starts and ends inside one segment++- **WHEN** a full backup's start and end positions both fall inside a single WAL segment, and that segment has been archived+- **THEN** the retention sweep leaves the segment in place+- **AND** a restore to any point in time after the backup's end position and covered by the archived WAL succeeds++#### Scenario: A segment entirely older than every retained backup++- **WHEN** an archived WAL segment ends at or before the start of the earliest retained full backup on its timeline+- **THEN** the retention sweep deletes it as unreferenced++#### Scenario: The anchoring backup is pruned++- **WHEN** retention removes the oldest full backup, and a later full backup remains whose start position falls inside a WAL segment that the removed backup used to anchor+- **THEN** that segment is still retained, because the remaining backup starts before the segment ends+- **AND** point-in-time restore from the remaining backup stays available++### Requirement: A completed full backup leaves an unbroken recovery chain++Once a physical full backup reports success and WAL streaming stays healthy, the system SHALL keep the archived WAL contiguous from that backup's end position onward, for as long as the backup is retained. No background maintenance SHALL introduce a gap immediately after a successful backup.++#### Scenario: Retention runs seconds after a full backup completes++- **WHEN** a full backup completes, its boundary WAL segment is archived, and the retention sweep runs before any further backup exists+- **THEN** the archived WAL is unbroken from the backup's end position to the newest archived segment+- **AND** the reported latest restorable point is later than the backup's end position rather than equal to it++#### Scenario: Restoring to a point shortly after the backup++- **WHEN** a user requests a restore to a point in time a few seconds after a successful full backup, and WAL covering that point has been archived+- **THEN** the restore proceeds instead of being refused for a WAL gap++#### Scenario: Restoring while a newer full backup is running++- **WHEN** a newer full backup has started but has not completed, and the archived WAL remains contiguous from the preceding completed full backup+- **THEN** a point-in-time restore during that interval uses the preceding completed full backup+- **AND** the restore set contains the full available contiguous WAL run, including WAL beyond the newer backup's start and stop positions++### Requirement: WAL archived during an active full backup is protected++No orphan or cascade deletion SHALL remove WAL for a database while that database has an active FULL backup claim. Scheduling the claim and every WAL-deleting maintenance path SHALL be serialized for each database, and each deletion transaction SHALL recheck the claim before removing a segment.++#### Scenario: All backups were deleted and a replacement full is running++- **WHEN** no completed full backup remains, a replacement FULL has an active claim, and WAL is archived during that backup+- **THEN** the orphan sweep leaves the WAL in place+- **AND** the WAL becomes anchored when the FULL completes++#### Scenario: An in-progress row has no active claim++- **WHEN** an `IN_PROGRESS` full backup row remains without an active FULL claim and no completed full backup anchors the WAL+- **THEN** the orphan sweep may reclaim the WAL++#### Scenario: Cascade deletion overlaps replacement full claim creation++- **WHEN** full removal or `FULL_BACKUPS` dependent cleanup starts while a replacement FULL claim is being committed+- **THEN** the cascade waits for claim creation to finish+- **AND** it leaves WAL in place after observing the active claim+- **AND** full removal leaves timeline history in place until the active FULL publishes a completed anchor+- **AND** the deletion preview reports zero deletable WAL and timeline history while that claim remains active++### Requirement: Deleting a full backup preserves WAL required by a surviving predecessor++Deleting a completed full backup SHALL retain its WAL when an older completed full backup remains on the same database and timeline. The older backup becomes the restore base after deletion and SHALL remain capable of replaying the shared WAL. The deletion preview SHALL report the same WAL count and size that the destructive operation can remove.++This rule applies when the FULL itself is removed. A `FULL_BACKUPS` retention policy may deliberately keep a FULL as a standalone restore point while removing its incrementals and owned WAL.++#### Scenario: A newer full backup is deleted++- **WHEN** a completed full backup is deleted while an older completed full backup remains on the same timeline+- **THEN** no WAL is deleted with the newer full backup+- **AND** point-in-time restore during the deleted backup's former interval succeeds from the older backup++#### Scenario: The earliest surviving full backup is deleted++- **WHEN** a completed full backup has no older completed predecessor on its timeline+- **THEN** deletion removes only WAL segments fully contained in its ownership span+- **AND** a segment crossing the next full backup's start position remains availablediff --git a/openspec/specs/postgresql-physical-backup-failover/spec.md b/openspec/specs/postgresql-physical-backup-failover/spec.mdnew file mode 100644index 00000000..107ed3f6--- /dev/null+++ b/openspec/specs/postgresql-physical-backup-failover/spec.md@@ -0,0 +1,110 @@+# PostgreSQL Physical Backup Failover Specification++## Purpose++Keep PostgreSQL physical backups recoverable when the source moves to a new timeline. Databasus rejects incremental backups that would extend an obsolete chain, creates a replacement FULL, and preserves the WAL needed during the transition.++## Requirements++### Requirement: Incremental backups use their root FULL timeline+The system SHALL start an incremental backup only when the live PostgreSQL timeline matches the timeline of the chain's root FULL.++#### Scenario: Timeline matches the root FULL+- **WHEN** an incremental backup starts and the live timeline matches its root FULL timeline+- **THEN** the system proceeds with the incremental backup++#### Scenario: Promotion happened before the incremental started+- **WHEN** the live timeline is newer than the root FULL timeline+- **THEN** the system refuses the incremental backup as `CHAIN_BROKEN` with reason `TIMELINE_SWITCH_DETECTED`++#### Scenario: Source timeline regressed+- **WHEN** the live timeline is older than the root FULL timeline+- **THEN** the system refuses the incremental backup with reason `TIMELINE_REGRESSION`++#### Scenario: Cataloged history is newer than the root FULL+- **WHEN** a history file for the live timeline is already cataloged but the root FULL belongs to an older timeline+- **THEN** the system still refuses the incremental backup as `TIMELINE_SWITCH_DETECTED`++### Requirement: Backup failures are rechecked for promotion+The system SHALL check PostgreSQL cluster identity again when a physical backup stream fails unexpectedly while the backup is still active.++#### Scenario: Promotion interrupts an incremental stream+- **WHEN** the timeline advances after incremental preflight and the stream fails+- **THEN** the system records the incremental as `CHAIN_BROKEN` with reason `TIMELINE_SWITCH_DETECTED`++#### Scenario: Promotion interrupts a FULL stream+- **WHEN** the timeline advances after FULL preflight and the stream fails+- **THEN** the system records the FULL as `ERROR` with reason `FAILOVER_DURING_BACKUP`++#### Scenario: Backup was canceled+- **WHEN** an operator or configuration change cancels the backup+- **THEN** the system preserves the cancellation result and does not classify it as a promotion++#### Scenario: Recheck cannot connect+- **WHEN** the second identity check fails+- **THEN** the system preserves the original backup result++#### Scenario: Cluster identity changed+- **WHEN** the second identity check reaches a cluster with a different system identifier+- **THEN** the system rejects that cluster and pauses automatic backup scheduling until an operator requests a FULL that completes against the expected cluster++#### Scenario: Operator requests a FULL after cluster identity changed+- **WHEN** automatic scheduling is paused by a system identifier mismatch and an operator requests a FULL+- **THEN** the system runs preflight again and refuses to write a backup unless the live system identifier matches the configured database++#### Scenario: Forced FULL still reaches a different cluster+- **WHEN** the operator-requested FULL is refused with `SYSTEM_IDENTIFIER_MISMATCH`+- **THEN** automatic backup scheduling remains paused++#### Scenario: Forced FULL fails without proving cluster identity+- **WHEN** an operator-requested FULL fails for another reason or is canceled+- **THEN** automatic backup scheduling remains paused because no later FULL has completed against the expected cluster++#### Scenario: Forced FULL reaches the expected cluster+- **WHEN** an operator-requested FULL completes against the configured system identifier+- **THEN** that newer completed FULL supersedes earlier mismatch results and normal automatic scheduling resumes++### Requirement: Timeline failures trigger one immediate replacement FULL+The system SHALL create one immediate replacement FULL after a backup ends with a timeline-related failure.++#### Scenario: Incremental chain broke after promotion+- **WHEN** an incremental ends as `CHAIN_BROKEN` with reason `TIMELINE_SWITCH_DETECTED` and no later FULL attempt exists+- **THEN** the system schedules a FULL without waiting for the configured incremental or FULL interval++#### Scenario: FULL was interrupted by promotion+- **WHEN** the latest FULL ends as `ERROR` with reason `FAILOVER_DURING_BACKUP`+- **THEN** the system schedules one replacement FULL without waiting for the configured interval++#### Scenario: Replacement FULL fails normally+- **WHEN** the immediate replacement FULL ends with a non-timeline error or is canceled+- **THEN** the system resumes its existing schedule and failed-FULL retry limits instead of scheduling a FULL on every scheduler tick++#### Scenario: Another promotion interrupts the replacement+- **WHEN** the timeline advances again during the replacement FULL+- **THEN** the system schedules one new immediate replacement FULL for the later promotion++#### Scenario: Two schedulers hold the same stale replacement decision+- **WHEN** two scheduler instances decide that the same terminal backup needs an immediate replacement+- **THEN** only the instance that confirms the failure is still current creates a FULL row++### Requirement: Active FULL backups retain archived WAL+The system SHALL keep orphan cleanup from deleting WAL for a database while a FULL backup for that database is active.++#### Scenario: Cleaner runs during the first FULL+- **WHEN** WAL is archived while the database's first FULL is still running and orphan cleanup starts+- **THEN** the WAL remains cataloged and stored for use after the FULL completes++#### Scenario: Cleaner finishes before a FULL starts+- **WHEN** orphan cleanup starts before the scheduler starts a FULL for the same database+- **THEN** the FULL starts only after cleanup finishes++### Requirement: Same-timeline PITR remains available during a newer FULL+The system SHALL allow a point-in-time restore from an older FULL when the target comes before a newer FULL completes on the same timeline.++#### Scenario: Target falls during the newer FULL+- **WHEN** the requested target is later than the older FULL completion and earlier than the newer FULL completion+- **THEN** the restore uses the older FULL and contiguous WAL through the target, even when that WAL crosses the newer FULL start position++#### Scenario: Target crosses a promotion+- **WHEN** reaching the target would require WAL from more than one timeline+- **THEN** the system reports that the target is not restorable until a FULL on the promoted timeline is available