Skip to content

PostgreSQL backup tool with Point-In-Time-Recovery and restore verification

REFACTOR (docs): Archive the stored file cleanup change

Rostislav Dugina405404a2f54
diff --git a/openspec/changes/ensure-storage-file-cleanup/.openspec.yaml b/openspec/changes/archive/2026-09-14-ensure-storage-file-cleanup/.openspec.yaml
similarity index 100%
rename from openspec/changes/ensure-storage-file-cleanup/.openspec.yaml
rename to openspec/changes/archive/2026-09-14-ensure-storage-file-cleanup/.openspec.yaml
diff --git a/openspec/changes/ensure-storage-file-cleanup/design.md b/openspec/changes/archive/2026-09-14-ensure-storage-file-cleanup/design.md
similarity index 100%
rename from openspec/changes/ensure-storage-file-cleanup/design.md
rename to openspec/changes/archive/2026-09-14-ensure-storage-file-cleanup/design.md
diff --git a/openspec/changes/ensure-storage-file-cleanup/proposal.md b/openspec/changes/archive/2026-09-14-ensure-storage-file-cleanup/proposal.md
similarity index 100%
rename from openspec/changes/ensure-storage-file-cleanup/proposal.md
rename to openspec/changes/archive/2026-09-14-ensure-storage-file-cleanup/proposal.md
diff --git a/openspec/changes/ensure-storage-file-cleanup/specs/local-storage-file-lifecycle/spec.md b/openspec/changes/archive/2026-09-14-ensure-storage-file-cleanup/specs/local-storage-file-lifecycle/spec.md
similarity index 100%
rename from openspec/changes/ensure-storage-file-cleanup/specs/local-storage-file-lifecycle/spec.md
rename to openspec/changes/archive/2026-09-14-ensure-storage-file-cleanup/specs/local-storage-file-lifecycle/spec.md
diff --git a/openspec/changes/ensure-storage-file-cleanup/specs/storage-file-cleanup/spec.md b/openspec/changes/archive/2026-09-14-ensure-storage-file-cleanup/specs/storage-file-cleanup/spec.md
similarity index 100%
rename from openspec/changes/ensure-storage-file-cleanup/specs/storage-file-cleanup/spec.md
rename to openspec/changes/archive/2026-09-14-ensure-storage-file-cleanup/specs/storage-file-cleanup/spec.md
diff --git a/openspec/changes/ensure-storage-file-cleanup/tasks.md b/openspec/changes/archive/2026-09-14-ensure-storage-file-cleanup/tasks.md
similarity index 100%
rename from openspec/changes/ensure-storage-file-cleanup/tasks.md
rename to openspec/changes/archive/2026-09-14-ensure-storage-file-cleanup/tasks.md
diff --git a/openspec/specs/local-storage-file-lifecycle/spec.md b/openspec/specs/local-storage-file-lifecycle/spec.md
index a2064f7b..cd6e47f8 100644
--- a/openspec/specs/local-storage-file-lifecycle/spec.md
+++ b/openspec/specs/local-storage-file-lifecycle/spec.md
@@ -58,18 +58,29 @@ Local storage SHALL open files from the configured data folder. It SHALL return
### Requirement: Local storage deletion is idempotent
-Local storage SHALL remove the requested path from the configured data folder. A missing path SHALL be treated as successfully deleted.
+Local storage SHALL remove both the requested path from the configured data folder and the matching staging path from the configured temporary folder. A missing path in either location SHALL be treated as successfully deleted only when the filesystem confirms that it does not exist.
#### Scenario: Published file is deleted
-- **WHEN** a caller deletes an existing file
+- **WHEN** a caller deletes an existing published file
- **THEN** local storage removes it from the data folder
+- **AND** it removes a matching staging file when one exists
+
+#### Scenario: Interrupted staging file is deleted
+
+- **WHEN** an interrupted write leaves a file only in the temporary folder
+- **THEN** local storage removes that staging file
#### Scenario: Missing file is deleted
-- **WHEN** a caller deletes a path that does not exist
+- **WHEN** neither the published path nor the staging path exists
- **THEN** local storage returns success
+#### Scenario: Filesystem lookup fails
+
+- **WHEN** local storage cannot determine whether either path exists
+- **THEN** deletion returns the filesystem error
+
### Requirement: Local storage connection tests exercise the temporary folder
The local-storage connection test SHALL create, close, and remove a probe file in the configured temporary folder. It SHALL return an error when any required operation fails.
diff --git a/openspec/specs/storage-file-cleanup/spec.md b/openspec/specs/storage-file-cleanup/spec.md
new file mode 100644
index 00000000..c3753264
--- /dev/null
+++ b/openspec/specs/storage-file-cleanup/spec.md
@@ -0,0 +1,275 @@
+# storage-file-cleanup Specification
+
+## Purpose
+Defines how Databasus publishes backup files and eventually removes every file that belongs to a failed, canceled, interrupted or deleted backup operation.
+
+## Requirements
+
+### Requirement: A stored file is kept only when a transaction claims it
+
+The system SHALL record a pending deletion for a unique file name before sending bytes to storage, and SHALL remove that record only as part of the transaction that publishes the file as part of a backup. Every other outcome SHALL leave the record in place, so the file is eventually removed.
+
+#### Scenario: File and backup are published together
+
+- **WHEN** storage accepts a file and the owning backup catalog change commits
+- **THEN** the system stops tracking the file for deletion
+- **AND** the file remains in storage
+
+#### Scenario: Catalog publication fails after upload
+
+- **WHEN** storage accepts a file but the owning catalog transaction fails
+- **THEN** the system keeps the catalog change uncommitted
+- **AND** it eventually removes the uploaded file
+
+#### Scenario: Application stops during a write
+
+- **WHEN** the application stops after recording a pending deletion but before publishing the file
+- **THEN** the record survives the restart
+- **AND** the system eventually removes every storage artifact created by that write
+
+#### Scenario: Provider rejects the write
+
+- **WHEN** a provider returns an error partway through a write
+- **THEN** the file becomes eligible for deletion immediately
+- **AND** the caller receives the provider error
+
+#### Scenario: Publication arrives too late
+
+- **WHEN** a caller tries to publish a file after cleanup has already taken it
+- **THEN** the publishing transaction fails instead of committing
+- **AND** the backup is not recorded as successful
+
+### Requirement: Every failed backup path discards its files
+
+The system SHALL leave the pending deletion in place for every file produced by an unsuccessful logical or physical backup operation, even when storage accepted the file before another stage failed. The owning backup flow SHALL identify the base file and each metadata, manifest, WAL or history sidecar that belongs to that operation.
+
+#### Scenario: Logical dump fails after producing output
+
+- **WHEN** a logical backup tool produces bytes and then exits with an error
+- **THEN** the backup is marked failed
+- **AND** the partial or completed storage file is eventually absent
+
+#### Scenario: Physical backup stream fails after producing output
+
+- **WHEN** a physical FULL or INCR stream writes bytes and then ends unsuccessfully
+- **THEN** the backup is not published as completed
+- **AND** its base file, manifest and metadata files are eventually absent
+
+#### Scenario: Sidecar creation fails
+
+- **WHEN** a base artifact is stored but its required metadata or manifest cannot be produced or published
+- **THEN** every file produced for that attempt is eventually absent
+
+#### Scenario: WAL or history catalog commit fails
+
+- **WHEN** a WAL segment or timeline history file reaches storage but its catalog transaction does not commit
+- **THEN** the artifact and its sidecar are eventually absent
+- **AND** a later retry uses a different per-attempt object name
+
+#### Scenario: Active backup is canceled
+
+- **WHEN** a user or scheduler cancels a backup that may still be writing
+- **THEN** every file from the canceled attempt is eventually absent
+- **AND** no file is removed while its write is still running
+
+### Requirement: Catalog removal and deletion requests are atomic
+
+The system SHALL commit the removal or terminal failure of catalog records together with durable deletion requests for all files those records own. It SHALL NOT perform remote storage deletion while holding that catalog transaction open.
+
+#### Scenario: Backup is removed while storage is unavailable
+
+- **WHEN** retention or a user removes a backup while its storage provider is unavailable
+- **THEN** the catalog removal and its deletion requests commit together
+- **AND** the system retries the remote deletion after storage becomes available
+
+#### Scenario: Deletion request cannot be persisted
+
+- **WHEN** the system cannot persist every required deletion request
+- **THEN** the related catalog removal or terminal state change rolls back
+- **AND** the catalog continues to identify the files that still exist
+
+#### Scenario: A cascade rolls back after part of it succeeded
+
+- **WHEN** a cascading removal fails partway and its transaction rolls back
+- **THEN** every catalog record it touched is restored
+- **AND** none of the files those records name have been removed from storage
+
+#### Scenario: A database with physical backups is deleted
+
+- **WHEN** database deletion cascades FULL, INCR, WAL, history, manifest or metadata catalog rows
+- **THEN** exact deletion requests for every referenced artifact are committed before the cascade removes the rows that name them
+- **AND** a failure to persist them stops the database removal
+
+#### Scenario: Deletion is requested more than once
+
+- **WHEN** the same storage identifier and file name are requested for deletion repeatedly
+- **THEN** the system retains one cleanup obligation
+- **AND** each request succeeds without creating competing deletions
+
+### Requirement: A deletion request is always accepted
+
+Requesting deletion of a file SHALL succeed whatever state that file is in, including while its writer is still running. A request made during an active write SHALL take the file away from that write, so the write cannot be published afterwards.
+
+#### Scenario: Deletion is requested during an active write
+
+- **WHEN** a catalog transaction requests deletion for a file whose writer has not stopped
+- **THEN** the request succeeds and the transaction can commit
+- **AND** the file is eventually absent
+
+#### Scenario: A cascading removal meets an in-flight upload
+
+- **WHEN** a cascading catalog removal covers a WAL segment that is uploading at that moment
+- **THEN** the removal commits without waiting for the upload
+- **AND** the uploaded object is eventually absent
+
+#### Scenario: A write completes after its file was requested for deletion
+
+- **WHEN** a write finishes successfully after deletion was requested for the same file
+- **THEN** the write reports that the file was taken for deletion
+- **AND** its caller cannot publish that file as part of a backup
+
+### Requirement: Cleanup never removes a file that is still being written or already published
+
+The system SHALL use an immutable, unique object name for each independent write attempt. It SHALL NOT delete a file whose write is still running in the application process, and it SHALL NOT start a write for a file name that already carries a pending deletion. A deletion attempt that has taken a file SHALL invalidate any outstanding permission to publish it, and an attempt that outlived its own timeout SHALL NOT overwrite the result of a newer attempt.
+
+#### Scenario: A long write outlives the cleanup interval
+
+- **WHEN** a write runs for longer than the cleanup worker's scan interval
+- **THEN** the worker does not delete that file while the write is running
+- **AND** the write completes normally
+
+#### Scenario: Cleanup takes a file that was uploaded but never published
+
+- **WHEN** cleanup takes a file whose upload finished but whose catalog transaction never committed
+- **THEN** any later attempt to publish that file fails
+- **AND** the file is removed from storage
+
+#### Scenario: A file name already carries a pending deletion
+
+- **WHEN** a write starts for a file name whose deletion was already requested
+- **THEN** the write fails without sending bytes to the provider
+- **AND** the existing cleanup obligation keeps its own schedule
+
+#### Scenario: An attempt outlives its own timeout
+
+- **WHEN** a deletion attempt finishes after its timeout and the file was already picked up again
+- **THEN** the late attempt does not reschedule or overwrite the newer attempt's state
+
+#### Scenario: Cleanup worker stops after remote deletion
+
+- **WHEN** the worker stops after deleting the remote file but before recording completion
+- **THEN** a later worker repeats the idempotent deletion
+- **AND** the cleanup obligation is removed only after deletion is confirmed
+
+### Requirement: Deletion removes the complete logical file
+
+Every storage provider SHALL treat deletion as removal of all physical state derived from the requested logical file name. Missing state SHALL count as success only when the provider confirms that it is absent; connection, authorization and lookup failures SHALL remain errors, except where a protocol gives one reply for both absence and refusal.
+
+#### Scenario: Provider write leaves partial state
+
+- **WHEN** a write fails after creating a temporary file, partial remote file, staged block, chunk object, manifest or multipart upload
+- **THEN** deletion removes every piece derived from the logical file name
+
+#### Scenario: S3 manifest is missing
+
+- **WHEN** an interrupted chunked S3 write leaves completed chunk objects or incomplete multipart uploads without a manifest
+- **THEN** deletion finds and removes only the exact chunks and uploads derived from that logical file name
+- **AND** similarly prefixed files remain untouched
+
+#### Scenario: File is already absent
+
+- **WHEN** the provider confirms that every representation of the requested file is missing
+- **THEN** deletion succeeds
+
+#### Scenario: Provider lookup fails
+
+- **WHEN** a connection, permission or other lookup error prevents the provider from confirming absence
+- **THEN** deletion returns an error
+- **AND** the cleanup obligation remains pending
+
+#### Scenario: A protocol cannot separate absence from refusal
+
+- **WHEN** a provider's only reply for a missing file is also its reply for a refused one
+- **THEN** deletion treats that reply as absence
+- **AND** every other failure the protocol can distinguish is still an error
+
+### Requirement: Failed deletions remain retryable and observable
+
+The system SHALL retry failed deletions with exponential delay, a maximum delay and jitter, grown from the number of deletion attempts alone. It SHALL persist that attempt count, the next attempt time and a sanitized bounded error message. Each worker pass SHALL have a stable `job_name` and a fresh `job_id`, and SHALL report how many obligations are pending, how many are overdue and the age of the oldest one. Every deletion attempt it logs SHALL identify the job, storage, file, attempt and next retry without exposing credentials.
+
+#### Scenario: Storage outage persists
+
+- **WHEN** repeated deletion attempts fail because storage is unavailable
+- **THEN** each failure remains recorded
+- **AND** retries continue at a bounded cadence
+
+#### Scenario: Storage credentials cannot delete
+
+- **WHEN** a provider rejects deletion because the configured credentials lack the permission
+- **THEN** the recorded error names the missing permission as the provider reported it
+- **AND** the obligation stays pending so it succeeds once the permission is granted
+
+#### Scenario: A file is requested for deletion more than once before any attempt
+
+- **WHEN** several callers request deletion of the same file before the worker has attempted it
+- **THEN** the file carries one obligation
+- **AND** its first failed attempt waits the base retry delay, because a request is not an attempt
+
+#### Scenario: Storage recovers
+
+- **WHEN** a later deletion attempt succeeds after earlier failures
+- **THEN** the remote file is absent
+- **AND** the system removes the cleanup obligation
+
+#### Scenario: Provider error contains sensitive material
+
+- **WHEN** a provider returns an error containing credentials or a signed address
+- **THEN** persisted state and logs omit the sensitive value
+
+#### Scenario: A deletion attempt is logged
+
+- **WHEN** the worker attempts or reschedules a deletion
+- **THEN** its logs retain `job_name`, `job_id`, `storage_id`, `file_name`, attempt count and next retry time after centralized redaction
+
+#### Scenario: An operator asks what cleanup is outstanding
+
+- **WHEN** cleanup obligations remain
+- **THEN** each worker pass reports the pending count, the overdue count and the age of the oldest obligation
+- **AND** the failing storage and file are identifiable from the recorded state
+
+### Requirement: Deleting a storage reports the files it leaves behind
+
+The system SHALL reject deleting a storage configuration while backup records still reference it, because those records are the only place its file names exist. When no backup records reference it, deletion SHALL attempt the storage's outstanding cleanup first and SHALL then succeed. It SHALL record how many files it could not remove and their names up to a fixed limit. That record SHALL be somewhere the user can read. Outstanding cleanup SHALL NOT make a storage or a workspace permanently undeletable.
+
+#### Scenario: Storage is deleted while backups still reference it
+
+- **WHEN** a user deletes a storage that still has backup records
+- **THEN** deletion is rejected with an actionable conflict
+- **AND** the backups stay readable and removable through their own deletion path
+
+#### Scenario: Storage with reachable pending cleanup is deleted
+
+- **WHEN** a storage with no backup records and reachable pending cleanup is deleted
+- **THEN** the pending files are removed from the provider before the configuration disappears
+- **AND** deletion succeeds
+
+#### Scenario: Storage with an unreachable provider is deleted
+
+- **WHEN** a storage is deleted while its provider cannot be reached
+- **THEN** deletion still succeeds
+- **AND** the audit log names the storage, how many files were left behind and their names up to a fixed limit
+
+#### Scenario: Workspace containing such a storage is deleted
+
+- **WHEN** a workspace is deleted whose storages still hold pending cleanup
+- **THEN** deletion follows the same path as deleting each storage
+- **AND** the caller never sees a raw database constraint error
+
+### Requirement: Cleanup state contains no provider credentials
+
+Durable cleanup state SHALL contain file identity, scheduling data, retry data and sanitized errors only. It SHALL NOT duplicate provider credentials, tokens, private keys, signed addresses or complete provider configuration.
+
+#### Scenario: Cleanup state is inspected at rest
+
+- **WHEN** an operator or database tool reads durable cleanup state
+- **THEN** the record contains no provider authentication material or reusable signed address