Skip to content

PostgreSQL backup tool with Point-In-Time-Recovery and restore verification

FIX (wal): Rebuild the replication slot for lag only when the running receiver fails to reduce it over 5 minutes of running time, so a receiver catching up after a back-pressure pause or a restart keeps the WAL chain

Rostislav Dugineea04b6d50bd
diff --git a/backend/internal/features/backups/backups/core/physical/wal_health/constants.go b/backend/internal/features/backups/backups/core/physical/wal_health/constants.go
index 7aa44dc3..4839e309 100644
--- a/backend/internal/features/backups/backups/core/physical/wal_health/constants.go
+++ b/backend/internal/features/backups/backups/core/physical/wal_health/constants.go
@@ -11,9 +11,17 @@ const (
SampleRetention = 7 * 24 * time.Hour
// A warning wal_status must persist this long before the WAL streamer alerts
- // on it, so a transient write burst stays quiet.
+ // on it, so a transient write burst stays quiet. A slot lag above the
+ // operator's threshold must go unreduced for this much receiver running time
+ // before the streamer rebuilds the slot, so a receiver catching up keeps the
+ // chain.
WarningSlotStatusHoldPeriod = 5 * time.Minute
+ // Over one hold period of running time, a lag above the threshold that shrinks
+ // by less than this share of the threshold counts as not reduced. It keeps a
+ // lag that wobbles on a busy source from passing as a drain.
+ SlotLagMinDrainPercentOfThreshold = 5
+
// A rarely-written database can legitimately hold its newest WAL locally for a
// while, but past this the recovery point has silently stopped tracking the
// source and only the operator can decide what to do about it.
diff --git a/backend/internal/features/backups/backups/core/physical/wal_health/verdict.go b/backend/internal/features/backups/backups/core/physical/wal_health/verdict.go
index 8480609e..38081901 100644
--- a/backend/internal/features/backups/backups/core/physical/wal_health/verdict.go
+++ b/backend/internal/features/backups/backups/core/physical/wal_health/verdict.go
@@ -348,9 +348,9 @@ func getSlotLimitApproachingReason(input VerdictInput, samples verdictSamples) *
}
}
-// Stays listed while the receiver is paused: the slot is rebuilt on the first
-// tick the receiver runs again above the rebuild lag, so a paused receiver
-// above it is exactly the risky case.
+// Stays listed while the receiver is paused: once it runs again above the
+// rebuild lag, the slot is rebuilt if the running receiver does not reduce the
+// lag, so a paused receiver above it is still at risk.
func getNearSlotRebuildReason(input VerdictInput, samples verdictSamples) *Reason {
if !samples.isNewestWithSourceValues || input.SlotRebuildLagBytes <= 0 {
return nil
diff --git a/backend/internal/features/backups/backups/usecases/physical/postgresql/lag_monitor.go b/backend/internal/features/backups/backups/usecases/physical/postgresql/lag_monitor.go
index ada9fe36..292c002a 100644
--- a/backend/internal/features/backups/backups/usecases/physical/postgresql/lag_monitor.go
+++ b/backend/internal/features/backups/backups/usecases/physical/postgresql/lag_monitor.go
@@ -47,12 +47,14 @@ const (
type slotBreakSample struct {
SlotState *SlotState
WalLagThresholdBytes int64
- IsReceiverRunning bool
+ // Zero when no receiver ran at the tick.
+ ReceiverRunStartedAt time.Time
ObservedAt time.Time
}
type slotBreakClassifier struct {
warningStatusSince time.Time
+ lagWindow slotLagWindow
}
// What the lag monitor keeps from one tick to the next within a session. The
@@ -96,8 +98,10 @@ func (m *lagMonitorState) rememberSourcePosition(probe slotProbe) {
// guaranteed whenever back pressure pauses us — and 'unreserved' means PG will
// trim that WAL at the next checkpoint, which already protects the primary. Both
// alert instead. The operator's WalLagThresholdBytes stays the explicit
-// "sacrifice the chain to protect the primary" knob, but never fires against lag
-// we caused by holding the receiver down ourselves.
+// "sacrifice the chain to protect the primary" knob, but fires only when the
+// running receiver fails to reduce the lag (slot_lag_window.go), never against
+// lag we caused by holding the receiver down ourselves. Lag seen during a
+// warning status never counts afterwards.
func (c *slotBreakClassifier) recordSampleAndClassifyBreak(
sample slotBreakSample,
) (walBreakReason, slotBreakAction) {
@@ -118,11 +122,14 @@ func (c *slotBreakClassifier) recordSampleAndClassifyBreak(
if state.WalStatus == "lost" {
c.warningStatusSince = time.Time{}
+ c.lagWindow.clear()
return breakReasonSlotLost, slotBreakActionRebuild
}
if state.WalStatus == "extended" || state.WalStatus == "unreserved" {
+ c.lagWindow.clear()
+
if c.warningStatusSince.IsZero() {
c.warningStatusSince = sample.ObservedAt
}
@@ -136,9 +143,7 @@ func (c *slotBreakClassifier) recordSampleAndClassifyBreak(
c.warningStatusSince = time.Time{}
- if sample.WalLagThresholdBytes > 0 &&
- state.LagBytes > sample.WalLagThresholdBytes &&
- sample.IsReceiverRunning {
+ if c.lagWindow.recordSampleAndDetectUnreducedLag(sample) {
return breakReasonWalLag, slotBreakActionRebuild
}
@@ -222,12 +227,7 @@ func (s *WalStreamSupervisor) runLagMonitorTick(
StalenessThreshold: s.spec.ArchiveStalenessThreshold,
}, lastCommittedWalAt)
- reason, action = monitorState.classifier.recordSampleAndClassifyBreak(slotBreakSample{
- SlotState: state,
- WalLagThresholdBytes: s.spec.WalLagThresholdBytes,
- IsReceiverRunning: s.isReceiverRunning.Load(),
- ObservedAt: now,
- })
+ reason, action = monitorState.classifier.recordSampleAndClassifyBreak(s.buildSlotBreakSample(state, now))
}
monitorState.recordProbeOutcome(probe, now)
@@ -246,7 +246,7 @@ func (s *WalStreamSupervisor) runLagMonitorTick(
ArchiveStalledSince: monitorState.stalenessTracker.fallingBehindSince,
LocalQueueBytes: s.backlogBytes(),
LocalQueuePauseBytes: s.highWatermarkBytes.Load(),
- IsReceiverRunning: s.isReceiverRunning.Load(),
+ IsReceiverRunning: s.isReceiverRunning(),
IsForcedRotationDenied: s.isForcedRotationDenied.Load(),
})
@@ -269,6 +269,15 @@ func (s *WalStreamSupervisor) runLagMonitorTick(
}
}
+func (s *WalStreamSupervisor) buildSlotBreakSample(state *SlotState, now time.Time) slotBreakSample {
+ return slotBreakSample{
+ SlotState: state,
+ WalLagThresholdBytes: s.spec.WalLagThresholdBytes,
+ ReceiverRunStartedAt: s.getReceiverRunStartedAt(),
+ ObservedAt: now,
+ }
+}
+
// Without a sample repository, as in tests that drive the streamer alone,
// nothing is recorded and the receiver counters stay for the test to read.
func (s *WalStreamSupervisor) recordWalStreamSample(
diff --git a/backend/internal/features/backups/backups/usecases/physical/postgresql/lag_monitor_test.go b/backend/internal/features/backups/backups/usecases/physical/postgresql/lag_monitor_test.go
index 2aac7cd1..64f7763f 100644
--- a/backend/internal/features/backups/backups/usecases/physical/postgresql/lag_monitor_test.go
+++ b/backend/internal/features/backups/backups/usecases/physical/postgresql/lag_monitor_test.go
@@ -3,6 +3,7 @@ package usecases_physical_postgresql
import (
"context"
"log/slog"
+ "slices"
"testing"
"time"
@@ -25,11 +26,96 @@ func runningSlotBreakSample(state *SlotState, walLagThresholdBytes int64, observ
return slotBreakSample{
SlotState: state,
WalLagThresholdBytes: walLagThresholdBytes,
- IsReceiverRunning: true,
+ ReceiverRunStartedAt: observedAt.Add(-time.Minute),
ObservedAt: observedAt,
}
}
+const lagWindowTestThresholdBytes int64 = 100 * 1024 * 1024
+
+var lagWindowTestStartedAt = time.Date(2026, time.October, 19, 12, 0, 0, 0, time.UTC)
+
+func getThresholdPercentBytes(percent float64) int64 {
+ return int64(float64(lagWindowTestThresholdBytes) * percent / 100)
+}
+
+// Samples one lag monitor tick apart, the first at lagWindowTestStartedAt.
+type lagSampleTimeline struct {
+ walLagThresholdBytes int64
+ samples []slotBreakSample
+}
+
+func newLagSampleTimeline() *lagSampleTimeline {
+ return &lagSampleTimeline{walLagThresholdBytes: lagWindowTestThresholdBytes}
+}
+
+func (l *lagSampleTimeline) getNextObservedAt() time.Time {
+ return lagWindowTestStartedAt.Add(time.Duration(len(l.samples)) * wal_health.SampleInterval)
+}
+
+// A zero runStartedAt is a tick with no receiver running.
+func (l *lagSampleTimeline) addSample(runStartedAt time.Time, walStatus string, lagBytes int64) {
+ l.samples = append(l.samples, slotBreakSample{
+ SlotState: &SlotState{WalStatus: walStatus, LagBytes: lagBytes},
+ WalLagThresholdBytes: l.walLagThresholdBytes,
+ ReceiverRunStartedAt: runStartedAt,
+ ObservedAt: l.getNextObservedAt(),
+ })
+}
+
+func (l *lagSampleTimeline) addRunningSample(runStartedAt time.Time, lagBytes int64) {
+ l.addSample(runStartedAt, "reserved", lagBytes)
+}
+
+func (l *lagSampleTimeline) addStoppedSample(lagBytes int64) {
+ l.addSample(time.Time{}, "reserved", lagBytes)
+}
+
+type classifiedBreak struct {
+ Reason walBreakReason
+ Action slotBreakAction
+}
+
+func classifyEachSample(classifier *slotBreakClassifier, samples []slotBreakSample) []classifiedBreak {
+ classifiedBreaks := make([]classifiedBreak, 0, len(samples))
+
+ for _, sample := range samples {
+ reason, action := classifier.recordSampleAndClassifyBreak(sample)
+ classifiedBreaks = append(classifiedBreaks, classifiedBreak{Reason: reason, Action: action})
+ }
+
+ return classifiedBreaks
+}
+
+func getFirstBreakIndex(classifiedBreaks []classifiedBreak, action slotBreakAction) int {
+ return slices.IndexFunc(classifiedBreaks, func(classified classifiedBreak) bool {
+ return classified.Action == action
+ })
+}
+
+// With one sample per tick, the hold of running time is covered by this many
+// intervals between samples of the same run.
+var lagWindowHoldIntervals = int(wal_health.WarningSlotStatusHoldPeriod / wal_health.SampleInterval)
+
+func requireLagRebuildAt(t *testing.T, classifiedBreaks []classifiedBreak, expectedIndex int, message string) {
+ t.Helper()
+
+ require.Equal(t, expectedIndex, getFirstBreakIndex(classifiedBreaks, slotBreakActionRebuild), message)
+ require.Equal(t, breakReasonWalLag, classifiedBreaks[expectedIndex].Reason)
+}
+
+func getFirstWalLagRebuildIndex(classifiedBreaks []classifiedBreak) int {
+ return slices.IndexFunc(classifiedBreaks, func(classified classifiedBreak) bool {
+ return classified.Action == slotBreakActionRebuild && classified.Reason == breakReasonWalLag
+ })
+}
+
+func requireNoRebuild(t *testing.T, classifiedBreaks []classifiedBreak, message string) {
+ t.Helper()
+
+ require.Equal(t, -1, getFirstBreakIndex(classifiedBreaks, slotBreakActionRebuild), message)
+}
+
func Test_RecordSampleAndClassifyBreak_WhenWalStatusLost_Rebuilds(t *testing.T) {
var classifier slotBreakClassifier
@@ -77,29 +163,308 @@ func Test_RecordSampleAndClassifyBreak_WhenWalStatusExtendedPastHold_AlertsWitho
require.Equal(t, breakReasonSlotRetention, reason)
}
-func Test_RecordSampleAndClassifyBreak_WhenLagOverThresholdAndReceiverRunning_Rebuilds(t *testing.T) {
+func Test_RecordSampleAndClassifyBreak_WhenReceiverResumesWithLagAboveThreshold_DoesNotRebuild(t *testing.T) {
+ timeline := newLagSampleTimeline()
+ timeline.addStoppedSample(2 * lagWindowTestThresholdBytes)
+
+ resumedAt := timeline.getNextObservedAt()
+ for sampleIndex := range 2*lagWindowHoldIntervals + 1 {
+ timeline.addRunningSample(
+ resumedAt,
+ 2*lagWindowTestThresholdBytes-int64(sampleIndex)*getThresholdPercentBytes(2),
+ )
+ }
+
+ var classifier slotBreakClassifier
+
+ requireNoRebuild(t, classifyEachSample(&classifier, timeline.samples),
+ "a receiver catching up after a back-pressure pause must not cost the chain")
+}
+
+func Test_RecordSampleAndClassifyBreak_WhenFirstSampleOfSessionShowsShrinkingLag_DoesNotRebuild(t *testing.T) {
+ timeline := newLagSampleTimeline()
+
+ runStartedAt := lagWindowTestStartedAt.Add(-time.Second)
+ for sampleIndex := range 2*lagWindowHoldIntervals + 1 {
+ timeline.addRunningSample(
+ runStartedAt,
+ 3*lagWindowTestThresholdBytes-int64(sampleIndex)*getThresholdPercentBytes(2),
+ )
+ }
+
var classifier slotBreakClassifier
- reason, action := classifier.recordSampleAndClassifyBreak(
- runningSlotBreakSample(&SlotState{WalStatus: "reserved", LagBytes: 101}, 100, time.Now().UTC()),
- )
+ requireNoRebuild(t, classifyEachSample(&classifier, timeline.samples),
+ "the backlog left by Databasus being down drains on its own and must not cost the chain")
+}
- require.Equal(t, slotBreakActionRebuild, action)
- require.Equal(t, breakReasonWalLag, reason)
+func Test_RecordSampleAndClassifyBreak_WhenSlowCatchUpOutlastsHold_DoesNotRebuild(t *testing.T) {
+ timeline := newLagSampleTimeline()
+
+ runStartedAt := lagWindowTestStartedAt
+ for sampleIndex := range 4*lagWindowHoldIntervals + 1 {
+ timeline.addRunningSample(
+ runStartedAt,
+ 2*lagWindowTestThresholdBytes-int64(sampleIndex)*getThresholdPercentBytes(0.6),
+ )
+ }
+
+ var classifier slotBreakClassifier
+
+ requireNoRebuild(t, classifyEachSample(&classifier, timeline.samples),
+ "a lag shrinking by more than 5% of the threshold every 5 minutes is a receiver draining")
}
-func Test_RecordSampleAndClassifyBreak_WhenLagOverThresholdAndReceiverStopped_DoesNotRebuild(t *testing.T) {
+func Test_RecordSampleAndClassifyBreak_WhenShortRunsDrainBetweenPausesThatGrowLag_DoesNotRebuild(t *testing.T) {
+ timeline := newLagSampleTimeline()
+
+ lagBytes := 2 * lagWindowTestThresholdBytes
+
+ for range 10 {
+ runStartedAt := timeline.getNextObservedAt()
+
+ for range 6 {
+ timeline.addRunningSample(runStartedAt, lagBytes)
+ lagBytes -= getThresholdPercentBytes(1)
+ }
+
+ for range 2 {
+ timeline.addStoppedSample(lagBytes)
+ lagBytes += getThresholdPercentBytes(5)
+ }
+ }
+
+ require.Greater(t, lagBytes, 2*lagWindowTestThresholdBytes, "the pauses grow the lag more than the runs drain it")
+
var classifier slotBreakClassifier
- _, action := classifier.recordSampleAndClassifyBreak(slotBreakSample{
- SlotState: &SlotState{WalStatus: "reserved", LagBytes: 101},
- WalLagThresholdBytes: 100,
- IsReceiverRunning: false,
- ObservedAt: time.Now().UTC(),
- })
+ requireNoRebuild(t, classifyEachSample(&classifier, timeline.samples),
+ "lag that grows while back pressure holds the receiver down must never count against it")
+}
+
+func Test_RecordSampleAndClassifyBreak_WhenRunningReceiverCannotKeepUp_RebuildsOnceHoldOfRunningTimePasses(
+ t *testing.T,
+) {
+ timeline := newLagSampleTimeline()
+
+ runStartedAt := lagWindowTestStartedAt
+ for sampleIndex := range lagWindowHoldIntervals + 1 {
+ timeline.addRunningSample(
+ runStartedAt,
+ lagWindowTestThresholdBytes+int64(sampleIndex+1)*getThresholdPercentBytes(1),
+ )
+ }
+
+ var classifier slotBreakClassifier
- require.Equal(t, slotBreakActionNone, action,
- "lag we caused by holding the receiver down must never cost the chain")
+ requireLagRebuildAt(t, classifyEachSample(&classifier, timeline.samples), lagWindowHoldIntervals,
+ "a running receiver falling behind is rebuilt on the first sample after 5 minutes of running time")
+}
+
+func Test_RecordSampleAndClassifyBreak_WhenLagFluctuatesAboveThreshold_RebuildsOnceHoldOfRunningTimePasses(
+ t *testing.T,
+) {
+ timeline := newLagSampleTimeline()
+
+ runStartedAt := lagWindowTestStartedAt
+ for sampleIndex := range lagWindowHoldIntervals + 1 {
+ swingBytes := int64(sampleIndex%2) * getThresholdPercentBytes(3)
+ trendBytes := int64(sampleIndex) * getThresholdPercentBytes(0.4)
+
+ timeline.addRunningSample(runStartedAt, 3*lagWindowTestThresholdBytes/2+swingBytes-trendBytes)
+ }
+
+ var classifier slotBreakClassifier
+
+ requireLagRebuildAt(t, classifyEachSample(&classifier, timeline.samples), lagWindowHoldIntervals,
+ "a lag that only wobbles is not drained, however often it dips")
+}
+
+func Test_RecordSampleAndClassifyBreak_WhenStalledReceiverIsRestartedEvery70Seconds_RebuildsAfterHoldOfRunTime(
+ t *testing.T,
+) {
+ const restartInterval = 70 * time.Second
+
+ timeline := newLagSampleTimeline()
+
+ for sampleIndex := range 30 {
+ sinceStart := timeline.getNextObservedAt().Sub(lagWindowTestStartedAt)
+ runStartedAt := lagWindowTestStartedAt.Add(sinceStart / restartInterval * restartInterval)
+
+ timeline.addRunningSample(
+ runStartedAt,
+ lagWindowTestThresholdBytes+int64(sampleIndex+1)*getThresholdPercentBytes(1),
+ )
+ }
+
+ var classifier slotBreakClassifier
+
+ // Samples at 0 s..480 s fall into runs starting every 70 s; the intervals
+ // inside one run add up to 300 s at the sample taken 480 s in.
+ requireLagRebuildAt(t, classifyEachSample(&classifier, timeline.samples), 16,
+ "restarts must not reset the running time, or a stalled receiver is never rebuilt")
+}
+
+func Test_RecordSampleAndClassifyBreak_WhenReceiverStoppedWithLagOverThreshold_DoesNotRebuildOrCountTheTime(
+ t *testing.T,
+) {
+ timeline := newLagSampleTimeline()
+
+ lagBytes := 2 * lagWindowTestThresholdBytes
+ for range 2 * lagWindowHoldIntervals {
+ timeline.addStoppedSample(lagBytes)
+ lagBytes += getThresholdPercentBytes(1)
+ }
+
+ stoppedSampleCount := len(timeline.samples)
+
+ runStartedAt := timeline.getNextObservedAt()
+ for range lagWindowHoldIntervals + 1 {
+ timeline.addRunningSample(runStartedAt, lagBytes)
+ lagBytes += getThresholdPercentBytes(1)
+ }
+
+ var classifier slotBreakClassifier
+
+ requireLagRebuildAt(t, classifyEachSample(&classifier, timeline.samples), stoppedSampleCount+lagWindowHoldIntervals,
+ "lag we caused by holding the receiver down must never cost the chain, nor shorten the hold afterwards")
+}
+
+func Test_RecordSampleAndClassifyBreak_WhenLagDipsBelowThreshold_RestartsHoldOfRunningTime(t *testing.T) {
+ timeline := newLagSampleTimeline()
+
+ runStartedAt := lagWindowTestStartedAt
+ lagBytes := lagWindowTestThresholdBytes + getThresholdPercentBytes(1)
+
+ for range lagWindowHoldIntervals - 1 {
+ timeline.addRunningSample(runStartedAt, lagBytes)
+ lagBytes += getThresholdPercentBytes(1)
+ }
+
+ timeline.addRunningSample(runStartedAt, lagWindowTestThresholdBytes)
+ dipIndex := len(timeline.samples) - 1
+
+ for range lagWindowHoldIntervals + 1 {
+ timeline.addRunningSample(runStartedAt, lagBytes)
+ lagBytes += getThresholdPercentBytes(1)
+ }
+
+ var classifier slotBreakClassifier
+
+ requireLagRebuildAt(t, classifyEachSample(&classifier, timeline.samples), dipIndex+1+lagWindowHoldIntervals,
+ "a lag at the threshold ends the window; the next one needs its own 5 minutes")
+}
+
+func Test_RecordSampleAndClassifyBreak_WhenThresholdIsZero_NeverRebuildsForLag(t *testing.T) {
+ timeline := newLagSampleTimeline()
+ timeline.walLagThresholdBytes = 0
+
+ runStartedAt := lagWindowTestStartedAt
+ for sampleIndex := range 4 * lagWindowHoldIntervals {
+ timeline.addRunningSample(runStartedAt, int64(sampleIndex+1)*lagWindowTestThresholdBytes)
+ }
+
+ var classifier slotBreakClassifier
+
+ requireNoRebuild(t, classifyEachSample(&classifier, timeline.samples), "a zero threshold turns the lag rule off")
+}
+
+func Test_RecordSampleAndClassifyBreak_WhenSlotStatusEpisodeEndsAboveThreshold_RestartsHoldOfRunningTime(
+ t *testing.T,
+) {
+ episodes := []struct {
+ walStatus string
+ sampleCount int
+ }{
+ {walStatus: "extended", sampleCount: 6},
+ {walStatus: "unreserved", sampleCount: 6},
+ {walStatus: "lost", sampleCount: 1},
+ }
+
+ for _, episode := range episodes {
+ t.Run(episode.walStatus, func(t *testing.T) {
+ timeline := newLagSampleTimeline()
+
+ runStartedAt := lagWindowTestStartedAt
+ lagBytes := lagWindowTestThresholdBytes + getThresholdPercentBytes(1)
+
+ for range lagWindowHoldIntervals - 2 {
+ timeline.addRunningSample(runStartedAt, lagBytes)
+ lagBytes += getThresholdPercentBytes(1)
+ }
+
+ for range episode.sampleCount {
+ timeline.addSample(runStartedAt, episode.walStatus, lagBytes)
+ lagBytes += getThresholdPercentBytes(1)
+ }
+
+ returnedToReservedIndex := len(timeline.samples)
+
+ for range lagWindowHoldIntervals + 1 {
+ timeline.addRunningSample(runStartedAt, lagBytes)
+ lagBytes += getThresholdPercentBytes(1)
+ }
+
+ var classifier slotBreakClassifier
+
+ classifiedBreaks := classifyEachSample(&classifier, timeline.samples)
+
+ require.Equal(t, returnedToReservedIndex+lagWindowHoldIntervals,
+ getFirstWalLagRebuildIndex(classifiedBreaks),
+ "lag seen before and during a slot status episode must not count once the slot returns to reserved")
+ })
+ }
+}
+
+func Test_RecordSampleAndClassifyBreak_WhenReceiverStallsAfterDrainingHold_RebuildsOneHoldAfterTheDrain(
+ t *testing.T,
+) {
+ timeline := newLagSampleTimeline()
+
+ runStartedAt := lagWindowTestStartedAt
+ lagBytes := 2 * lagWindowTestThresholdBytes
+
+ for range lagWindowHoldIntervals + 1 {
+ timeline.addRunningSample(runStartedAt, lagBytes)
+ lagBytes -= getThresholdPercentBytes(2)
+ }
+
+ drainedHoldEndIndex := len(timeline.samples) - 1
+
+ for range lagWindowHoldIntervals + 1 {
+ timeline.addRunningSample(runStartedAt, lagBytes)
+ lagBytes += getThresholdPercentBytes(1)
+ }
+
+ var classifier slotBreakClassifier
+
+ requireLagRebuildAt(t, classifyEachSample(&classifier, timeline.samples),
+ drainedHoldEndIndex+lagWindowHoldIntervals,
+ "a drained hold starts the window again, so a stall after it is judged over its own 5 minutes")
+}
+
+func Test_RecordSampleAndClassifyBreak_WhenExtendedSlotLagsAboveThreshold_AlertsWithoutRebuilding(t *testing.T) {
+ timeline := newLagSampleTimeline()
+
+ runStartedAt := lagWindowTestStartedAt
+ for sampleIndex := range 2*lagWindowHoldIntervals + 1 {
+ timeline.addSample(
+ runStartedAt,
+ "extended",
+ lagWindowTestThresholdBytes+int64(sampleIndex+1)*getThresholdPercentBytes(1),
+ )
+ }
+
+ var classifier slotBreakClassifier
+
+ classifiedBreaks := classifyEachSample(&classifier, timeline.samples)
+
+ requireNoRebuild(t, classifiedBreaks, "an extended slot alerts instead of costing the chain")
+
+ firstAlertIndex := getFirstBreakIndex(classifiedBreaks, slotBreakActionAlert)
+ require.Equal(t, lagWindowHoldIntervals+1, firstAlertIndex,
+ "the retention alert comes on the first sample after the status has lasted 5 minutes")
+ require.Equal(t, breakReasonSlotRetention, classifiedBreaks[firstAlertIndex].Reason)
}
func Test_RecordSampleAndClassifyBreak_WhenForeignConsumerHoldsSlot_Rebuilds(t *testing.T) {
@@ -446,7 +811,7 @@ func Test_RunLagMonitorTick_WhenTickRebuildsSlot_RecordsSlotAsProbedBeforeRebuil
WaitUntilSlotLag(t, adminConn, slotName, 2*walLagThresholdBytes, 30*time.Second)
supervisor := newLagMonitorTickSupervisor(t, fixture, fixture.DB.PostgresqlPhysical, walLagThresholdBytes)
- supervisor.isReceiverRunning.Store(true)
+ supervisor.receiverRunStartedAtUnixNano.Store(lagMonitorTickStartedAt.UnixNano())
sampleCountAtRebuild := -1
supervisor.spec.OnSlotRebuilt = func(context.Context, string) error {
@@ -455,21 +820,27 @@ func Test_RunLagMonitorTick_WhenTickRebuildsSlot_RecordsSlotAsProbedBeforeRebuil
return nil
}
+ // No receiver drains the slot, so the lag stays above the threshold until
+ // the hold of running time ends on the last tick.
+ tickCount := int(wal_health.WarningSlotStatusHoldPeriod/wal_health.SampleInterval) + 1
+
runLagMonitorTicks(
t,
supervisor,
supervisor.newLagMonitorState(logger.GetLogger()),
lagMonitorTickStartedAt,
- 1,
+ tickCount,
nil,
)
- require.Equal(t, 1, sampleCountAtRebuild, "the tick records its sample before it rebuilds the slot")
+ require.Equal(t, tickCount, sampleCountAtRebuild, "the tick records its sample before it rebuilds the slot")
samples := getRecordedSamples(t, fixture)
- require.Len(t, samples, 1)
- require.NotNil(t, samples[0].SlotLagBytes)
- assert.Greater(t, *samples[0].SlotLagBytes, int64(walLagThresholdBytes),
+ require.Len(t, samples, tickCount)
+
+ rebuildSample := samples[len(samples)-1]
+ require.NotNil(t, rebuildSample.SlotLagBytes)
+ assert.Greater(t, *rebuildSample.SlotLagBytes, int64(walLagThresholdBytes),
"the sample shows the slot that caused the rebuild")
}
diff --git a/backend/internal/features/backups/backups/usecases/physical/postgresql/receivewal_process.go b/backend/internal/features/backups/backups/usecases/physical/postgresql/receivewal_process.go
index 9ba003af..e3067f0d 100644
--- a/backend/internal/features/backups/backups/usecases/physical/postgresql/receivewal_process.go
+++ b/backend/internal/features/backups/backups/usecases/physical/postgresql/receivewal_process.go
@@ -93,11 +93,12 @@ func (s *WalStreamSupervisor) spawnAndSupervise(
return receiverRunResult{Exit: receiverRetryable}
}
- s.isReceiverRunning.Store(true)
- defer s.isReceiverRunning.Store(false)
+ startedAt := time.Now().UTC()
+
+ s.receiverRunStartedAtUnixNano.Store(startedAt.UnixNano())
+ defer s.receiverRunStartedAtUnixNano.Store(0)
stderr := newStderrCapture(stderrPipe)
- startedAt := time.Now().UTC()
logger.InfoContext(ctx, "pg_receivewal started", "watch_dir", s.watchDir)
diff --git a/backend/internal/features/backups/backups/usecases/physical/postgresql/slot_lag_window.go b/backend/internal/features/backups/backups/usecases/physical/postgresql/slot_lag_window.go
new file mode 100644
index 00000000..f4cdcc03
--- /dev/null
+++ b/backend/internal/features/backups/backups/usecases/physical/postgresql/slot_lag_window.go
@@ -0,0 +1,76 @@
+package usecases_physical_postgresql
+
+import (
+ "time"
+
+ "databasus-backend/internal/features/backups/backups/core/physical/wal_health"
+)
+
+// Adds up the lag change over the time the receiver spends running, so the lag
+// threshold judges only what a running receiver fails to reduce. A pair of
+// samples counts only inside one receiver run: the lag that builds while no
+// receiver runs, or across a restart, is lag we caused by holding the receiver
+// down. Restarts do not clear the sums, so a receiver the slot-LSN watcher or
+// back pressure keeps restarting still reaches the rule when its runs do not
+// reduce the lag.
+type slotLagWindow struct {
+ runningTime time.Duration
+ lagChangeBytes int64
+
+ hasPreviousSample bool
+ previousRunStartedAt time.Time
+ previousObservedAt time.Time
+ previousLagBytes int64
+}
+
+func (w *slotLagWindow) recordSampleAndDetectUnreducedLag(sample slotBreakSample) bool {
+ lagBytes := sample.SlotState.LagBytes
+ thresholdBytes := sample.WalLagThresholdBytes
+
+ if thresholdBytes <= 0 || lagBytes <= thresholdBytes {
+ w.clear()
+
+ return false
+ }
+
+ if sample.ReceiverRunStartedAt.IsZero() {
+ w.hasPreviousSample = false
+
+ return false
+ }
+
+ if w.hasPreviousSample && w.previousRunStartedAt.Equal(sample.ReceiverRunStartedAt) {
+ w.runningTime += sample.ObservedAt.Sub(w.previousObservedAt)
+ w.lagChangeBytes += lagBytes - w.previousLagBytes
+ }
+
+ w.rememberPreviousSample(sample)
+
+ if w.runningTime < wal_health.WarningSlotStatusHoldPeriod {
+ return false
+ }
+
+ minDrainBytes := thresholdBytes * wal_health.SlotLagMinDrainPercentOfThreshold / 100
+
+ if -w.lagChangeBytes < minDrainBytes {
+ w.clear()
+
+ return true
+ }
+
+ w.runningTime = 0
+ w.lagChangeBytes = 0
+
+ return false
+}
+
+func (w *slotLagWindow) rememberPreviousSample(sample slotBreakSample) {
+ w.hasPreviousSample = true
+ w.previousRunStartedAt = sample.ReceiverRunStartedAt
+ w.previousObservedAt = sample.ObservedAt
+ w.previousLagBytes = sample.SlotState.LagBytes
+}
+
+func (w *slotLagWindow) clear() {
+ *w = slotLagWindow{}
+}
diff --git a/backend/internal/features/backups/backups/usecases/physical/postgresql/slot_lsn_watcher.go b/backend/internal/features/backups/backups/usecases/physical/postgresql/slot_lsn_watcher.go
index bc47e77e..1d8700f9 100644
--- a/backend/internal/features/backups/backups/usecases/physical/postgresql/slot_lsn_watcher.go
+++ b/backend/internal/features/backups/backups/usecases/physical/postgresql/slot_lsn_watcher.go
@@ -155,7 +155,7 @@ func (s *WalStreamSupervisor) runSlotLsnWatcher(ctx context.Context, logger *slo
sample := slotLivenessSample{
RestartLSN: state.RestartLSN,
LagBytes: state.LagBytes,
- IsReceiverRunning: s.isReceiverRunning.Load(),
+ IsReceiverRunning: s.isReceiverRunning(),
ObservedAt: time.Now().UTC(),
}
diff --git a/backend/internal/features/backups/backups/usecases/physical/postgresql/wal_stream.go b/backend/internal/features/backups/backups/usecases/physical/postgresql/wal_stream.go
index 7c36a10b..22de9f4a 100644
--- a/backend/internal/features/backups/backups/usecases/physical/postgresql/wal_stream.go
+++ b/backend/internal/features/backups/backups/usecases/physical/postgresql/wal_stream.go
@@ -155,7 +155,8 @@ type WalStreamSpec struct {
// recovery can re-process finalized-but-not-uploaded segments.
WatchDirRoot string
- // A slot lag over this many bytes triggers a rebuild (lag_monitor.go).
+ // A slot lag over this many bytes that the running receiver does not reduce
+ // triggers a rebuild (slot_lag_window.go).
WalLagThresholdBytes int64
// How often a source that is being written to is asked to close its current WAL
@@ -216,9 +217,12 @@ type WalStreamSupervisor struct {
// drop+recreate the slot without the receiver re-attaching.
isPaused atomic.Bool
- // Lets the slot-LSN watcher and the lag monitor tell a real problem from lag
- // we are causing ourselves by holding the receiver down.
- isReceiverRunning atomic.Bool
+ // When the current pg_receivewal run started, as Unix nanoseconds; zero while
+ // no receiver runs. One value carries both "running" and "since when", so the
+ // slot-LSN watcher and the lag monitor can tell a real problem from lag we are
+ // causing ourselves by holding the receiver down, and never see a running
+ // receiver without its start time.
+ receiverRunStartedAtUnixNano atomic.Int64
// Set by the back-pressure monitor when it stops the receiver, so the respawn
// that follows counts as a pause rather than a restart.
@@ -532,6 +536,20 @@ func (s *WalStreamSupervisor) drainRestartSignal() {
}
}
+// Zero while no receiver runs.
+func (s *WalStreamSupervisor) getReceiverRunStartedAt() time.Time {
+ startedAtUnixNano := s.receiverRunStartedAtUnixNano.Load()
+ if startedAtUnixNano == 0 {
+ return time.Time{}
+ }
+
+ return time.Unix(0, startedAtUnixNano).UTC()
+}
+
+func (s *WalStreamSupervisor) isReceiverRunning() bool {
+ return s.receiverRunStartedAtUnixNano.Load() != 0
+}
+
func walSegmentSizeBytes(sourceDB *postgresql_physical.PostgresqlPhysicalDatabase) int64 {
if sourceDB.WalSegmentSizeBytes != nil && *sourceDB.WalSegmentSizeBytes > 0 {
return *sourceDB.WalSegmentSizeBytes
diff --git a/backend/internal/features/backups/backups/usecases/physical/postgresql/wal_stream_test.go b/backend/internal/features/backups/backups/usecases/physical/postgresql/wal_stream_test.go
index e7d77a0e..096c1010 100644
--- a/backend/internal/features/backups/backups/usecases/physical/postgresql/wal_stream_test.go
+++ b/backend/internal/features/backups/backups/usecases/physical/postgresql/wal_stream_test.go
@@ -966,19 +966,19 @@ func Test_WalStream_WhenQueueFillsAndReceiverExitsOnItsOwn_CountsOnePauseAndOneR
supervisor := streamer.Supervisor
- require.Eventually(t, supervisor.isReceiverRunning.Load, 60*time.Second, 200*time.Millisecond)
+ require.Eventually(t, supervisor.isReceiverRunning, 60*time.Second, 200*time.Millisecond)
// Written once: WAL written during the pause would refill the queue as soon
// as the receiver resumes and pause it a second time.
ForceReplicationLag(t, adminConn, 4*segmentSizeBytes)
require.Eventually(t, func() bool {
- return !supervisor.isReceiverRunning.Load() && supervisor.backlogBytes() >= supervisor.highWatermarkBytes.Load()
+ return !supervisor.isReceiverRunning() && supervisor.backlogBytes() >= supervisor.highWatermarkBytes.Load()
}, 90*time.Second, 200*time.Millisecond, "a full local queue must pause the receiver")
store.stopFailingSaves()
- require.Eventually(t, supervisor.isReceiverRunning.Load, 90*time.Second, 200*time.Millisecond,
+ require.Eventually(t, supervisor.isReceiverRunning, 90*time.Second, 200*time.Millisecond,
"the receiver resumes once the queue drains")
var restarts, pauses int
@@ -1032,3 +1032,42 @@ func terminateWalReceiverBackend(t *testing.T, adminConn *pgx.Conn, slotName str
return err == nil && terminatedCount == 1
}, 30*time.Second, 200*time.Millisecond, "the receiver must be attached to the slot before it can be cut off")
}
+
+func Test_WalStream_WhileReceiverRunsAndAfterItExits_SlotBreakSampleCarriesItsRunStartTime(t *testing.T) {
+ if testing.Short() {
+ t.Skip("streamer integration test runs pg_receivewal; skipped in -short")
+ }
+
+ fixture := SetupPhysicalDBForBackup(t)
+ t.Cleanup(func() {
+ _ = physical_repositories.GetWalStreamerRepository().DeleteByDatabaseID(fixture.DB.ID)
+ })
+
+ streamerStartedAt := time.Now().UTC()
+
+ streamer := StartWalStreamerForTest(t, WalStreamerTestSpec{
+ Fixture: fixture,
+ Storage: newMockWalStorage(),
+ WatchDirRoot: t.TempDir(),
+ })
+ t.Cleanup(streamer.Stop)
+
+ supervisor := streamer.Supervisor
+
+ var runningSample slotBreakSample
+
+ require.Eventually(t, func() bool {
+ runningSample = supervisor.buildSlotBreakSample(&SlotState{}, time.Now().UTC())
+
+ return !runningSample.ReceiverRunStartedAt.IsZero()
+ }, 60*time.Second, 200*time.Millisecond, "a running receiver gives the sample its start time")
+
+ assert.False(t, runningSample.ReceiverRunStartedAt.Before(streamerStartedAt))
+ assert.False(t, runningSample.ReceiverRunStartedAt.After(runningSample.ObservedAt))
+
+ streamer.Stop()
+
+ require.Eventually(t, func() bool {
+ return supervisor.buildSlotBreakSample(&SlotState{}, time.Now().UTC()).ReceiverRunStartedAt.IsZero()
+ }, 30*time.Second, 200*time.Millisecond, "an exited receiver leaves the sample without a start time")
+}
diff --git a/openspec/changes/archive/2026-10-19-fix-wal-lag-rebuild-after-backpressure/.openspec.yaml b/openspec/changes/archive/2026-10-19-fix-wal-lag-rebuild-after-backpressure/.openspec.yaml
new file mode 100644
index 00000000..5c881ba7
--- /dev/null
+++ b/openspec/changes/archive/2026-10-19-fix-wal-lag-rebuild-after-backpressure/.openspec.yaml
@@ -0,0 +1,2 @@
+schema: spec-driven
+created: 2026-10-18
diff --git a/openspec/changes/archive/2026-10-19-fix-wal-lag-rebuild-after-backpressure/design.md b/openspec/changes/archive/2026-10-19-fix-wal-lag-rebuild-after-backpressure/design.md
new file mode 100644
index 00000000..f7df2ba5
--- /dev/null
+++ b/openspec/changes/archive/2026-10-19-fix-wal-lag-rebuild-after-backpressure/design.md
@@ -0,0 +1,77 @@
+## Context
+
+See proposal.md for the motivation. Paths below are under `backend/internal/features/backups/backups/usecases/physical/postgresql/`.
+
+- The lag monitor ticks every 30 seconds and passes one `slotBreakSample` to `slotBreakClassifier.recordSampleAndClassifyBreak` (`lag_monitor.go:121-170`). The sample carries the slot state, `WalLagThresholdBytes`, `IsReceiverRunning` and the observation time (`lag_monitor.go:54-59`).
+- The classifier checks, in order: a foreign consumer, `lost`, `extended`/`unreserved` with a 5-minute hold, and then the lag threshold. The lag rule is one comparison on the current sample: `LagBytes > WalLagThresholdBytes && IsReceiverRunning` (`lag_monitor.go:110-113`). The comment above it promises that lag Databasus caused by holding the receiver down never costs the chain (`lag_monitor.go:64-71`).
+- `isReceiverRunning` is set right after `pg_receivewal` starts and cleared when it exits (`receivewal_process.go:96-97`). Back pressure stops the receiver and the supervision loop waits for the queue to drain before it spawns a new one (`wal_backpressure.go:41-64`, `wal_stream.go:340`). The first tick after the spawn therefore sees the lag built during the pause, with the receiver running.
+- The slot-LSN watcher reads the same flag for its stall check and restarts a receiver whose `restart_lsn` has been frozen for 60 seconds (`slot_lsn_watcher.go:112-150`). Back pressure restarts it at the high watermark (`wal_backpressure.go:34-35`). This change keeps both, and the lag rule must survive their restarts.
+- The classifier is covered by pure unit tests on single samples (`lag_monitor_test.go:20-150`). `Test_RecordSampleAndClassifyBreak_WhenLagOverThresholdAndReceiverRunning_Rebuilds` pins the current one-sample behavior.
+
+Constraints from `backend/AGENTS.md`: unit tests are allowed for complex model logic with no API surface, which the classifier is; test names follow `Test_WhatWeDo_WhichConditions_WhatWeExpect`; `time.Now().UTC()`.
+
+## Goals / Non-Goals
+
+**Goals:**
+
+- The lag rule fires only for lag the running receiver does not reduce.
+- The decision stays a pure function of the samples, testable without a database.
+
+**Non-Goals:**
+
+- Changing the `lost`, foreign consumer or `extended`/`unreserved` rules.
+- Persisting classifier state across sessions. A new session starts with an empty window, which only delays a rebuild.
+
+## Decisions
+
+### D1. One run-start timestamp instead of the running flag
+
+`isReceiverRunning atomic.Bool` (`wal_stream.go:208`) becomes one atomic run-start time, set from the existing `startedAt` in `receivewal_process.go:100` and cleared when the process exits. Zero means stopped. The lag monitor and the slot-LSN watcher derive "running" from a non-zero value, so the two can never disagree. `slotBreakSample` replaces `IsReceiverRunning` with `ReceiverRunStartedAt`; a zero value means the receiver was stopped at that tick.
+
+The sample is built by a small method on the supervisor, so a test can check the wiring through what the classifier receives rather than through the atomic itself.
+
+Alternative rejected: *a second atomic next to the flag.* Two fields stored separately let a reader see "running" with no start time.
+
+### D2. A lag window over running time
+
+The classifier keeps a lag window: the added-up running time, the added-up lag change, and the previous sample's run start and lag. On each sample that passes the earlier rules:
+
+1. When the threshold is zero, or the lag is at or below the threshold, the window is cleared.
+2. When the receiver is stopped, nothing is added, and the previous sample is forgotten, so the next pair starts inside a run.
+3. When the previous sample belongs to the same run, the time between the two samples is added to the running time and the lag difference to the change. A pair that spans two runs adds nothing, because the lag between them grew while no receiver ran.
+4. When the running time reaches 5 minutes:
+ - if the lag shrank by less than 5% of the threshold, the classifier asks for a rebuild with `breakReasonWalLag` and clears the window;
+ - otherwise the receiver is draining, and the window starts again from this sample.
+
+The `lost` and `extended`/`unreserved` branches clear the window as well, so lag seen during such an episode never counts afterwards.
+
+Restarts no longer reset the window, so a receiver the slot-LSN watcher restarts every 60 to 70 seconds (`slot_lsn_watcher.go:112-150`), or that back pressure stops often, still reaches the rule when its runs do not reduce the lag. Back pressure's own runs drain the queue at network speed, so their lag shrinks and the window keeps restarting.
+
+The 5% tolerance makes "not reducing" robust to a lag that wobbles on a busy source. For the default 256 MiB threshold it is about 13 MiB over 5 minutes, roughly 43 KiB/s of drain, which is effectively a receiver that only keeps pace.
+
+Alternatives rejected:
+
+- *A window per receiver run.* A watcher or back-pressure restart every minute would reset it, so a stuck receiver would never be rebuilt.
+- *A fixed grace period after each receiver start.* A backlog that takes longer than the grace to drain still costs the chain, and the right grace depends on the source's write rate and the network.
+- *Rebuilding only when the lag grew between two samples.* A receiver that keeps exact pace while far behind holds the excess WAL forever, and the operator's threshold says that excess is too much.
+- *Restarting the window on any decrease.* A lag that wobbles by a few kilobytes would restart it forever.
+
+### D3. The hold reuses the slot status hold
+
+The window length is `warningSlotStatusHoldPeriod`, 5 minutes, the hold the classifier already applies to `extended` and `unreserved`. One hold for both rules keeps "how long Databasus waits before acting on a slot condition" a single value. The 5% tolerance is a new named constant next to it.
+
+Alternative rejected: *a separate hold constant.* No observed need for a different value, and two holds of the same length drift apart in later edits.
+
+### D4. Rebuild requests keep their existing guards
+
+The classifier only asks; `rebuildSlot` still applies the cap of three attempts an hour (`slot_rebuild.go:17-27`) and refuses to drop a slot held by a consumer it cannot attribute to its own receiver (`slot_rebuild.go:41-53`). This change touches neither.
+
+## Risks / Trade-offs
+
+- [A receiver that genuinely cannot keep up is rebuilt after 5 minutes of running time instead of at once.] → The source holds at most that much more WAL than the threshold. `max_slot_wal_keep_size` and the slot retention alert still bound the worst case.
+- [While back pressure keeps pausing the receiver and each run drains the lag, the lag rule never fires and the source keeps WAL until archiving catches up.] → This is what the comment always promised. The source is protected by `max_slot_wal_keep_size` and warned by the slot retention alert, and the Monitoring tab shows the backlog.
+- [A lag that shrinks slowly but by more than 5% of the threshold per 5 minutes is never rebuilt.] → It ends by itself. If it slows below that, the next window asks for a rebuild.
+
+## Migration Plan
+
+No data or schema change. The classifier state lives in memory per session, so a deploy simply starts with empty windows. A rollback restores the old one-sample rule.
diff --git a/openspec/changes/archive/2026-10-19-fix-wal-lag-rebuild-after-backpressure/proposal.md b/openspec/changes/archive/2026-10-19-fix-wal-lag-rebuild-after-backpressure/proposal.md
new file mode 100644
index 00000000..1f89e1e3
--- /dev/null
+++ b/openspec/changes/archive/2026-10-19-fix-wal-lag-rebuild-after-backpressure/proposal.md
@@ -0,0 +1,41 @@
+## Why
+
+Databasus drops and recreates the replication slot when the slot falls behind by more than `WalLagThresholdBytes`, 256 MiB by default. The rebuild leaves a gap in the WAL chain and forces a new full backup. The code intends this as the operator's "sacrifice the chain to protect the primary" knob, and promises in a comment that it "never fires against lag we caused by holding the receiver down ourselves" (`backups/backups/usecases/physical/postgresql/lag_monitor.go:64-71`).
+
+The check does not keep that promise. It only asks whether the receiver is running at the moment of the 30-second tick (`lag_monitor.go:110-113`). Back pressure stops the receiver while the local queue drains, and the slot falls behind during the pause. When the receiver starts again, the next tick sees a lag above the threshold with the receiver running, and drops the slot, although the receiver is just about to catch up. The same happens after Databasus was down for a few minutes in front of a busy source. A busy database writing to a distant storage, the case `parallel-wal-segment-uploads` was built for, can therefore lose its chain on every long pause.
+
+## What Changes
+
+- The lag threshold asks for a rebuild only when the running receiver fails to reduce the lag: over 5 minutes of time the receiver spent running, the lag stayed above the threshold and shrank by less than 5% of it. Time while the receiver is stopped does not count, and receiver restarts do not reset the count.
+- A receiver that is catching up after a back-pressure pause, a receiver restart or a Databasus restart no longer costs the chain, however large the lag it starts from.
+- A receiver that runs and still falls behind, or only keeps pace while far behind, is rebuilt as before after 5 minutes of running time, also when the stall watcher keeps restarting it.
+- A lost slot and a slot held by a foreign consumer are still handled on the first sample. Every rebuild keeps the existing limit of attempts per hour.
+
+## Capabilities
+
+### New Capabilities
+
+- `physical-backups/wal-slot-rebuild`: when Databasus drops and recreates the WAL streaming replication slot, and when it must not.
+
+### Modified Capabilities
+
+None.
+
+## Impact
+
+- Module rules: `AGENTS.md` and `backend/AGENTS.md`.
+- Backend:
+ - the slot break classifier in `lag_monitor.go` gains a lag window over running time;
+ - the receiver's running flag becomes the start time of its current run, zero while stopped;
+ - the classifier tests change, because one sample above the threshold no longer rebuilds.
+- The Monitoring tab from `wal-stream-monitoring-tab`, already implemented, estimates the time to a slot rebuild only up to the moment the lag reaches the threshold (`fillEstimates` in `wal_health/keeping_up.go`). After this change the rebuild comes at least 5 minutes of running time later, and only when the lag does not shrink, so the estimate runs early until its follow-up lands.
+- Source disk: while back pressure keeps pausing the receiver, the threshold no longer drops the slot, so the source keeps the WAL until the receiver catches up. `max_slot_wal_keep_size` on the source and the existing slot retention alert remain the protection for that case, as the comment already intends.
+
+## Out of scope
+
+- Changing the default threshold or making the hold period configurable.
+- Showing the threshold in the backup settings UI.
+- The slot stall watcher (`slot_lsn_watcher.go`) and the rebuild attempt cap.
+- Back pressure itself and the watermark sizes.
+- Requesting a full backup after the silent slot recreate at session start.
+- Follow-up: adding the hold of running time to the Monitoring tab's slot rebuild estimate (`EstimatedSlotRebuildSeconds`) and its `rebuildConsequence` text.
diff --git a/openspec/changes/archive/2026-10-19-fix-wal-lag-rebuild-after-backpressure/specs/physical-backups/wal-slot-rebuild/spec.md b/openspec/changes/archive/2026-10-19-fix-wal-lag-rebuild-after-backpressure/specs/physical-backups/wal-slot-rebuild/spec.md
new file mode 100644
index 00000000..ed518e30
--- /dev/null
+++ b/openspec/changes/archive/2026-10-19-fix-wal-lag-rebuild-after-backpressure/specs/physical-backups/wal-slot-rebuild/spec.md
@@ -0,0 +1,104 @@
+## Purpose
+
+Defines when Databasus asks to drop and recreate the replication slot of a WAL streaming database. A rebuild always leaves a gap in the WAL chain and forces a new full backup, so Databasus asks for one only when keeping the slot would harm the source or cannot work.
+
+## ADDED Requirements
+
+### Requirement: The lag threshold asks for a rebuild only when the running receiver does not reduce the lag
+
+When a lag threshold is configured, Databasus SHALL judge the slot lag only over time the receiver spends running. It SHALL compare consecutive samples taken within the same receiver run, and add up their duration and their lag change. Time while the receiver is stopped, and the lag that builds up during it, SHALL count for nothing. A receiver restart SHALL NOT discard what was added up before it.
+
+Databasus SHALL ask for a rebuild with the lag threshold as the reason when, over at least 5 minutes of added-up running time:
+
+- every sample had a lag above the threshold;
+- the lag shrank in total by less than 5% of the threshold.
+
+When the lag shrank by 5% of the threshold or more over 5 minutes of running time, the receiver is draining. The added-up time and change then SHALL start again from zero.
+
+They SHALL also start again from zero when a sample has a lag at or below the threshold, or when the slot's status is `lost`, `extended` or `unreserved`. A lag threshold of zero SHALL never ask for a rebuild.
+
+A rebuild that Databasus asks for SHALL be subject to the existing limit on rebuild attempts per hour.
+
+#### Scenario: Resume after a back-pressure pause
+
+- **WHEN** the receiver was paused until the local queue drained, the slot lag reached twice the threshold meanwhile, and the receiver now runs and the lag shrinks
+- **THEN** no rebuild is asked for
+
+#### Scenario: Start after Databasus was down
+
+- **WHEN** Databasus starts after 10 minutes of downtime in front of a busy source, and the receiver begins with a lag above the threshold that shrinks
+- **THEN** no rebuild is asked for
+
+#### Scenario: Slow catch-up longer than 5 minutes
+
+- **WHEN** the receiver has run for 20 minutes with the lag above the threshold, and the lag shrank by more than 5% of the threshold in every 5 minutes
+- **THEN** no rebuild is asked for
+
+#### Scenario: Short runs between pauses
+
+- **WHEN** back pressure pauses the receiver every 3 minutes, the lag grows during each pause, and it shrinks during each run
+- **THEN** no rebuild is asked for
+
+#### Scenario: Receiver cannot keep up
+
+- **WHEN** the receiver runs and the lag stays above the threshold and grows
+- **THEN** a rebuild with the lag threshold as the reason is asked for on the first sample after 5 minutes of running time, and not before
+
+#### Scenario: Lag fluctuates above the threshold
+
+- **WHEN** the receiver runs and the lag moves up and down by less than 5% of the threshold around a level above the threshold
+- **THEN** a rebuild is asked for on the first sample after 5 minutes of running time
+
+#### Scenario: Stalled receiver restarted repeatedly
+
+- **WHEN** the receiver is restarted every 70 seconds because it stopped confirming WAL, and the lag stays above the threshold and grows
+- **THEN** a rebuild is asked for once the sampled intervals inside those runs add up to 5 minutes
+- **AND** with 30-second samples, that is roughly 7 to 12 minutes of wall time
+
+#### Scenario: Receiver stopped
+
+- **WHEN** the lag is above the threshold and the receiver is not running
+- **THEN** no rebuild is asked for, and the time does not count towards the 5 minutes
+
+#### Scenario: Lag drops below the threshold
+
+- **WHEN** the lag stays above the threshold for 4 minutes of running time, drops below it for one sample, and rises above it again
+- **THEN** no rebuild is asked for until 5 more minutes of running time pass with the lag above the threshold
+
+#### Scenario: Extended episode ends above the threshold
+
+- **WHEN** the slot's status was `extended` and returns to `reserved` with the lag still above the threshold
+- **THEN** no rebuild is asked for until 5 minutes of running time pass after the return, with the lag above the threshold and not shrinking by 5% of the threshold
+
+#### Scenario: Hourly limit reached
+
+- **WHEN** the lag rule asks for a rebuild after three rebuild attempts in the last hour
+- **THEN** the slot is not dropped
+
+#### Scenario: No threshold
+
+- **WHEN** the lag threshold is zero
+- **THEN** no rebuild is ever asked for because of lag
+
+### Requirement: A lost slot or a slot held by another consumer is handled on the first sample
+
+On the first sample that reports the slot's status as `lost`, Databasus SHALL ask for a rebuild with the lost slot as the reason, whatever the receiver's state. On the first sample that shows the slot held by a backend that is not Databasus's own receiver, it SHALL ask for a rebuild with the stolen slot as the reason. It SHALL NOT drop a slot while a consumer that is not its own holds it. Both are subject to the hourly limit on rebuild attempts.
+
+A slot whose status is `extended` or `unreserved` SHALL raise the existing slot retention alert after 5 minutes. Databasus SHALL NOT ask for a rebuild because of that status or because of lag while the status lasts.
+
+#### Scenario: Lost slot
+
+- **WHEN** a sample reports the slot's status as `lost`
+- **THEN** a rebuild with the lost slot as the reason is asked for in that tick, whatever the receiver's state
+
+#### Scenario: Foreign consumer
+
+- **WHEN** a sample shows the slot held by a backend that is not Databasus's receiver
+- **THEN** a rebuild with the stolen slot as the reason is asked for in that tick
+- **AND** the slot is not dropped while that backend holds it
+
+#### Scenario: Extended slot above the lag threshold
+
+- **WHEN** the slot's status is `extended`, the receiver has run for 10 minutes and the lag is above the threshold and growing
+- **THEN** no rebuild is asked for
+- **AND** the slot retention alert is raised once the status has lasted 5 minutes
diff --git a/openspec/changes/archive/2026-10-19-fix-wal-lag-rebuild-after-backpressure/tasks.md b/openspec/changes/archive/2026-10-19-fix-wal-lag-rebuild-after-backpressure/tasks.md
new file mode 100644
index 00000000..cce85271
--- /dev/null
+++ b/openspec/changes/archive/2026-10-19-fix-wal-lag-rebuild-after-backpressure/tasks.md
@@ -0,0 +1,33 @@
+Backend commands run in `backend/`. `<pg>` stands for `./internal/features/backups/backups/usecases/physical/postgresql/...`.
+
+## 1. Run start time
+
+- [x] 1.1 Replace `isReceiverRunning` with one atomic run-start time, set from `startedAt` in `receivewal_process.go` and cleared on exit, and derive "running" from it in the lag monitor and the slot-LSN watcher (design D1). Build the classifier's sample in a small supervisor method, and replace `IsReceiverRunning` in `slotBreakSample` with `ReceiverRunStartedAt`; the lag rule keeps its current behavior, reading a non-zero start as running. Update the existing classifier tests to set `ReceiverRunStartedAt`. Verify with `just test-package <pg>` passing, including a new test in `wal_stream_test.go` that runs a receiver against the test source and checks through the sample-building method that the sample carries a start time while it runs and, waiting with `require.Eventually` up to a limit, a zero one after it exits.
+
+## 2. Reproduce the bug
+
+- [x] 2.1 Add `Test_RecordSampleAndClassifyBreak_WhenReceiverResumesWithLagAboveThreshold_DoesNotRebuild` in `lag_monitor_test.go` for the spec scenario "Resume after a back-pressure pause": one sample with a zero `ReceiverRunStartedAt` and the lag at twice the threshold, then samples with a new non-zero start time and the lag shrinking. Verify that `just test-package <pg>` fails on this test before the fix, because the first running sample asks for a rebuild.
+
+## 3. Fix
+
+- [x] 3.1 Replace the one-sample lag rule with the lag window over running time from design D2, using `warningSlotStatusHoldPeriod` as its length and a new 5% tolerance constant (D3), and update the comment above it. Rewrite `Test_RecordSampleAndClassifyBreak_WhenLagOverThresholdAndReceiverRunning_Rebuilds`, which pins the old behavior, as the "Receiver cannot keep up" case. Verify with `just test-package <pg>` passing, the test from 2.1 included, and with new classifier tests in `lag_monitor_test.go` covering the spec scenarios:
+ - "Start after Databasus was down" and "Slow catch-up longer than 5 minutes";
+ - "Short runs between pauses": a new start time every 3 minutes, lag growing between runs and shrinking within them;
+ - "Receiver cannot keep up": the rebuild comes on the first sample after 5 minutes of running time, not before;
+ - "Lag fluctuates above the threshold": swings below 5% of the threshold, rebuild on the first sample after 5 minutes of running time;
+ - "Stalled receiver restarted repeatedly": a new start time every 70 seconds with 30-second samples and the lag growing reaches a rebuild on the sample where the intervals between samples of the same run add up to 5 minutes, and not before;
+ - "Receiver stopped", "Lag drops below the threshold" and "No threshold";
+ - "Extended episode ends above the threshold" and "Extended slot above the lag threshold";
+ - "Lost slot" and "Foreign consumer", whose existing tests must still pass.
+
+ The scenarios "Hourly limit reached" and the foreign consumer's refusal to drop are pinned by the existing `Test_WalStream_RebuildAttemptCap_StopsFourthAttemptInHour` and the `Test_IsOwnedReceiverBackend_*` tests in `slot_rebuild_test.go`, which must still pass; this change does not touch either path (design D4).
+
+## 4. Wrap-up
+
+- [x] 4.1 Run the formatter and the linters. Verify with `just format` and `just lint` at the repository root finishing clean.
+
+## Change verification
+
+```sh
+just test-package ./internal/features/backups/backups/usecases/physical/postgresql/...
+```
diff --git a/openspec/specs/physical-backups/wal-slot-rebuild/spec.md b/openspec/specs/physical-backups/wal-slot-rebuild/spec.md
new file mode 100644
index 00000000..8cf75622
--- /dev/null
+++ b/openspec/specs/physical-backups/wal-slot-rebuild/spec.md
@@ -0,0 +1,106 @@
+# WAL Slot Rebuild Specification
+
+## Purpose
+
+Defines when Databasus asks to drop and recreate the replication slot of a WAL streaming database. A rebuild always leaves a gap in the WAL chain and forces a new full backup, so Databasus asks for one only when keeping the slot would harm the source or cannot work.
+
+## Requirements
+
+### Requirement: The lag threshold asks for a rebuild only when the running receiver does not reduce the lag
+
+When a lag threshold is configured, Databasus SHALL judge the slot lag only over time the receiver spends running. It SHALL compare consecutive samples taken within the same receiver run, and add up their duration and their lag change. Time while the receiver is stopped, and the lag that builds up during it, SHALL count for nothing. A receiver restart SHALL NOT discard what was added up before it.
+
+Databasus SHALL ask for a rebuild with the lag threshold as the reason when, over at least 5 minutes of added-up running time:
+
+- every sample had a lag above the threshold;
+- the lag shrank in total by less than 5% of the threshold.
+
+When the lag shrank by 5% of the threshold or more over 5 minutes of running time, the receiver is draining. The added-up time and change then SHALL start again from zero.
+
+They SHALL also start again from zero when a sample has a lag at or below the threshold, or when the slot's status is `lost`, `extended` or `unreserved`. A lag threshold of zero SHALL never ask for a rebuild.
+
+A rebuild that Databasus asks for SHALL be subject to the existing limit on rebuild attempts per hour.
+
+#### Scenario: Resume after a back-pressure pause
+
+- **WHEN** the receiver was paused until the local queue drained, the slot lag reached twice the threshold meanwhile, and the receiver now runs and the lag shrinks
+- **THEN** no rebuild is asked for
+
+#### Scenario: Start after Databasus was down
+
+- **WHEN** Databasus starts after 10 minutes of downtime in front of a busy source, and the receiver begins with a lag above the threshold that shrinks
+- **THEN** no rebuild is asked for
+
+#### Scenario: Slow catch-up longer than 5 minutes
+
+- **WHEN** the receiver has run for 20 minutes with the lag above the threshold, and the lag shrank by more than 5% of the threshold in every 5 minutes
+- **THEN** no rebuild is asked for
+
+#### Scenario: Short runs between pauses
+
+- **WHEN** back pressure pauses the receiver every 3 minutes, the lag grows during each pause, and it shrinks during each run
+- **THEN** no rebuild is asked for
+
+#### Scenario: Receiver cannot keep up
+
+- **WHEN** the receiver runs and the lag stays above the threshold and grows
+- **THEN** a rebuild with the lag threshold as the reason is asked for on the first sample after 5 minutes of running time, and not before
+
+#### Scenario: Lag fluctuates above the threshold
+
+- **WHEN** the receiver runs and the lag moves up and down by less than 5% of the threshold around a level above the threshold
+- **THEN** a rebuild is asked for on the first sample after 5 minutes of running time
+
+#### Scenario: Stalled receiver restarted repeatedly
+
+- **WHEN** the receiver is restarted every 70 seconds because it stopped confirming WAL, and the lag stays above the threshold and grows
+- **THEN** a rebuild is asked for once the sampled intervals inside those runs add up to 5 minutes
+- **AND** with 30-second samples, that is roughly 7 to 12 minutes of wall time
+
+#### Scenario: Receiver stopped
+
+- **WHEN** the lag is above the threshold and the receiver is not running
+- **THEN** no rebuild is asked for, and the time does not count towards the 5 minutes
+
+#### Scenario: Lag drops below the threshold
+
+- **WHEN** the lag stays above the threshold for 4 minutes of running time, drops below it for one sample, and rises above it again
+- **THEN** no rebuild is asked for until 5 more minutes of running time pass with the lag above the threshold
+
+#### Scenario: Extended episode ends above the threshold
+
+- **WHEN** the slot's status was `extended` and returns to `reserved` with the lag still above the threshold
+- **THEN** no rebuild is asked for until 5 minutes of running time pass after the return, with the lag above the threshold and not shrinking by 5% of the threshold
+
+#### Scenario: Hourly limit reached
+
+- **WHEN** the lag rule asks for a rebuild after three rebuild attempts in the last hour
+- **THEN** the slot is not dropped
+
+#### Scenario: No threshold
+
+- **WHEN** the lag threshold is zero
+- **THEN** no rebuild is ever asked for because of lag
+
+### Requirement: A lost slot or a slot held by another consumer is handled on the first sample
+
+On the first sample that reports the slot's status as `lost`, Databasus SHALL ask for a rebuild with the lost slot as the reason, whatever the receiver's state. On the first sample that shows the slot held by a backend that is not Databasus's own receiver, it SHALL ask for a rebuild with the stolen slot as the reason. It SHALL NOT drop a slot while a consumer that is not its own holds it. Both are subject to the hourly limit on rebuild attempts.
+
+A slot whose status is `extended` or `unreserved` SHALL raise the existing slot retention alert after 5 minutes. Databasus SHALL NOT ask for a rebuild because of that status or because of lag while the status lasts.
+
+#### Scenario: Lost slot
+
+- **WHEN** a sample reports the slot's status as `lost`
+- **THEN** a rebuild with the lost slot as the reason is asked for in that tick, whatever the receiver's state
+
+#### Scenario: Foreign consumer
+
+- **WHEN** a sample shows the slot held by a backend that is not Databasus's receiver
+- **THEN** a rebuild with the stolen slot as the reason is asked for in that tick
+- **AND** the slot is not dropped while that backend holds it
+
+#### Scenario: Extended slot above the lag threshold
+
+- **WHEN** the slot's status is `extended`, the receiver has run for 10 minutes and the lag is above the threshold and growing
+- **THEN** no rebuild is asked for
+- **AND** the slot retention alert is raised once the status has lasted 5 minutes