Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(cohorts): upload checkpoints on request instead of on every stop (#122841)

Matheus Batistacd76b5eb81e0
diff --git a/docs/internal/realtime-cohorts/processor-runtime.md b/docs/internal/realtime-cohorts/processor-runtime.md
index fd23e6ea8478..0a4cb46dd74e 100644
--- a/docs/internal/realtime-cohorts/processor-runtime.md
+++ b/docs/internal/realtime-cohorts/processor-runtime.md
@@ -218,7 +218,7 @@ Change `COHORT_POD_COUNT` only after every pod runs a build with stamps.
With durable restore enabled, the pod also refuses to start without checkpoints, because each parent deletes the slices it gives away and its child would start without them.
- A pod whose ordinal is below the old count is a parent.
- It reopens its store, and boot deletes the slices it gave away.
+ It reopens its store, and boot deletes the slices it gave away and drops their files.
- Every other pod is a child, and takes its partitions over from the parent at its ordinal modulo the old count.
It waits, with the `parent_running` reason, until the parent has stamped a partition under the new split.
It then deletes a live store that holds no pending restore, and restores the newest checkpoint in the parent's S3 lineage that covers it, counted in `checkpoint_restore_total{source="parent_s3"}`.
@@ -227,16 +227,20 @@ With durable restore enabled, the pod also refuses to start without checkpoints,
To reshard:
-1. Scale the processor to 0, and wait until every pod has stopped after its final checkpoint.
+1. On each parent, run `kill -USR1 1` from a shell in the pod, and wait for `checkpoint uploaded to S3` with `trigger` `requested`.
+ If a warning with that `trigger` comes instead, the upload failed, so send the signal again.
+ Within a minute or two, scale the processor to 0, and wait until every pod has stopped.
+ A stop's final checkpoint stays local, so each child replays only what its parent processed after that upload.
2. Set `COHORT_POD_COUNT` to the new count, `COHORT_RESHARD_FROM_POD_COUNT` to the old count, and the replicas to the new count.
3. Check `owned_partitions` on each pod, `ownership_fence_blocked` at 0, and one `parent_s3` restore for each child.
4. Remove `COHORT_RESHARD_FROM_POD_COUNT` in a later change, and before any offset reset of the events group.
While it is set, a reset that clears the stamp from every events partition of a child makes the child take its partitions over again and delete its store.
-If step 1 is skipped, each child waits until its parent restarts, and the old split keeps every partition until then.
-Deleting the parent pods moves the reshard on.
+Scaling to 0 is required.
+The StatefulSet updates from the highest ordinal and waits for each pod to turn ready, and a child is not ready while it waits for its parent, so a reshard applied in one edit deadlocks.
After its takeover, a child searches only its own lineage.
So a child that loses its volume before its first upload finishes creates its store behind the coverage fence, even though its parent's checkpoint still covers it.
+That upload waits until the child is ready and has compacted the partitions it does not own.
### Lowering the pod count
@@ -261,7 +265,7 @@ At boot the processor:
A checkpoint restore that cannot finish blocks here and retries, as [a blocked restore](state-store-and-durability.md#a-blocked-restore) describes.
A store written under a split that does not cover the pod's refuses to open, as [ownership](#ownership-and-the-single-pod-deployment) explains,
7. stamps every owned partition on every input group, and reads the stamps back,
-8. starts the timer loops and the events consumer, and starts the followers and the checkpoint loop once the catalog has loaded and boot recovery has ended.
+8. starts the timer loops and the events consumer, starts the followers once the catalog has loaded and boot recovery has ended, and takes the first checkpoint once the pod is ready.
By default the store is wiped at boot, so there is nothing to recover, and the events consumer dispatches from its first poll.
@@ -298,9 +302,10 @@ They commit only for the partitions recorded at the drain.
A follower that waits longer than 35 seconds commits for the partitions it owned when shutdown began.
With checkpoints enabled, a final checkpoint follows once every consumer and follower has finished.
-It captures positions for the partitions recorded at the drain and uploads within `CHECKPOINT_FINAL_UPLOAD_TIMEOUT_SECS` of the drain, after shutdown cancels any periodic upload in flight.
-A plan that would upload in full stays local, because it cannot finish in that budget, and the next start uploads in full anyway.
+It captures positions for the partitions recorded at the drain, after shutdown cancels any upload in flight.
+It stays local, and the next start's first upload covers what changed since the last upload.
It is skipped when boot never ended or the drain never finished, because positions are not settled then.
+The binary must run as PID 1, from an exec-form command, so that SIGTERM and SIGUSR1 reach it.
The events consumer gets 30 seconds to drain, each follower 45 seconds, and the final checkpoint 60 seconds after them, inside the process's 110-second ceiling.
Kubernetes kills the pod when its termination grace period ends, and that period also covers the chart's 5-second preStop sleep before the shutdown starts.
diff --git a/docs/internal/realtime-cohorts/state-store-and-durability.md b/docs/internal/realtime-cohorts/state-store-and-durability.md
index 78c4ad527c5d..d0d42a538f86 100644
--- a/docs/internal/realtime-cohorts/state-store-and-durability.md
+++ b/docs/internal/realtime-cohorts/state-store-and-durability.md
@@ -235,7 +235,7 @@ It renames the store to the staging path first, so a deletion cut short never le
## Checkpoints
With `CHECKPOINT_ENABLED`, the processor takes a whole-store RocksDB checkpoint into `CHECKPOINT_LOCAL_DIR` every `CHECKPOINT_INTERVAL_MS`, and uploads one in every few to object storage, at `CHECKPOINT_S3_UPLOAD_INTERVAL_MS`.
-The loop starts when boot ends.
+The first checkpoint waits until the pod is ready, so its upload does not compete for the disk with the workers' eviction-queue rebuilds.
Checkpoints require durable restore, a bucket, and an absolute checkpoint directory without `..` on the store's filesystem but outside the store and its `<STORE_PATH>.restore` staging directory, and the pod fails at start without them.
- **One lineage per pod.**
@@ -256,23 +256,41 @@ Checkpoints require durable restore, a bucket, and an absolute checkpoint direct
Once a chain is older than `CHECKPOINT_FULL_UPLOAD_INTERVAL_SECS`, the next upload is full, which bounds the age of the oldest object a restore needs.
- **Paced uploads.**
One upload reads at most `CHECKPOINT_UPLOAD_MAX_BYTES_PER_SEC` from the store's disk across all its files, so a full upload does not starve the live path.
-- **A final checkpoint.**
- A graceful stop cancels a periodic upload in flight, takes one more checkpoint once every consumer has made its final commit, and uploads it when the plan is incremental.
- A full plan stays local, because it cannot finish in the budget, and the next start uploads in full anyway.
- The upload is cancelled `CHECKPOINT_FINAL_UPLOAD_TIMEOUT_SECS` after the drain, but an S3 request in flight runs to its end, so the checkpoint component's shutdown window is the hard limit.
+- **A final checkpoint that stays local.**
+ A graceful stop cancels an upload in flight, and takes one more checkpoint once every consumer has made its final commit.
+ It does not upload, so a stop usually takes seconds, and the next start's first upload covers what changed since the last upload.
+ An S3 request in flight when the stop cancels an upload still runs to its end, retries included, and while object storage fails those retries take tens of seconds.
+ The checkpoint component's shutdown window is the hard limit, and a stop that reaches it skips the final checkpoint.
+ It is still the freshest local checkpoint when the volume survives and the store does not.
An upload stopped before its `metadata.json` is not restorable, so the last finished upload stays the newest restorable one.
-- **No expiry.**
- Objects are not expired.
- An expiry, when one is set, must exceed the full-upload interval plus the import window.
+- **An upload on request.**
+ `SIGUSR1` asks the checkpoint loop for a checkpoint that uploads, whatever the cadence: run `kill -USR1 1` from a shell in the pod, such as the Argo terminal.
+ The image has no `kill` binary, so `kubectl exec` needs `sh -c 'kill -USR1 1'`.
+ Before the loop starts, the request waits for it.
+ The pod logs `checkpoint upload requested (SIGUSR1)`, and then `checkpoint uploaded to S3` with `trigger` `requested` and the `checkpoint_id`.
+ A requested checkpoint that fails at any step logs a warning with `trigger` `requested` instead, and is not retried, so send the signal again.
+ A request that arrives during a checkpoint waits for it, and requests that arrive meanwhile count as one.
+- **Compaction of unowned partitions.**
+ Boot deletes the slices of the partitions the pod does not own, and drops the SST files that lie wholly inside them.
+ The other files, such as those that also hold an owned partition, keep the deleted data until a compaction rewrites them.
+ So once the pod is ready, and before its first checkpoint, the checkpoint task compacts every unowned partition that still holds more than 16 MiB in a column family, one at a time.
+ It logs the bytes above that size before and after, and how long the pass took.
+ A routine boot finds none.
+ A compaction cannot be interrupted, so a stop waits for the one in progress.
+- **Expiry by age.**
+ The bucket's lifecycle rule expires objects by age.
+ That age must exceed `CHECKPOINT_FULL_UPLOAD_INTERVAL_SECS` plus `CHECKPOINT_IMPORT_WINDOW_HOURS`, because a restore can pick any upload in the import window, and that upload can name objects as old as its chain's full upload.
The uploader is built on the first upload, so an outage of object storage never stops a pod that has its store.
A failed build counts `checkpoint_uploads_total{result="unavailable"}`, and the next upload tries again.
These metrics follow checkpoints:
-- `checkpoint_last_upload_timestamp_seconds` is the last successful upload, and starts at process start, so a staleness alert fires only once its threshold passes,
+- `checkpoint_last_upload_timestamp_seconds` is the capture time of the newest successful upload.
+ It starts at the upload that `uploaded.json` records, or at process start without one, so a restart does not make an old upload look new,
- `checkpoint_last_capture_timestamp_seconds` and `checkpoint_last_full_upload_timestamp_seconds` are the last captured manifest and the start of the newest upload's chain,
-- `checkpoint_uploads_total{result, trigger}` and `checkpoint_uploaded_bytes_total{kind}` count uploads and the bytes they sent,
+- `checkpoint_size_bytes` and `checkpoint_file_count` are the size and the file count of the newest local checkpoint,
+- `checkpoint_uploads_total{result, trigger}` and `checkpoint_uploaded_bytes_total{kind}` count uploads and the bytes they sent, where `trigger` is `periodic` or `requested`,
- `checkpoint_capture_failures_total{reason}` counts ticks skipped before the checkpoint,
- `checkpoint_restore_total{source}` counts where each boot's store came from: `reopen_live`, `pending`, `local`, `s3` or `cold`,
- `checkpoint_restore_candidates_total{verdict}` counts the candidates a restore tried,
diff --git a/rust/cohort-stream-processor/clippy.toml b/rust/cohort-stream-processor/clippy.toml
index 5de12f755f8c..ce3c752f52c5 100644
--- a/rust/cohort-stream-processor/clippy.toml
+++ b/rust/cohort-stream-processor/clippy.toml
@@ -31,6 +31,7 @@ disallowed-methods = [
{ path = "cohort_stream_processor::store::rocks::CohortStore::flush_wal_sync", reason = "async code must use StoreHandle::flush_wal_sync; sync contexts need a module-level allow — see store/handle.rs" },
{ path = "cohort_stream_processor::store::rocks::CohortStore::reset_slices", reason = "async code must not write the store on a runtime thread; sync contexts need a module-level allow — see store/handle.rs" },
{ path = "cohort_stream_processor::store::rocks::CohortStore::create_checkpoint", reason = "async code must go through the checkpoint sweeper's own offload; sync contexts need a module-level allow — see store/handle.rs" },
+ { path = "cohort_stream_processor::store::rocks::CohortStore::compact_unowned", reason = "async code must go through the checkpoint sweeper's own offload; sync contexts need a module-level allow — see store/handle.rs" },
{ path = "cohort_stream_processor::store::rocks::CohortStore::stats_snapshot", reason = "async code must use StoreHandle::stats_snapshot; sync contexts need a module-level allow — see store/handle.rs" },
{ path = "rdkafka::consumer::Consumer::commit", reason = "every offset commit must carry this pod's stamp; commit a StampedList — see partitions/stamp.rs" },
{ path = "rdkafka::consumer::Consumer::commit_consumer_state", reason = "every offset commit must carry this pod's stamp; commit a StampedList — see partitions/stamp.rs" },
diff --git a/rust/cohort-stream-processor/src/config.rs b/rust/cohort-stream-processor/src/config.rs
index 57e325039c00..a991464dcf3b 100644
--- a/rust/cohort-stream-processor/src/config.rs
+++ b/rust/cohort-stream-processor/src/config.rs
@@ -570,7 +570,7 @@ pub struct Config {
pub checkpoint_max_upload_buffers: usize,
/// Retries per S3 operation before giving up.
- #[envconfig(default = "3")]
+ #[envconfig(default = "10")]
pub checkpoint_s3_max_retries: usize,
/// Total timeout for one S3 operation including retries (secs).
@@ -613,13 +613,6 @@ pub struct Config {
/// Average rate one upload may read from the store's disk (bytes per second).
#[envconfig(default = "67108864")]
pub checkpoint_upload_max_bytes_per_sec: u64,
-
- /// Budget for the final checkpoint on a graceful stop, from the drain to the end of its upload
- /// (secs). Past it the upload is cancelled, but an S3 request in flight runs to its end, so the
- /// checkpoint component's shutdown window is the hard limit. An upload stopped before its
- /// `metadata.json` leaves the last finished upload the newest restorable one.
- #[envconfig(default = "45")]
- pub checkpoint_final_upload_timeout_secs: u64,
}
/// librdkafka consumer fetch-queue bounds: an aggregate byte cap across all partitions and a
@@ -828,7 +821,6 @@ impl Config {
local_checkpoint_max_staleness: self.checkpoint_local_max_staleness(),
full_upload_interval: Duration::from_secs(self.checkpoint_full_upload_interval_secs),
upload_max_bytes_per_sec: self.checkpoint_upload_max_bytes_per_sec,
- final_upload_timeout: Duration::from_secs(self.checkpoint_final_upload_timeout_secs),
}
}
@@ -951,7 +943,7 @@ impl Config {
!self.checkpoint_s3_bucket.is_empty(),
"CHECKPOINT_ENABLED requires CHECKPOINT_S3_BUCKET.",
);
- let knobs: [(&str, u64); 14] = [
+ let knobs: [(&str, u64); 13] = [
("CHECKPOINT_INTERVAL_MS", self.checkpoint_interval_ms),
(
"CHECKPOINT_S3_UPLOAD_INTERVAL_MS",
@@ -997,10 +989,6 @@ impl Config {
"CHECKPOINT_IMPORT_TIMEOUT_SECS",
self.checkpoint_import_timeout_secs,
),
- (
- "CHECKPOINT_FINAL_UPLOAD_TIMEOUT_SECS",
- self.checkpoint_final_upload_timeout_secs,
- ),
(
"CHECKPOINT_UPLOAD_MAX_BYTES_PER_SEC",
self.checkpoint_upload_max_bytes_per_sec,
@@ -1367,7 +1355,7 @@ mod tests {
checkpoint_max_concurrent_uploads: 40,
checkpoint_max_concurrent_downloads: 40,
checkpoint_max_upload_buffers: 40,
- checkpoint_s3_max_retries: 3,
+ checkpoint_s3_max_retries: 10,
checkpoint_s3_operation_timeout_secs: 120,
checkpoint_s3_attempt_timeout_secs: 20,
checkpoint_local_max_staleness_secs: 7200,
@@ -1376,7 +1364,6 @@ mod tests {
checkpoint_import_timeout_secs: 7200,
checkpoint_full_upload_interval_secs: 86_400,
checkpoint_upload_max_bytes_per_sec: 64 * 1024 * 1024,
- checkpoint_final_upload_timeout_secs: 45,
cohort_seed_consumer_enabled: false,
cohort_stream_seed_events_topic: "cohort_stream_seed_events".to_string(),
kafka_seed_consumer_group: "cohort-stream-seeds".to_string(),
@@ -2102,7 +2089,7 @@ mod tests {
enabled().validate_startup().unwrap();
type Misconfigure = fn(&mut Config);
- let cases: [(&str, Misconfigure, &str); 10] = [
+ let cases: [(&str, Misconfigure, &str); 9] = [
(
"no bucket",
|c| c.checkpoint_s3_bucket.clear(),
@@ -2113,11 +2100,6 @@ mod tests {
|c| c.checkpoint_interval_ms = 0,
"CHECKPOINT_INTERVAL_MS",
),
- (
- "a zero final upload budget",
- |c| c.checkpoint_final_upload_timeout_secs = 0,
- "CHECKPOINT_FINAL_UPLOAD_TIMEOUT_SECS",
- ),
(
"a relative checkpoint dir",
|c| c.checkpoint_local_dir = "checkpoints".to_string(),
@@ -2252,7 +2234,6 @@ mod tests {
config.checkpoint_import_timeout_secs = 90;
config.checkpoint_full_upload_interval_secs = 3601;
config.checkpoint_upload_max_bytes_per_sec = 1024;
- config.checkpoint_final_upload_timeout_secs = 12;
let durability = config.durability_config();
assert_eq!(durability.local_checkpoint_dir, "/data/ckpt");
@@ -2283,7 +2264,6 @@ mod tests {
);
assert_eq!(durability.full_upload_interval, Duration::from_secs(3601));
assert_eq!(durability.upload_max_bytes_per_sec, 1024);
- assert_eq!(durability.final_upload_timeout, Duration::from_secs(12));
}
#[test]
diff --git a/rust/cohort-stream-processor/src/consumers/events.rs b/rust/cohort-stream-processor/src/consumers/events.rs
index adf4ed36f449..3324c37d083c 100644
--- a/rust/cohort-stream-processor/src/consumers/events.rs
+++ b/rust/cohort-stream-processor/src/consumers/events.rs
@@ -659,13 +659,14 @@ impl EventDispatcher {
counter!(PARTITIONS_ASSIGNED_TOTAL).increment(1);
}
- /// Delete the on-disk slices of every partition `ownership` does not own. The store opens only
- /// under the ownership it was written under, or adopts it, so at one pod this deletes nothing.
- pub(crate) async fn reclaim_unowned_slices(&self, ownership: &PartitionOwnership) {
+ /// Delete the on-disk slices of every partition `ownership` does not own, and return how many
+ /// held any key. The store opens only under the ownership it was written under, or adopts it,
+ /// so at one pod this deletes nothing.
+ pub(crate) async fn reclaim_unowned_slices(&self, ownership: &PartitionOwnership) -> u64 {
let mut wiped = 0u64;
for partition in (0..ownership.partition_count()).filter(|&p| !ownership.owns(p)) {
match self.handle.delete_partition(partition).await {
- Ok(()) => wiped += 1,
+ Ok(held_keys) => wiped += u64::from(held_keys),
Err(err) => warn!(
partition,
error = %err, "failed to reclaim unowned-partition state at boot"
@@ -681,6 +682,7 @@ impl EventDispatcher {
partition_count = ownership.partition_count(),
"durable restore: boot staleness sweep complete"
);
+ wiped
}
/// Re-produce every staged `cf_pending_transfers` entry for the owned partitions at boot, before
@@ -1860,7 +1862,7 @@ mod tests {
PartitionOwnership::new(NonZeroU16::new(2).unwrap(), PodOrdinal::STANDALONE, 4)
.unwrap();
- dispatcher.reclaim_unowned_slices(&pod_zero_of_two).await;
+ assert_eq!(dispatcher.reclaim_unowned_slices(&pod_zero_of_two).await, 2);
assert_eq!(
(0..4u16)
@@ -1868,6 +1870,7 @@ mod tests {
.collect::<Vec<_>>(),
[true, false, true, false],
);
+ assert_eq!(dispatcher.reclaim_unowned_slices(&pod_zero_of_two).await, 0);
}
#[tokio::test]
diff --git a/rust/cohort-stream-processor/src/consumers/readiness.rs b/rust/cohort-stream-processor/src/consumers/readiness.rs
index 2a3d5a19b269..aeff2dd76d9f 100644
--- a/rust/cohort-stream-processor/src/consumers/readiness.rs
+++ b/rust/cohort-stream-processor/src/consumers/readiness.rs
@@ -6,10 +6,9 @@
//! answer would put two pods into catch-up at once.
use std::fmt;
-use std::sync::atomic::{AtomicUsize, Ordering};
use std::sync::Arc;
-use tokio::sync::SetOnce;
+use tokio::sync::{watch, SetOnce};
use crate::filters::manager::CatalogHandle;
@@ -39,7 +38,7 @@ impl fmt::Display for NotReady {
pub struct BootReadiness {
catalog: Arc<CatalogHandle>,
live: SetOnce<()>,
- rebuilding: AtomicUsize,
+ rebuilding: watch::Sender<usize>,
}
impl BootReadiness {
@@ -47,7 +46,7 @@ impl BootReadiness {
Arc::new(Self {
catalog,
live: SetOnce::new(),
- rebuilding: AtomicUsize::new(0),
+ rebuilding: watch::Sender::new(0),
})
}
@@ -58,7 +57,8 @@ impl BootReadiness {
if !self.live.initialized() {
return Err(NotReady::Booting);
}
- match self.rebuilding.load(Ordering::Acquire) {
+ let rebuilding = *self.rebuilding.borrow();
+ match rebuilding {
0 => Ok(()),
partitions => Err(NotReady::Rebuilding { partitions }),
}
@@ -77,8 +77,19 @@ impl BootReadiness {
self.live.wait().await;
}
+ /// Resolve once [`Self::check`] reads ready. Boot registers every worker's rebuild before it
+ /// ends, so after that the count only falls.
+ pub async fn wait_until_ready(&self) {
+ self.wait_until_dispatchable().await;
+ self.rebuilding
+ .subscribe()
+ .wait_for(|&rebuilding| rebuilding == 0)
+ .await
+ .expect("the sender lives in self, so the channel never closes");
+ }
+
pub fn rebuild_started(self: &Arc<Self>) -> RebuildGuard {
- self.rebuilding.fetch_add(1, Ordering::AcqRel);
+ self.rebuilding.send_modify(|rebuilding| *rebuilding += 1);
RebuildGuard(self.clone())
}
}
@@ -89,12 +100,16 @@ pub struct RebuildGuard(Arc<BootReadiness>);
impl Drop for RebuildGuard {
fn drop(&mut self) {
- self.0.rebuilding.fetch_sub(1, Ordering::AcqRel);
+ self.0.rebuilding.send_modify(|rebuilding| *rebuilding -= 1);
}
}
#[cfg(test)]
mod tests {
+ use std::pin::pin;
+
+ use futures::FutureExt;
+
use super::*;
use crate::filters::FilterCatalog;
@@ -111,6 +126,8 @@ mod tests {
let readiness =
BootReadiness::new(Arc::new(CatalogHandle::from_catalog(FilterCatalog::new())));
assert_eq!(readiness.check(), Err(NotReady::Booting));
+ let mut ready = pin!(readiness.wait_until_ready());
+ assert_eq!(ready.as_mut().now_or_never(), None);
let finished = readiness.rebuild_started();
let panicked = readiness.rebuild_started();
@@ -125,6 +142,7 @@ mod tests {
readiness.check(),
Err(NotReady::Rebuilding { partitions: 1 })
);
+ assert_eq!(ready.as_mut().now_or_never(), None);
let rebuild = tokio::spawn(async move {
let _guard = panicked;
@@ -132,5 +150,6 @@ mod tests {
});
assert!(rebuild.await.unwrap_err().is_panic());
assert_eq!(readiness.check(), Ok(()));
+ assert_eq!(ready.now_or_never(), Some(()));
}
}
diff --git a/rust/cohort-stream-processor/src/main.rs b/rust/cohort-stream-processor/src/main.rs
index 03c2427f1b92..8c2fed874e45 100644
--- a/rust/cohort-stream-processor/src/main.rs
+++ b/rust/cohort-stream-processor/src/main.rs
@@ -8,6 +8,7 @@ use envconfig::Envconfig;
use lifecycle::{ComponentOptions, Handle, Manager};
use rdkafka::consumer::{Consumer, ConsumerContext, StreamConsumer};
use tokio::net::TcpListener;
+use tokio::signal::unix::{signal, SignalKind};
use tracing::{info, warn};
use tracing_subscriber::layer::SubscriberExt;
use tracing_subscriber::util::SubscriberInitExt;
@@ -84,6 +85,13 @@ async fn async_main(config: Config) -> Result<()> {
.checkpoint_enabled
.then(|| config.checkpoint_lineage())
.transpose()?;
+ // `kill -USR1 1` uploads a checkpoint ahead of a planned stop, such as a reshard, because the
+ // final checkpoint stays local. The handler goes in before the restore, so a request made
+ // during boot waits for the checkpoint loop: without a handler, PID 1 discards the signal.
+ let upload_requests = lineage
+ .map(|_| signal(SignalKind::user_defined1()))
+ .transpose()
+ .context("installing the SIGUSR1 handler")?;
// Covers the 45 s follower windows, then the checkpoint component's 60 s. The pod's termination
// grace period must exceed it plus the chart's 5 s preStop sleep, so above 115 s, or Kubernetes
@@ -588,9 +596,9 @@ async fn async_main(config: Config) -> Result<()> {
));
// Whole-DB checkpoints to the local volume and S3, from boot to the final one on a graceful stop.
- // The loop waits for boot: a capture before the events rewind would record the broker's old
- // offsets of a pending restore.
- if let Some(checkpoint_handle) = checkpoint_handle {
+ // The task waits for boot: a capture before the events rewind would record the broker's old
+ // offsets of a pending restore. Its first checkpoint then waits for the pod to be ready.
+ if let Some((checkpoint_handle, mut upload_requests)) = checkpoint_handle.zip(upload_requests) {
let sweeper = Arc::new(CheckpointSweeper::new(
store_for_checkpoint,
dispatcher.clone(),
@@ -603,6 +611,20 @@ async fn async_main(config: Config) -> Result<()> {
),
consumer_handle.shutdown_token(),
));
+ let requests_stop = consumer_handle.shutdown_token();
+ let requested = sweeper.clone();
+ tokio::spawn(async move {
+ loop {
+ tokio::select! {
+ biased;
+ _ = requests_stop.cancelled() => break,
+ Some(()) = upload_requests.recv() => {
+ info!("checkpoint upload requested (SIGUSR1)");
+ requested.request_upload();
+ }
+ }
+ }
+ });
let stop = consumer_handle.shutdown_token();
let interval = config.checkpoint_interval();
let checkpoint_readiness = boot_readiness.clone();
@@ -612,7 +634,8 @@ async fn async_main(config: Config) -> Result<()> {
// Boot never ended, so there are no settled positions to checkpoint.
_ = stop.cancelled() => checkpoint_handle.work_completed(),
_ = checkpoint_readiness.wait_until_dispatchable() => {
- run_checkpoints(sweeper, interval, checkpoint_handle).await;
+ run_checkpoints(sweeper, &checkpoint_readiness, interval, checkpoint_handle)
+ .await;
}
}
});
diff --git a/rust/cohort-stream-processor/src/observability/metrics.rs b/rust/cohort-stream-processor/src/observability/metrics.rs
index a006b3092ebd..effe1945d184 100644
--- a/rust/cohort-stream-processor/src/observability/metrics.rs
+++ b/rust/cohort-stream-processor/src/observability/metrics.rs
@@ -72,8 +72,8 @@ pub const WAL_FSYNC_ERRORS_TOTAL: &str = "wal_fsync_errors_total";
/// On-disk partition slices kept (reopen-live) across a durable restart, because still assigned to
/// this pod (counter).
pub const DURABLE_RESTORE_PARTITIONS_KEPT_TOTAL: &str = "durable_restore_partitions_kept_total";
-/// On-disk partition slices wiped because no longer assigned to this pod — stale state to cold-rebuild
-/// from the committed offset (counter).
+/// On-disk partition slices wiped at boot because another pod owns them, counting only slices that
+/// held a key (counter). A boot that follows a reshard wipes some, and other boots wipe none.
pub const DURABLE_RESTORE_PARTITIONS_WIPED_STALE_TOTAL: &str =
"durable_restore_partitions_wiped_stale_total";
/// Partitions adopted as complete by opening a store without coverage records (counter).
@@ -93,13 +93,13 @@ pub const DURABLE_RESTORE_PENDING_TRANSFERS_RECOVERED_PARTITIONS_TOTAL: &str =
/// Wall-clock duration of one whole-DB `Checkpoint::create_checkpoint` (histogram, seconds).
/// Whole-DB checkpoint, so these are per-pod, not per-partition.
pub const CHECKPOINT_DURATION_SECONDS: &str = "checkpoint_duration_seconds";
-/// On-disk size of a freshly-taken checkpoint directory (histogram, bytes).
+/// On-disk size of the newest local checkpoint directory (gauge, bytes).
pub const CHECKPOINT_SIZE_BYTES: &str = "checkpoint_size_bytes";
-/// File count in a freshly-taken checkpoint directory (histogram).
+/// File count in the newest local checkpoint directory (gauge).
pub const CHECKPOINT_FILE_COUNT: &str = "checkpoint_file_count";
/// Checkpoint S3 uploads, labelled by `result` (`success`|`error`|`cancelled`|`unavailable`) and
-/// `trigger` (`periodic`|`final`), plus checkpoint ticks skipped by a failed WAL fsync
-/// (`result=flush_failed`) (counter). A cancelled final upload ran out of its budget.
+/// `trigger` (`periodic`|`requested`), plus checkpoint ticks skipped by a failed WAL fsync
+/// (`result=flush_failed`) (counter). A shutdown cancels an upload in flight.
pub const CHECKPOINT_UPLOADS_TOTAL: &str = "checkpoint_uploads_total";
/// Bytes a successful upload sent, labelled by `kind` (`full`|`incremental`) (counter). Sizes the
/// bucket's growth for its expiry setting.
@@ -108,8 +108,9 @@ pub const CHECKPOINT_UPLOADED_BYTES_TOTAL: &str = "checkpoint_uploaded_bytes_tot
/// staleness alert fires only after its threshold passes.
pub const CHECKPOINT_LAST_CAPTURE_TIMESTAMP_SECONDS: &str =
"checkpoint_last_capture_timestamp_seconds";
-/// Unix time of the last successful upload (gauge, seconds). Starts at the sweeper's start.
-/// **Alert on staleness**: no restorable checkpoint is getting newer.
+/// Capture time of the newest successful upload, in Unix seconds (gauge). Starts at the upload
+/// `uploaded.json` records, else at the sweeper's start. **Alert on staleness**: no restorable
+/// checkpoint is getting newer.
pub const CHECKPOINT_LAST_UPLOAD_TIMESTAMP_SECONDS: &str =
"checkpoint_last_upload_timestamp_seconds";
/// `full_upload_at` of the newest upload: when its chain of incrementals began (gauge, seconds).
diff --git a/rust/cohort-stream-processor/src/store/durability/checkpoint.rs b/rust/cohort-stream-processor/src/store/durability/checkpoint.rs
index f6d797162cbd..fda9dd455906 100644
--- a/rust/cohort-stream-processor/src/store/durability/checkpoint.rs
+++ b/rust/cohort-stream-processor/src/store/durability/checkpoint.rs
@@ -2,9 +2,9 @@
//!
//! [`CheckpointSweeper`] takes a frozen whole-DB RocksDB checkpoint to the local volume on every
//! tick, beside an offset manifest read from the consumer groups, and uploads it to S3 every Nth
-//! tick and once more on a graceful stop, unless that upload would be full. [`run_checkpoints`]
-//! drives it. One `create_checkpoint` at a time, never two racing: the final checkpoint runs on the
-//! periodic loop's task, after the loop.
+//! tick and whenever an operator requests one. A graceful stop takes a final checkpoint that stays
+//! local. [`run_checkpoints`] drives it. One `create_checkpoint` at a time, never two racing:
+//! requested and final checkpoints run on the periodic loop's task.
//!
//! ## Interior mutability without a lock across `.await`
//!
@@ -35,8 +35,8 @@ use async_trait::async_trait;
use chrono::{DateTime, Utc};
use lifecycle::Handle;
use serde::{Deserialize, Serialize};
-use tokio::sync::OnceCell;
-use tokio::time::Instant;
+use tokio::sync::{Notify, OnceCell};
+use tokio::time::{Instant, MissedTickBehavior};
use tokio_util::sync::CancellationToken;
use tracing::{error, info, warn};
@@ -46,20 +46,26 @@ use super::{
OffsetManifest, S3Uploader, METADATA_FILENAME,
};
use crate::consumers::events::{DrainWait, DrainedOwnership};
-use crate::consumers::EventDispatcher;
+use crate::consumers::{BootReadiness, EventDispatcher};
use crate::observability::metrics::{
CHECKPOINT_CAPTURE_FAILURES_TOTAL, CHECKPOINT_FILE_COUNT,
CHECKPOINT_LAST_CAPTURE_TIMESTAMP_SECONDS, CHECKPOINT_LAST_FULL_UPLOAD_TIMESTAMP_SECONDS,
CHECKPOINT_LAST_UPLOAD_TIMESTAMP_SECONDS, CHECKPOINT_SIZE_BYTES,
- CHECKPOINT_UPLOADED_BYTES_TOTAL, CHECKPOINT_UPLOADS_TOTAL,
+ CHECKPOINT_UPLOADED_BYTES_TOTAL, CHECKPOINT_UPLOADS_TOTAL, SWEEP_CYCLES_TOTAL,
+ SWEEP_CYCLE_DURATION_SECONDS,
};
use crate::partitions::{ClaimedSplit, InputGroups, PositionsError};
use crate::store::{CohortStore, DbIdentity};
-use crate::sweep::{run_sweep_loop, Sweeper};
+use crate::sweep::Sweeper;
/// The loop-name label for the checkpoint sweep cycle metrics.
pub const CHECKPOINT_LOOP_NAME: &str = "checkpoint";
+/// Size past which a range of an unowned partition is compacted before the first checkpoint. A
+/// compacted range keeps about one partial block per straddling file, far below it, so a routine
+/// boot compacts nothing. The straddling files of a reshard leave far more.
+const UNOWNED_COMPACTION_THRESHOLD_BYTES: u64 = 16 * 1024 * 1024;
+
/// The last successful upload's [`UploadBaseline`], in the lineage's local directory. A restarted
/// process builds its first upload on it when the store's DB id and the bucket still match. A
/// restore rewrites it, because a restored DB reuses the SST numbers written after its checkpoint.
@@ -132,18 +138,20 @@ pub fn upload_cadence(checkpoint_interval_ms: u64, s3_upload_interval_ms: u64) -
/// Why a checkpoint runs.
#[derive(Debug)]
pub(crate) enum Trigger {
- Periodic,
- /// A graceful stop, after every consumer's final commit. Its upload is cancelled at `deadline`.
- Final {
- drained: DrainedOwnership,
- deadline: Instant,
- },
+ /// A tick of the periodic loop. Every `upload_every_n`-th tick uploads.
+ Periodic { upload: bool },
+ /// An operator asked for an upload now, ahead of a planned stop.
+ Requested,
+ /// A graceful stop, after every consumer's final commit. It stays local, so the stop takes
+ /// seconds, and the next start uploads what changed.
+ Final { drained: DrainedOwnership },
}
impl Trigger {
fn label(&self) -> &'static str {
match self {
- Self::Periodic => "periodic",
+ Self::Periodic { .. } => "periodic",
+ Self::Requested => "requested",
Self::Final { .. } => "final",
}
}
@@ -152,8 +160,8 @@ impl Trigger {
/// because closing the events consumer empties the live set.
fn owned(&self, live: impl FnOnce() -> Vec<i32>) -> BTreeSet<u16> {
let partitions = match self {
- Self::Periodic => live(),
- Self::Final { drained, .. } => drained.partitions().iter().copied().collect(),
+ Self::Periodic { .. } | Self::Requested => live(),
+ Self::Final { drained } => drained.partitions().iter().copied().collect(),
};
partitions
.into_iter()
@@ -161,13 +169,11 @@ impl Trigger {
.collect()
}
- /// A final checkpoint uploads off cadence, so a planned stop leaves nothing to replay past it.
- /// It uploads only an incremental plan: a full one cannot finish within its budget, and the
- /// next start uploads in full anyway.
- fn uploads(&self, tick: u64, every_n: u64, kind: &UploadKind<'_>) -> bool {
+ fn uploads(&self) -> bool {
match self {
- Self::Periodic => should_upload(tick, every_n),
- Self::Final { .. } => matches!(kind, UploadKind::Incremental(_)),
+ Self::Periodic { upload } => *upload,
+ Self::Requested => true,
+ Self::Final { .. } => false,
}
}
}
@@ -246,7 +252,10 @@ pub struct CheckpointSweeper {
upload_every_n: u64,
/// Baseline for the next incremental upload. Updated only after a successful upload.
baseline: Mutex<Option<CheckpointMetadata>>,
- /// Cancels a periodic upload at shutdown, so the final checkpoint does not wait behind it.
+ /// Holds at most one permit, so requests made while a checkpoint runs become one upload after
+ /// it.
+ upload_requested: Notify,
+ /// Cancels an upload in flight at shutdown, so the final checkpoint does not wait behind it.
stop: CancellationToken,
}
@@ -269,9 +278,14 @@ impl CheckpointSweeper {
.join(UPLOADED_FILENAME),
&config.s3_bucket,
);
- let started = Utc::now().timestamp() as f64;
- metrics::gauge!(CHECKPOINT_LAST_CAPTURE_TIMESTAMP_SECONDS).set(started);
- metrics::gauge!(CHECKPOINT_LAST_UPLOAD_TIMESTAMP_SECONDS).set(started);
+ let started = Utc::now();
+ metrics::gauge!(CHECKPOINT_LAST_CAPTURE_TIMESTAMP_SECONDS).set(started.timestamp() as f64);
+ // A restart must not make the last upload look new.
+ let last_upload = baseline
+ .as_ref()
+ .map_or(started, |uploaded| uploaded.attempt_timestamp);
+ metrics::gauge!(CHECKPOINT_LAST_UPLOAD_TIMESTAMP_SECONDS)
+ .set(last_upload.timestamp() as f64);
Self {
store,
dispatcher,
@@ -283,6 +297,7 @@ impl CheckpointSweeper {
tick: AtomicU64::new(0),
upload_every_n,
baseline: Mutex::new(baseline),
+ upload_requested: Notify::new(),
stop,
}
}
@@ -292,16 +307,17 @@ impl CheckpointSweeper {
.local_dir(Path::new(&self.config.local_checkpoint_dir))
}
+ /// Asks the checkpoint loop for a checkpoint that uploads at once, whatever the cadence.
+ pub fn request_upload(&self) {
+ self.upload_requested.notify_one();
+ }
+
/// The checkpoint a graceful stop takes once every consumer has made its final commit. Skipped
/// when the worker drain never finished, because the positions are then not settled.
pub async fn final_checkpoint(&self) {
- // The budget starts before the WAL flush and the checkpoint, so the upload's cancel lands
- // inside this component's shutdown window.
- let deadline = Instant::now() + self.config.final_upload_timeout;
match self.dispatcher.wait_for_worker_drain(Duration::ZERO).await {
DrainWait::Drained(drained) => {
- self.checkpoint_once(Trigger::Final { drained, deadline })
- .await;
+ self.checkpoint_once(Trigger::Final { drained }).await;
}
DrainWait::TimedOut => warn!(
"final checkpoint skipped: the worker drain did not finish, so positions are unsettled",
@@ -309,22 +325,82 @@ impl CheckpointSweeper {
}
}
- async fn checkpoint_once(&self, trigger: Trigger) {
- let tick = self.tick.fetch_add(1, Ordering::SeqCst);
+ /// Boot drops the whole files of the partitions this pod does not own, but a file that also
+ /// holds an owned partition keeps their data until a compaction rewrites it. Compacting those
+ /// ranges before the first checkpoint keeps the data out of the uploads. A stop waits for the
+ /// range in progress.
+ async fn compact_unowned_ranges(&self) {
+ let store = self.store.clone();
+ let ownership = self.claimed.ownership();
+ let stop = self.stop.clone();
+ let started = Instant::now();
+ let compaction = tokio::task::spawn_blocking(move || {
+ store.compact_unowned(&ownership, UNOWNED_COMPACTION_THRESHOLD_BYTES, &stop)
+ })
+ .await;
+ match compaction {
+ Ok(Ok(pass)) if pass.ranges == 0 => {}
+ Ok(Ok(pass)) => info!(
+ ranges = pass.ranges,
+ compacted = pass.compacted,
+ bytes_before = pass.bytes_before,
+ bytes_after = pass.bytes_after,
+ elapsed_secs = started.elapsed().as_secs_f64(),
+ "compacted the unowned ranges above the threshold",
+ ),
+ Ok(Err(e)) => warn!(
+ error = %e,
+ "compacting the unowned ranges failed; the uploads carry their data",
+ ),
+ Err(join_err) => error!(
+ error = %join_err,
+ "compacting the unowned ranges panicked; the uploads carry their data",
+ ),
+ }
+ }
+
+ /// One task takes the periodic and the requested checkpoints, so two never race.
+ async fn run_loop(&self, interval: Duration) {
+ let mut ticker = tokio::time::interval(interval);
+ ticker.set_missed_tick_behavior(MissedTickBehavior::Skip);
+ info!(
+ interval_ms = interval.as_millis(),
+ "checkpoint loop started"
+ );
+ loop {
+ tokio::select! {
+ biased;
+ _ = self.stop.cancelled() => break,
+ // Before the ticker: a checkpoint that outlasts the interval leaves a tick ready
+ // at once, so a request polled after it would wait behind every such tick.
+ _ = self.upload_requested.notified() => {
+ self.checkpoint_once(Trigger::Requested).await;
+ }
+ _ = ticker.tick() => {
+ let started = Instant::now();
+ self.run_once().await;
+ metrics::histogram!(SWEEP_CYCLE_DURATION_SECONDS, "loop" => CHECKPOINT_LOOP_NAME)
+ .record(started.elapsed().as_secs_f64());
+ metrics::counter!(SWEEP_CYCLES_TOTAL, "loop" => CHECKPOINT_LOOP_NAME).increment(1);
+ }
+ }
+ }
+ }
+ async fn checkpoint_once(&self, trigger: Trigger) {
// 1. fsync the WAL before the snapshot. The checkpoint must hold at least what the commits
// read in step 3 cover, and each of those followed its own fsync.
let flush_store = self.store.clone();
match tokio::task::spawn_blocking(move || flush_store.flush_wal_sync()).await {
Ok(Ok(())) => {}
Ok(Err(e)) => {
- warn!(error = %e, "checkpoint tick: WAL fsync failed; skipping tick");
+ warn!(error = %e, trigger = trigger.label(), "checkpoint tick: WAL fsync failed; skipping tick");
metrics::counter!(CHECKPOINT_UPLOADS_TOTAL, "result" => "flush_failed")
.increment(1);
return;
}
Err(join_err) => {
- error!(error = %join_err, "checkpoint tick: WAL fsync task panicked; skipping tick");
+ error!(error = %join_err, trigger = trigger.label(), "checkpoint tick: WAL fsync task panicked; skipping tick");
metrics::counter!(CHECKPOINT_UPLOADS_TOTAL, "result" => "flush_failed")
.increment(1);
return;
@@ -357,13 +433,13 @@ impl CheckpointSweeper {
PositionsError::Foreign { .. } => "foreign",
PositionsError::Kafka(_) | PositionsError::Negative(_) => "positions",
};
- warn!(error = %e, "checkpoint tick: reading input positions failed; skipping tick");
+ warn!(error = %e, trigger = trigger.label(), "checkpoint tick: reading input positions failed; skipping tick");
metrics::counter!(CHECKPOINT_CAPTURE_FAILURES_TOTAL, "reason" => reason)
.increment(1);
return;
}
Err(join_err) => {
- error!(error = %join_err, "checkpoint tick: reading input positions panicked; skipping tick");
+ error!(error = %join_err, trigger = trigger.label(), "checkpoint tick: reading input positions panicked; skipping tick");
metrics::counter!(CHECKPOINT_CAPTURE_FAILURES_TOTAL, "reason" => "positions")
.increment(1);
return;
@@ -374,7 +450,7 @@ impl CheckpointSweeper {
let manifest = match OffsetManifest::capture(&owned, positions) {
Ok(manifest) => manifest,
Err(e) => {
- warn!(error = %e, "checkpoint tick: incomplete manifest; skipping tick");
+ warn!(error = %e, trigger = trigger.label(), "checkpoint tick: incomplete manifest; skipping tick");
metrics::counter!(CHECKPOINT_CAPTURE_FAILURES_TOTAL, "reason" => "incomplete")
.increment(1);
return;
@@ -388,7 +464,7 @@ impl CheckpointSweeper {
let attempt_dir = self.local_dir().join(&checkpoint_id);
// `create_checkpoint` requires the leaf to NOT exist; it creates it and errors if it does.
if let Err(e) = tokio::fs::create_dir_all(self.local_dir()).await {
- warn!(error = %e, dir = %self.local_dir().display(), "checkpoint tick: cannot create attempt parent dir; skipping tick");
+ warn!(error = %e, dir = %self.local_dir().display(), trigger = trigger.label(), "checkpoint tick: cannot create attempt parent dir; skipping tick");
return;
}
let checkpoint_store = self.store.clone();
@@ -400,12 +476,12 @@ impl CheckpointSweeper {
match create_result {
Ok(Ok(())) => {}
Ok(Err(e)) => {
- warn!(error = %e, "checkpoint tick: create_checkpoint failed; skipping tick");
+ warn!(error = %e, trigger = trigger.label(), "checkpoint tick: create_checkpoint failed; skipping tick");
drop(tokio::fs::remove_dir_all(&attempt_dir).await);
return;
}
Err(join_err) => {
- error!(error = %join_err, "checkpoint tick: create_checkpoint task panicked; skipping tick");
+ error!(error = %join_err, trigger = trigger.label(), "checkpoint tick: create_checkpoint task panicked; skipping tick");
drop(tokio::fs::remove_dir_all(&attempt_dir).await);
return;
}
@@ -414,7 +490,7 @@ impl CheckpointSweeper {
// 6. Write offsets.json after create_checkpoint (so it is never frozen mid-write) but before
// planning, so the planner tracks it as a non-SST file and the S3 upload carries it.
if let Err(e) = manifest.write_to_dir(&attempt_dir) {
- warn!(error = %e, "checkpoint tick: writing offsets.json failed; skipping tick");
+ warn!(error = %e, trigger = trigger.label(), "checkpoint tick: writing offsets.json failed; skipping tick");
drop(tokio::fs::remove_dir_all(&attempt_dir).await);
return;
}
@@ -423,7 +499,7 @@ impl CheckpointSweeper {
let db = match self.store.db_identity() {
Ok(db) => db,
Err(e) => {
- warn!(error = %e, "checkpoint tick: reading the DB id failed; skipping tick");
+ warn!(error = %e, trigger = trigger.label(), "checkpoint tick: reading the DB id failed; skipping tick");
drop(tokio::fs::remove_dir_all(&attempt_dir).await);
return;
}
@@ -449,7 +525,7 @@ impl CheckpointSweeper {
) {
Ok(plan) => plan,
Err(e) => {
- warn!(error = %e, "checkpoint tick: planning failed; skipping tick");
+ warn!(error = %e, trigger = trigger.label(), "checkpoint tick: planning failed; skipping tick");
drop(tokio::fs::remove_dir_all(&attempt_dir).await);
return;
}
@@ -459,15 +535,15 @@ impl CheckpointSweeper {
.metadata
.save(&attempt_dir.join(METADATA_FILENAME))
{
- warn!(error = %e, "checkpoint tick: writing metadata.json failed; skipping tick");
+ warn!(error = %e, trigger = trigger.label(), "checkpoint tick: writing metadata.json failed; skipping tick");
drop(tokio::fs::remove_dir_all(&attempt_dir).await);
return;
}
// 8. Emit local-checkpoint size and file-count metrics.
let (size_bytes, file_count) = dir_size_and_count(&attempt_dir);
- metrics::histogram!(CHECKPOINT_SIZE_BYTES).record(size_bytes as f64);
- metrics::histogram!(CHECKPOINT_FILE_COUNT).record(file_count as f64);
+ metrics::gauge!(CHECKPOINT_SIZE_BYTES).set(size_bytes as f64);
+ metrics::gauge!(CHECKPOINT_FILE_COUNT).set(file_count as f64);
metrics::gauge!(CHECKPOINT_LAST_CAPTURE_TIMESTAMP_SECONDS)
.set(manifest.captured_at().timestamp() as f64);
info!(
@@ -480,7 +556,7 @@ impl CheckpointSweeper {
);
// 9. Upload when due, and advance the baseline on success.
- if trigger.uploads(tick, self.upload_every_n, &kind) {
+ if trigger.uploads() {
match &kind {
UploadKind::Full(reason) => {
info!(checkpoint_id, ?reason, "uploading a full checkpoint")
@@ -492,11 +568,10 @@ impl CheckpointSweeper {
),
}
self.upload(&plan, kind.label(), &trigger).await;
- } else if let (Trigger::Final { .. }, UploadKind::Full(reason)) = (&trigger, &kind) {
- warn!(
+ } else if matches!(trigger, Trigger::Final { .. }) {
+ info!(
checkpoint_id,
- ?reason,
- "final checkpoint kept local: a full upload cannot finish within its budget",
+ "final checkpoint kept local; the next start uploads what changed",
);
}
@@ -519,45 +594,18 @@ impl CheckpointSweeper {
Err(e) => {
metrics::counter!(CHECKPOINT_UPLOADS_TOTAL, "result" => "unavailable", "trigger" => trigger.label())
.increment(1);
- warn!(error = %format!("{e:#}"), "checkpoint S3 uploader unavailable; retrying on the next upload");
+ warn!(error = %format!("{e:#}"), trigger = trigger.label(), "checkpoint S3 uploader unavailable; retrying on the next upload");
return;
}
};
// A cancelled upload never writes metadata.json, so the last finished upload stays the newest
// restorable one, and the baseline stays where it was.
- let result = match trigger {
- Trigger::Periodic => {
- exporter
- .export_checkpoint_with_plan_cancellable(
- plan,
- Some(&self.stop),
- trigger.label(),
- )
- .await
- }
- Trigger::Final { deadline, .. } => {
- let cancel = CancellationToken::new();
- let upload = exporter.export_checkpoint_with_plan_cancellable(
- plan,
- Some(&cancel),
- trigger.label(),
- );
- tokio::pin!(upload);
- tokio::select! {
- result = &mut upload => result,
- // Cancel, then await, instead of dropping the upload with a timeout. A
- // cancelled upload aborts its multipart writes, and S3 keeps the parts of a
- // dropped one.
- _ = tokio::time::sleep_until(*deadline) => {
- cancel.cancel();
- upload.await
- }
- }
- }
- };
- if let Err(e) = result {
- warn!(error = %format!("{e:#}"), "checkpoint S3 upload failed; baseline unchanged");
+ if let Err(e) = exporter
+ .export_checkpoint_with_plan_cancellable(plan, Some(&self.stop), trigger.label())
+ .await
+ {
+ warn!(error = %format!("{e:#}"), trigger = trigger.label(), "checkpoint S3 upload failed; baseline unchanged");
return;
}
@@ -569,7 +617,7 @@ impl CheckpointSweeper {
.sum();
metrics::counter!(CHECKPOINT_UPLOADED_BYTES_TOTAL, "kind" => kind).increment(bytes);
metrics::gauge!(CHECKPOINT_LAST_UPLOAD_TIMESTAMP_SECONDS)
- .set(Utc::now().timestamp() as f64);
+ .set(plan.info.metadata.attempt_timestamp.timestamp() as f64);
metrics::gauge!(CHECKPOINT_LAST_FULL_UPLOAD_TIMESTAMP_SECONDS)
.set(plan.info.metadata.full_upload_at.timestamp() as f64);
@@ -586,6 +634,8 @@ impl CheckpointSweeper {
.lock()
.expect("the baseline lock is never held across a panic") = Some(uploaded);
info!(
+ checkpoint_id = %plan.info.metadata.id,
+ trigger = trigger.label(),
kind,
uploaded_files = plan.files_to_upload.len(),
bytes,
@@ -623,17 +673,31 @@ impl CheckpointSweeper {
#[async_trait]
impl Sweeper for CheckpointSweeper {
async fn run_once(&self) {
- self.checkpoint_once(Trigger::Periodic).await;
+ let tick = self.tick.fetch_add(1, Ordering::Relaxed);
+ let upload = should_upload(tick, self.upload_every_n);
+ self.checkpoint_once(Trigger::Periodic { upload }).await;
}
}
-/// Takes periodic checkpoints until the sweeper's stop token fires, then a final one once
-/// `handle`'s shutdown phase begins. Register `handle` in a later shutdown phase than every
-/// consumer, so the final checkpoint runs after their final commits.
-pub async fn run_checkpoints(sweeper: Arc<CheckpointSweeper>, interval: Duration, handle: Handle) {
+/// Waits until the pod is ready, so the first upload does not compete with the eviction rebuilds
+/// for the disk. Then takes periodic and requested checkpoints until the sweeper's stop token
+/// fires, and a final one once `handle`'s shutdown phase begins. Register `handle` in a later
+/// shutdown phase than every consumer, so the final checkpoint runs after their final commits.
+pub async fn run_checkpoints(
+ sweeper: Arc<CheckpointSweeper>,
+ readiness: &BootReadiness,
+ interval: Duration,
+ handle: Handle,
+) {
let _scope = handle.process_scope();
- let stop = sweeper.stop.clone();
- run_sweep_loop(sweeper.clone(), interval, CHECKPOINT_LOOP_NAME, stop).await;
+ tokio::select! {
+ biased;
+ _ = sweeper.stop.cancelled() => {}
+ _ = readiness.wait_until_ready() => {
+ sweeper.compact_unowned_ranges().await;
+ sweeper.run_loop(interval).await;
+ }
+ }
handle.shutdown_recv().await;
sweeper.final_checkpoint().await;
}
@@ -699,28 +763,25 @@ mod tests {
}
#[test]
- fn the_final_trigger_captures_the_drained_ownership_and_uploads_only_an_incremental_plan() {
+ fn the_final_trigger_captures_the_drained_ownership_and_stays_local_while_requested_uploads() {
let final_trigger = Trigger::Final {
drained: DrainedOwnership::for_test([3, 7]),
- deadline: Instant::now(),
};
- let baseline = CheckpointMetadata::new(
- PodOrdinal::STANDALONE,
- Utc::now(),
- DbIdentity::for_test("db"),
- );
- let incremental = UploadKind::Incremental(&baseline);
- let full = UploadKind::Full(FullReason::ChainExpired);
assert_eq!(final_trigger.owned(Vec::new), BTreeSet::from([3, 7]));
- assert!(final_trigger.uploads(1, 3, &incremental));
- assert!(!final_trigger.uploads(0, 3, &full));
-
assert_eq!(
- Trigger::Periodic.owned(|| vec![1, 2]),
+ Trigger::Requested.owned(|| vec![1, 2]),
BTreeSet::from([1, 2])
);
- assert!(!Trigger::Periodic.uploads(1, 3, &incremental));
- assert!(Trigger::Periodic.uploads(0, 3, &full));
+
+ let uploads = [
+ (final_trigger, false),
+ (Trigger::Requested, true),
+ (Trigger::Periodic { upload: true }, true),
+ (Trigger::Periodic { upload: false }, false),
+ ];
+ for (trigger, expected) in uploads {
+ assert_eq!(trigger.uploads(), expected, "{trigger:?}");
+ }
}
#[test]
diff --git a/rust/cohort-stream-processor/src/store/durability/config.rs b/rust/cohort-stream-processor/src/store/durability/config.rs
index 8e20cec49f97..f1b299ec9e24 100644
--- a/rust/cohort-stream-processor/src/store/durability/config.rs
+++ b/rust/cohort-stream-processor/src/store/durability/config.rs
@@ -81,9 +81,6 @@ pub struct DurabilityConfig {
/// Average read rate one upload may take from the store's disk.
pub upload_max_bytes_per_sec: u64,
-
- /// Budget for the final checkpoint on a graceful stop, from the drain to the end of its upload.
- pub final_upload_timeout: Duration,
}
impl Default for DurabilityConfig {
@@ -100,7 +97,7 @@ impl Default for DurabilityConfig {
checkpoint_import_window_hours: 20,
s3_operation_timeout: Duration::from_secs(120),
s3_attempt_timeout: Duration::from_secs(20),
- s3_max_retries: 3,
+ s3_max_retries: 10,
checkpoint_import_attempt_depth: 10,
max_concurrent_checkpoint_file_downloads: 40,
max_concurrent_checkpoint_file_uploads: 40,
@@ -111,7 +108,6 @@ impl Default for DurabilityConfig {
),
full_upload_interval: Duration::from_secs(86_400),
upload_max_bytes_per_sec: 64 * 1024 * 1024,
- final_upload_timeout: Duration::from_secs(45),
}
}
}
diff --git a/rust/cohort-stream-processor/src/store/durability/s3_client.rs b/rust/cohort-stream-processor/src/store/durability/s3_client.rs
index 9b197d5ef432..d19d686743eb 100644
--- a/rust/cohort-stream-processor/src/store/durability/s3_client.rs
+++ b/rust/cohort-stream-processor/src/store/durability/s3_client.rs
@@ -8,7 +8,7 @@ use futures::StreamExt;
use object_store::aws::{AmazonS3, AmazonS3Builder};
use object_store::limit::LimitStore;
use object_store::path::Path as ObjectPath;
-use object_store::{ClientOptions, ObjectStore};
+use object_store::{BackoffConfig, ClientOptions, ObjectStore};
use std::sync::Arc;
use std::time::Duration;
use tracing::{debug, info};
@@ -32,9 +32,17 @@ pub async fn create_s3_client(
.with_bucket_name(&config.s3_bucket)
.with_client_options(ClientOptions::new().with_timeout(config.s3_attempt_timeout))
.with_retry(object_store::RetryConfig {
+ // From 1 s instead of object_store's 100 ms, so ten fast failures spread over tens of
+ // seconds and outlast a broken connection. Each wait is a random draw between 1 s and
+ // twice the last wait, capped at 15 s, so the total stays below about 105 s. Slow
+ // attempts reach `retry_timeout` first.
+ backoff: BackoffConfig {
+ init_backoff: Duration::from_secs(1),
+ max_backoff: Duration::from_secs(15),
+ base: 2.0,
+ },
max_retries: config.s3_max_retries,
retry_timeout: config.s3_operation_timeout,
- ..Default::default()
});
if let Some(ref region) = config.aws_region {
diff --git a/rust/cohort-stream-processor/src/store/handle.rs b/rust/cohort-stream-processor/src/store/handle.rs
index f6087f1a3b37..5f7746c13984 100644
--- a/rust/cohort-stream-processor/src/store/handle.rs
+++ b/rust/cohort-stream-processor/src/store/handle.rs
@@ -54,9 +54,9 @@ const LANE_SECTION: &str = "section";
pub enum OffloadMode {
/// Every op runs inline on the caller's thread. The operator kill switch.
Off,
- /// Maintenance-lane reads, `flush_wal_sync`, `run_section`, and `stats_snapshot` offload;
- /// Event-lane reads and every other write run inline. Keeps the event path off the blocking pool
- /// while still isolating maintenance storms and the fsync syscall.
+ /// Maintenance-lane reads, `flush_wal_sync`, `delete_partition`, `run_section`, and
+ /// `stats_snapshot` offload; Event-lane reads and every other write run inline. Keeps the event
+ /// path off the blocking pool while still isolating maintenance storms and the fsync syscall.
Maintenance,
/// Every op offloads. The default operating point.
#[default]
@@ -169,9 +169,9 @@ impl StoreHandle {
}
/// Run a write op. Writes take no permit. Under Maintenance every write runs inline *except*
- /// `flush_wal_sync` (`is_fsync`): the fsync is the expensive syscall, memtable writes are cheap,
- /// so the fsync offloads while ordinary commits stay on the caller's thread. Under Off all
- /// inline, under All all offloaded.
+ /// the ones that fsync (`is_fsync`): the fsync is the expensive syscall, memtable writes are
+ /// cheap, so the fsync offloads while ordinary commits stay on the caller's thread. Under Off
+ /// all inline, under All all offloaded.
#[hotpath::skip]
async fn write<T, F>(&self, op: &'static str, is_fsync: bool, f: F) -> Result<T, StoreError>
where
@@ -499,9 +499,9 @@ impl StoreHandle {
.await
}
- /// Reclaim all state for one partition. No permit.
- pub async fn delete_partition(&self, partition_id: u16) -> Result<(), StoreError> {
- self.write("delete_partition", false, move |store| {
+ /// Reclaim all state for one partition, and return whether it held any key. No permit.
+ pub async fn delete_partition(&self, partition_id: u16) -> Result<bool, StoreError> {
+ self.write("delete_partition", true, move |store| {
store.delete_partition(partition_id)
})
.await
diff --git a/rust/cohort-stream-processor/src/store/rocks.rs b/rust/cohort-stream-processor/src/store/rocks.rs
index 50ffc595bf2f..9981973392dd 100644
--- a/rust/cohort-stream-processor/src/store/rocks.rs
+++ b/rust/cohort-stream-processor/src/store/rocks.rs
@@ -18,11 +18,12 @@ use metrics::{counter, histogram};
use rocksdb::properties::{self, PropName};
use rocksdb::statistics::{StatsLevel, Ticker};
use rocksdb::{
- Cache, ColumnFamily, DBWithThreadMode, Direction, FlushOptions, IteratorMode, Options,
- ReadOptions, SingleThreaded, WriteBatch, WriteOptions,
+ BottommostLevelCompaction, Cache, ColumnFamily, CompactOptions, DBWithThreadMode, Direction,
+ FlushOptions, IteratorMode, Options, ReadOptions, SingleThreaded, WriteBatch, WriteOptions,
};
use serde::{Deserialize, Serialize};
use thiserror::Error;
+use tokio_util::sync::CancellationToken;
use tracing::{info, warn};
use cohort_core::partitioner::COHORT_PARTITION_COUNT;
@@ -342,6 +343,27 @@ impl Drop for Stage2DirtyTrackingGuard {
}
}
+/// One partitioned CF's range of a partition another pod owns, with its estimated SST size.
+#[derive(Debug)]
+struct UnownedRange {
+ cf: Cf,
+ partition: u16,
+ bytes: u64,
+}
+
+/// What one [`CohortStore::compact_unowned`] pass did. Sizes are RocksDB's estimates.
+#[derive(Debug)]
+pub struct UnownedCompaction {
+ /// The unowned ranges above the threshold when the pass began.
+ pub ranges: usize,
+ /// The bytes of those ranges.
+ pub bytes_before: u64,
+ /// The ranges compacted. Fewer than `ranges` when the stop fired.
+ pub compacted: usize,
+ /// The bytes of the ranges still above the threshold after the pass.
+ pub bytes_after: u64,
+}
+
/// Handle to the per-process state store.
///
/// Writes have two layers. The closure-based [`Self::write_batch`] is for synchronous callers that
@@ -1165,13 +1187,73 @@ impl CohortStore {
Ok(out)
}
- /// Reclaim all state for one partition on rebalance. Non-partitioned CFs (`cf_meta`) are skipped:
- /// their short literal keys collide with an arbitrary partition's byte range, so a range delete
- /// would wipe store-wide guards like the schema stamp.
- pub fn delete_partition(&self, partition_id: u16) -> Result<(), StoreError> {
+ /// Reclaim all state for one partition on rebalance, and return whether it held any key.
+ /// Non-partitioned CFs (`cf_meta`) are skipped: their short literal keys collide with an
+ /// arbitrary partition's byte range, so a range delete would wipe store-wide guards like the
+ /// schema stamp.
+ ///
+ /// The SST files that lie wholly inside the partition are dropped only after the range deletes
+ /// are durable. A file that straddles the partition boundary can hold older versions of keys
+ /// that a dropped file overwrote, and only the range deletes keep those versions hidden.
+ /// RocksDB syncs a file drop to its MANIFEST at once, so the WAL fsync comes first: a host crash
+ /// must not keep the drop and lose the range deletes, because the partition can be owned again
+ /// later.
+ pub fn delete_partition(&self, partition_id: u16) -> Result<bool, StoreError> {
+ let held_keys = self.slice_holds_keys(partition_id)?;
let mut batch = WriteBatch::default();
self.delete_partition_ranges(&mut batch, partition_id)?;
- self.commit(batch, OP_DELETE_PARTITION)
+ self.commit(batch, OP_DELETE_PARTITION)?;
+ self.flush_wal_sync()?;
+ let (start, end) = keys::partition_range(partition_id);
+ for cf in Cf::ALL {
+ if !cf.partitioned() {
+ continue;
+ }
+ self.db
+ .delete_file_in_range_cf(self.cf(cf)?, &start, &end)
+ .map_err(|source| {
+ counter!(STORE_ERRORS_TOTAL, "op" => OP_DELETE_PARTITION).increment(1);
+ StoreError::Backend {
+ op: OP_DELETE_PARTITION,
+ source,
+ }
+ })?;
+ }
+ Ok(held_keys)
+ }
+
+ /// Total-order seek, because `cf_behavioral`'s prefix extractor would otherwise end the seek at
+ /// the first person boundary.
+ fn slice_holds_keys(&self, partition_id: u16) -> Result<bool, StoreError> {
+ let (start, end) = keys::partition_range(partition_id);
+ for cf in Cf::ALL {
+ if !cf.partitioned() {
+ continue;
+ }
+ let mut read_opts = ReadOptions::default();
+ read_opts.set_iterate_upper_bound(end.clone());
+ read_opts.set_total_order_seek(true);
+ let first = self
+ .db
+ .iterator_cf_opt(
+ self.cf(cf)?,
+ read_opts,
+ IteratorMode::From(&start, Direction::Forward),
+ )
+ .next()
+ .transpose()
+ .map_err(|source| {
+ counter!(STORE_ERRORS_TOTAL, "op" => OP_SCAN).increment(1);
+ StoreError::Backend {
+ op: OP_SCAN,
+ source,
+ }
+ })?;
+ if first.is_some() {
+ return Ok(true);
+ }
+ }
+ Ok(false)
}
/// Delete the slices a checkpoint restore cannot replay, with their coverage records, so each
@@ -1205,6 +1287,84 @@ impl CohortStore {
Ok(())
}
+ /// Compact through the bottom level every range of a partition outside `ownership` that holds
+ /// more than `threshold` bytes of SST data in a partitioned CF, which drops the data its range
+ /// deletes cover. A manual compaction cannot be interrupted, so the ranges compact one at a
+ /// time, and none starts after `stop` fires.
+ pub fn compact_unowned(
+ &self,
+ ownership: &PartitionOwnership,
+ threshold: u64,
+ stop: &CancellationToken,
+ ) -> Result<UnownedCompaction, StoreError> {
+ let ranges = self.unowned_ranges_above(ownership, threshold)?;
+ let mut opts = CompactOptions::default();
+ opts.set_bottommost_level_compaction(BottommostLevelCompaction::ForceOptimized);
+ let mut compacted = 0;
+ for range in &ranges {
+ if stop.is_cancelled() {
+ break;
+ }
+ let (start, end) = keys::partition_range(range.partition);
+ self.db
+ .compact_range_cf_opt(self.cf(range.cf)?, Some(start), Some(end), &opts);
+ compacted += 1;
+ }
+ // rust-rocksdb discards the compaction's error, so only a second measure shows what the
+ // pass dropped.
+ let bytes_after = self
+ .unowned_ranges_above(ownership, threshold)?
+ .iter()
+ .map(|range| range.bytes)
+ .sum();
+ Ok(UnownedCompaction {
+ ranges: ranges.len(),
+ bytes_before: ranges.iter().map(|range| range.bytes).sum(),
+ compacted,
+ bytes_after,
+ })
+ }
+
+ /// The ranges of the partitions outside `ownership` that hold more than `threshold` bytes of
+ /// SST data in a partitioned CF. RocksDB estimates the size from index blocks, and range deletes
+ /// do not shrink the estimate, so it counts deleted data that no compaction has dropped yet.
+ fn unowned_ranges_above(
+ &self,
+ ownership: &PartitionOwnership,
+ threshold: u64,
+ ) -> Result<Vec<UnownedRange>, StoreError> {
+ let partitions: Vec<u16> = (0..ownership.partition_count())
+ .filter(|&partition| !ownership.owns(partition))
+ .collect();
+ let bounds: Vec<(Vec<u8>, Vec<u8>)> = partitions
+ .iter()
+ .map(|&partition| keys::partition_range(partition))
+ .collect();
+ let ranges: Vec<rocksdb::Range<'_>> = bounds
+ .iter()
+ .map(|(start, end)| rocksdb::Range::new(start, end))
+ .collect();
+ let mut above = Vec::new();
+ for cf in Cf::ALL {
+ if !cf.partitioned() {
+ continue;
+ }
+ let sizes = self.db.get_approximate_sizes_cf(self.cf(cf)?, &ranges);
+ above.extend(
+ partitions
+ .iter()
+ .zip(sizes)
+ .filter(|&(_, bytes)| bytes > threshold)
+ .map(|(&partition, bytes)| UnownedRange {
+ cf,
+ partition,
+ bytes,
+ }),
+ );
+ }
+ Ok(above)
+ }
+
/// Flush all CF memtables to SST. Checkpoint path and tests only.
pub fn flush(&self) -> Result<(), StoreError> {
let handles: Vec<&ColumnFamily> = Cf::ALL
@@ -3147,6 +3307,109 @@ mod tests {
);
}
+ #[test]
+ fn deleting_unowned_partitions_drops_their_files_and_compaction_removes_the_rest() {
+ const KEYS: u128 = 1_000;
+ const THRESHOLD: u64 = 64 * 1024;
+ // Distinct in every 16-byte chunk, so LZ4 keeps each value near its 256 bytes and the size
+ // estimates stay far above `THRESHOLD`.
+ let value = |partition: u16, person: u128| -> Vec<u8> {
+ (0..16u128)
+ .flat_map(|chunk| {
+ ((u128::from(partition) << 96) | (person << 8) | chunk)
+ .wrapping_mul(0x2545_F491_4F6C_DD1D_9E37_79B9_7F4A_7C15)
+ .to_le_bytes()
+ })
+ .collect()
+ };
+ let dir = TempDir::new().unwrap();
+ let store = CohortStore::open(&StoreConfig {
+ path: dir.path().join("db"),
+ ..StoreConfig::default()
+ })
+ .unwrap();
+ // Partitions 0 and 1 each get a file of their own, and partitions 2 and 3 share one.
+ for partitions in [0..1u16, 1..2, 2..4] {
+ store
+ .write_batch(|batch| {
+ for partition in partitions.clone() {
+ for person in 0..KEYS {
+ batch.put::<Behavioral>(
+ &behavioral_key_for(partition, person),
+ &value(partition, person),
+ );
+ }
+ }
+ })
+ .unwrap();
+ store.flush().unwrap();
+ store.db.compact_range_cf(
+ store.cf(Cf::Behavioral).unwrap(),
+ Some(keys::partition_range(partitions.start).0),
+ Some(keys::partition_range(partitions.end - 1).1),
+ );
+ }
+ let behavioral_files = || -> Vec<rocksdb::LiveFile> {
+ store
+ .db
+ .live_files()
+ .unwrap()
+ .into_iter()
+ .filter(|file| file.column_family_name == Cf::Behavioral.as_str())
+ .collect()
+ };
+ assert_eq!(behavioral_files().len(), 3);
+ let pod_zero_of_two = PartitionOwnership::new(
+ std::num::NonZeroU16::new(2).unwrap(),
+ PodOrdinal::STANDALONE,
+ 4,
+ )
+ .unwrap();
+
+ assert!(store.delete_partition(1).unwrap());
+ assert!(store.delete_partition(3).unwrap());
+ assert_eq!(behavioral_files().len(), 2);
+ assert_eq!(
+ store
+ .unowned_ranges_above(&pod_zero_of_two, THRESHOLD)
+ .unwrap()
+ .iter()
+ .map(|range| (range.cf, range.partition))
+ .collect::<Vec<_>>(),
+ [(Cf::Behavioral, 3)],
+ );
+ let stopped = CancellationToken::new();
+ stopped.cancel();
+ let skipped = store
+ .compact_unowned(&pod_zero_of_two, THRESHOLD, &stopped)
+ .unwrap();
+ assert_eq!(
+ (skipped.ranges, skipped.compacted, skipped.bytes_after),
+ (1, 0, skipped.bytes_before),
+ );
+ let pass = store
+ .compact_unowned(&pod_zero_of_two, THRESHOLD, &CancellationToken::new())
+ .unwrap();
+ assert_eq!((pass.ranges, pass.compacted, pass.bytes_after), (1, 1, 0));
+
+ let held: Vec<usize> = (0..4)
+ .map(|partition| {
+ store
+ .scan_behavioral(partition, None, usize::MAX)
+ .unwrap()
+ .len()
+ })
+ .collect();
+ assert_eq!(held, [KEYS as usize, 0, KEYS as usize, 0]);
+ assert_eq!(
+ behavioral_files()
+ .iter()
+ .map(|file| file.num_entries)
+ .sum::<u64>(),
+ 2 * KEYS as u64,
+ );
+ }
+
#[test]
fn person_record_ttl_drops_ancient_keeps_fresh_and_malformed_and_never_touches_behavioral() {
use crate::stage1::person_record::{PersonRecord, Stamp};