:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(cohorts): retry failed checkpoint files on upload and restore (#123684)
Matheus Batistab14a1953841f
diff --git a/docs/internal/realtime-cohorts/state-store-and-durability.md b/docs/internal/realtime-cohorts/state-store-and-durability.mdindex 3341b4970318..1223a199e25f 100644--- a/docs/internal/realtime-cohorts/state-store-and-durability.md+++ b/docs/internal/realtime-cohorts/state-store-and-durability.md@@ -163,6 +163,8 @@ The restore judges each candidate from its `metadata.json` and `offsets.json` beA candidate that passes is materialized in `<STORE_PATH>.restore`, a sibling of the store on the same mount.A local checkpoint is hard-linked there, SSTs only, with every other file copied.A remote one is downloaded there, within `CHECKPOINT_IMPORT_TIMEOUT_SECS`.+A file that fails to download is downloaded again from its start, up to three tries, while the other files continue.+A file that fails every try fails the candidate.A download that times out fails, and the search stops there, because older candidates are about as large.The stage must open read-only with this build's column families and schema, and RocksDB's open checks that every SST its MANIFEST names is present at the recorded size.The replay window is then checked again, because a download can outlast part of it.@@ -256,6 +258,9 @@ Checkpoints require durable restore, a bucket, and an absolute checkpoint directOnce a chain is older than `CHECKPOINT_FULL_UPLOAD_INTERVAL_SECS`, the next upload is full, which bounds the age of the oldest object a restore needs.- **Paced uploads.**One upload reads at most `CHECKPOINT_UPLOAD_MAX_BYTES_PER_SEC` from the store's disk across all its files, so a full upload does not starve the live path.+- **Failed files.**+ A file that fails to upload is sent again from its start, up to three tries, while the other files continue.+ A file that fails every try fails the upload.- **A final checkpoint that stays local.**A graceful stop cancels an upload in flight, and takes one more checkpoint once every consumer has made its final commit.It does not upload, so a stop usually takes seconds, and the next start's first upload covers what changed since the last upload.diff --git a/rust/cohort-stream-processor/src/store/durability/flaky_store.rs b/rust/cohort-stream-processor/src/store/durability/flaky_store.rsnew file mode 100644index 000000000000..aa67f1bb04fe--- /dev/null+++ b/rust/cohort-stream-processor/src/store/durability/flaky_store.rs@@ -0,0 +1,133 @@+use std::collections::HashMap;+use std::fmt;+use std::sync::atomic::{AtomicBool, Ordering};+use std::sync::Mutex;++use async_trait::async_trait;+use futures::stream::{self, BoxStream, StreamExt};+use object_store::memory::InMemory;+use object_store::path::Path;+use object_store::{+ CopyOptions, GetOptions, GetResult, GetResultPayload, ListResult, MultipartUpload, ObjectMeta,+ ObjectStore, PutMultipartOptions, PutOptions, PutPayload, PutResult, Result,+};++/// An in-memory store whose first put or get of `flaky` fails. The failed get sends half the object+/// first.+#[derive(Debug)]+pub(super) struct FlakyStore {+ inner: InMemory,+ flaky: Path,+ tripped: AtomicBool,+ gets: Mutex<HashMap<Path, usize>>,+}++impl FlakyStore {+ /// Objects already in `inner` skip the failed put.+ pub(super) fn new(inner: InMemory, flaky: &str) -> Self {+ Self {+ inner,+ flaky: Path::from(flaky),+ tripped: AtomicBool::new(false),+ gets: Mutex::default(),+ }+ }++ pub(super) fn tripped(&self) -> bool {+ self.tripped.load(Ordering::SeqCst)+ }++ pub(super) fn gets(&self, key: &str) -> usize {+ self.gets+ .lock()+ .unwrap()+ .get(&Path::from(key))+ .copied()+ .unwrap_or_default()+ }++ fn trips(&self, location: &Path) -> bool {+ *location == self.flaky && !self.tripped.swap(true, Ordering::SeqCst)+ }+}++fn proxy_error() -> object_store::Error {+ object_store::Error::Generic {+ store: "S3",+ source: "error writing a body to connection: unexpected EOF".into(),+ }+}++impl fmt::Display for FlakyStore {+ fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {+ write!(f, "FlakyStore({})", self.flaky)+ }+}++#[async_trait]+impl ObjectStore for FlakyStore {+ async fn put_opts(+ &self,+ location: &Path,+ payload: PutPayload,+ opts: PutOptions,+ ) -> Result<PutResult> {+ if self.trips(location) {+ return Err(proxy_error());+ }+ self.inner.put_opts(location, payload, opts).await+ }++ async fn put_multipart_opts(+ &self,+ location: &Path,+ opts: PutMultipartOptions,+ ) -> Result<Box<dyn MultipartUpload>> {+ self.inner.put_multipart_opts(location, opts).await+ }++ async fn get_opts(&self, location: &Path, options: GetOptions) -> Result<GetResult> {+ *self+ .gets+ .lock()+ .unwrap()+ .entry(location.clone())+ .or_default() += 1;+ let result = self.inner.get_opts(location, options).await?;+ if !self.trips(location) {+ return Ok(result);+ }+ let (meta, range, attributes) = (+ result.meta.clone(),+ result.range.clone(),+ result.attributes.clone(),+ );+ let mut half = result.bytes().await?;+ half.truncate(half.len() / 2);+ Ok(GetResult {+ payload: GetResultPayload::Stream(stream::iter([Ok(half), Err(proxy_error())]).boxed()),+ meta,+ range,+ attributes,+ })+ }++ fn delete_stream(+ &self,+ locations: BoxStream<'static, Result<Path>>,+ ) -> BoxStream<'static, Result<Path>> {+ self.inner.delete_stream(locations)+ }++ fn list(&self, prefix: Option<&Path>) -> BoxStream<'static, Result<ObjectMeta>> {+ self.inner.list(prefix)+ }++ async fn list_with_delimiter(&self, prefix: Option<&Path>) -> Result<ListResult> {+ self.inner.list_with_delimiter(prefix).await+ }++ async fn copy_opts(&self, from: &Path, to: &Path, options: CopyOptions) -> Result<()> {+ self.inner.copy_opts(from, to, options).await+ }+}diff --git a/rust/cohort-stream-processor/src/store/durability/import.rs b/rust/cohort-stream-processor/src/store/durability/import.rsindex bfb60e97b5c4..d03f7d9f472d 100644--- a/rust/cohort-stream-processor/src/store/durability/import.rs+++ b/rust/cohort-stream-processor/src/store/durability/import.rs@@ -5,7 +5,6 @@ use std::path::Path;use std::time::{Duration, Instant};use chrono::{DateTime, Utc};-use tokio_util::sync::CancellationToken;use tracing::info;use super::manifest::{ManifestError, OffsetManifest, MANIFEST_FILENAME};@@ -96,11 +95,7 @@ impl CheckpointImporter {.iter().map(|file| file.remote_filepath.clone()).collect();- // Without a token, the files left after a failed one would all still download.- let siblings = CancellationToken::new();- let download = self- .downloader- .download_files_cancellable(&keys, stage, Some(&siblings));+ let download = self.downloader.download_files(&keys, stage);let result = match tokio::time::timeout(self.import_timeout, download).await {Ok(Ok(())) => Ok(()),Ok(Err(err)) => Err(ImportError::Download(err)),diff --git a/rust/cohort-stream-processor/src/store/durability/mod.rs b/rust/cohort-stream-processor/src/store/durability/mod.rsindex 3194859958f8..43c3be56f3d8 100644--- a/rust/cohort-stream-processor/src/store/durability/mod.rs+++ b/rust/cohort-stream-processor/src/store/durability/mod.rs@@ -14,6 +14,8 @@ pub mod config;pub mod downloader;pub mod error;pub mod export;+#[cfg(test)]+mod flaky_store;pub mod import;pub mod lineage;pub mod manifest;diff --git a/rust/cohort-stream-processor/src/store/durability/s3_client.rs b/rust/cohort-stream-processor/src/store/durability/s3_client.rsindex d19d686743eb..61e6b2993cf9 100644--- a/rust/cohort-stream-processor/src/store/durability/s3_client.rs+++ b/rust/cohort-stream-processor/src/store/durability/s3_client.rs@@ -9,15 +9,57 @@ use object_store::aws::{AmazonS3, AmazonS3Builder};use object_store::limit::LimitStore;use object_store::path::Path as ObjectPath;use object_store::{BackoffConfig, ClientOptions, ObjectStore};+use std::future::Future;use std::sync::Arc;use std::time::Duration;-use tracing::{debug, info};+use tokio_util::sync::CancellationToken;+use tracing::{debug, info, warn};use super::config::DurabilityConfig;/// Short timeout for bucket validation so misconfiguration fails fast.const VALIDATION_TIMEOUT: Duration = Duration::from_secs(10);+/// object_store never retries an error of kind `Unknown`, such as a TLS EOF on a proxied+/// connection, so a whole file is tried again instead.+const FILE_ATTEMPTS: u32 = 3;+const FILE_RETRY_WAIT: Duration = Duration::from_secs(1);++/// Runs `attempt` up to `FILE_ATTEMPTS` times, and stops once `cancel` fires.+pub(super) async fn retry_file<T, Fut>(+ key: &str,+ cancel: &CancellationToken,+ mut attempt: impl FnMut() -> Fut,+) -> Result<T>+where+ Fut: Future<Output = Result<T>>,+{+ let mut result = attempt().await;+ for retry in 1..FILE_ATTEMPTS {+ let Err(e) = &result else { break };+ if cancel.is_cancelled() {+ break;+ }+ let wait = FILE_RETRY_WAIT * retry;+ warn!(+ key,+ retry,+ wait_secs = wait.as_secs(),+ error = %format!("{e:#}"),+ "checkpoint file failed; trying it again",+ );+ if cancel+ .run_until_cancelled(tokio::time::sleep(wait))+ .await+ .is_none()+ {+ break;+ }+ result = attempt().await;+ }+ result+}+/// Build an S3 client wrapped in a `LimitStore` for bounded concurrency, and check that it can list/// `lineage_dir`.///diff --git a/rust/cohort-stream-processor/src/store/durability/s3_downloader.rs b/rust/cohort-stream-processor/src/store/durability/s3_downloader.rsindex bace4467668e..2190c859af8a 100644--- a/rust/cohort-stream-processor/src/store/durability/s3_downloader.rs+++ b/rust/cohort-stream-processor/src/store/durability/s3_downloader.rs@@ -9,7 +9,7 @@ use super::downloader::CheckpointDownloader;use super::error::DownloadCancelledError;use super::lineage::CheckpointLineage;use super::metadata::{CheckpointMetadata, METADATA_FILENAME};-use super::s3_client::create_s3_client;+use super::s3_client::{create_s3_client, retry_file};use crate::observability::metrics::{CHECKPOINT_FILES_DOWNLOADED_TOTAL, CHECKPOINT_FILES_FETCH_DURATION_SECONDS,CHECKPOINT_FILE_FETCH_DURATION_SECONDS, CHECKPOINT_FILE_FETCH_STORE_DURATION_SECONDS,@@ -21,7 +21,6 @@ use async_trait::async_trait;use chrono::{DateTime, Duration, Utc};use futures::stream;use futures::{StreamExt, TryStreamExt};-use object_store::limit::LimitStore;use object_store::path::Path as ObjectPath;use object_store::{ObjectStore, ObjectStoreExt};use tokio::io::AsyncWriteExt;@@ -104,7 +103,7 @@ fn s3_error_kind(e: &object_store::Error) -> &'static str {/// memory.#[derive(Debug)]pub struct S3Downloader {- store: Arc<LimitStore<object_store::aws::AmazonS3>>,+ store: Arc<dyn ObjectStore>,s3_bucket: String,/// The lineage's remote directory: every attempt this downloader lists lives under it.remote_dir: String,@@ -318,11 +317,22 @@ impl CheckpointDownloader for S3Downloader {}).collect();+ // Stops the siblings of a failed file even without a caller token.+ let siblings = cancel_token.cloned().unwrap_or_default();let mut stream = stream::iter(download_tasks)- .map(|(remote_key, local_filepath)| async move {- self.download_and_store_file_cancellable(&remote_key, &local_filepath, cancel_token)+ .map(|(remote_key, local_filepath)| {+ let siblings = &siblings;+ async move {+ retry_file(&remote_key, siblings, || {+ self.download_and_store_file_cancellable(+ &remote_key,+ &local_filepath,+ Some(siblings),+ )+ }).await.with_context(|| format!("Failed to download: {remote_key}"))+ }}).buffer_unordered(self.max_concurrent_file_downloads);@@ -331,9 +341,7 @@ impl CheckpointDownloader for S3Downloader {while let Some(result) = stream.next().await {if let Err(e) = result {first_error = Some(e);- if let Some(token) = cancel_token {- token.cancel();- }+ siblings.cancel();break;}}@@ -388,6 +396,55 @@ impl CheckpointDownloader for S3Downloader {mod tests {use super::*;use chrono::TimeZone;+ use object_store::memory::InMemory;+ use tempfile::TempDir;++ use crate::store::durability::flaky_store::FlakyStore;++ #[tokio::test(start_paused = true)]+ async fn a_file_that_fails_mid_stream_downloads_again_without_its_siblings() {+ let dir = "819f7b67/checkpoints/cohort_stream_state/0";+ let attempt = format!("{dir}/2026-10-25T10-49-00-000Z");+ let objects: [(&str, &[u8]); 3] = [+ ("000007.sst", b"first sst"),+ ("000008.sst", b"second sst"),+ ("MANIFEST-000005", b"manifest"),+ ];+ let inner = InMemory::new();+ let mut keys = Vec::new();+ for (name, body) in objects {+ let key = format!("{attempt}/{name}");+ inner+ .put(&ObjectPath::from(key.as_str()), body.to_vec().into())+ .await+ .unwrap();+ keys.push(key);+ }+ let store = Arc::new(FlakyStore::new(inner, &keys[0]));+ let downloader = S3Downloader {+ store: store.clone(),+ s3_bucket: String::new(),+ remote_dir: dir.to_owned(),+ checkpoint_import_window_hours: 20,+ max_concurrent_file_downloads: 3,+ };+ let stage = TempDir::new().unwrap();++ downloader+ .download_files_cancellable(&keys, stage.path(), Some(&CancellationToken::new()))+ .await+ .unwrap();++ for (name, body) in objects {+ assert_eq!(+ std::fs::read(stage.path().join(name)).unwrap(),+ body,+ "{name}"+ );+ }+ let gets: Vec<usize> = keys.iter().map(|key| store.gets(key)).collect();+ assert_eq!(gets, [2, 1, 1]);+ }#[test]fn the_listing_keeps_attempts_inside_the_window_newest_first() {diff --git a/rust/cohort-stream-processor/src/store/durability/s3_uploader.rs b/rust/cohort-stream-processor/src/store/durability/s3_uploader.rsindex 7bce98eb9317..215c75a9f5a9 100644--- a/rust/cohort-stream-processor/src/store/durability/s3_uploader.rs+++ b/rust/cohort-stream-processor/src/store/durability/s3_uploader.rs@@ -17,7 +17,7 @@ use tracing::{info, warn};use super::config::DurabilityConfig;use super::error::UploadCancelledError;use super::lineage::CheckpointLineage;-use super::s3_client::create_s3_client;+use super::s3_client::{create_s3_client, retry_file};use super::uploader::CheckpointUploader;use crate::observability::metrics::CHECKPOINT_FILES_UPLOADED_TOTAL;@@ -114,8 +114,9 @@ impl S3Uploader {Ok(Self { store, config })}- /// Upload a file with cancellation support.- /// On cancellation or error, calls `abort()` to clean up multipart upload parts.+ /// Upload a file with cancellation support. A failure aborts the multipart upload, except one+ /// inside `shutdown`, which `BufWriter` cannot abort; the bucket's lifecycle rule removes it+ /// after a day.async fn upload_file_cancellable(&self,local_path: &Path,@@ -197,11 +198,11 @@ impl S3Uploader {Ok(())}- async fn upload_bytes(&self, s3_key: &str, data: Vec<u8>) -> Result<()> {+ async fn upload_bytes(&self, s3_key: &str, payload: PutPayload) -> Result<()> {let path = ObjectPath::from(s3_key);self.store- .put(&path, PutPayload::from(data))+ .put(&path, payload).await.with_context(|| format!("Failed to upload to S3 key: {s3_key}"))?;@@ -260,8 +261,10 @@ impl CheckpointUploader for S3Uploader {let token = upload_token.clone();let pace = &pace;async move {- self.upload_file_cancellable(&src, &dest, &token, pace)- .await?;+ retry_file(&dest, &token, || {+ self.upload_file_cancellable(&src, &dest, &token, pace)+ })+ .await?;Ok::<String, anyhow::Error>(dest)}})@@ -298,11 +301,13 @@ impl CheckpointUploader for S3Uploader {}}- let metadata_json = plan.info.metadata.to_json()?;+ let metadata = PutPayload::from(plan.info.metadata.to_json()?);let metadata_key = plan.info.get_metadata_key();- self.upload_bytes(&metadata_key, metadata_json.into_bytes())- .await- .with_context(|| format!("Failed to upload metadata to S3 key: {metadata_key}"))?;+ retry_file(&metadata_key, &upload_token, || {+ self.upload_bytes(&metadata_key, metadata.clone())+ })+ .await+ .with_context(|| format!("Failed to upload metadata to S3 key: {metadata_key}"))?;info!("Uploaded {} files and metadata file to s3://{}/{}",@@ -324,9 +329,62 @@ impl CheckpointUploader for S3Uploader {#[cfg(test)]mod tests {use super::*;+ use chrono::{TimeZone, Utc};+ use object_store::memory::InMemory;use tempfile::TempDir;use tokio::io::AsyncWriteExt;+ use crate::store::durability::flaky_store::FlakyStore;+ use crate::store::durability::{plan_checkpoint, CheckpointMetadata, PodOrdinal};+ use crate::store::DbIdentity;++ #[tokio::test(start_paused = true)]+ async fn a_file_whose_put_fails_once_is_sent_again_and_the_upload_finishes() {+ let attempt = TempDir::new().unwrap();+ std::fs::write(attempt.path().join("000007.sst"), b"sst bytes").unwrap();+ std::fs::write(attempt.path().join("MANIFEST-000005"), b"manifest").unwrap();+ let plan = plan_checkpoint(+ attempt.path(),+ CheckpointMetadata::new(+ PodOrdinal::STANDALONE,+ Utc.with_ymd_and_hms(2026, 10, 25, 10, 49, 0).unwrap(),+ DbIdentity::for_test("db"),+ ),+ "checkpoints".to_string(),+ None,+ None,+ )+ .unwrap();+ let metadata = plan.info.metadata.to_json().unwrap();+ let cases = [+ (+ plan.info.get_file_key("000007.sst"),+ b"sst bytes".as_slice(),+ ),+ (plan.info.get_metadata_key(), metadata.as_bytes()),+ ];++ for (flaky_key, body) in cases {+ let store = Arc::new(FlakyStore::new(InMemory::new(), &flaky_key));+ let uploader = S3Uploader {+ store: store.clone(),+ config: DurabilityConfig::default(),+ };++ uploader.upload_checkpoint_with_plan(&plan).await.unwrap();++ assert!(store.tripped(), "{flaky_key}");+ let uploaded = store+ .get(&ObjectPath::from(flaky_key.as_str()))+ .await+ .unwrap()+ .bytes()+ .await+ .unwrap();+ assert_eq!(uploaded.as_ref(), body, "{flaky_key}");+ }+ }+#[tokio::test]async fn read_chunk_cancellable_returns_data_without_token() {let tmp_dir = TempDir::new().unwrap();