Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(ai-research-v4): write recorder watermark files per write day (#128133)

Robbie54c1506ecea5
diff --git a/products/ai_training/docs/replay-data.md b/products/ai_training/docs/replay-data.md
index b7979e310896..dac9aaa2f865 100644
--- a/products/ai_training/docs/replay-data.md
+++ b/products/ai_training/docs/replay-data.md
@@ -513,6 +513,7 @@ Fetched images and fonts are partitioned by the key month of their reference. A
| `rrweb-v4` | `index/images/dt={day}/` | Parquet, one row per stored inline image |
| `rrweb-v4` | `index/blocks/_watermarks/` | JSON, one object per Kafka partition: how far the block index is complete |
| `rrweb-v4` | `index/images/_watermarks/` | JSON, one object per Kafka partition: how far the image index is complete |
+| `rrweb-v4` | `index/_recorder_watermarks/dt={day}/` | JSON, one object per input Kafka partition: how far the recorder is complete |
| `images-v4` | `images/packs/dt={day}/{team_id}/` | One object per recorder flush and team: sealed images, each found by byte range |
| `images-v4` | `images/url/month={YYYY-MM}/{team_id}/{hash}` | One sealed fetched image per URL reference |
| `images-v4` | `fonts/url/month={YYYY-MM}/{team_id}/{hash}` | One sealed fetched font per font URL reference |
@@ -552,6 +553,7 @@ The order holds only within a file. The blocks of a session spread over many fil
A reader that needs whole sessions selects them by the start time in the session UUIDv7, not by write day:
- Read the write days from the UTC day of `start - 1 day` onward. The recorder admits a session from 1 day before its start until 14 days after it, so no block of a session has an earlier write day.
+- The [recorder watermark files](#recorder-watermark-files) give the last write day to read for every block captured before a cutoff. A recorder that lags writes blocks to write days after their capture day.
- Use a session only while its start is less than 29 days old. Every block lives at least 30 days after its write, so until then no block has expired.
- A session can get more blocks until its start is 14 days old. A reader that needs final sessions uses only sessions with a start between 14 and 29 days old.
@@ -648,7 +650,7 @@ Each write is a plain PUT that replaces the file.
A failed write does not fail the flush. The sink counts it in `ai_research_v4_parquet_sink_watermark_writes_total`, logs it, and writes the file again after the next interval. A write that the sink aborts because it lost the partition counts as `outcome="cancelled"`.
S3 expires a file 30 days after its last write, as it does every object in the bucket.
-To learn how far an index is complete, a reader:
+To learn how far the sink has written an index, a reader:
1. Reads every file under that index's `_watermarks/`.
2. Takes the minimum `complete_up_to_capture_ms` across the partitions.
@@ -659,9 +661,66 @@ Some limits apply:
- A partition that delivers no records keeps its last value, as the gauge does, and the sink keeps rewriting its file. So a quiet partition holds the minimum down.
- When the sink loses a partition, it aborts the partition's watermark writes that are still in flight, retries included. A write that S3 already received can still land. The new owner writes its first file only after its first flush, so the two overlap only if S3 takes longer to apply that write than the new owner takes to flush. The file then holds an older value until the new owner writes again. That value is still true. A full fence is not possible, because the bucket does not let the sink read the file that it replaces.
- A partition that the sink has not flushed since it started to write these files has no file. A reader that needs every partition compares the number of files with the partition count of the topic.
-- The value covers only the rows that reached the sink. The recorder sends a row only after its own flush stores the block or image, so a row with an earlier capture time can still be in the recorder. A sink that has read all of its topic does not make the index complete.
+- The value covers only the rows that reached the sink. The recorder sends a row only after its own flush stores the block or image, so a row with an earlier capture time can still be in the recorder. A sink that has read all of its topic does not make the index complete. The [recorder watermark files](#recorder-watermark-files) cover the recorder.
- A reader that lists `index/blocks/` or `index/images/` for Parquet files must skip `_watermarks/`. Hive-style readers such as Spark, Athena and pyarrow datasets ignore paths that start with `_`, and a `*.parquet` glob does not match the JSON files.
+#### Recorder watermark files
+
+The recorder sets the write day of a flush from its own clock when the flush starts.
+A recorder that lags therefore puts blocks and index rows in a write day after their capture day.
+The sink's watermark files cannot show this, so the recorder writes its own files to the rrweb bucket:
+
+- `index/_recorder_watermarks/dt={day}/topic={topic}/partition={NNNN}.json`
+
+There is one file per write day, input topic and Kafka partition.
+The input topics are the replay topic and its overflow topic.
+`{NNNN}` is the partition number with four digits.
+Each file holds one object:
+
+| Field | Meaning |
+| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `topic` | The input topic. |
+| `partition` | The input partition. |
+| `complete_up_to_capture_ms` | Every record of the partition with an earlier capture time has its blocks and images stored in this write day or an earlier one. Each index row of the record was delivered to `ai_research_v4_index` or failed. |
+| `index_end_offsets` | The high watermark of each partition of `ai_research_v4_index`, by partition number. The recorder reads them after it takes the value, so each delivered row that the value covers is below them. |
+| `input_partitions` | The partition count of each input topic, by topic name. Any one file tells a reader which files a day must have. |
+| `written_at_ms` | When the recorder wrote the file. |
+
+The capture time of a record is its Kafka timestamp, the same time as the `capture-timestamp-ms` header of its index rows.
+The recorder sets the value this way:
+
+- A record that the recorder has read but not finished holds the value at or below its capture time. A record is finished when its flush has stored the blocks and the producer has delivered or failed every index row of the flush.
+- A record that the recorder has not read yet can have an earlier capture time than one that it has read, by up to the delivery timeout of the record's producer. So the value is at least `CAPTURE_ORDER_SLACK_MS` (`watermark.rs`) before the capture time of the last record read, unless the partition is read to its end.
+- A partition that the recorder has read to its end moves with the clock, also when no records arrive. Its value can rise to `CAPTURE_ORDER_SLACK_MS` before the time that the recorder read the end offset, so a quiet overflow partition does not stop a reader. A partition that holds no record counts as read to its end, also when it has no committed offset.
+- After an assignment, the recorder writes no file for the partition until the index delivery timeout plus one minute has passed. By then, each index row that the previous owner sent is delivered or failed.
+
+The recorder writes the files every `WATERMARK_FILE_INTERVAL` (`watermark.rs`), from the consume loop, for each partition that it owns.
+Each write goes to the UTC day of the write and is a plain PUT that replaces the file of that day.
+So the files of a day change until the day ends, and the last write of the day stays.
+A file of a running recorder is at most the interval plus one flush old. A file of the current day that is more than about 15 minutes old shows a partition that no consume loop moves forward.
+A failed write does not stop the recorder. It counts the write in `ai_research_v4_recorder_watermark_files_total` and writes the file again in the next round.
+A round that cannot read the Kafka offsets writes no files and counts `outcome="kafka_failed"` in `ai_research_v4_recorder_watermark_rounds_total`.
+S3 expires a file 30 days after its last write, as it does every object in the bucket.
+
+To find the write days that hold every record captured before a cutoff `C`, a reader:
+
+1. Starts at the UTC day of `C` and uses only days that have ended, because the files of the current day still change.
+2. Finds the first day `X` with a file for every partition of every topic in `input_partitions`, each with `complete_up_to_capture_ms` at or after `C`. Each block and index row of a record captured before `C` then has a write day of `X` or earlier. A day with a missing file proves nothing, but a later day can.
+3. Takes the highest value of `index_end_offsets` for each index partition, over the files of day `X`.
+4. Waits until, for each index partition, one of the sink's [watermark files](#watermark-files) of that partition has a `committed_offset` at or past that value. The block and image files of a partition carry its one committed offset, so either file shows it, and a reader of one index does not wait for the other's file to refresh. An index partition whose value is 0 has never received a row, so it needs no sink file. Every index row of those records is then in an index file in S3.
+5. Reads the write days up to and including `X`.
+
+For example, take a cutoff of 2026-10-27 00:00 and a recorder that is 60 hours behind.
+The recorder reaches the cutoff at about 2026-10-29 12:00, so 2026-10-29 is the first day whose files reach the cutoff.
+A reader of the sessions that start from 2026-10-12 then reads the write days from 2026-10-11 to 2026-10-29.
+
+Some limits apply:
+
+- An index row that the producer failed to deliver stays out of the index, as above. The value counts its record as finished.
+- A write of an earlier owner can land after a newer file of the same day. Its value is older, but it is still true.
+- The value assumes that a record reaches its partition no later than `CAPTURE_ORDER_SLACK_MS` after its capture time. A record that arrives later can have a capture time before a value that the recorder already wrote.
+- A reader that lists `index/` must skip `_recorder_watermarks/`.
+
### Image index
`index/images/` has one row per inline image in a pack: `team_id`, `key_month`, `image_ref`, `pack_s3_key`, `byte_start`, `byte_end`, `mime` and `byte_size`.
diff --git a/rust/ai-research/replay-recorder-v4/src/consumer.rs b/rust/ai-research/replay-recorder-v4/src/consumer.rs
index 3578cedd4ceb..155ff6921ccf 100644
--- a/rust/ai-research/replay-recorder-v4/src/consumer.rs
+++ b/rust/ai-research/replay-recorder-v4/src/consumer.rs
@@ -23,8 +23,13 @@ use crate::flush::{write_flush, FlushWork, Writer, PRODUCER_FLUSH_ON_SHUTDOWN};
use crate::limits::{Restrictions, SessionLimiter};
use crate::process::{admit, sampled, Admitted, Headers, Skip};
use crate::teams::TeamCache;
+use crate::watermark::{
+ assignment_grace, watermark_files, FlushDone, KafkaEnds, Progress, WatermarkFiles,
+ WATERMARK_FILE_INTERVAL,
+};
const KAFKA_TIMEOUT: Duration = Duration::from_secs(10);
+const WATERMARK_READS_TIMEOUT: Duration = Duration::from_secs(30);
const HEARTBEAT: Duration = Duration::from_secs(5);
/// How long offset commits can keep failing before the process exits. A rebalance can wait for one member for the whole
@@ -43,6 +48,7 @@ const POLL_BATCH: usize = 1000;
#[derive(Default)]
struct Assignments {
epochs: HashMap<Tp, (u64, bool)>,
+ assigned_at: HashMap<Tp, Instant>,
revoked: Vec<Tp>,
/// Partitions without a known committed offset, with the first offset read once there is one. It stays here until
/// it commits or the partition is revoked.
@@ -70,6 +76,20 @@ impl Assignments {
.collect()
}
+ fn owned_since(&self, now: Instant, grace: Duration) -> Vec<(Tp, u64)> {
+ self.epochs
+ .iter()
+ .filter(|(tp, (_, owned))| {
+ *owned
+ && self
+ .assigned_at
+ .get(tp)
+ .is_some_and(|at| now.saturating_duration_since(*at) >= grace)
+ })
+ .map(|(tp, (epoch, _))| (*tp, *epoch))
+ .collect()
+ }
+
fn starts_committed(&mut self, starts: &[(Tp, i64)], committed: bool, now: Instant) {
if committed {
for (tp, offset) in starts {
@@ -208,11 +228,13 @@ impl RebalanceContext {
if let Some(tp) = self.tp(element.topic(), element.partition()) {
if assigned {
state.epochs.entry(tp).or_default().1 = true;
+ state.assigned_at.insert(tp, Instant::now());
} else {
let entry = state.epochs.entry(tp).or_default();
*entry = (entry.0 + 1, false);
state.revoked.push(tp);
state.uncommitted.remove(&tp);
+ state.assigned_at.remove(&tp);
}
}
}
@@ -258,6 +280,7 @@ impl ConsumerContext for RebalanceContext {
}
}
+#[derive(Clone)]
pub struct RecorderConsumer {
consumer: Arc<StreamConsumer<RebalanceContext>>,
state: Arc<Mutex<Assignments>>,
@@ -406,6 +429,70 @@ impl RecorderConsumer {
}
}
+ fn read_committed(&self, owned: &[Tp]) -> Result<HashMap<Tp, i64>, KafkaError> {
+ let mut list = TopicPartitionList::new();
+ for tp in owned {
+ list.add_partition(tp.topic, tp.partition);
+ }
+ Ok(self
+ .consumer
+ .committed_offsets(list, KAFKA_TIMEOUT)?
+ .elements()
+ .iter()
+ .filter_map(|e| match e.offset() {
+ Offset::Offset(offset) => Some((
+ self.consumer.context().tp(e.topic(), e.partition())?,
+ offset,
+ )),
+ _ => None,
+ })
+ .collect())
+ }
+
+ fn read_ends(&self, owned: &[Tp], index_topic: &str) -> Result<KafkaEnds, KafkaError> {
+ let deadline = Instant::now() + WATERMARK_READS_TIMEOUT;
+ let remaining = || deadline.saturating_duration_since(Instant::now());
+ let reads_started_at_ms = SystemTime::now()
+ .duration_since(UNIX_EPOCH)
+ .map_or(0, |d| d.as_millis() as i64);
+ let mut input_offsets = HashMap::new();
+ for tp in owned {
+ let (low, high) =
+ self.consumer
+ .fetch_watermarks(tp.topic, tp.partition, remaining())?;
+ input_offsets.insert(*tp, low..high);
+ }
+ let mut input_partitions = BTreeMap::new();
+ for &topic in &self.consumer.context().topics {
+ input_partitions.insert(topic, self.partition_count(topic, remaining())?);
+ }
+ let mut index_end_offsets = Vec::new();
+ for partition in 0..self.partition_count(index_topic, remaining())? {
+ let (_, end) =
+ self.consumer
+ .fetch_watermarks(index_topic, partition as i32, remaining())?;
+ index_end_offsets.push(end);
+ }
+ Ok(KafkaEnds {
+ reads_started_at_ms,
+ input_offsets,
+ input_partitions,
+ index_end_offsets,
+ })
+ }
+
+ fn partition_count(&self, topic: &str, timeout: Duration) -> Result<usize, KafkaError> {
+ let metadata = self.consumer.fetch_metadata(Some(topic), timeout)?;
+ match metadata.topics().first() {
+ Some(found) if found.error().is_none() && !found.partitions().is_empty() => {
+ Ok(found.partitions().len())
+ }
+ _ => Err(KafkaError::MetadataFetch(
+ RDKafkaErrorCode::UnknownTopicOrPartition,
+ )),
+ }
+ }
+
async fn retry_commit(&self) -> Result<(), String> {
self.commit(&HashMap::new()).await
}
@@ -490,7 +577,24 @@ pub async fn run(
writer: Option<Writer>,
handle: Handle,
) {
- consume(&consumer, &settings, &sources, writer.as_ref(), &handle).await;
+ let watermarks = writer.as_ref().map(|writer| {
+ Arc::new(WatermarkFiles {
+ progress: Mutex::new(Progress::default()),
+ store: writer.store.clone(),
+ bucket: writer.rrweb_bucket.clone(),
+ index_topic: writer.index_topic.clone(),
+ grace: assignment_grace(writer.index_delivery_timeout),
+ })
+ });
+ consume(
+ &consumer,
+ &settings,
+ &sources,
+ writer.as_ref(),
+ watermarks.as_ref(),
+ &handle,
+ )
+ .await;
// The flushes do not wait for their index rows, so the producer can still hold some.
if let Some(writer) = &writer {
writer.flush_producers(PRODUCER_FLUSH_ON_SHUTDOWN).await;
@@ -502,6 +606,7 @@ async fn consume(
settings: &Settings,
sources: &Sources,
writer: Option<&Writer>,
+ watermarks: Option<&Arc<WatermarkFiles>>,
handle: &Handle,
) {
let shutdown = handle.shutdown_signal();
@@ -509,6 +614,8 @@ async fn consume(
let mut batch = Batch::default();
let mut inspectors = Inspectors::new(settings.session_state_max_bytes);
let mut tick = tokio::time::interval(Duration::from_secs(1));
+ let mut next_watermark_round = Instant::now() + WATERMARK_FILE_INTERVAL;
+ let mut watermark_round: Option<tokio::task::JoinHandle<()>> = None;
loop {
tokio::select! {
_ = &mut shutdown => break,
@@ -527,7 +634,8 @@ async fn consume(
},
|| consumer.consumer.recv().now_or_never(),
);
- handle_messages(&messages, consumer, settings, sources, &mut batch).await;
+ handle_messages(&messages, consumer, settings, sources, watermarks, &mut batch)
+ .await;
drop(messages);
if let Some(error) = failed {
if receive_failed(consumer, handle, error) {
@@ -547,6 +655,27 @@ async fn consume(
if !revoked.is_empty() {
inspectors.drop_partitions(&revoked);
batch.retain(|tp, epoch| consumer.current_epoch(tp) == Some(epoch));
+ if let Some(watermarks) = watermarks {
+ watermarks
+ .progress
+ .lock()
+ .unwrap()
+ .retain(|tp, epoch| consumer.current_epoch(tp) == Some(epoch));
+ }
+ }
+ // Rounds start here, so a consume loop that stops leaves its files to go stale.
+ if let Some(watermarks) = watermarks {
+ if Instant::now() >= next_watermark_round
+ && watermark_round
+ .as_ref()
+ .is_none_or(|round| round.is_finished())
+ {
+ next_watermark_round = Instant::now() + WATERMARK_FILE_INTERVAL;
+ watermark_round = Some(tokio::spawn(write_watermarks(
+ consumer.clone(),
+ watermarks.clone(),
+ )));
+ }
}
if batch.should_flush(settings.flush_interval, settings.flush_max_bytes) {
let work = flush(
@@ -555,6 +684,7 @@ async fn consume(
consumer,
settings,
writer,
+ watermarks.map(|w| &**w),
handle,
settings.flush_retries,
);
@@ -598,6 +728,7 @@ async fn consume(
consumer,
settings,
writer,
+ watermarks.map(|w| &**w),
handle,
0,
),
@@ -634,6 +765,7 @@ async fn handle_messages(
consumer: &RecorderConsumer,
settings: &Settings,
sources: &Sources,
+ watermarks: Option<&Arc<WatermarkFiles>>,
batch: &mut Batch,
) {
let mut admitted = Vec::with_capacity(messages.len());
@@ -669,6 +801,7 @@ async fn handle_messages(
} else {
sources.limiter.blocked(&sessions).await
};
+ let mut progress = watermarks.map(|w| w.progress.lock().unwrap());
for (message, tp, epoch, session) in admitted {
let session = session.and_then(|session| {
if blocked.contains(&session) {
@@ -677,6 +810,15 @@ async fn handle_messages(
Ok(session)
}
});
+ if let Some(progress) = progress.as_mut() {
+ progress.handled(
+ tp,
+ epoch,
+ message.offset(),
+ message.timestamp().to_millis(),
+ session.is_ok(),
+ );
+ }
match session {
Ok(session) => {
counter!("ai_research_v4_recorder_messages_total", "outcome" => "admitted")
@@ -743,11 +885,12 @@ async fn flush(
consumer: &RecorderConsumer,
settings: &Settings,
writer: Option<&Writer>,
+ watermarks: Option<&WatermarkFiles>,
handle: &Handle,
retries: u32,
) -> Result<(), String> {
let work = flush_steps(
- batch, inspectors, consumer, settings, writer, handle, retries,
+ batch, inspectors, consumer, settings, writer, watermarks, handle, retries,
);
tokio::time::timeout(FLUSH_DEADLINE, work)
.await
@@ -761,10 +904,14 @@ async fn flush_steps(
consumer: &RecorderConsumer,
settings: &Settings,
writer: Option<&Writer>,
+ watermarks: Option<&WatermarkFiles>,
handle: &Handle,
retries: u32,
) -> Result<(), String> {
let flush_started = Instant::now();
+ let done = watermarks.map_or_else(FlushDone::default, |w| {
+ w.progress.lock().unwrap().flush_started()
+ });
let offsets: HashMap<Tp, (u64, i64)> = batch
.spans
.iter()
@@ -812,7 +959,7 @@ async fn flush_steps(
timeout,
with_heartbeat(
handle,
- write_flush(&blocks, writer, settings.flush_concurrency, &work),
+ write_flush(&blocks, writer, settings.flush_concurrency, &work, &done),
),
)
.await;
@@ -844,6 +991,66 @@ async fn flush_steps(
Ok(())
}
+/// The committed offsets are read before the progress: a partition that has handled nothing under its assignment by then has committed nothing under it, so its committed offset is past no undelivered index row. The end offsets are read after the progress, so that every index row it counts as delivered is below the index end offsets.
+async fn write_watermarks(consumer: RecorderConsumer, watermarks: Arc<WatermarkFiles>) {
+ let owned = |consumer: &RecorderConsumer| -> Vec<(Tp, u64)> {
+ consumer
+ .state
+ .lock()
+ .unwrap()
+ .owned_since(Instant::now(), watermarks.grace)
+ };
+ let partitions = |owned: &[(Tp, u64)]| -> Vec<Tp> { owned.iter().map(|(tp, _)| *tp).collect() };
+ let candidates = partitions(&owned(&consumer));
+ if candidates.is_empty() {
+ return;
+ }
+ let reader = consumer.clone();
+ let Some(committed) = read_kafka(move || reader.read_committed(&candidates)).await else {
+ return;
+ };
+ // Read again, so that a partition revoked since the committed offsets were read is left out.
+ let owned = owned(&consumer);
+ let progress: Vec<_> = {
+ let progress = watermarks.progress.lock().unwrap();
+ owned
+ .iter()
+ .map(|&(tp, epoch)| (tp, progress.of(tp, epoch)))
+ .collect()
+ };
+ let read = partitions(&owned);
+ let index_topic = watermarks.index_topic.clone();
+ let Some(ends) = read_kafka(move || consumer.read_ends(&read, &index_topic)).await else {
+ return;
+ };
+ match watermark_files(&progress, &committed, &ends, chrono::Utc::now()) {
+ Ok(files) => {
+ counter!("ai_research_v4_recorder_watermark_rounds_total", "outcome" => "written")
+ .increment(1);
+ watermarks.put(files).await;
+ }
+ Err(error) => tracing::error!(%error, "encoding the watermark files failed"),
+ }
+}
+
+async fn read_kafka<T: Send + 'static>(
+ read: impl FnOnce() -> Result<T, KafkaError> + Send + 'static,
+) -> Option<T> {
+ let read = tokio::task::spawn_blocking(read)
+ .await
+ .map_err(|error| error.to_string())
+ .and_then(|read| read.map_err(|error| error.to_string()));
+ match read {
+ Ok(value) => Some(value),
+ Err(error) => {
+ tracing::warn!(%error, "reading the offsets for the watermark files failed");
+ counter!("ai_research_v4_recorder_watermark_rounds_total", "outcome" => "kafka_failed")
+ .increment(1);
+ None
+ }
+ }
+}
+
/// The blocks this process may still write: none once the group took its assignment away, and otherwise those of the
/// partitions it still owns under the assignment the flush read them in.
fn still_owned(
@@ -929,6 +1136,21 @@ mod tests {
assert!(state.uncommitted.is_empty());
}
+ #[test]
+ fn a_partition_gets_watermark_files_only_once_its_assignment_is_older_than_the_grace() {
+ let mut state = Assignments::default();
+ let assigned = Instant::now();
+ let grace = Duration::from_secs(360);
+ state.epochs.insert(tp(0), (1, true));
+ state.assigned_at.insert(tp(0), assigned);
+ state.epochs.insert(tp(1), (1, true));
+ state
+ .assigned_at
+ .insert(tp(1), assigned + Duration::from_secs(1));
+ state.epochs.insert(tp(2), (2, false));
+ assert_eq!(state.owned_since(assigned + grace, grace), [(tp(0), 1)]);
+ }
+
#[test]
fn a_failed_commit_waits_for_the_next_flush_while_the_partition_stays_owned() {
let mut state = Assignments::default();
diff --git a/rust/ai-research/replay-recorder-v4/src/flush.rs b/rust/ai-research/replay-recorder-v4/src/flush.rs
index c7614a2313ae..3777ef271205 100644
--- a/rust/ai-research/replay-recorder-v4/src/flush.rs
+++ b/rust/ai-research/replay-recorder-v4/src/flush.rs
@@ -27,6 +27,7 @@ use crate::history::{mark_fresh, FetchHistory};
use crate::index::{Index, IndexMessage};
use crate::store::BlobStore;
use crate::urls::{fetch_jobs, records, FetchJobs, FlushUrls};
+use crate::watermark::FlushDone;
use crate::write::{
block_index_rows, compress_block, new_images, object_key, seal_blocking, seal_compressed,
session_keyed_frames, write_blocks, write_images, CompressedBlock, FlushAttempt, WriteError,
@@ -54,6 +55,7 @@ pub struct Writer {
/// The flush does not wait for index rows, so they get their own producer with a delivery timeout of minutes.
pub index_producer: FutureProducer<KafkaContext>,
pub index_topic: String,
+ pub index_delivery_timeout: Duration,
/// Remote image references this process sent to the fetcher, or found fresh in its history, by UTC day. A
/// reference goes out again each day unless its stored image needs no refresh, so the fetcher refreshes it within
/// every refresh interval and keeps the image as long as the blocks that reference it.
@@ -156,6 +158,7 @@ pub async fn write_flush(
writer: &Writer,
concurrency: usize,
work: &Arc<FlushWork>,
+ flush_done: &FlushDone,
) -> Result<(), FlushError> {
let now = Utc::now();
let mut resolved = work.keys.lock().await;
@@ -308,7 +311,13 @@ pub async fn write_flush(
index.push(IndexMessage::block(&row)?);
}
produce_jobs(writer, jobs, &day, now.timestamp_millis()).await;
- send_index(&writer.index_producer, &writer.index_topic, &day, index);
+ send_index(
+ &writer.index_producer,
+ &writer.index_topic,
+ &day,
+ index,
+ flush_done.clone(),
+ );
Ok(())
}
@@ -381,14 +390,14 @@ async fn produce_jobs(writer: &Writer, jobs: FetchJobs, day: &str, now_ms: i64)
metrics::gauge!("ai_research_v4_recorder_sent_urls").set(writer.sent_urls.entry_count() as f64);
}
-/// Hands the index rows to the producer and returns without waiting for their delivery. A task counts the outcomes.
-/// `write_day` is the `dt=` partition of the blocks and packs the rows point at, so the sink files them with those.
+/// Hands the index rows to the producer and returns without waiting for their delivery. The returned task counts the outcomes, then finishes `done`. `write_day` is the `dt=` partition of the blocks and packs the rows point at, so the sink files them with those.
pub fn send_index(
producer: &FutureProducer<KafkaContext>,
topic: &str,
write_day: &str,
messages: Vec<IndexMessage>,
-) {
+ done: FlushDone,
+) -> tokio::task::JoinHandle<()> {
let mut deliveries = Vec::with_capacity(messages.len());
let mut failed = Failures::default();
for message in &messages {
@@ -426,7 +435,8 @@ pub fn send_index(
}
}
failed.report();
- });
+ done.finish();
+ })
}
/// The index rows of one flush that did not reach the topic. They are logged once per flush, not once per row.
@@ -483,7 +493,8 @@ mod tests {
async fn sends_one_record_per_row_with_its_index_and_write_day() {
let (cluster, producer) = create_mock_kafka().await;
cluster.create_topic(TOPIC, 1, 1).unwrap();
- send_index(
+ let done = FlushDone::default();
+ let delivered = send_index(
&producer,
TOPIC,
"2026-10-15",
@@ -491,10 +502,15 @@ mod tests {
message(Index::Blocks, "session-1", Some(1_000)),
message(Index::Images, "image:v4:7:2026-10:abc", None),
],
+ done.clone(),
);
+ // The test runtime has one thread, so the delivery task has not run yet.
+ assert!(!done.is_finished());
producer
.flush(Timeout::After(Duration::from_secs(5)))
.unwrap();
+ delivered.await.unwrap();
+ assert!(done.is_finished());
let consumer: BaseConsumer = ClientConfig::new()
.set("bootstrap.servers", cluster.bootstrap_servers())
diff --git a/rust/ai-research/replay-recorder-v4/src/lib.rs b/rust/ai-research/replay-recorder-v4/src/lib.rs
index 13f32d19bf56..ac5b7d58c7cc 100644
--- a/rust/ai-research/replay-recorder-v4/src/lib.rs
+++ b/rust/ai-research/replay-recorder-v4/src/lib.rs
@@ -14,4 +14,5 @@ pub mod scrub;
pub mod store;
pub mod teams;
pub mod urls;
+pub mod watermark;
pub mod write;
diff --git a/rust/ai-research/replay-recorder-v4/src/main.rs b/rust/ai-research/replay-recorder-v4/src/main.rs
index 8f3c337a8576..8accc1d533f7 100644
--- a/rust/ai-research/replay-recorder-v4/src/main.rs
+++ b/rust/ai-research/replay-recorder-v4/src/main.rs
@@ -246,6 +246,7 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
)
.await?,
index_topic: config.index_topic.clone(),
+ index_delivery_timeout: Duration::from_millis(config.index_delivery_timeout_ms.into()),
sent_urls: replay_recorder_v4::urls::sent_urls(500_000),
fetch_history: (!config.image_fetch_history_table.is_empty()).then(
|| -> Arc<dyn FetchHistory> {
diff --git a/rust/ai-research/replay-recorder-v4/src/store.rs b/rust/ai-research/replay-recorder-v4/src/store.rs
index 7c01635373b8..fc902c6303e1 100644
--- a/rust/ai-research/replay-recorder-v4/src/store.rs
+++ b/rust/ai-research/replay-recorder-v4/src/store.rs
@@ -11,6 +11,8 @@ pub trait BlobStore: Send + Sync {
/// Writes an object that must not exist yet. The recorder names every object uniquely, so an existing one means
/// the SDK retried this same write, and the first write stands.
async fn put_new(&self, bucket: &str, key: &str, body: Vec<u8>) -> Result<(), StoreError>;
+
+ async fn put(&self, bucket: &str, key: &str, body: Vec<u8>) -> Result<(), StoreError>;
}
pub struct S3Store {
@@ -48,4 +50,16 @@ impl BlobStore for S3Store {
Err(error) => Err(StoreError(error.to_string())),
}
}
+
+ async fn put(&self, bucket: &str, key: &str, body: Vec<u8>) -> Result<(), StoreError> {
+ self.client
+ .put_object()
+ .bucket(bucket)
+ .key(key)
+ .body(body.into())
+ .send()
+ .await
+ .map(|_| ())
+ .map_err(|error| StoreError(error.to_string()))
+ }
}
diff --git a/rust/ai-research/replay-recorder-v4/src/watermark.rs b/rust/ai-research/replay-recorder-v4/src/watermark.rs
new file mode 100644
index 000000000000..1242566b6240
--- /dev/null
+++ b/rust/ai-research/replay-recorder-v4/src/watermark.rs
@@ -0,0 +1,406 @@
+//! The recorder's watermark files. The format is in `products/ai_training/docs/replay-data.md`, section "Recorder watermark files".
+
+use std::collections::{BTreeMap, HashMap};
+use std::ops::Range;
+use std::sync::atomic::{AtomicBool, Ordering};
+use std::sync::{Arc, Mutex};
+use std::time::Duration;
+
+use chrono::{DateTime, Utc};
+use metrics::counter;
+use replay_core_v4::keys::rows::write_day;
+use serde::Serialize;
+
+use crate::batch::Tp;
+use crate::store::BlobStore;
+
+pub const WATERMARK_FILE_INTERVAL: Duration = Duration::from_secs(180);
+
+/// A later offset can hold a record captured earlier, by up to its producer's delivery timeout. This is well over the delivery timeouts of capture and of the replay lane that moves records to the overflow topic.
+pub const CAPTURE_ORDER_SLACK_MS: i64 = 5 * 60 * 1000;
+
+const PUT_TIMEOUT: Duration = Duration::from_secs(10);
+
+/// The previous owner's last index rows are delivered or failed within the index delivery timeout of the assignment. The extra minute covers clock differences between processes, which pick write days by their own clocks.
+pub fn assignment_grace(index_delivery_timeout: Duration) -> Duration {
+ index_delivery_timeout + Duration::from_secs(60)
+}
+
+#[derive(Default)]
+pub struct Progress {
+ partitions: HashMap<Tp, Partition>,
+ flushes: Vec<PendingFlush>,
+}
+
+struct Partition {
+ epoch: u64,
+ next_offset: i64,
+ last_captured_at_ms: Option<i64>,
+ buffered_earliest_ms: Option<i64>,
+}
+
+impl Partition {
+ fn new(epoch: u64, first_offset: i64) -> Self {
+ Self {
+ epoch,
+ next_offset: first_offset,
+ last_captured_at_ms: None,
+ buffered_earliest_ms: None,
+ }
+ }
+}
+
+struct PendingFlush {
+ earliest_ms: HashMap<Tp, (u64, i64)>,
+ done: FlushDone,
+}
+
+/// Set once a flush has stored its blocks and each of its index rows was delivered or failed.
+#[derive(Clone, Default)]
+pub struct FlushDone(Arc<AtomicBool>);
+
+impl FlushDone {
+ pub fn finish(&self) {
+ self.0.store(true, Ordering::Release);
+ }
+
+ pub fn is_finished(&self) -> bool {
+ self.0.load(Ordering::Acquire)
+ }
+}
+
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct PartitionProgress {
+ pub next_offset: i64,
+ pub last_captured_at_ms: Option<i64>,
+ pub earliest_unfinished_ms: Option<i64>,
+}
+
+impl Progress {
+ pub fn handled(
+ &mut self,
+ tp: Tp,
+ epoch: u64,
+ offset: i64,
+ captured_at_ms: Option<i64>,
+ admitted: bool,
+ ) {
+ let partition = self
+ .partitions
+ .entry(tp)
+ .or_insert_with(|| Partition::new(epoch, offset));
+ if partition.epoch != epoch {
+ *partition = Partition::new(epoch, offset);
+ }
+ partition.next_offset = partition.next_offset.max(offset + 1);
+ let Some(captured_at_ms) = captured_at_ms.filter(|ms| *ms > 0) else {
+ return;
+ };
+ partition.last_captured_at_ms = Some(captured_at_ms);
+ if admitted {
+ partition.buffered_earliest_ms = Some(
+ partition
+ .buffered_earliest_ms
+ .map_or(captured_at_ms, |earliest| earliest.min(captured_at_ms)),
+ );
+ }
+ }
+
+ pub fn flush_started(&mut self) -> FlushDone {
+ self.flushes.retain(|flush| !flush.done.is_finished());
+ let done = FlushDone::default();
+ let earliest_ms: HashMap<Tp, (u64, i64)> = self
+ .partitions
+ .iter_mut()
+ .filter_map(|(tp, p)| Some((*tp, (p.epoch, p.buffered_earliest_ms.take()?))))
+ .collect();
+ if !earliest_ms.is_empty() {
+ self.flushes.push(PendingFlush {
+ earliest_ms,
+ done: done.clone(),
+ });
+ }
+ done
+ }
+
+ /// A flush that lost its partitions never finishes, so this is also what removes it.
+ pub fn retain(&mut self, keep: impl Fn(Tp, u64) -> bool) {
+ self.partitions.retain(|tp, p| keep(*tp, p.epoch));
+ for flush in &mut self.flushes {
+ flush.earliest_ms.retain(|tp, (epoch, _)| keep(*tp, *epoch));
+ }
+ self.flushes
+ .retain(|flush| !flush.earliest_ms.is_empty() && !flush.done.is_finished());
+ }
+
+ pub fn of(&self, tp: Tp, epoch: u64) -> Option<PartitionProgress> {
+ let partition = self.partitions.get(&tp).filter(|p| p.epoch == epoch)?;
+ let unfinished = self
+ .flushes
+ .iter()
+ .filter(|flush| !flush.done.is_finished())
+ .filter_map(|flush| match flush.earliest_ms.get(&tp) {
+ Some(&(flush_epoch, earliest)) if flush_epoch == epoch => Some(earliest),
+ _ => None,
+ });
+ Some(PartitionProgress {
+ next_offset: partition.next_offset,
+ last_captured_at_ms: partition.last_captured_at_ms,
+ earliest_unfinished_ms: unfinished.chain(partition.buffered_earliest_ms).min(),
+ })
+ }
+}
+
+/// `offsets` runs from the partition's low watermark to its high watermark, read at `offsets_read_at_ms`.
+pub fn complete_up_to_capture_ms(
+ progress: Option<PartitionProgress>,
+ committed_offset: Option<i64>,
+ offsets: &Range<i64>,
+ offsets_read_at_ms: i64,
+) -> Option<i64> {
+ // A partition that holds no record has nothing to read, wherever the consumer starts.
+ let next_offset = progress
+ .map(|p| p.next_offset)
+ .or(committed_offset)
+ .or(offsets.is_empty().then_some(offsets.end))?;
+ // Every record appended before the read is handled, and one appended after it was captured at most the slack before the read.
+ let read_to_end_at_ms = (next_offset >= offsets.end).then_some(offsets_read_at_ms);
+ let newest_ms = progress
+ .and_then(|p| p.last_captured_at_ms)
+ .max(read_to_end_at_ms)?;
+ let frontier_ms = newest_ms - CAPTURE_ORDER_SLACK_MS;
+ Some(
+ progress
+ .and_then(|p| p.earliest_unfinished_ms)
+ .map_or(frontier_ms, |earliest| earliest.min(frontier_ms)),
+ )
+}
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct KafkaEnds {
+ pub reads_started_at_ms: i64,
+ pub input_offsets: HashMap<Tp, Range<i64>>,
+ pub input_partitions: BTreeMap<&'static str, usize>,
+ pub index_end_offsets: Vec<i64>,
+}
+
+#[derive(Serialize)]
+struct WatermarkFile<'a> {
+ topic: &'a str,
+ partition: i32,
+ complete_up_to_capture_ms: i64,
+ index_end_offsets: &'a [i64],
+ input_partitions: &'a BTreeMap<&'static str, usize>,
+ written_at_ms: i64,
+}
+
+pub fn watermark_key(write_day: &str, tp: Tp) -> String {
+ format!(
+ "index/_recorder_watermarks/dt={write_day}/topic={}/partition={:04}.json",
+ tp.topic, tp.partition
+ )
+}
+
+/// `written_at` must be read after the Kafka reads, so that the file's day is no earlier than the write day of any flush that the progress counts.
+pub fn watermark_files(
+ owned: &[(Tp, Option<PartitionProgress>)],
+ committed_offsets: &HashMap<Tp, i64>,
+ ends: &KafkaEnds,
+ written_at: DateTime<Utc>,
+) -> Result<Vec<(String, Vec<u8>)>, serde_json::Error> {
+ let day = write_day(written_at);
+ let mut files = Vec::with_capacity(owned.len());
+ for &(tp, progress) in owned {
+ let Some(offsets) = ends.input_offsets.get(&tp) else {
+ continue;
+ };
+ let Some(complete_up_to_capture_ms) = complete_up_to_capture_ms(
+ progress,
+ committed_offsets.get(&tp).copied(),
+ offsets,
+ ends.reads_started_at_ms,
+ ) else {
+ continue;
+ };
+ let body = serde_json::to_vec(&WatermarkFile {
+ topic: tp.topic,
+ partition: tp.partition,
+ complete_up_to_capture_ms,
+ index_end_offsets: &ends.index_end_offsets,
+ input_partitions: &ends.input_partitions,
+ written_at_ms: written_at.timestamp_millis(),
+ })?;
+ files.push((watermark_key(&day, tp), body));
+ }
+ Ok(files)
+}
+
+pub struct WatermarkFiles {
+ pub progress: Mutex<Progress>,
+ pub store: Arc<dyn BlobStore>,
+ pub bucket: String,
+ pub index_topic: String,
+ pub grace: Duration,
+}
+
+impl WatermarkFiles {
+ pub async fn put(&self, files: Vec<(String, Vec<u8>)>) {
+ let writes = files.into_iter().map(|(key, body)| async move {
+ match tokio::time::timeout(PUT_TIMEOUT, self.store.put(&self.bucket, &key, body)).await
+ {
+ Ok(Ok(())) => Ok(()),
+ Ok(Err(error)) => Err(error.to_string()),
+ Err(_) => Err(format!("timed out after {PUT_TIMEOUT:?}")),
+ }
+ });
+ let mut failed = 0;
+ for written in futures::future::join_all(writes).await {
+ let outcome = match written {
+ Ok(()) => "written",
+ Err(error) => {
+ if failed == 0 {
+ tracing::warn!(%error, "writing a watermark file failed");
+ }
+ failed += 1;
+ "failed"
+ }
+ };
+ counter!("ai_research_v4_recorder_watermark_files_total", "outcome" => outcome)
+ .increment(1);
+ }
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::*;
+
+ const MAIN: Tp = Tp {
+ topic: "main",
+ partition: 7,
+ };
+
+ fn progress(
+ next_offset: i64,
+ last_captured_at_ms: Option<i64>,
+ earliest_unfinished_ms: Option<i64>,
+ ) -> Option<PartitionProgress> {
+ Some(PartitionProgress {
+ next_offset,
+ last_captured_at_ms,
+ earliest_unfinished_ms,
+ })
+ }
+
+ #[test]
+ fn the_value_is_the_earliest_unfinished_record_or_the_reading_frontier() {
+ const SLACK: i64 = CAPTURE_ORDER_SLACK_MS;
+ // (progress, committed offset, low and high watermarks, expected), all read at 10_000_000.
+ let cases = [
+ (
+ progress(10, Some(9_000_000), Some(4_000_000)),
+ None,
+ 0..20,
+ Some(4_000_000),
+ ),
+ (
+ progress(10, Some(9_000_000), None),
+ None,
+ 0..20,
+ Some(9_000_000 - SLACK),
+ ),
+ (
+ progress(20, Some(9_000_000), None),
+ None,
+ 0..20,
+ Some(10_000_000 - SLACK),
+ ),
+ (
+ progress(20, Some(9_000_000), Some(4_000_000)),
+ None,
+ 0..20,
+ Some(4_000_000),
+ ),
+ (None, Some(20), 0..20, Some(10_000_000 - SLACK)),
+ (None, Some(10), 0..20, None),
+ (None, None, 0..20, None),
+ (None, None, 20..20, Some(10_000_000 - SLACK)),
+ (progress(10, None, None), None, 0..20, None),
+ ];
+ for (progress, committed, offsets, expected) in cases {
+ assert_eq!(
+ complete_up_to_capture_ms(progress, committed, &offsets, 10_000_000),
+ expected,
+ "{progress:?} {committed:?} {offsets:?}"
+ );
+ }
+ }
+
+ #[test]
+ fn an_admitted_record_holds_its_partition_until_its_flush_finishes() {
+ let mut state = Progress::default();
+ state.handled(MAIN, 1, 10, Some(1_000), false);
+ state.handled(MAIN, 1, 11, Some(2_000), true);
+ state.handled(MAIN, 1, 12, Some(3_000), true);
+ assert_eq!(state.of(MAIN, 1), progress(13, Some(3_000), Some(2_000)));
+ assert_eq!(state.of(MAIN, 2), None);
+
+ let first = state.flush_started();
+ state.handled(MAIN, 1, 13, Some(4_000), true);
+ let second = state.flush_started();
+ second.finish();
+ assert_eq!(state.of(MAIN, 1), progress(14, Some(4_000), Some(2_000)));
+ first.finish();
+ assert_eq!(state.of(MAIN, 1), progress(14, Some(4_000), None));
+
+ // A round can read the progress before the consume loop drops the earlier assignment.
+ state.handled(MAIN, 1, 14, Some(5_000), true);
+ let _lost = state.flush_started();
+ state.handled(MAIN, 2, 14, Some(6_000), false);
+ assert_eq!(state.of(MAIN, 2), progress(15, Some(6_000), None));
+ }
+
+ #[test]
+ fn a_file_names_its_day_and_partition_and_carries_the_index_end_offsets() {
+ let overflow = Tp {
+ topic: "overflow",
+ partition: 3,
+ };
+ let ends = KafkaEnds {
+ reads_started_at_ms: 10_000_000,
+ input_offsets: HashMap::from([(MAIN, 0..20), (overflow, 0..9)]),
+ input_partitions: BTreeMap::from([("main", 64), ("overflow", 8)]),
+ index_end_offsets: vec![100, 200],
+ };
+ let written_at = DateTime::from_timestamp_millis(1_793_059_200_123).unwrap();
+ let files = watermark_files(
+ &[
+ (MAIN, progress(10, Some(9_000_000), None)),
+ (overflow, None),
+ ],
+ &HashMap::from([(overflow, 5)]),
+ &ends,
+ written_at,
+ )
+ .unwrap();
+ let files: Vec<(String, serde_json::Value)> = files
+ .into_iter()
+ .map(|(key, body)| (key, serde_json::from_slice(&body).unwrap()))
+ .collect();
+ assert_eq!(
+ files,
+ [(
+ "index/_recorder_watermarks/dt=2026-10-27/topic=main/partition=0007.json"
+ .to_owned(),
+ serde_json::json!({
+ "topic": "main",
+ "partition": 7,
+ "complete_up_to_capture_ms": 9_000_000 - CAPTURE_ORDER_SLACK_MS,
+ "index_end_offsets": [100, 200],
+ "input_partitions": {"main": 64, "overflow": 8},
+ "written_at_ms": 1_793_059_200_123i64,
+ })
+ )]
+ );
+ }
+}
diff --git a/rust/ai-research/replay-recorder-v4/src/write.rs b/rust/ai-research/replay-recorder-v4/src/write.rs
index 0c0c33ff0633..b9a2556edbb0 100644
--- a/rust/ai-research/replay-recorder-v4/src/write.rs
+++ b/rust/ai-research/replay-recorder-v4/src/write.rs
@@ -764,6 +764,14 @@ mod tests {
.or_insert(body);
Ok(())
}
+
+ async fn put(&self, bucket: &str, key: &str, body: Vec<u8>) -> Result<(), StoreError> {
+ self.objects
+ .lock()
+ .unwrap()
+ .insert((bucket.to_owned(), key.to_owned()), body);
+ Ok(())
+ }
}
fn origin(partition: i32, start_offset: i64, end_offset: i64) -> Origin {