:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(ai-research): write v4 index watermark files to s3 (#125019)
Robbie654dc82e9368
diff --git a/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-batcher.test.ts b/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-batcher.test.tsindex 76b8eaeb4d98..ea23f1a5109b 100644--- a/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-batcher.test.ts+++ b/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-batcher.test.ts@@ -3,13 +3,23 @@ import { ParquetReader } from '@dsnp/parquetjs'import { Message } from 'node-rdkafka'import { register } from 'prom-client'-import { OffsetStore } from '~/ingestion/pipelines/sessionreplay/shared/capture-watermark'+import { parseJSON } from '~/common/utils/json-parse'+import {+ CAPTURE_TIMESTAMP_HEADER,+ CaptureWatermark,+ OffsetStore,+ capturedRecords,+ releasingOffsetStore,+} from '~/ingestion/pipelines/sessionreplay/shared/capture-watermark'import { IndexBatcher, IndexBatcherOptions } from './index-batcher'import { BLOCK_INDEX_COLUMNS, V4Index } from './index-columns'+import { indexCapturedData } from './index-message'import { IndexParquetStore } from './index-parquet-store'+import { IndexWatermarkFiles, WATERMARK_FILE_INTERVAL_MS } from './index-watermark-files'const TOPIC = 'ai_research_v4_index'+const WALL_CLOCK_MS = 1_790_000_000_000function blockRow(sessionId: string): Record<string, unknown> {const row: Record<string, unknown> = {}@@ -62,6 +72,11 @@ function message(} as unknown as Message}+const captured = (capturedAtMs: number, message: Message): Message => ({+ ...message,+ headers: [...(message.headers ?? []), { [CAPTURE_TIMESTAMP_HEADER]: Buffer.from(String(capturedAtMs)) }],+})+async function readColumn(body: Buffer, column: string): Promise<unknown[]> {const reader = await ParquetReader.openBuffer(body)const cursor = reader.getCursor()@@ -78,22 +93,41 @@ async function rejected(reason: string): Promise<number> {return metric.values.find((value) => value.labels.reason === reason)?.value ?? 0}+async function failedWatermarkWrites(): Promise<number> {+ const metric = await register.getSingleMetric('ai_research_v4_parquet_sink_watermark_writes_total')!.get()+ return (+ metric.values.find((value) => value.labels.index === 'blocks' && value.labels.outcome === 'failed')?.value ?? 0+ )+}+describe('IndexBatcher', () => {let puts: PutObjectCommandInput[]let send: jest.Mocklet offsets: jest.Mocked<OffsetStore>+ let watermark: CaptureWatermarkconst makeBatcher = (options: Partial<IndexBatcherOptions> = {}) =>new IndexBatcher(new IndexParquetStore({ send } as unknown as S3Client, 'rrweb-v4', 1_000),- offsets,+ releasingOffsetStore(offsets, watermark),+ new IndexWatermarkFiles({ send } as unknown as S3Client, 'rrweb-v4', watermark),{ flushIntervalMs: 600_000, maxRows: 1_000, maxBytes: 1024 * 1024, ...options },0)+ const deliver = (batcher: IndexBatcher, messages: Message[], nowMs: number): Promise<void> => {+ watermark.hold(capturedRecords(messages, indexCapturedData))+ return batcher.handleBatch(messages, nowMs)+ }+const keysOf = (index: V4Index): string[] =>puts.map((put) => put.Key!).filter((key) => key.startsWith(`index/${index}/`))+ const watermarkPuts = (): PutObjectCommandInput[] => puts.filter((put) => put.Key!.includes('/_watermarks/'))++ const watermarkFiles = (): Record<string, unknown> =>+ Object.fromEntries(watermarkPuts().map((put) => [put.Key!, parseJSON(put.Body as string)]))+beforeEach(() => {puts = []send = jest.fn((command: { input: PutObjectCommandInput }) => {@@ -101,6 +135,11 @@ describe('IndexBatcher', () => {return Promise.resolve({})})offsets = { offsetsStore: jest.fn() }+ watermark = new CaptureWatermark('index-batcher-test', 0)+ })++ afterEach(() => {+ jest.restoreAllMocks()})it('writes one file per index and write day, then stores the next offset of each partition', async () => {@@ -283,4 +322,175 @@ describe('IndexBatcher', () => {expect(await readColumn(puts[0].Body as Buffer, 'session_id')).toEqual(['kept'])expect(offsets.offsetsStore).toHaveBeenCalledWith([{ topic: TOPIC, partition: 0, offset: 2 }])})++ it('writes the watermark files after the index files and the offsets, and never past a row still buffered', async () => {+ jest.spyOn(Date, 'now').mockReturnValue(WALL_CLOCK_MS)+ const steps: string[] = []+ const record = (command: { input: PutObjectCommandInput }): void => {+ puts.push(command.input)+ steps.push(command.input.Key!.replace(/\d+-[0-9a-f]{16}\.parquet$/, '{file}'))+ }+ let finishUpload: ((value: unknown) => void) | undefined+ send.mockImplementation((command: { input: PutObjectCommandInput }) => {+ record(command)+ return Promise.resolve({})+ }).mockImplementationOnce((command: { input: PutObjectCommandInput }) => {+ record(command)+ return new Promise((resolve) => (finishUpload = resolve))+ })+ offsets.offsetsStore.mockImplementation(() => steps.push('store offsets'))+ const batcher = makeBatcher()+ await deliver(+ batcher,+ [+ captured(5_000, message(10, 'blocks', '2026-10-15', blockRow('s1'))),+ captured(4_000, message(11, 'images', '2026-10-15', imageRow('image:v4:7:2026-10:a'))),+ ],+ 1+ )++ const flushed = batcher.flush(2)+ while (!finishUpload) {+ await new Promise((resolve) => setImmediate(resolve))+ }+ // The shutdown flush runs while batches still arrive, so this row is buffered but not in the files being written.+ await deliver(batcher, [captured(3_000, message(12, 'blocks', '2026-10-15', blockRow('s2')))], 3)+ finishUpload({})+ await flushed++ expect(steps).toEqual([+ 'index/blocks/dt=2026-10-15/{file}',+ 'index/images/dt=2026-10-15/{file}',+ 'store offsets',+ 'index/blocks/_watermarks/partition=0000.json',+ 'index/images/_watermarks/partition=0000.json',+ ])+ expect(watermarkFiles()).toEqual({+ 'index/blocks/_watermarks/partition=0000.json': {+ partition: 0,+ complete_up_to_capture_ms: 3_000,+ committed_offset: 12,+ written_at_ms: WALL_CLOCK_MS,+ },+ 'index/images/_watermarks/partition=0000.json': {+ partition: 0,+ complete_up_to_capture_ms: 4_000,+ committed_offset: 12,+ written_at_ms: WALL_CLOCK_MS,+ },+ })+ expect(watermarkPuts().map((put) => put.IfNoneMatch)).toEqual([undefined, undefined])+ })++ it('rewrites the watermark files once per interval, also on idle polls, and only for partitions it owns', async () => {+ const clock = jest.spyOn(Date, 'now')+ const batcher = makeBatcher()+ const flushAt = async (atMs: number, messages: Message[]): Promise<void> => {+ clock.mockReturnValue(WALL_CLOCK_MS + atMs)+ await deliver(batcher, messages, atMs)+ await batcher.flush(atMs)+ }++ await flushAt(0, [+ captured(1_000, message(1, 'blocks', '2026-10-15', blockRow('s1'))),+ captured(2_000, message(1, 'blocks', '2026-10-15', blockRow('s2'), 1)),+ ])+ batcher.release([{ topic: TOPIC, partition: 1 }])+ await flushAt(WATERMARK_FILE_INTERVAL_MS - 1, [+ captured(3_000, message(2, 'blocks', '2026-10-15', blockRow('s1'))),+ ])+ expect(watermarkPuts()).toHaveLength(2)+ clock.mockReturnValue(WALL_CLOCK_MS + WATERMARK_FILE_INTERVAL_MS)+ await batcher.handleBatch([], WATERMARK_FILE_INTERVAL_MS)++ expect(watermarkPuts().map((put) => put.Key)).toEqual([+ 'index/blocks/_watermarks/partition=0000.json',+ 'index/blocks/_watermarks/partition=0001.json',+ 'index/blocks/_watermarks/partition=0000.json',+ ])+ expect(watermarkFiles()['index/blocks/_watermarks/partition=0000.json']).toEqual({+ partition: 0,+ complete_up_to_capture_ms: 3_000,+ committed_offset: 3,+ written_at_ms: WALL_CLOCK_MS + WATERMARK_FILE_INTERVAL_MS,+ })+ })++ it('aborts the watermark write of a partition revoked while the write is in flight, and keeps the others', async () => {+ const failedBefore = await failedWatermarkWrites()+ const finishWrite = new Map<string, () => void>()+ send.mockImplementation((command: { input: PutObjectCommandInput }, options: { abortSignal: AbortSignal }) => {+ if (!command.input.Key!.includes('/_watermarks/')) {+ puts.push(command.input)+ return Promise.resolve({})+ }+ return new Promise((resolve, reject) => {+ options.abortSignal.addEventListener('abort', () => reject(new Error('aborted')))+ finishWrite.set(command.input.Key!, () => {+ if (!options.abortSignal.aborted) {+ puts.push(command.input)+ resolve({})+ }+ })+ })+ })+ const batcher = makeBatcher()+ await deliver(+ batcher,+ [+ captured(1_000, message(1, 'blocks', '2026-10-15', blockRow('s1'))),+ captured(2_000, message(1, 'blocks', '2026-10-15', blockRow('s2'), 1)),+ ],+ 0+ )++ const flushed = batcher.flush(0)+ while (finishWrite.size < 2) {+ await new Promise((resolve) => setImmediate(resolve))+ }+ batcher.release([{ topic: TOPIC, partition: 1 }])+ for (const finish of finishWrite.values()) {+ finish()+ }+ await flushed++ expect(watermarkPuts().map((put) => put.Key)).toEqual(['index/blocks/_watermarks/partition=0000.json'])+ expect(await failedWatermarkWrites()).toBe(failedBefore)+ })++ it('keeps a flush successful when a watermark write fails, and writes the file on a poll after the interval', async () => {+ const clock = jest.spyOn(Date, 'now').mockReturnValue(WALL_CLOCK_MS)+ const failedBefore = await failedWatermarkWrites()+ send.mockImplementation((command: { input: PutObjectCommandInput }) => {+ if (command.input.Key!.includes('/_watermarks/')) {+ return Promise.reject(new Error('s3 down'))+ }+ puts.push(command.input)+ return Promise.resolve({})+ })+ const batcher = makeBatcher()+ await deliver(batcher, [captured(1_000, message(1, 'blocks', '2026-10-15', blockRow('s1')))], 1)++ await expect(batcher.flush(2)).resolves.toBeUndefined()+ expect(offsets.offsetsStore).toHaveBeenCalledWith([{ topic: TOPIC, partition: 0, offset: 2 }])+ expect(await failedWatermarkWrites()).toBe(failedBefore + 1)++ send.mockImplementation((command: { input: PutObjectCommandInput }) => {+ puts.push(command.input)+ return Promise.resolve({})+ })+ clock.mockReturnValue(WALL_CLOCK_MS + WATERMARK_FILE_INTERVAL_MS)+ await batcher.handleBatch([], 2 + WATERMARK_FILE_INTERVAL_MS - 1)+ expect(watermarkPuts()).toHaveLength(0)+ await batcher.handleBatch([], 2 + WATERMARK_FILE_INTERVAL_MS)++ expect(watermarkFiles()).toEqual({+ 'index/blocks/_watermarks/partition=0000.json': {+ partition: 0,+ complete_up_to_capture_ms: 1_000,+ committed_offset: 2,+ written_at_ms: WALL_CLOCK_MS + WATERMARK_FILE_INTERVAL_MS,+ },+ })+ })})diff --git a/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-batcher.ts b/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-batcher.tsindex 1ff5677a1e09..90b5dff896b3 100644--- a/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-batcher.ts+++ b/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-batcher.ts@@ -1,6 +1,7 @@/*** Buffers v4 index rows across Kafka batches and writes one Parquet file per index and write day on each flush. Offsets- * are stored only after every file that holds their rows is written, so a failed write replays (at-least-once).+ * are stored only after every file that holds their rows is written, so a failed write replays (at-least-once). A flush+ * that succeeds then writes the watermark files of the partitions that this consumer still owns.*/import { Message, TopicPartition, TopicPartitionOffset } from 'node-rdkafka'@@ -10,6 +11,7 @@ import { OffsetStore } from '~/ingestion/pipelines/sessionreplay/shared/capture-import { V4Index, V4IndexRow } from './index-columns'import { parseIndexMessage } from './index-message'+import { WATERMARK_FILE_INTERVAL_MS } from './index-watermark-files'import { V4ParquetSinkMetrics } from './metrics'export type IndexFileStore = {@@ -17,6 +19,11 @@ export type IndexFileStore = {put(index: V4Index, writeDay: string, body: Buffer, nowMs: number): Promise<string>}+export type IndexWatermarkWriter = {+ write(ownedOffsets: readonly TopicPartitionOffset[], nowMs: number): Promise<void>+ cancelWrites(partitions: readonly TopicPartition[]): void+}+export interface IndexBatcherOptions {flushIntervalMs: numbermaxRows: number@@ -47,6 +54,8 @@ export class IndexBatcher {private rowCount = 0private bufferedBytes = 0private pendingOffsets = new Map<string, TopicPartitionOffset>()+ private readonly storedOffsets = new Map<string, TopicPartitionOffset>()+ private lastWatermarkWriteMs: number | undefined/** Partitions revoked since their last message. A flush that started before the revocation must not store their offsets. */private readonly revoked = new Set<string>()private lastFlushMs: number@@ -55,6 +64,7 @@ export class IndexBatcher {constructor(private readonly store: IndexFileStore,private readonly offsetStore: OffsetStore,+ private readonly watermarkFiles: IndexWatermarkWriter,private readonly options: IndexBatcherOptions,nowMs: number) {@@ -80,19 +90,25 @@ export class IndexBatcher {}if (this.shouldFlush(nowMs)) {await this.flush(nowMs)+ } else if (this.watermarkFilesDue(nowMs)) {+ // A watermark can rise with no new rows, for example when its finished records expire, so idle polls rewrite the files too.+ await this.enqueue(() => this.writeWatermarkFilesIfDue(nowMs))}}/**- * Drops the buffered rows and offsets of revoked partitions. Their new owner reads them again from the committed- * offset, so writing them here would only duplicate them, and this consumer can no longer store their offsets.+ * Drops the buffered rows and offsets of revoked partitions, and aborts their watermark writes. Their new owner reads+ * them again from the committed offset, so writing them here would only duplicate them, and this consumer can no+ * longer store their offsets.*/public release(partitions: readonly TopicPartition[]): void {+ this.watermarkFiles.cancelWrites(partitions)const keys = new Set(partitions.map(partitionKey))let released = 0for (const key of keys) {this.revoked.add(key)this.pendingOffsets.delete(key)+ this.storedOffsets.delete(key)}for (const [groupKey, group] of this.groups) {const kept = group.rows.filter((buffered) => !keys.has(buffered.partition))@@ -131,10 +147,29 @@ export class IndexBatcher {}public flush(nowMs: number): Promise<void> {+ return this.enqueue(() => this.flushNow(nowMs))+ }++ private enqueue(task: () => Promise<void>): Promise<void> {// A later flush's offsets also cover rows that an earlier flush is still writing, so flushes run one at a time and in order.- const flushed = this.flushQueue.then(() => this.flushNow(nowMs))- this.flushQueue = flushed.catch(() => undefined)- return flushed+ const done = this.flushQueue.then(task)+ this.flushQueue = done.catch(() => undefined)+ return done+ }++ private watermarkFilesDue(nowMs: number): boolean {+ return (+ this.storedOffsets.size > 0 &&+ (this.lastWatermarkWriteMs === undefined || nowMs - this.lastWatermarkWriteMs >= WATERMARK_FILE_INTERVAL_MS)+ )+ }++ private async writeWatermarkFilesIfDue(nowMs: number): Promise<void> {+ if (!this.watermarkFilesDue(nowMs)) {+ return+ }+ this.lastWatermarkWriteMs = nowMs+ await this.watermarkFiles.write([...this.storedOffsets.values()], Date.now())}private async flushNow(nowMs: number): Promise<void> {@@ -146,6 +181,7 @@ export class IndexBatcher {heartbeats?.unref()try {await this.writeTaken(nowMs)+ await this.writeWatermarkFilesIfDue(nowMs)} finally {if (heartbeats) {clearInterval(heartbeats)@@ -187,6 +223,9 @@ export class IndexBatcher {const assigned = [...offsets].filter(([partition]) => !this.revoked.has(partition))if (assigned.length > 0) {this.offsetStore.offsetsStore(assigned.map(([, offset]) => offset))+ for (const [partition, offset] of assigned) {+ this.storedOffsets.set(partition, offset)+ }}if (offsets.size > 0) {V4ParquetSinkMetrics.incFlush(wroteFiles ? 'written' : 'empty')diff --git a/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-message.ts b/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-message.tsindex f867c1f0c407..78553a6accd4 100644--- a/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-message.ts+++ b/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-message.ts@@ -4,6 +4,7 @@ import { Message } from 'node-rdkafka'import { parseKafkaHeaders } from '~/common/kafka/consumer/consumer-v1'import { parseJSON } from '~/common/utils/json-parse'import { logger } from '~/common/utils/logger'+import { CapturedData } from '~/ingestion/pipelines/sessionreplay/shared/capture-watermark'import { V4Index, V4IndexRow, selectIndexRow } from './index-columns'import { V4ParquetSinkMetrics, V4RowRejection } from './metrics'@@ -19,10 +20,21 @@ export interface V4IndexMessage {row: V4IndexRow}+/** Block rows feed the event metadata watermark and image rows the inline image one, as in v3. */+export const INDEX_CAPTURED_DATA: Readonly<Record<V4Index, CapturedData>> = {+ blocks: 'event_metadata',+ images: 'inline_images',+}+function isIndex(value: string | undefined): value is V4Index {return value === 'blocks' || value === 'images'}+export function indexCapturedData(message: Message): CapturedData | undefined {+ const index = parseKafkaHeaders(message.headers)[INDEX_HEADER]+ return isIndex(index) ? INDEX_CAPTURED_DATA[index] : undefined+}+function isWriteDay(value: string | undefined): value is string {if (!value || !/^\d{4}-\d{2}-\d{2}$/.test(value)) {return falsediff --git a/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-watermark-files.ts b/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-watermark-files.tsnew file mode 100644index 000000000000..c589cbc3f568--- /dev/null+++ b/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-watermark-files.ts@@ -0,0 +1,120 @@+/**+ * Writes `index/{index}/_watermarks/partition={NNNN}.json` to the rrweb bucket after a flush: the capture watermark of+ * one Kafka partition, so a reader of the bucket can see how far the index is complete without the metrics store.+ */+import { PutObjectCommand, S3Client } from '@aws-sdk/client-s3'+import { TopicPartition, TopicPartitionOffset } from 'node-rdkafka'++import { logger } from '~/common/utils/logger'+import { CaptureWatermark, CapturedData } from '~/ingestion/pipelines/sessionreplay/shared/capture-watermark'++import { V4Index } from './index-columns'+import { INDEX_CAPTURED_DATA } from './index-message'+import { V4ParquetSinkMetrics } from './metrics'++/**+ * The batcher rewrites the files on the first flush or poll after this interval, so a file of a running sink is at most+ * this interval plus one poll old. Readers call a file older than about 15 minutes stalled.+ */+export const WATERMARK_FILE_INTERVAL_MS = 3 * 60 * 1000+const WATERMARK_PUT_TIMEOUT_MS = 10_000++interface IndexWatermarkFile {+ partition: number+ complete_up_to_capture_ms: number+ committed_offset: number+ written_at_ms: number+}++function watermarkFileKey(index: V4Index, partition: number): string {+ return `index/${index}/_watermarks/partition=${String(partition).padStart(4, '0')}.json`+}++const partitionKey = (topic: string, partition: number): string => `${topic}:${partition}`+const dataKey = (topic: string, partition: number, data: CapturedData): string =>+ `${partitionKey(topic, partition)}:${data}`++export class IndexWatermarkFiles {+ private readonly inFlightWrites = new Map<string, AbortController>()++ constructor(+ private readonly s3Client: S3Client,+ private readonly bucket: string,+ private readonly watermark: CaptureWatermark+ ) {}++ /**+ * `ownedOffsets` holds the last stored offset of each partition that the consumer still owns. Never throws, so a+ * failed write cannot fail the flush that called it.+ */+ public async write(ownedOffsets: readonly TopicPartitionOffset[], nowMs: number): Promise<void> {+ // Rows that are not in S3 yet hold the watermark down, because the batcher stores their offsets only after it writes them.+ const capturedAtMs = new Map<string, number>()+ for (const row of this.watermark.snapshot(nowMs)) {+ capturedAtMs.set(dataKey(row.topic, row.partition, row.data), row.capturedAtMs)+ }+ const writes: Promise<void>[] = []+ const writingPartitions: string[] = []+ for (const { topic, partition, offset } of ownedOffsets) {+ const cancellation = new AbortController()+ this.inFlightWrites.set(partitionKey(topic, partition), cancellation)+ writingPartitions.push(partitionKey(topic, partition))+ for (const index of ['blocks', 'images'] as const) {+ const completeUpToCaptureMs = capturedAtMs.get(dataKey(topic, partition, INDEX_CAPTURED_DATA[index]))+ if (completeUpToCaptureMs === undefined) {+ continue+ }+ writes.push(+ this.put(+ index,+ watermarkFileKey(index, partition),+ {+ partition,+ complete_up_to_capture_ms: completeUpToCaptureMs,+ committed_offset: offset,+ written_at_ms: nowMs,+ },+ cancellation.signal+ )+ )+ }+ }+ await Promise.all(writes)+ for (const partition of writingPartitions) {+ this.inFlightWrites.delete(partition)+ }+ }++ /**+ * Aborts the writes still in flight for partitions that this consumer lost, retries included, so that a slow write+ * cannot replace a file that the new owner wrote after it.+ */+ public cancelWrites(partitions: readonly TopicPartition[]): void {+ for (const { topic, partition } of partitions) {+ this.inFlightWrites.get(partitionKey(topic, partition))?.abort()+ }+ }++ private async put(index: V4Index, key: string, file: IndexWatermarkFile, cancelled: AbortSignal): Promise<void> {+ try {+ await this.s3Client.send(+ new PutObjectCommand({+ Bucket: this.bucket,+ Key: key,+ Body: JSON.stringify(file),+ ContentType: 'application/json',+ }),+ { abortSignal: AbortSignal.any([cancelled, AbortSignal.timeout(WATERMARK_PUT_TIMEOUT_MS)]) }+ )+ } catch (error) {+ if (cancelled.aborted) {+ V4ParquetSinkMetrics.incWatermarkWrite(index, 'cancelled')+ return+ }+ V4ParquetSinkMetrics.incWatermarkWrite(index, 'failed')+ logger.warn('🪶', 'ai_research_v4_parquet_watermark_write_failed', { key, error: String(error) })+ return+ }+ V4ParquetSinkMetrics.incWatermarkWrite(index, 'written')+ }+}diff --git a/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/metrics.ts b/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/metrics.tsindex 4e3d504a2653..cab8268577f0 100644--- a/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/metrics.ts+++ b/nodejs/src/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/metrics.ts@@ -48,6 +48,11 @@ export class V4ParquetSinkMetrics {help: 'v4 sink flushes by outcome: wrote files, or committed offsets of messages that gave no rows',labelNames: ['outcome'],})+ private static readonly watermarkWrites = new Counter({+ name: 'ai_research_v4_parquet_sink_watermark_writes_total',+ help: 'v4 index watermark file writes by index and outcome; a failed write leaves its flush successful and is tried again after the next interval, and a cancelled one was for a revoked partition',+ labelNames: ['index', 'outcome'],+ })public static incRowsParsed(index: V4Index): void {this.rowsParsed.labels(index).inc()@@ -72,4 +77,7 @@ export class V4ParquetSinkMetrics {public static incFlush(outcome: 'written' | 'empty'): void {this.flushes.labels(outcome).inc()}+ public static incWatermarkWrite(index: V4Index, outcome: 'written' | 'failed' | 'cancelled'): void {+ this.watermarkWrites.labels(index, outcome).inc()+ }}diff --git a/nodejs/src/servers/ai-research-v4-parquet-sink-server.ts b/nodejs/src/servers/ai-research-v4-parquet-sink-server.tsindex 808077e4bd6b..8450521c4b52 100644--- a/nodejs/src/servers/ai-research-v4-parquet-sink-server.ts+++ b/nodejs/src/servers/ai-research-v4-parquet-sink-server.ts@@ -1,16 +1,14 @@-import { Message } from 'node-rdkafka'-import { KAFKA_AI_RESEARCH_V4_INDEX } from '~/common/config/kafka-topics'-import { KafkaConsumer, KafkaConsumerConfig, parseKafkaHeaders } from '~/common/kafka/consumer/consumer-v1'+import { KafkaConsumer, KafkaConsumerConfig } from '~/common/kafka/consumer/consumer-v1'import { logger } from '~/common/utils/logger'import { AiResearchV4ServerConfig } from '~/ingestion/pipelines/sessionreplay/ai-research/config'import { IndexBatcher } from '~/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-batcher'-import { INDEX_HEADER } from '~/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-message'+import { indexCapturedData } from '~/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-message'import { IndexParquetStore } from '~/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-parquet-store'+import { IndexWatermarkFiles } from '~/ingestion/pipelines/sessionreplay/ai-research/parquet-sink-v4/index-watermark-files'import { AiResearchV4ConsumerServer } from '~/ingestion/pipelines/sessionreplay/ai-research/server/v4-consumer-server'import {CaptureWatermark,- CapturedData,capturedRecords,releasingOffsetStore,} from '~/ingestion/pipelines/sessionreplay/shared/capture-watermark'@@ -43,12 +41,6 @@ export function buildV4SinkConsumerConfig(config: AiResearchV4ServerConfig): Kaf}}-/** Block rows feed the event metadata watermark and image rows the inline image one, as in v3. */-function indexData(message: Message): CapturedData | undefined {- const index = parseKafkaHeaders(message.headers)[INDEX_HEADER]- return index === 'blocks' ? 'event_metadata' : index === 'images' ? 'inline_images' : undefined-}-export class AiResearchV4ParquetSinkServer extends AiResearchV4ConsumerServer {protected async startServices(): Promise<void> {const consumerConfig = buildV4SinkConsumerConfig(this.config)@@ -70,6 +62,7 @@ export class AiResearchV4ParquetSinkServer extends AiResearchV4ConsumerServer {const batcher = new IndexBatcher(new IndexParquetStore(s3Client, bucket, rowGroupRows),releasingOffsetStore(consumer, watermark),+ new IndexWatermarkFiles(s3Client, bucket, watermark),{flushIntervalMs: this.config.AI_RESEARCH_V4_PARQUET_FLUSH_INTERVAL_MS,maxRows: this.config.AI_RESEARCH_V4_PARQUET_MAX_ROWS,@@ -81,7 +74,7 @@ export class AiResearchV4ParquetSinkServer extends AiResearchV4ConsumerServer {await consumer.connect((messages) => {consumer.heartbeat()- watermark.hold(capturedRecords(messages, indexData))+ watermark.hold(capturedRecords(messages, indexCapturedData))return batcher.handleBatch(messages, Date.now())},(partitions) => {diff --git a/products/ai_training/docs/replay-data.md b/products/ai_training/docs/replay-data.mdindex 7ee00cbf8b27..2f5a04a69ac9 100644--- a/products/ai_training/docs/replay-data.md+++ b/products/ai_training/docs/replay-data.md@@ -486,6 +486,8 @@ Fetched images are partitioned by the key month of their reference.| `rrweb-v4` | `blocks/dt={day}/` | One object per recorder flush: sealed blocks, each found by byte range || `rrweb-v4` | `index/blocks/dt={day}/` | Parquet, one row per block || `rrweb-v4` | `index/images/dt={day}/` | Parquet, one row per stored inline image |+| `rrweb-v4` | `index/blocks/_watermarks/` | JSON, one object per Kafka partition: how far the block index is complete |+| `rrweb-v4` | `index/images/_watermarks/` | JSON, one object per Kafka partition: how far the image index is complete || `images-v4` | `images/packs/dt={day}/{team_id}/` | One object per recorder flush and team: sealed images, each found by byte range || `images-v4` | `images/url/month={YYYY-MM}/{team_id}/{hash}` | One sealed fetched image per URL reference |@@ -566,6 +568,46 @@ This holds for the blocks that one recorder process inspected. After a rebalance- `url_fingerprints` holds up to 10 distinct page URLs of the block as the first 16 hexadecimal characters of an HMAC-SHA256 of the URL. The key is derived from the session key with HKDF-SHA256, empty salt and info `ml-url-fingerprint-key`, so fingerprints can be compared only within one session. They count distinct URLs, where `navigations` counts URL changes.- `cold_start` is true when the recorder first sees a session after it started, for example after a partition rebalance, or after it dropped the session's DOM state to stay within its memory budget. Its playability columns then describe only the part of the session the recorder saw.+#### Watermark files++After a flush, the sink writes one JSON file per index and Kafka partition of `ai_research_v4_index`, so a reader can learn from the bucket how far the index is complete:++- `index/blocks/_watermarks/partition={NNNN}.json` for block rows.+- `index/images/_watermarks/partition={NNNN}.json` for image rows.++`{NNNN}` is the partition number with four digits, for example `partition=0007.json`.+Each file holds one object:++| Field | Meaning |+| --------------------------- | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| `partition` | The Kafka partition. |+| `complete_up_to_capture_ms` | Every row that the sink read from the partition with an earlier capture time is in an index file in S3. It is the value of `ml_replay_capture_watermark_timestamp_seconds` for the partition, in milliseconds: `data="event_metadata"` for blocks and `data="inline_images"` for images. |+| `committed_offset` | The next offset to read on the partition. The sink stored it for commit after it wrote every row below it. |+| `written_at_ms` | When the sink wrote the file. |++The capture time of a row is the Kafka timestamp of the replay record that it came from, which the recorder sends in the `capture-timestamp-ms` header.+The sink writes a file only for a partition that it still owns, and only after a flush wrote the partition's rows and stored its offset.+A row that the sink holds but has not written keeps the value at or below its capture time.+The sink rewrites its files once every `WATERMARK_FILE_INTERVAL_MS` (`index-watermark-files.ts`), on the first flush or poll after the interval, also when no rows arrive.+A value can rise without new rows, because the watermark also counts the rows that it finished in the last `FINISHED_WINDOW_MS` (`capture-watermark.ts`).+Each write is a plain PUT that replaces the file.+A failed write does not fail the flush. The sink counts it in `ai_research_v4_parquet_sink_watermark_writes_total`, logs it, and writes the file again after the next interval. A write that the sink aborts because it lost the partition counts as `outcome="cancelled"`.+S3 expires a file 30 days after its last write, as it does every object in the bucket.++To learn how far an index is complete, a reader:++1. Reads every file under that index's `_watermarks/`.+2. Takes the minimum `complete_up_to_capture_ms` across the partitions.+3. Treats a file whose `written_at_ms` is more than about 15 minutes old as a partition that is not progressing: no sink writes it. Its value is still true, but it does not move. A running sink keeps each file it owns at most the rewrite interval plus one poll old.++Some limits apply:++- A partition that delivers no records keeps its last value, as the gauge does, and the sink keeps rewriting its file. So a quiet partition holds the minimum down.+- When the sink loses a partition, it aborts the partition's watermark writes that are still in flight, retries included. A write that S3 already received can still land. The new owner writes its first file only after its first flush, so the two overlap only if S3 takes longer to apply that write than the new owner takes to flush. The file then holds an older value until the new owner writes again. That value is still true. A full fence is not possible, because the bucket does not let the sink read the file that it replaces.+- A partition that the sink has not flushed since it started to write these files has no file. A reader that needs every partition compares the number of files with the partition count of the topic.+- The value covers only the rows that reached the sink. The recorder sends a row only after its own flush stores the block or image, so a row with an earlier capture time can still be in the recorder. A sink that has read all of its topic does not make the index complete.+- A reader that lists `index/blocks/` or `index/images/` for Parquet files must skip `_watermarks/`. Hive-style readers such as Spark, Athena and pyarrow datasets ignore paths that start with `_`, and a `*.parquet` glob does not match the JSON files.+### Image index`index/images/` has one row per inline image in a pack: `team_id`, `key_month`, `image_ref`, `pack_s3_key`, `byte_start`, `byte_end`, `mime` and `byte_size`.@@ -573,6 +615,7 @@ This holds for the blocks that one recorder process inspected. After a rebalance- Each frame in a pack is sealed under the team month key of the write month, as a fetched image is. A session deletion therefore does not make inline images unreadable (see [Deletion](#deletion)).- `image_ref` uses the team month image content key, so the same image has the same reference in every session of the team month. A pack holds one frame per distinct reference, so an image that two sessions show in one flush is stored once. A later flush stores it again.- A reader resolves an image of a block by (`team_id`, `image_ref`) and opens it with the team month key of `key_month`.+- The sink writes [watermark files](#watermark-files) for this index too, under `index/images/_watermarks/`.### Remote images