:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(cohorts): skip seeder tiles that a newer day already covers (#125076)
Matheus Batistad4fd2eead7d7
diff --git a/docs/internal/realtime-cohorts/invariants.md b/docs/internal/realtime-cohorts/invariants.mdindex f160af507d13..9c2a545a2d0c 100644--- a/docs/internal/realtime-cohorts/invariants.md+++ b/docs/internal/realtime-cohorts/invariants.md@@ -206,6 +206,8 @@ Most of these are not enforced by a single test, and a violation usually shows uEvery outcome write is fenced by the dispatch epoch, and `reconcile_observed_at` is written last.- Limit: the proof covers planned days, not persons.Raising the seeder's band count while a run is seeding leaves persons unscanned, and every chunk still confirms.+- Limit: a chunk skips a person's tile on a condition where only the newest day counts, once a newer day's chunk of the run has scanned that person.+ If the newer chunk fails and the person's events on its day are deleted before the retry, neither day sends the person, and every chunk still confirms.- See [completion and readiness](completion-and-readiness.md).**26. A readiness stamp vouches only for the definition its run replayed.**diff --git a/docs/internal/realtime-cohorts/seeder.md b/docs/internal/realtime-cohorts/seeder.mdindex 0b70669e3cd7..8c46bbba98a5 100644--- a/docs/internal/realtime-cohorts/seeder.md+++ b/docs/internal/realtime-cohorts/seeder.md@@ -155,7 +155,14 @@ At the end of the chunk, each count becomes one **day tile**:Tiles are keyed `"{team_id}:{person_id}"`, so each lands on the partition of the worker that owns the person.A shared rate limiter paces production, and a cap bounds the tiles in flight.-The chunk then moves to `produced`, which means every tile was handed to the producer, waits for every acknowledgment, and moves to `confirmed`.+On a condition where every leaf of the run keeps only the newest matching day, as a `performed_event` leaf does, an older tile changes nothing.+Each run keeps an in-memory index of the newest day it has scanned for each person on such a condition.+A chunk records its days right after its scan, and skips a tile when the index holds a strictly newer day.+Losing the index on a restart only sends more tiles.+`SEEDER_NEWEST_DAY_INDEX_MAX_ENTRIES` caps the entries of all indexes together, and past the cap a new pair sends every day.+`seeder_tiles_skipped_total` counts the skipped tiles, and `seeder_newest_day_index_entries` shows how full the cap is.++The chunk then moves to `produced`, which means every tile it sends was handed to the producer, waits for every acknowledgment, and moves to `confirmed`.### Chunk states and retriesdiff --git a/rust/Cargo.lock b/rust/Cargo.lockindex 82126aeade9c..33bc36d55957 100644--- a/rust/Cargo.lock+++ b/rust/Cargo.lock@@ -2815,6 +2815,7 @@ dependencies = ["common-kafka","common-liveness","common-types",+ "dashmap 6.1.0","envconfig","futures","governor",diff --git a/rust/cohort-seeder/Cargo.toml b/rust/cohort-seeder/Cargo.tomlindex 3f85ad4ec7b3..0edb4e687aae 100644--- a/rust/cohort-seeder/Cargo.toml+++ b/rust/cohort-seeder/Cargo.toml@@ -30,6 +30,7 @@ common-database = { path = "../common/database" }common-kafka = { path = "../common/kafka" }common-liveness = { path = "../common/liveness" }common-types = { path = "../common/types" }+dashmap = { workspace = true }envconfig = { workspace = true }futures = { workspace = true }governor = { workspace = true }diff --git a/rust/cohort-seeder/src/app/deliver.rs b/rust/cohort-seeder/src/app/deliver.rsindex 37f1ae4f2e27..894c58f695fb 100644--- a/rust/cohort-seeder/src/app/deliver.rs+++ b/rust/cohort-seeder/src/app/deliver.rs@@ -1,9 +1,9 @@//! App layer, Kafka produce sequencing: the two-phase enqueue/await split. Depends on `domain`,//! `kafka`, `store`, and `app::settings`; never imported by a lower layer.//!-//! The sequence `enqueue_tiles → store.mark_produced → await_deliveries` (composed by `execute`) is-//! byte-equivalent to the former monolithic produce: enqueue every tile honoring the in-flight bound-//! and draining overflow acks before the mark, then drain the remaining acks after it, folding all+//! The sequence `enqueue_tiles → store.mark_produced → await_deliveries` (composed by+//! `execute`) enqueues every tile its run has no newer day for, honoring the in-flight bound and+//! draining overflow acks before the mark, then drains the remaining acks after it, folding all//! delivery high-water marks into one accumulator carried across the mark by [`InFlight`].use std::collections::VecDeque;@@ -14,7 +14,9 @@ use rdkafka::error::KafkaError;use rdkafka::producer::DeliveryFuture;use tokio_util::sync::CancellationToken;-use crate::domain::{CancelCause, EnqueuedChunk, Halted, ProduceHwms, ProducedChunk, ScannedChunk};+use crate::domain::{+ CancelCause, EnqueuedChunk, Halted, ProduceHwms, ProducedChunk, ScannedChunk, StreamedChunk,+};use crate::kafka::pacing::TilePacer;use crate::kafka::producer::{EnqueueError, SeedTileProducer};use crate::observability::metrics::{@@ -22,6 +24,7 @@ use crate::observability::metrics::{};use crate::store::chunks::ChunkStoreError;+use super::newest_day::NewestDayIndex;use super::settings::ProducerSettings;/// The ceiling the queue-full backoff doubles up to; also clamps [`ProducerSettings`] at construction.@@ -96,19 +99,29 @@ pub(super) async fn enqueue_one(Ok(())}-/// Phase one (pre-mark): pace and enqueue every tile honoring the in-flight bound, draining overflow-/// acks into the HWMs. Returns the still-`scanning` chunk (for the store's mark) plus the in-flight-/// deliveries. Every error hands the [`ScannedChunk`] back so the caller can release or fail it.+/// Phase one (pre-mark): pace and enqueue every tile `newest_day` has no newer day for, honoring the+/// in-flight bound and draining overflow acks into the HWMs. Returns the still-`scanning` chunk (for+/// the store's mark) with the count sent, plus the in-flight deliveries. Every error hands the+/// [`ScannedChunk`] back so the caller can release or fail it.pub(super) async fn enqueue_tiles(producer: &SeedTileProducer,chunk: ScannedChunk,+ newest_day: &NewestDayIndex,pacer: &TilePacer,settings: ProducerSettings,lease_cancel: &CancellationToken,shutdown: &CancellationToken,-) -> Result<(ScannedChunk, InFlight), Halted<ScannedChunk, ProduceError>> {+) -> Result<(StreamedChunk, InFlight), Halted<ScannedChunk, ProduceError>> {let mut inflight = InFlight::new(settings.max_inflight.get());+ let mut sent = 0;for tile in chunk.tiles() {+ // Checked before the pacer, so a newer day another chunk records while this one waits+ // still skips this chunk's remaining tiles for that person. A skipped tile takes no pacer+ // token, so the task budget makes it yield.+ if newest_day.has_newer_day(tile) {+ tokio::task::coop::consume_budget().await;+ continue;+ }if let Err(stop) = enqueue_one(|| producer.enqueue(tile),&mut inflight,@@ -121,8 +134,9 @@ pub(super) async fn enqueue_tiles({return Err(stop.into_halted(chunk));}+ sent += 1;}- Ok((chunk, inflight))+ Ok((chunk.into_streamed(sent), inflight))}/// Phase two (post-mark): drain the remaining acks into the HWMs and fold them onto the markeddiff --git a/rust/cohort-seeder/src/app/execute.rs b/rust/cohort-seeder/src/app/execute.rsindex f855774ba563..a263108a18b3 100644--- a/rust/cohort-seeder/src/app/execute.rs+++ b/rust/cohort-seeder/src/app/execute.rs@@ -1,10 +1,11 @@-//! App layer: one chunk's `scan → enqueue → mark → await → confirm` pipeline and the single recovery-//! resolver. Depends on `clickhouse`, `kafka`, `store`, `domain`, and its `app` siblings.+//! App layer: one chunk's `scan → record → enqueue → mark → await → confirm` pipeline and the single+//! recovery resolver. Depends on `clickhouse`, `kafka`, `store`, `domain`, and its `app` siblings.//!-//! The store's `mark_produced` sits between the two deliver phases: deliver owns Kafka, execute owns-//! the store gating. Every halt funnels through [`resolve_halt`], which applies the exact recovery-//! matrix — a pre-mark halt during shutdown unclaims (no attempt spent); everything else fails for-//! retry — encoded once in [`FailureDisposition::resolve`] and pinned by its unit test.+//! The store's `mark_produced` sits between the two deliver phases: deliver owns Kafka,+//! execute owns the store gating. Every halt funnels through [`resolve_halt`], which applies the+//! exact recovery matrix — a pre-mark halt during shutdown unclaims (no attempt spent); everything+//! else fails for retry — encoded once in [`FailureDisposition::resolve`] and pinned by its unit+//! test.use std::sync::Arc;use std::time::Duration;@@ -22,7 +23,7 @@ use crate::domain::{use crate::kafka::pacing::TilePacer;use crate::kafka::producer::SeedTileProducer;use crate::observability::metrics::{- CHUNKS_CONFIRMED, CHUNKS_FAILED, CLICKHOUSE_RESOURCE_ERRORS, TILES_PRODUCED,+ CHUNKS_CONFIRMED, CHUNKS_FAILED, CLICKHOUSE_RESOURCE_ERRORS, TILES_PRODUCED, TILES_SKIPPED,};use crate::store::chunks::{ChunkStoreError, PgChunkStore};use crate::store::lease::LeaseHandle;@@ -30,6 +31,7 @@ use crate::store::runs::RunKind;use crate::store::RenderedError;use super::deliver::{self, ProduceError};+use super::newest_day::NewestDayIndex;use super::person_execute::PersonChunkStats;use super::prepare::PreparedBehavioral;use super::settings::ProducerSettings;@@ -39,6 +41,7 @@ pub(super) struct ChunkTaskContext {pub(super) chunk: ClaimedChunk,pub(super) lease: LeaseHandle,pub(super) prepared: Arc<PreparedBehavioral>,+ pub(super) newest_day: Arc<NewestDayIndex>,pub(super) store: PgChunkStore,pub(super) scanner: ChunkScanner,pub(super) producer: SeedTileProducer,@@ -55,6 +58,7 @@ pub(super) async fn execute_chunk(chunk,lease,prepared,+ newest_day,store,scanner,producer,@@ -78,10 +82,16 @@ pub(super) async fn execute_chunk(Ok(scanned) => scanned,Err(halt) => return resolve_halt(&store, halt, &shutdown, retry_backoff).await,};- // PreMark: pace and enqueue every tile, holding the in-flight deliveries.- let (scanned, inflight) = match deliver::enqueue_tiles(+ // PreMark: record this scan's days before enqueuing, so an older day's chunk skips its tiles.+ // A large chunk takes seconds, so the worker's other tasks move to another thread meanwhile.+ tokio::task::block_in_place(|| newest_day.record(scanned.tiles()));+ let scanned_tiles = scanned.tiles().len() as u64;+ // PreMark: pace and enqueue every tile not superseded by a newer day, holding the in-flight+ // deliveries.+ let (streamed, inflight) = match deliver::enqueue_tiles(&producer,scanned,+ &newest_day,&pacer,producer_settings,&lease_cancel,@@ -93,7 +103,7 @@ pub(super) async fn execute_chunk(Err(halt) => return resolve_halt(&store, halt, &shutdown, retry_backoff).await,};// PreMark: CAS `scanning`→`produced` — the row is still `scanning` on failure.- let enqueued = match store.mark_produced(scanned).await {+ let enqueued = match store.mark_produced(streamed).await {Ok(enqueued) => enqueued,Err(halt) => {return resolve_halt(&store, mark_produced_halt(halt), &shutdown, retry_backoff).await@@ -103,6 +113,7 @@ pub(super) async fn execute_chunk(let produced = match deliver::await_deliveries(enqueued, inflight, &lease_cancel).await {Ok(produced) => {counter!(TILES_PRODUCED).increment(produced.tiles_produced());+ counter!(TILES_SKIPPED).increment(scanned_tiles - produced.tiles_produced());produced}Err(halt) => return resolve_halt(&store, halt, &shutdown, retry_backoff).await,diff --git a/rust/cohort-seeder/src/app/mod.rs b/rust/cohort-seeder/src/app/mod.rsindex 62cea176bd8d..fd542f9197f2 100644--- a/rust/cohort-seeder/src/app/mod.rs+++ b/rust/cohort-seeder/src/app/mod.rs@@ -5,6 +5,7 @@ mod breaker;pub mod completion;mod deliver;mod execute;+mod newest_day;pub mod observe;mod orchestrator;mod person_execute;diff --git a/rust/cohort-seeder/src/app/newest_day.rs b/rust/cohort-seeder/src/app/newest_day.rsnew file mode 100644index 000000000000..eab30fe4e743--- /dev/null+++ b/rust/cohort-seeder/src/app/newest_day.rs@@ -0,0 +1,289 @@+//! App layer: the per-run index of the newest day scanned for each (person, condition) pair, which+//! lets an older day's chunk skip the tiles the processor would merge away. Depends on `domain` and+//! `app::prepare`.+//!+//! The state is process-local, like the run breakers. Losing an index, on a restart or when an idle+//! run drops out of one refresh's eligible set, only sends more tiles.++use std::collections::HashMap;+use std::num::NonZeroUsize;+use std::sync::atomic::{AtomicUsize, Ordering};+use std::sync::Arc;++use dashmap::mapref::entry::Entry;+use dashmap::DashMap;+use metrics::gauge;+use uuid::Uuid;++use crate::domain::{ConditionSlot, DayIdx, NewestDayConditions, PinnedRun, RunId, SeedTile};+use crate::observability::metrics::NEWEST_DAY_INDEX_ENTRIES;++use super::prepare::PreparedRun;++type Key = (Uuid, ConditionSlot);++/// The newest-day indexes of the runs this process seeds, and the entry budget they share.+pub(super) struct NewestDayIndexes {+ budget: IndexBudget,+ by_run: HashMap<RunId, Arc<NewestDayIndex>>,+}++impl NewestDayIndexes {+ pub(super) fn new(max_entries: NonZeroUsize) -> Self {+ Self {+ budget: IndexBudget {+ used: Arc::default(),+ max: max_entries,+ },+ by_run: HashMap::new(),+ }+ }++ /// Built on the run's first claim. A run's frozen catalog only loses leaves after the run is+ /// created, so a `NewestDayConditions` built then can miss a skip but never allow a wrong one.+ pub(super) fn for_run(&mut self, run: &PinnedRun) -> Arc<NewestDayIndex> {+ let budget = &self.budget;+ self.by_run+ .entry(run.run_id)+ .or_insert_with(|| {+ let conditions = NewestDayConditions::of(&run.conditions, &run.filters);+ Arc::new(NewestDayIndex::new(conditions, budget.clone()))+ })+ .clone()+ }++ /// Keeps an index while a chunk task holds it, because a discovery error yields an empty+ /// eligible set, and a second index for the run would split its days and its budget. The gauge+ /// is set only here, so two chunk tasks can never set it out of order.+ pub(super) fn release_idle(&mut self, eligible: &HashMap<RunId, PreparedRun>) {+ self.by_run+ .retain(|run_id, index| eligible.contains_key(run_id) || Arc::strong_count(index) > 1);+ gauge!(NEWEST_DAY_INDEX_ENTRIES).set(self.budget.used.load(Ordering::Relaxed) as f64);+ }+}++/// One run's newest recorded day for each person on each qualifying condition.+///+/// It must stay per run: an index shared across runs would let one run's readiness depend on a+/// tile that only another run sends.+pub(super) struct NewestDayIndex {+ conditions: NewestDayConditions,+ newest: DashMap<Key, DayIdx>,+ budget: IndexBudget,+}++impl NewestDayIndex {+ fn new(conditions: NewestDayConditions, budget: IndexBudget) -> Self {+ Self {+ conditions,+ newest: DashMap::new(),+ budget,+ }+ }++ /// Max-merges each qualifying tile's day. A new pair takes one budget entry, and past the+ /// budget it is not recorded, so its older days are sent.+ pub(super) fn record(&self, tiles: &[SeedTile]) {+ for tile in tiles {+ let Some(key) = self.key(tile) else {+ continue;+ };+ match self.newest.entry(key) {+ Entry::Occupied(mut newest) => {+ let newest = newest.get_mut();+ *newest = (*newest).max(tile.day_idx());+ }+ Entry::Vacant(vacant) => {+ if self.budget.try_take() {+ vacant.insert(tile.day_idx());+ }+ }+ }+ }+ }++ /// Only a strictly newer day counts, so a retried chunk never skips its own tiles, and the+ /// chunk holding a pair's newest day always sends it, unless events deleted before its retry+ /// remove the pair.+ pub(super) fn has_newer_day(&self, tile: &SeedTile) -> bool {+ self.key(tile).is_some_and(|key| {+ self.newest+ .get(&key)+ .is_some_and(|newest| *newest > tile.day_idx())+ })+ }++ fn key(&self, tile: &SeedTile) -> Option<Key> {+ self.conditions+ .slot(tile.condition_hash())+ .map(|slot| (tile.person_id(), slot))+ }+}++/// Gives this index's entries back, so a finished run frees its share of the budget.+impl Drop for NewestDayIndex {+ fn drop(&mut self) {+ self.budget+ .used+ .fetch_sub(self.newest.len(), Ordering::Relaxed);+ }+}++/// The entries all indexes may hold together. Every entry an index holds took one from here.+#[derive(Clone)]+struct IndexBudget {+ used: Arc<AtomicUsize>,+ max: NonZeroUsize,+}++impl IndexBudget {+ fn try_take(&self) -> bool {+ self.used+ .fetch_update(Ordering::Relaxed, Ordering::Relaxed, |used| {+ (used < self.max.get()).then_some(used + 1)+ })+ .is_ok()+ }+}++#[cfg(test)]+mod tests {+ use std::num::NonZeroU32;++ use chrono_tz::UTC;+ use cohort_core::filters::{CohortId, TeamId};++ use super::*;+ use crate::domain::row_filter::test_catalog::{catalog, event_is, hash, Leaf};+ use crate::domain::{+ Boundary, ClaimEpoch, EventNameSet, Lookback, PinnedCondition, SChunkMs, TriggerKind,+ UtcMillis,+ };++ const SLIDING: &str = "aaaaaaaaaaaaaaaa";+ /// Pinned, but absent from the frozen catalog, so it does not qualify.+ const UNCATALOGUED: &str = "bbbbbbbbbbbbbbbb";++ fn budget(max: usize) -> IndexBudget {+ IndexBudget {+ used: Arc::default(),+ max: NonZeroUsize::new(max).unwrap(),+ }+ }++ fn run(id: u128) -> PinnedRun {+ PinnedRun {+ run_id: RunId(Uuid::from_u128(id)),+ team_id: TeamId(2),+ trigger: TriggerKind::TeamEnablement,+ tz: UTC,+ boundary: Boundary::new(UtcMillis::new(0), UTC),+ conditions: [SLIDING, UNCATALOGUED]+ .map(|value| PinnedCondition {+ cohort_id: CohortId(1),+ hash: hash(value),+ event_name: "$pageview".to_owned(),+ lookback: Lookback::SlidingDays(7),+ })+ .to_vec(),+ event_names: EventNameSet::new(["$pageview".to_owned()]),+ filters: catalog(&[Leaf {+ cohort: 1,+ key: "$pageview",+ hash: SLIDING,+ body: event_is("$pageview"),+ }]),+ }+ }++ fn index(budget: &IndexBudget) -> NewestDayIndex {+ let run = run(0);+ let conditions = NewestDayConditions::of(&run.conditions, &run.filters);+ NewestDayIndex::new(conditions, budget.clone())+ }++ fn tile(person: u128, condition: &str, day: DayIdx) -> SeedTile {+ SeedTile::new(+ TeamId(2),+ Uuid::from_u128(person),+ hash(condition),+ NonZeroU32::MIN,+ day,+ SChunkMs(0),+ RunId(Uuid::nil()),+ ClaimEpoch(1),+ )+ }++ #[test]+ fn only_a_strictly_newer_day_for_the_same_person_and_condition_skips_a_tile() {+ let index = index(&budget(10));+ index.record(&[tile(1, SLIDING, 10), tile(1, UNCATALOGUED, 10)]);+ index.record(&[tile(1, SLIDING, 9)]);++ for (case, tile, skipped) in [+ ("the recorded day", tile(1, SLIDING, 10), false),+ ("an older day", tile(1, SLIDING, 9), true),+ ("another person's older day", tile(2, SLIDING, 9), false),+ (+ "an older day on a non-qualifying condition",+ tile(1, UNCATALOGUED, 9),+ false,+ ),+ ] {+ assert_eq!(index.has_newer_day(&tile), skipped, "{case}");+ }+ }++ #[test]+ fn a_full_budget_sends_new_pairs_until_another_index_gives_its_entries_back() {+ let budget = budget(1);+ let first = index(&budget);+ first.record(&[tile(1, SLIDING, 10), tile(2, SLIDING, 10)]);+ assert!(+ !first.has_newer_day(&tile(2, SLIDING, 9)),+ "a pair past the budget was recorded"+ );+ first.record(&[tile(1, SLIDING, 11)]);+ assert!(+ first.has_newer_day(&tile(1, SLIDING, 10)),+ "a recorded pair stopped updating at a full budget"+ );++ drop(first);+ let second = index(&budget);+ second.record(&[tile(2, SLIDING, 10)]);+ assert!(+ second.has_newer_day(&tile(2, SLIDING, 9)),+ "a dropped index kept its budget"+ );+ }++ #[test]+ fn a_run_keeps_one_index_until_it_drops_out_and_no_chunk_task_holds_it() {+ let mut indexes = NewestDayIndexes::new(NonZeroUsize::MIN);+ let (held, idle) = (run(1), run(2));+ let task = indexes.for_run(&held);+ task.record(&[tile(1, SLIDING, 10)]);+ assert!(+ indexes.for_run(&held).has_newer_day(&tile(1, SLIDING, 9)),+ "a run's next claim got a new index"+ );+ let idle_index = Arc::downgrade(&indexes.for_run(&idle));+ assert!(+ !indexes.for_run(&idle).has_newer_day(&tile(1, SLIDING, 9)),+ "another run's day skipped a tile"+ );++ // A discovery error yields an empty eligible set.+ indexes.release_idle(&HashMap::new());+ assert!(+ Arc::ptr_eq(&indexes.for_run(&held), &task),+ "a held index was released, so the run's next chunk gets a second one"+ );+ assert!(+ idle_index.upgrade().is_none(),+ "an idle index outlived its run and kept its budget"+ );+ }+}diff --git a/rust/cohort-seeder/src/app/orchestrator.rs b/rust/cohort-seeder/src/app/orchestrator.rsindex 5d2d40f75af2..164f9558a19f 100644--- a/rust/cohort-seeder/src/app/orchestrator.rs+++ b/rust/cohort-seeder/src/app/orchestrator.rs@@ -32,6 +32,7 @@ use super::completion::CompletionDriver;use super::execute::{execute_chunk, record_task_result, BreakerSignal, ChunkOutcome, ChunkTaskContext,};+use super::newest_day::NewestDayIndexes;use super::person_execute::{execute_person_chunk, PersonChunkTaskContext};use super::person_plan::{plan_person_run, PersonPlanAttempt, PersonPlanRequest};use super::prepare::{refresh_runs, run_ids_of_kind, PreparedRun, RefreshOutcome};@@ -101,6 +102,7 @@ impl PlanningState {struct ClaimState {inflight: InflightTasks,breakers: RunBreakers,+ newest_day: NewestDayIndexes,}pub struct SeederOrchestrator {@@ -160,6 +162,7 @@ impl SeederOrchestrator {let mut claims = ClaimState {inflight: InflightTasks::default(),breakers: RunBreakers::new(self.settings.breaker),+ newest_day: NewestDayIndexes::new(self.settings.newest_day_index_max_entries),};let mut eligible_runs = HashMap::new();let mut reported_runs = HashSet::new();@@ -234,6 +237,7 @@ impl SeederOrchestrator {self.complete_trailing_runs(&eligible_runs).await;claims.breakers.expire_idle(Instant::now());publish_open_breakers(&eligible_runs, &claims.breakers);+ claims.newest_day.release_idle(&eligible_runs);self.fill_claim_slots(&eligible_runs,&mut tasks,@@ -518,6 +522,7 @@ impl SeederOrchestrator {chunk,lease,prepared: prepared.clone(),+ newest_day: claims.newest_day.for_run(&prepared.run),store: self.store.clone(),scanner: self.scanner.clone(),producer: self.producer.clone(),diff --git a/rust/cohort-seeder/src/app/person_execute.rs b/rust/cohort-seeder/src/app/person_execute.rsindex 0b5b82e17218..7fb051c69d5b 100644--- a/rust/cohort-seeder/src/app/person_execute.rs+++ b/rust/cohort-seeder/src/app/person_execute.rs@@ -97,7 +97,7 @@ pub(super) async fn execute_person_chunk(Err(halt) => return resolve_halt(&store, halt, &shutdown, retry_backoff).await,};// PreMark: CAS `scanning`→`produced` — the row is still `scanning` on failure.- let enqueued = match store.mark_produced_streamed(streamed).await {+ let enqueued = match store.mark_produced(streamed).await {Ok(enqueued) => enqueued,Err(halt) => {return resolve_halt(&store, mark_produced_halt(halt), &shutdown, retry_backoff).awaitdiff --git a/rust/cohort-seeder/src/app/prime.rs b/rust/cohort-seeder/src/app/prime.rsindex 08599614c21f..531987fdff49 100644--- a/rust/cohort-seeder/src/app/prime.rs+++ b/rust/cohort-seeder/src/app/prime.rs@@ -28,7 +28,7 @@ use crate::clickhouse::{ResourceError, ScanSkipReason};use crate::observability::metrics::{CHUNKS_POISONED, CLICKHOUSE_RESOURCE_ERRORS, EVENTS_SKIPPED, HOGVM_ERRORS,PERSON_CONDITIONS_SHORTCUT, PERSON_ROWS_PRUNED, PERSON_SCAN_FILTER, PRUNE_REASON_IRRELEVANT,- RUN_BREAKER_TRIPS,+ RUN_BREAKER_TRIPS, TILES_SKIPPED,};use crate::store::runs::RunKind;@@ -60,6 +60,7 @@ pub fn prime_zero_series() {for outcome in ScanFilterOutcome::ALL {counter!(PERSON_SCAN_FILTER, "outcome" => outcome.as_str()).increment(0);}+ counter!(TILES_SKIPPED).increment(0);}#[cfg(test)]@@ -78,7 +79,8 @@ mod tests {// The readings a validation run is gated on: malformed blobs skipped, HogVM aborts on an// unresolvable reference, chunks dead-lettered at the attempt cap, persons pruned as- // irrelevant, and whether the person key filter applied or was refused as too long.+ // irrelevant, whether the person key filter applied or was refused as too long, and+ // behavioral tiles skipped for a newer day.for series in [format!("{EVENTS_SKIPPED}{{reason=\"globals_parse_error\"}} 0"),format!("{HOGVM_ERRORS}{{class=\"unknown_ref\"}} 0"),@@ -90,6 +92,7 @@ mod tests {format!("{PERSON_SCAN_FILTER}{{outcome=\"none\"}} 0"),format!("{PERSON_SCAN_FILTER}{{outcome=\"too_long\"}} 0"),format!("{PERSON_CONDITIONS_SHORTCUT} 0"),+ format!("{TILES_SKIPPED} 0"),] {assert!(rendered.contains(&series),diff --git a/rust/cohort-seeder/src/app/settings.rs b/rust/cohort-seeder/src/app/settings.rsindex 4d23d2b09296..0213b9c5b780 100644--- a/rust/cohort-seeder/src/app/settings.rs+++ b/rust/cohort-seeder/src/app/settings.rs@@ -67,6 +67,7 @@ pub struct OrchestratorSettings {pub(super) plan_caps: PlanCaps,pub(super) producer: ProducerSettings,pub(super) person: Option<PersonSettings>,+ pub(super) newest_day_index_max_entries: NonZeroUsize,}impl OrchestratorSettings {@@ -84,6 +85,7 @@ impl OrchestratorSettings {trailing_day_grace: Duration,producer: ProducerSettings,person: Option<PersonSettings>,+ newest_day_index_max_entries: usize,) -> Result<Self, OrchestratorSettingsError> {if run_poll_interval.is_zero() {return Err(OrchestratorSettingsError::ZeroPollInterval);@@ -106,6 +108,8 @@ impl OrchestratorSettings {let bands_per_day = NonZeroU16::new(bands_per_day).filter(|bands| i16::try_from(bands.get()).is_ok()).ok_or(OrchestratorSettingsError::BandsPerDayOutOfRange)?;+ let newest_day_index_max_entries = NonZeroUsize::new(newest_day_index_max_entries)+ .ok_or(OrchestratorSettingsError::ZeroNewestDayIndexEntries)?;Ok(Self {run_poll_interval,max_concurrent_chunks,@@ -121,6 +125,7 @@ impl OrchestratorSettings {},producer,person,+ newest_day_index_max_entries,})}@@ -203,6 +208,7 @@ impl TryFrom<&Config> for OrchestratorSettings {Duration::from_secs(config.seeder_trailing_day_grace_secs),producer,person,+ config.seeder_newest_day_index_max_entries,)?)}}@@ -235,6 +241,8 @@ pub enum OrchestratorSettingsError {ZeroPersonsPerChunk,#[error("person max concurrent chunks must be greater than zero")]ZeroPersonConcurrency,+ #[error("newest-day index max entries must be greater than zero")]+ ZeroNewestDayIndexEntries,#[error("one person chunk needs ~{projected_secs}s at the configured rate and concurrency, over \the {budget_secs}s scan budget (half of SEEDER_CH_MAX_EXECUTION_TIME_SECS); shrink \@@ -344,6 +352,7 @@ mod tests {Duration::ZERO,producer_settings(),None,+ 1,),OrchestratorSettingsError::PollIntervalExceedsLivenessDeadline,),@@ -361,6 +370,7 @@ mod tests {Duration::ZERO,producer_settings(),None,+ 1,),OrchestratorSettingsError::ZeroPollInterval,),@@ -378,6 +388,7 @@ mod tests {Duration::ZERO,producer_settings(),None,+ 1,),OrchestratorSettingsError::ZeroConcurrency,),@@ -395,6 +406,7 @@ mod tests {Duration::ZERO,producer_settings(),None,+ 1,),OrchestratorSettingsError::LeaseTooShort,),@@ -412,6 +424,7 @@ mod tests {Duration::ZERO,producer_settings(),None,+ 1,),OrchestratorSettingsError::ZeroMaxAttempts,),@@ -429,6 +442,7 @@ mod tests {Duration::ZERO,producer_settings(),None,+ 1,),OrchestratorSettingsError::BandsPerDayOutOfRange,),@@ -446,6 +460,7 @@ mod tests {Duration::ZERO,producer_settings(),None,+ 1,),OrchestratorSettingsError::BandsPerDayOutOfRange,),diff --git a/rust/cohort-seeder/src/config.rs b/rust/cohort-seeder/src/config.rsindex 35b59c026ce6..7ef52da4e524 100644--- a/rust/cohort-seeder/src/config.rs+++ b/rust/cohort-seeder/src/config.rs@@ -213,6 +213,14 @@ pub struct Config {#[envconfig(default = "4000")]pub seeder_max_inflight_tiles: usize,+ /// Entries all newest-day indexes may hold together. A run skips a person's tile for a day+ /// older than one it already scanned, on conditions where every leaf keeps only the newest day,+ /// and its index holds one entry per (person, condition) pair, about 30-55 bytes, so the default+ /// takes up to about 550 MB. Runs take entries first come, first served, and free them when they+ /// end. Past the cap new pairs are not recorded, so their tiles are sent for every day.+ #[envconfig(default = "10000000")]+ pub seeder_newest_day_index_max_entries: usize,+#[envconfig(default = "400")]pub seeder_max_lookback_days: u32,diff --git a/rust/cohort-seeder/src/domain/chunk.rs b/rust/cohort-seeder/src/domain/chunk.rsindex 5e62d23c8915..c2394cffbbce 100644--- a/rust/cohort-seeder/src/domain/chunk.rs+++ b/rust/cohort-seeder/src/domain/chunk.rs@@ -236,8 +236,9 @@ impl ClaimedChunk {}}-/// A person chunk whose seeds were interleaved scan→enqueue — never buffered — awaiting the-/// `scanning`→`produced` CAS.+/// A chunk whose seeds are all enqueued, awaiting the `scanning`→`produced` CAS. Only the count and+/// the scan volume remain: a person chunk streams its seeds as it scans, and a behavioral chunk+/// frees its tiles once they are enqueued.#[derive(Debug)]pub struct StreamedChunk {spec: ChunkSpec,@@ -286,6 +287,17 @@ impl ScannedChunk {pub const fn volume(&self) -> ScanVolume {self.volume}++ /// Pure transition once every tile is enqueued or skipped, with `seeds_produced` counting the+ /// enqueued ones. Dropping the vector here frees it before the mark and the ack wait. No DB+ /// effect.+ pub fn into_streamed(self, seeds_produced: u64) -> StreamedChunk {+ StreamedChunk {+ spec: self.spec,+ seeds_produced,+ volume: self.volume,+ }+ }}impl std::fmt::Debug for ScannedChunk {diff --git a/rust/cohort-seeder/src/domain/mod.rs b/rust/cohort-seeder/src/domain/mod.rsindex b9f9e2fd0be3..b32175b7bc88 100644--- a/rust/cohort-seeder/src/domain/mod.rs+++ b/rust/cohort-seeder/src/domain/mod.rs@@ -18,6 +18,7 @@ pub mod condition;pub mod condition_analysis;pub mod ids;pub mod ledger;+pub mod newest_day;pub mod partition;pub mod person;pub mod person_analysis;@@ -62,6 +63,7 @@ pub use ids::{ScannedAtMs, UtcMillis, UtcMsRange, UtcRangeError,};pub use ledger::{MarkerFold, MarkerLedger, SettledVerdict, WithheldCohort};+pub use newest_day::{ConditionSlot, NewestDayConditions};pub use partition::{SeedPartition, SeedPartitionCountError, SeedPartitions};pub use person::{person_chunk_sentinel_day, tile_ranges, EvaluatedConditions, PersonChunkSpec,diff --git a/rust/cohort-seeder/src/domain/newest_day.rs b/rust/cohort-seeder/src/domain/newest_day.rsnew file mode 100644index 000000000000..037934202272--- /dev/null+++ b/rust/cohort-seeder/src/domain/newest_day.rs@@ -0,0 +1,175 @@+//! Domain layer: the conditions of a run on which only each person's newest scanned day changes+//! stage-1 state, so an older day's tile can be skipped. Depends on `condition`, `ids`, and+//! `cohort-core`.++use std::collections::{BTreeSet, HashMap};++use cohort_core::filters::{LeafStateMeta, TeamFilters};+use cohort_core::{EvictionWindow, StateVariant};++use super::condition::PinnedCondition;+use super::ids::ConditionHash;++/// A condition on which every leaf of the run keeps only its newest matching day. Only+/// [`NewestDayConditions::slot`] hands one out, so an index can only key a qualifying condition.+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]+pub struct ConditionSlot(u32);++/// The qualifying conditions of one run, each with a dense slot.+#[derive(Debug)]+pub struct NewestDayConditions(HashMap<ConditionHash, ConditionSlot>);++impl NewestDayConditions {+ /// The processor routes a tile to every leaf on its hash in the team's live catalog. A hash+ /// qualifies only when every leaf on it in this run's frozen catalog does. A live leaf outside+ /// the run gets every day from its own run.+ pub fn of(conditions: &[PinnedCondition], filters: &TeamFilters) -> Self {+ let hashes: BTreeSet<ConditionHash> =+ conditions.iter().map(|condition| condition.hash).collect();+ let qualifying = hashes+ .into_iter()+ .filter(|hash| every_leaf_keeps_only_newest_day(*hash, filters));+ Self(qualifying.zip((0..).map(ConditionSlot)).collect())+ }++ pub fn slot(&self, hash: ConditionHash) -> Option<ConditionSlot> {+ self.0.get(&hash).copied()+ }+}++fn every_leaf_keeps_only_newest_day(hash: ConditionHash, filters: &TeamFilters) -> bool {+ filters+ .by_condition_to_lsk+ .get(&hash.as_bytes())+ .is_some_and(|lsks| {+ !lsks.is_empty()+ && lsks+ .iter()+ .all(|lsk| filters.by_lsk.get(lsk).is_some_and(keeps_only_newest_day))+ })+}++/// Whether a leaf's merged state depends on the newest seeded day alone.+///+/// `merge_single` in `cohort-stream-processor/src/workers/seed_path.rs` is the source of truth, and+/// this rule must change with it. It keeps `max(last_event_at, end of the tile's day)`, so an older+/// day's tile leaves the state unchanged. Its drops are monotone in the day: when the newest tile+/// falls before `from_day` or outside its window, every older tile does too. An explicit upper+/// bound breaks that: the newest day can fall after `to_day` and drop, and the leaf then needs the+/// older in-range day. Count variants sum every day.+fn keeps_only_newest_day(meta: &LeafStateMeta) -> bool {+ match (meta.variant, meta.window) {+ (+ StateVariant::BehavioralSingle,+ Some(+ EvictionWindow::RelativeDays { .. }+ | EvictionWindow::RelativeSeconds { .. }+ | EvictionWindow::Explicit { to_day: None, .. },+ ),+ ) => true,+ (+ StateVariant::BehavioralSingle,+ Some(EvictionWindow::Explicit {+ to_day: Some(_), ..+ })+ | None,+ ) => false,+ (+ StateVariant::BehavioralDailyBuckets+ | StateVariant::BehavioralCompressedHistory+ | StateVariant::PersonProperty,+ _,+ ) => false,+ }+}++#[cfg(test)]+mod tests {+ use chrono_tz::UTC;+ use cohort_core::filters::{CohortId, TeamFiltersBuilder, TeamId};+ use serde_json::{json, Value};++ use super::*;+ use crate::domain::Lookback;++ const HASH: &str = "aaaaaaaaaaaaaaaa";+ const EVENT: &str = "$pageview";++ fn leaf(shape: Value) -> Value {+ let mut leaf = json!({+ "type": "behavioral",+ "key": EVENT,+ "conditionHash": HASH,+ "bytecode": ["_H", 1, 32, EVENT, 32, "event", 1, 1, 11],+ });+ leaf.as_object_mut()+ .unwrap()+ .extend(shape.as_object().unwrap().clone());+ leaf+ }++ fn sliding(days: u32) -> Value {+ leaf(json!({ "value": "performed_event", "time_value": days, "time_interval": "day" }))+ }++ /// Each leaf is its own cohort, because the processor routes a tile to every cohort's leaves.+ fn qualifies(leaves: &[Value]) -> bool {+ let mut builder = TeamFiltersBuilder::default();+ for (cohort, leaf) in (1..).zip(leaves) {+ builder+ .add_cohort(+ CohortId(cohort),+ TeamId(2),+ &json!({ "properties": { "type": "AND", "values": [leaf] } }),+ )+ .unwrap();+ }+ let filters = builder.freeze(UTC);+ let hash = ConditionHash::parse(HASH).unwrap();+ let conditions = [PinnedCondition {+ cohort_id: CohortId(1),+ hash,+ event_name: EVENT.to_owned(),+ lookback: Lookback::SlidingDays(7),+ }];+ NewestDayConditions::of(&conditions, &filters)+ .slot(hash)+ .is_some()+ }++ #[test]+ fn a_hash_qualifies_only_when_every_leaf_on_it_keeps_only_the_newest_day() {+ let count = leaf(json!({+ "value": "performed_event_multiple",+ "time_value": 14,+ "time_interval": "day",+ "operator": "gte",+ "operator_value": 2,+ }));+ let bounded = leaf(json!({+ "value": "performed_event",+ "explicit_datetime": "2026-01-01",+ "explicit_datetime_to": "2026-02-01",+ }));+ let open_ended = leaf(json!({+ "value": "performed_event",+ "explicit_datetime": "2026-01-01",+ }));+ for (case, leaves, expected) in [+ ("sliding leaves only", vec![sliding(7), sliding(30)], true),+ ("plus a count leaf", vec![sliding(7), count], false),+ (+ "plus an explicit upper bound",+ vec![sliding(7), bounded],+ false,+ ),+ (+ "plus an explicit range with no upper bound",+ vec![sliding(7), open_ended],+ true,+ ),+ ] {+ assert_eq!(qualifies(&leaves), expected, "{case}");+ }+ }+}diff --git a/rust/cohort-seeder/src/observability/metrics.rs b/rust/cohort-seeder/src/observability/metrics.rsindex 613d9651eb03..1b724df3bd76 100644--- a/rust/cohort-seeder/src/observability/metrics.rs+++ b/rust/cohort-seeder/src/observability/metrics.rs@@ -131,6 +131,13 @@ pub const CONDITIONS_EVALUATED: &str = "seeder_conditions_evaluated_total";pub const HOGVM_ERRORS: &str = "seeder_hogvm_errors_total";pub const AGGREGATE_ENTRIES: &str = "seeder_aggregate_entries";pub const TILES_PRODUCED: &str = "seeder_tiles_produced_total";+/// Behavioral tiles not sent because their run already recorded a newer day for the same person+/// and condition (counter).+pub const TILES_SKIPPED: &str = "seeder_tiles_skipped_total";+/// (person, condition) pairs all newest-day indexes hold together (gauge). At+/// `SEEDER_NEWEST_DAY_INDEX_MAX_ENTRIES` the budget is full, and tiles for new pairs are sent for+/// every day.+pub const NEWEST_DAY_INDEX_ENTRIES: &str = "seeder_newest_day_index_entries";pub const TILE_PRODUCE_QUEUE_FULL: &str = "seeder_tile_produce_queue_full_total";pub const TILE_PRODUCE_ERRORS: &str = "seeder_tile_produce_errors_total";pub const PRODUCE_ACK_SECONDS: &str = "seeder_produce_ack_seconds";diff --git a/rust/cohort-seeder/src/store/chunks.rs b/rust/cohort-seeder/src/store/chunks.rsindex f1f57d7a0d5c..b12153990315 100644--- a/rust/cohort-seeder/src/store/chunks.rs+++ b/rust/cohort-seeder/src/store/chunks.rs@@ -25,7 +25,7 @@ use crate::domain::{bands_for_day, person_chunk_sentinel_day, AttemptCount, Band, BandSpec, BandSpecError, ChunkId,ChunkLease, ChunkSpec, ChunkStatus, ClaimEpoch, ClaimKind, ClaimedChunk, DaySchedule,EnqueuedChunk, Halted, PersonRange, PersonRangeError, PlannedDay, ProduceHwms, ProducedChunk,- RunId, SChunkMs, ScanVolume, ScannedChunk, StreamedChunk, UtcMillis,+ RunId, SChunkMs, ScanVolume, StreamedChunk, UtcMillis,};use super::lease::LeaseHandle;@@ -259,6 +259,10 @@ impl PgChunkStore {-- A trailing day is scanned only once it has ended and its run is `trailing`.-- Claimed while the run still seeds, it would lose its lease when the-- reconciling CAS fires, because the heartbeat joins the same run statuses.+ -- The newest-day index depends on this too. Readiness does not wait for a+ -- trailing day. A trailing day scanned while a historical chunk still sends+ -- would make that chunk skip its tiles, and readiness could finish before the+ -- trailing day's tile lands.AND (c.claimable_after IS NULLOR (c.claimable_after <= now() AND r.status = 'trailing'))AND ((c.status = $7 AND c.attempts < $4)@@ -392,9 +396,9 @@ impl PgChunkStore {Ok(rows.into_iter().map(ExhaustedRun::from).collect())}- /// The person path's mark: seeds were already streamed to Kafka during the scan, so only the- /// count crosses the `scanning`→`produced` CAS.- pub async fn mark_produced_streamed(+ /// The mark for both paths: every seed is already enqueued, so only the count sent crosses the+ /// `scanning`→`produced` CAS.+ pub async fn mark_produced(&self,chunk: StreamedChunk,) -> Result<EnqueuedChunk, Halted<StreamedChunk, ChunkStoreError>> {@@ -410,30 +414,6 @@ impl PgChunkStore {Ok(EnqueuedChunk::new(spec, seeds_produced))}- pub async fn mark_produced(- &self,- chunk: ScannedChunk,- ) -> Result<EnqueuedChunk, Halted<ScannedChunk, ChunkStoreError>> {- let spec = chunk.spec();- let tiles_produced = match u64::try_from(chunk.tiles().len()) {- Ok(count) => count,- Err(_) => {- return Err(Halted::failed(- chunk,- ChunkStoreError::TilesProducedOutOfRange(u64::MAX),- ));- }- };- let volume = chunk.volume();- if let Err(error) = self- .mark_produced_raw(spec.lease, tiles_produced, volume)- .await- {- return Err(Halted::failed(chunk, error));- }- Ok(EnqueuedChunk::new(spec, tiles_produced))- }-pub async fn confirm(&self,chunk: ProducedChunk,diff --git a/rust/cohort-stream-processor/src/workers/seed_path.rs b/rust/cohort-stream-processor/src/workers/seed_path.rsindex 03e227dee8d3..2c75f1f6a424 100644--- a/rust/cohort-stream-processor/src/workers/seed_path.rs+++ b/rust/cohort-stream-processor/src/workers/seed_path.rs@@ -1464,6 +1464,32 @@ mod tests {prop_assert!(!replay_merged, "replay merged again");}}++ /// The seeder sends a single leaf's tile only for each person's newest day+ /// (`cohort-seeder/src/domain/newest_day.rs`). That is sound only while the newest tile+ /// alone reaches the state of every tile in any order. If this fails, change that rule too.+ #[test]+ fn single_state_is_the_newest_tiles_alone(+ tiles in prop::collection::vec((0i32..=12, 1u32..=5), 1..10),+ window in prop_oneof![+ (1u32..=14).prop_map(|days| EvictionWindow::RelativeDays { days }),+ (1i64..=172_800).prop_map(|seconds| EvictionWindow::RelativeSeconds { seconds }),+ prop::option::of(0i32..=12).prop_map(|back| EvictionWindow::Explicit {+ from_day: back.map(|back| now_day() - back),+ to_day: None,+ }),+ ],+ ) {+ let meta = single_meta(window);+ let newest = *tiles+ .iter()+ .min_by_key(|(offset, _)| *offset)+ .expect("the strategy draws at least one tile");+ prop_assert_eq!(+ apply_all(&meta, &tiles, now_day(), NOW_MS),+ apply_all(&meta, &[newest], now_day(), NOW_MS),+ );+ }}// ---- live-equivalence: tile (d, N) on empty ≡ N same-day live events ----