Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(cohorts): skip seeder tiles that a newer day already covers (#125076)

Matheus Batistad4fd2eead7d7
diff --git a/docs/internal/realtime-cohorts/invariants.md b/docs/internal/realtime-cohorts/invariants.md
index f160af507d13..9c2a545a2d0c 100644
--- a/docs/internal/realtime-cohorts/invariants.md
+++ b/docs/internal/realtime-cohorts/invariants.md
@@ -206,6 +206,8 @@ Most of these are not enforced by a single test, and a violation usually shows u
Every outcome write is fenced by the dispatch epoch, and `reconcile_observed_at` is written last.
- Limit: the proof covers planned days, not persons.
Raising the seeder's band count while a run is seeding leaves persons unscanned, and every chunk still confirms.
+- Limit: a chunk skips a person's tile on a condition where only the newest day counts, once a newer day's chunk of the run has scanned that person.
+ If the newer chunk fails and the person's events on its day are deleted before the retry, neither day sends the person, and every chunk still confirms.
- See [completion and readiness](completion-and-readiness.md).
**26. A readiness stamp vouches only for the definition its run replayed.**
diff --git a/docs/internal/realtime-cohorts/seeder.md b/docs/internal/realtime-cohorts/seeder.md
index 0b70669e3cd7..8c46bbba98a5 100644
--- a/docs/internal/realtime-cohorts/seeder.md
+++ b/docs/internal/realtime-cohorts/seeder.md
@@ -155,7 +155,14 @@ At the end of the chunk, each count becomes one **day tile**:
Tiles are keyed `"{team_id}:{person_id}"`, so each lands on the partition of the worker that owns the person.
A shared rate limiter paces production, and a cap bounds the tiles in flight.
-The chunk then moves to `produced`, which means every tile was handed to the producer, waits for every acknowledgment, and moves to `confirmed`.
+On a condition where every leaf of the run keeps only the newest matching day, as a `performed_event` leaf does, an older tile changes nothing.
+Each run keeps an in-memory index of the newest day it has scanned for each person on such a condition.
+A chunk records its days right after its scan, and skips a tile when the index holds a strictly newer day.
+Losing the index on a restart only sends more tiles.
+`SEEDER_NEWEST_DAY_INDEX_MAX_ENTRIES` caps the entries of all indexes together, and past the cap a new pair sends every day.
+`seeder_tiles_skipped_total` counts the skipped tiles, and `seeder_newest_day_index_entries` shows how full the cap is.
+
+The chunk then moves to `produced`, which means every tile it sends was handed to the producer, waits for every acknowledgment, and moves to `confirmed`.
### Chunk states and retries
diff --git a/rust/Cargo.lock b/rust/Cargo.lock
index 82126aeade9c..33bc36d55957 100644
--- a/rust/Cargo.lock
+++ b/rust/Cargo.lock
@@ -2815,6 +2815,7 @@ dependencies = [
"common-kafka",
"common-liveness",
"common-types",
+ "dashmap 6.1.0",
"envconfig",
"futures",
"governor",
diff --git a/rust/cohort-seeder/Cargo.toml b/rust/cohort-seeder/Cargo.toml
index 3f85ad4ec7b3..0edb4e687aae 100644
--- a/rust/cohort-seeder/Cargo.toml
+++ b/rust/cohort-seeder/Cargo.toml
@@ -30,6 +30,7 @@ common-database = { path = "../common/database" }
common-kafka = { path = "../common/kafka" }
common-liveness = { path = "../common/liveness" }
common-types = { path = "../common/types" }
+dashmap = { workspace = true }
envconfig = { workspace = true }
futures = { workspace = true }
governor = { workspace = true }
diff --git a/rust/cohort-seeder/src/app/deliver.rs b/rust/cohort-seeder/src/app/deliver.rs
index 37f1ae4f2e27..894c58f695fb 100644
--- a/rust/cohort-seeder/src/app/deliver.rs
+++ b/rust/cohort-seeder/src/app/deliver.rs
@@ -1,9 +1,9 @@
//! App layer, Kafka produce sequencing: the two-phase enqueue/await split. Depends on `domain`,
//! `kafka`, `store`, and `app::settings`; never imported by a lower layer.
//!
-//! The sequence `enqueue_tiles → store.mark_produced → await_deliveries` (composed by `execute`) is
-//! byte-equivalent to the former monolithic produce: enqueue every tile honoring the in-flight bound
-//! and draining overflow acks before the mark, then drain the remaining acks after it, folding all
+//! The sequence `enqueue_tiles → store.mark_produced → await_deliveries` (composed by
+//! `execute`) enqueues every tile its run has no newer day for, honoring the in-flight bound and
+//! draining overflow acks before the mark, then drains the remaining acks after it, folding all
//! delivery high-water marks into one accumulator carried across the mark by [`InFlight`].
use std::collections::VecDeque;
@@ -14,7 +14,9 @@ use rdkafka::error::KafkaError;
use rdkafka::producer::DeliveryFuture;
use tokio_util::sync::CancellationToken;
-use crate::domain::{CancelCause, EnqueuedChunk, Halted, ProduceHwms, ProducedChunk, ScannedChunk};
+use crate::domain::{
+ CancelCause, EnqueuedChunk, Halted, ProduceHwms, ProducedChunk, ScannedChunk, StreamedChunk,
+};
use crate::kafka::pacing::TilePacer;
use crate::kafka::producer::{EnqueueError, SeedTileProducer};
use crate::observability::metrics::{
@@ -22,6 +24,7 @@ use crate::observability::metrics::{
};
use crate::store::chunks::ChunkStoreError;
+use super::newest_day::NewestDayIndex;
use super::settings::ProducerSettings;
/// The ceiling the queue-full backoff doubles up to; also clamps [`ProducerSettings`] at construction.
@@ -96,19 +99,29 @@ pub(super) async fn enqueue_one(
Ok(())
}
-/// Phase one (pre-mark): pace and enqueue every tile honoring the in-flight bound, draining overflow
-/// acks into the HWMs. Returns the still-`scanning` chunk (for the store's mark) plus the in-flight
-/// deliveries. Every error hands the [`ScannedChunk`] back so the caller can release or fail it.
+/// Phase one (pre-mark): pace and enqueue every tile `newest_day` has no newer day for, honoring the
+/// in-flight bound and draining overflow acks into the HWMs. Returns the still-`scanning` chunk (for
+/// the store's mark) with the count sent, plus the in-flight deliveries. Every error hands the
+/// [`ScannedChunk`] back so the caller can release or fail it.
pub(super) async fn enqueue_tiles(
producer: &SeedTileProducer,
chunk: ScannedChunk,
+ newest_day: &NewestDayIndex,
pacer: &TilePacer,
settings: ProducerSettings,
lease_cancel: &CancellationToken,
shutdown: &CancellationToken,
-) -> Result<(ScannedChunk, InFlight), Halted<ScannedChunk, ProduceError>> {
+) -> Result<(StreamedChunk, InFlight), Halted<ScannedChunk, ProduceError>> {
let mut inflight = InFlight::new(settings.max_inflight.get());
+ let mut sent = 0;
for tile in chunk.tiles() {
+ // Checked before the pacer, so a newer day another chunk records while this one waits
+ // still skips this chunk's remaining tiles for that person. A skipped tile takes no pacer
+ // token, so the task budget makes it yield.
+ if newest_day.has_newer_day(tile) {
+ tokio::task::coop::consume_budget().await;
+ continue;
+ }
if let Err(stop) = enqueue_one(
|| producer.enqueue(tile),
&mut inflight,
@@ -121,8 +134,9 @@ pub(super) async fn enqueue_tiles(
{
return Err(stop.into_halted(chunk));
}
+ sent += 1;
}
- Ok((chunk, inflight))
+ Ok((chunk.into_streamed(sent), inflight))
}
/// Phase two (post-mark): drain the remaining acks into the HWMs and fold them onto the marked
diff --git a/rust/cohort-seeder/src/app/execute.rs b/rust/cohort-seeder/src/app/execute.rs
index f855774ba563..a263108a18b3 100644
--- a/rust/cohort-seeder/src/app/execute.rs
+++ b/rust/cohort-seeder/src/app/execute.rs
@@ -1,10 +1,11 @@
-//! App layer: one chunk's `scan → enqueue → mark → await → confirm` pipeline and the single recovery
-//! resolver. Depends on `clickhouse`, `kafka`, `store`, `domain`, and its `app` siblings.
+//! App layer: one chunk's `scan → record → enqueue → mark → await → confirm` pipeline and the single
+//! recovery resolver. Depends on `clickhouse`, `kafka`, `store`, `domain`, and its `app` siblings.
//!
-//! The store's `mark_produced` sits between the two deliver phases: deliver owns Kafka, execute owns
-//! the store gating. Every halt funnels through [`resolve_halt`], which applies the exact recovery
-//! matrix — a pre-mark halt during shutdown unclaims (no attempt spent); everything else fails for
-//! retry — encoded once in [`FailureDisposition::resolve`] and pinned by its unit test.
+//! The store's `mark_produced` sits between the two deliver phases: deliver owns Kafka,
+//! execute owns the store gating. Every halt funnels through [`resolve_halt`], which applies the
+//! exact recovery matrix — a pre-mark halt during shutdown unclaims (no attempt spent); everything
+//! else fails for retry — encoded once in [`FailureDisposition::resolve`] and pinned by its unit
+//! test.
use std::sync::Arc;
use std::time::Duration;
@@ -22,7 +23,7 @@ use crate::domain::{
use crate::kafka::pacing::TilePacer;
use crate::kafka::producer::SeedTileProducer;
use crate::observability::metrics::{
- CHUNKS_CONFIRMED, CHUNKS_FAILED, CLICKHOUSE_RESOURCE_ERRORS, TILES_PRODUCED,
+ CHUNKS_CONFIRMED, CHUNKS_FAILED, CLICKHOUSE_RESOURCE_ERRORS, TILES_PRODUCED, TILES_SKIPPED,
};
use crate::store::chunks::{ChunkStoreError, PgChunkStore};
use crate::store::lease::LeaseHandle;
@@ -30,6 +31,7 @@ use crate::store::runs::RunKind;
use crate::store::RenderedError;
use super::deliver::{self, ProduceError};
+use super::newest_day::NewestDayIndex;
use super::person_execute::PersonChunkStats;
use super::prepare::PreparedBehavioral;
use super::settings::ProducerSettings;
@@ -39,6 +41,7 @@ pub(super) struct ChunkTaskContext {
pub(super) chunk: ClaimedChunk,
pub(super) lease: LeaseHandle,
pub(super) prepared: Arc<PreparedBehavioral>,
+ pub(super) newest_day: Arc<NewestDayIndex>,
pub(super) store: PgChunkStore,
pub(super) scanner: ChunkScanner,
pub(super) producer: SeedTileProducer,
@@ -55,6 +58,7 @@ pub(super) async fn execute_chunk(
chunk,
lease,
prepared,
+ newest_day,
store,
scanner,
producer,
@@ -78,10 +82,16 @@ pub(super) async fn execute_chunk(
Ok(scanned) => scanned,
Err(halt) => return resolve_halt(&store, halt, &shutdown, retry_backoff).await,
};
- // PreMark: pace and enqueue every tile, holding the in-flight deliveries.
- let (scanned, inflight) = match deliver::enqueue_tiles(
+ // PreMark: record this scan's days before enqueuing, so an older day's chunk skips its tiles.
+ // A large chunk takes seconds, so the worker's other tasks move to another thread meanwhile.
+ tokio::task::block_in_place(|| newest_day.record(scanned.tiles()));
+ let scanned_tiles = scanned.tiles().len() as u64;
+ // PreMark: pace and enqueue every tile not superseded by a newer day, holding the in-flight
+ // deliveries.
+ let (streamed, inflight) = match deliver::enqueue_tiles(
&producer,
scanned,
+ &newest_day,
&pacer,
producer_settings,
&lease_cancel,
@@ -93,7 +103,7 @@ pub(super) async fn execute_chunk(
Err(halt) => return resolve_halt(&store, halt, &shutdown, retry_backoff).await,
};
// PreMark: CAS `scanning`→`produced` — the row is still `scanning` on failure.
- let enqueued = match store.mark_produced(scanned).await {
+ let enqueued = match store.mark_produced(streamed).await {
Ok(enqueued) => enqueued,
Err(halt) => {
return resolve_halt(&store, mark_produced_halt(halt), &shutdown, retry_backoff).await
@@ -103,6 +113,7 @@ pub(super) async fn execute_chunk(
let produced = match deliver::await_deliveries(enqueued, inflight, &lease_cancel).await {
Ok(produced) => {
counter!(TILES_PRODUCED).increment(produced.tiles_produced());
+ counter!(TILES_SKIPPED).increment(scanned_tiles - produced.tiles_produced());
produced
}
Err(halt) => return resolve_halt(&store, halt, &shutdown, retry_backoff).await,
diff --git a/rust/cohort-seeder/src/app/mod.rs b/rust/cohort-seeder/src/app/mod.rs
index 62cea176bd8d..fd542f9197f2 100644
--- a/rust/cohort-seeder/src/app/mod.rs
+++ b/rust/cohort-seeder/src/app/mod.rs
@@ -5,6 +5,7 @@ mod breaker;
pub mod completion;
mod deliver;
mod execute;
+mod newest_day;
pub mod observe;
mod orchestrator;
mod person_execute;
diff --git a/rust/cohort-seeder/src/app/newest_day.rs b/rust/cohort-seeder/src/app/newest_day.rs
new file mode 100644
index 000000000000..eab30fe4e743
--- /dev/null
+++ b/rust/cohort-seeder/src/app/newest_day.rs
@@ -0,0 +1,289 @@
+//! App layer: the per-run index of the newest day scanned for each (person, condition) pair, which
+//! lets an older day's chunk skip the tiles the processor would merge away. Depends on `domain` and
+//! `app::prepare`.
+//!
+//! The state is process-local, like the run breakers. Losing an index, on a restart or when an idle
+//! run drops out of one refresh's eligible set, only sends more tiles.
+
+use std::collections::HashMap;
+use std::num::NonZeroUsize;
+use std::sync::atomic::{AtomicUsize, Ordering};
+use std::sync::Arc;
+
+use dashmap::mapref::entry::Entry;
+use dashmap::DashMap;
+use metrics::gauge;
+use uuid::Uuid;
+
+use crate::domain::{ConditionSlot, DayIdx, NewestDayConditions, PinnedRun, RunId, SeedTile};
+use crate::observability::metrics::NEWEST_DAY_INDEX_ENTRIES;
+
+use super::prepare::PreparedRun;
+
+type Key = (Uuid, ConditionSlot);
+
+/// The newest-day indexes of the runs this process seeds, and the entry budget they share.
+pub(super) struct NewestDayIndexes {
+ budget: IndexBudget,
+ by_run: HashMap<RunId, Arc<NewestDayIndex>>,
+}
+
+impl NewestDayIndexes {
+ pub(super) fn new(max_entries: NonZeroUsize) -> Self {
+ Self {
+ budget: IndexBudget {
+ used: Arc::default(),
+ max: max_entries,
+ },
+ by_run: HashMap::new(),
+ }
+ }
+
+ /// Built on the run's first claim. A run's frozen catalog only loses leaves after the run is
+ /// created, so a `NewestDayConditions` built then can miss a skip but never allow a wrong one.
+ pub(super) fn for_run(&mut self, run: &PinnedRun) -> Arc<NewestDayIndex> {
+ let budget = &self.budget;
+ self.by_run
+ .entry(run.run_id)
+ .or_insert_with(|| {
+ let conditions = NewestDayConditions::of(&run.conditions, &run.filters);
+ Arc::new(NewestDayIndex::new(conditions, budget.clone()))
+ })
+ .clone()
+ }
+
+ /// Keeps an index while a chunk task holds it, because a discovery error yields an empty
+ /// eligible set, and a second index for the run would split its days and its budget. The gauge
+ /// is set only here, so two chunk tasks can never set it out of order.
+ pub(super) fn release_idle(&mut self, eligible: &HashMap<RunId, PreparedRun>) {
+ self.by_run
+ .retain(|run_id, index| eligible.contains_key(run_id) || Arc::strong_count(index) > 1);
+ gauge!(NEWEST_DAY_INDEX_ENTRIES).set(self.budget.used.load(Ordering::Relaxed) as f64);
+ }
+}
+
+/// One run's newest recorded day for each person on each qualifying condition.
+///
+/// It must stay per run: an index shared across runs would let one run's readiness depend on a
+/// tile that only another run sends.
+pub(super) struct NewestDayIndex {
+ conditions: NewestDayConditions,
+ newest: DashMap<Key, DayIdx>,
+ budget: IndexBudget,
+}
+
+impl NewestDayIndex {
+ fn new(conditions: NewestDayConditions, budget: IndexBudget) -> Self {
+ Self {
+ conditions,
+ newest: DashMap::new(),
+ budget,
+ }
+ }
+
+ /// Max-merges each qualifying tile's day. A new pair takes one budget entry, and past the
+ /// budget it is not recorded, so its older days are sent.
+ pub(super) fn record(&self, tiles: &[SeedTile]) {
+ for tile in tiles {
+ let Some(key) = self.key(tile) else {
+ continue;
+ };
+ match self.newest.entry(key) {
+ Entry::Occupied(mut newest) => {
+ let newest = newest.get_mut();
+ *newest = (*newest).max(tile.day_idx());
+ }
+ Entry::Vacant(vacant) => {
+ if self.budget.try_take() {
+ vacant.insert(tile.day_idx());
+ }
+ }
+ }
+ }
+ }
+
+ /// Only a strictly newer day counts, so a retried chunk never skips its own tiles, and the
+ /// chunk holding a pair's newest day always sends it, unless events deleted before its retry
+ /// remove the pair.
+ pub(super) fn has_newer_day(&self, tile: &SeedTile) -> bool {
+ self.key(tile).is_some_and(|key| {
+ self.newest
+ .get(&key)
+ .is_some_and(|newest| *newest > tile.day_idx())
+ })
+ }
+
+ fn key(&self, tile: &SeedTile) -> Option<Key> {
+ self.conditions
+ .slot(tile.condition_hash())
+ .map(|slot| (tile.person_id(), slot))
+ }
+}
+
+/// Gives this index's entries back, so a finished run frees its share of the budget.
+impl Drop for NewestDayIndex {
+ fn drop(&mut self) {
+ self.budget
+ .used
+ .fetch_sub(self.newest.len(), Ordering::Relaxed);
+ }
+}
+
+/// The entries all indexes may hold together. Every entry an index holds took one from here.
+#[derive(Clone)]
+struct IndexBudget {
+ used: Arc<AtomicUsize>,
+ max: NonZeroUsize,
+}
+
+impl IndexBudget {
+ fn try_take(&self) -> bool {
+ self.used
+ .fetch_update(Ordering::Relaxed, Ordering::Relaxed, |used| {
+ (used < self.max.get()).then_some(used + 1)
+ })
+ .is_ok()
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use std::num::NonZeroU32;
+
+ use chrono_tz::UTC;
+ use cohort_core::filters::{CohortId, TeamId};
+
+ use super::*;
+ use crate::domain::row_filter::test_catalog::{catalog, event_is, hash, Leaf};
+ use crate::domain::{
+ Boundary, ClaimEpoch, EventNameSet, Lookback, PinnedCondition, SChunkMs, TriggerKind,
+ UtcMillis,
+ };
+
+ const SLIDING: &str = "aaaaaaaaaaaaaaaa";
+ /// Pinned, but absent from the frozen catalog, so it does not qualify.
+ const UNCATALOGUED: &str = "bbbbbbbbbbbbbbbb";
+
+ fn budget(max: usize) -> IndexBudget {
+ IndexBudget {
+ used: Arc::default(),
+ max: NonZeroUsize::new(max).unwrap(),
+ }
+ }
+
+ fn run(id: u128) -> PinnedRun {
+ PinnedRun {
+ run_id: RunId(Uuid::from_u128(id)),
+ team_id: TeamId(2),
+ trigger: TriggerKind::TeamEnablement,
+ tz: UTC,
+ boundary: Boundary::new(UtcMillis::new(0), UTC),
+ conditions: [SLIDING, UNCATALOGUED]
+ .map(|value| PinnedCondition {
+ cohort_id: CohortId(1),
+ hash: hash(value),
+ event_name: "$pageview".to_owned(),
+ lookback: Lookback::SlidingDays(7),
+ })
+ .to_vec(),
+ event_names: EventNameSet::new(["$pageview".to_owned()]),
+ filters: catalog(&[Leaf {
+ cohort: 1,
+ key: "$pageview",
+ hash: SLIDING,
+ body: event_is("$pageview"),
+ }]),
+ }
+ }
+
+ fn index(budget: &IndexBudget) -> NewestDayIndex {
+ let run = run(0);
+ let conditions = NewestDayConditions::of(&run.conditions, &run.filters);
+ NewestDayIndex::new(conditions, budget.clone())
+ }
+
+ fn tile(person: u128, condition: &str, day: DayIdx) -> SeedTile {
+ SeedTile::new(
+ TeamId(2),
+ Uuid::from_u128(person),
+ hash(condition),
+ NonZeroU32::MIN,
+ day,
+ SChunkMs(0),
+ RunId(Uuid::nil()),
+ ClaimEpoch(1),
+ )
+ }
+
+ #[test]
+ fn only_a_strictly_newer_day_for_the_same_person_and_condition_skips_a_tile() {
+ let index = index(&budget(10));
+ index.record(&[tile(1, SLIDING, 10), tile(1, UNCATALOGUED, 10)]);
+ index.record(&[tile(1, SLIDING, 9)]);
+
+ for (case, tile, skipped) in [
+ ("the recorded day", tile(1, SLIDING, 10), false),
+ ("an older day", tile(1, SLIDING, 9), true),
+ ("another person's older day", tile(2, SLIDING, 9), false),
+ (
+ "an older day on a non-qualifying condition",
+ tile(1, UNCATALOGUED, 9),
+ false,
+ ),
+ ] {
+ assert_eq!(index.has_newer_day(&tile), skipped, "{case}");
+ }
+ }
+
+ #[test]
+ fn a_full_budget_sends_new_pairs_until_another_index_gives_its_entries_back() {
+ let budget = budget(1);
+ let first = index(&budget);
+ first.record(&[tile(1, SLIDING, 10), tile(2, SLIDING, 10)]);
+ assert!(
+ !first.has_newer_day(&tile(2, SLIDING, 9)),
+ "a pair past the budget was recorded"
+ );
+ first.record(&[tile(1, SLIDING, 11)]);
+ assert!(
+ first.has_newer_day(&tile(1, SLIDING, 10)),
+ "a recorded pair stopped updating at a full budget"
+ );
+
+ drop(first);
+ let second = index(&budget);
+ second.record(&[tile(2, SLIDING, 10)]);
+ assert!(
+ second.has_newer_day(&tile(2, SLIDING, 9)),
+ "a dropped index kept its budget"
+ );
+ }
+
+ #[test]
+ fn a_run_keeps_one_index_until_it_drops_out_and_no_chunk_task_holds_it() {
+ let mut indexes = NewestDayIndexes::new(NonZeroUsize::MIN);
+ let (held, idle) = (run(1), run(2));
+ let task = indexes.for_run(&held);
+ task.record(&[tile(1, SLIDING, 10)]);
+ assert!(
+ indexes.for_run(&held).has_newer_day(&tile(1, SLIDING, 9)),
+ "a run's next claim got a new index"
+ );
+ let idle_index = Arc::downgrade(&indexes.for_run(&idle));
+ assert!(
+ !indexes.for_run(&idle).has_newer_day(&tile(1, SLIDING, 9)),
+ "another run's day skipped a tile"
+ );
+
+ // A discovery error yields an empty eligible set.
+ indexes.release_idle(&HashMap::new());
+ assert!(
+ Arc::ptr_eq(&indexes.for_run(&held), &task),
+ "a held index was released, so the run's next chunk gets a second one"
+ );
+ assert!(
+ idle_index.upgrade().is_none(),
+ "an idle index outlived its run and kept its budget"
+ );
+ }
+}
diff --git a/rust/cohort-seeder/src/app/orchestrator.rs b/rust/cohort-seeder/src/app/orchestrator.rs
index 5d2d40f75af2..164f9558a19f 100644
--- a/rust/cohort-seeder/src/app/orchestrator.rs
+++ b/rust/cohort-seeder/src/app/orchestrator.rs
@@ -32,6 +32,7 @@ use super::completion::CompletionDriver;
use super::execute::{
execute_chunk, record_task_result, BreakerSignal, ChunkOutcome, ChunkTaskContext,
};
+use super::newest_day::NewestDayIndexes;
use super::person_execute::{execute_person_chunk, PersonChunkTaskContext};
use super::person_plan::{plan_person_run, PersonPlanAttempt, PersonPlanRequest};
use super::prepare::{refresh_runs, run_ids_of_kind, PreparedRun, RefreshOutcome};
@@ -101,6 +102,7 @@ impl PlanningState {
struct ClaimState {
inflight: InflightTasks,
breakers: RunBreakers,
+ newest_day: NewestDayIndexes,
}
pub struct SeederOrchestrator {
@@ -160,6 +162,7 @@ impl SeederOrchestrator {
let mut claims = ClaimState {
inflight: InflightTasks::default(),
breakers: RunBreakers::new(self.settings.breaker),
+ newest_day: NewestDayIndexes::new(self.settings.newest_day_index_max_entries),
};
let mut eligible_runs = HashMap::new();
let mut reported_runs = HashSet::new();
@@ -234,6 +237,7 @@ impl SeederOrchestrator {
self.complete_trailing_runs(&eligible_runs).await;
claims.breakers.expire_idle(Instant::now());
publish_open_breakers(&eligible_runs, &claims.breakers);
+ claims.newest_day.release_idle(&eligible_runs);
self.fill_claim_slots(
&eligible_runs,
&mut tasks,
@@ -518,6 +522,7 @@ impl SeederOrchestrator {
chunk,
lease,
prepared: prepared.clone(),
+ newest_day: claims.newest_day.for_run(&prepared.run),
store: self.store.clone(),
scanner: self.scanner.clone(),
producer: self.producer.clone(),
diff --git a/rust/cohort-seeder/src/app/person_execute.rs b/rust/cohort-seeder/src/app/person_execute.rs
index 0b5b82e17218..7fb051c69d5b 100644
--- a/rust/cohort-seeder/src/app/person_execute.rs
+++ b/rust/cohort-seeder/src/app/person_execute.rs
@@ -97,7 +97,7 @@ pub(super) async fn execute_person_chunk(
Err(halt) => return resolve_halt(&store, halt, &shutdown, retry_backoff).await,
};
// PreMark: CAS `scanning`→`produced` — the row is still `scanning` on failure.
- let enqueued = match store.mark_produced_streamed(streamed).await {
+ let enqueued = match store.mark_produced(streamed).await {
Ok(enqueued) => enqueued,
Err(halt) => {
return resolve_halt(&store, mark_produced_halt(halt), &shutdown, retry_backoff).await
diff --git a/rust/cohort-seeder/src/app/prime.rs b/rust/cohort-seeder/src/app/prime.rs
index 08599614c21f..531987fdff49 100644
--- a/rust/cohort-seeder/src/app/prime.rs
+++ b/rust/cohort-seeder/src/app/prime.rs
@@ -28,7 +28,7 @@ use crate::clickhouse::{ResourceError, ScanSkipReason};
use crate::observability::metrics::{
CHUNKS_POISONED, CLICKHOUSE_RESOURCE_ERRORS, EVENTS_SKIPPED, HOGVM_ERRORS,
PERSON_CONDITIONS_SHORTCUT, PERSON_ROWS_PRUNED, PERSON_SCAN_FILTER, PRUNE_REASON_IRRELEVANT,
- RUN_BREAKER_TRIPS,
+ RUN_BREAKER_TRIPS, TILES_SKIPPED,
};
use crate::store::runs::RunKind;
@@ -60,6 +60,7 @@ pub fn prime_zero_series() {
for outcome in ScanFilterOutcome::ALL {
counter!(PERSON_SCAN_FILTER, "outcome" => outcome.as_str()).increment(0);
}
+ counter!(TILES_SKIPPED).increment(0);
}
#[cfg(test)]
@@ -78,7 +79,8 @@ mod tests {
// The readings a validation run is gated on: malformed blobs skipped, HogVM aborts on an
// unresolvable reference, chunks dead-lettered at the attempt cap, persons pruned as
- // irrelevant, and whether the person key filter applied or was refused as too long.
+ // irrelevant, whether the person key filter applied or was refused as too long, and
+ // behavioral tiles skipped for a newer day.
for series in [
format!("{EVENTS_SKIPPED}{{reason=\"globals_parse_error\"}} 0"),
format!("{HOGVM_ERRORS}{{class=\"unknown_ref\"}} 0"),
@@ -90,6 +92,7 @@ mod tests {
format!("{PERSON_SCAN_FILTER}{{outcome=\"none\"}} 0"),
format!("{PERSON_SCAN_FILTER}{{outcome=\"too_long\"}} 0"),
format!("{PERSON_CONDITIONS_SHORTCUT} 0"),
+ format!("{TILES_SKIPPED} 0"),
] {
assert!(
rendered.contains(&series),
diff --git a/rust/cohort-seeder/src/app/settings.rs b/rust/cohort-seeder/src/app/settings.rs
index 4d23d2b09296..0213b9c5b780 100644
--- a/rust/cohort-seeder/src/app/settings.rs
+++ b/rust/cohort-seeder/src/app/settings.rs
@@ -67,6 +67,7 @@ pub struct OrchestratorSettings {
pub(super) plan_caps: PlanCaps,
pub(super) producer: ProducerSettings,
pub(super) person: Option<PersonSettings>,
+ pub(super) newest_day_index_max_entries: NonZeroUsize,
}
impl OrchestratorSettings {
@@ -84,6 +85,7 @@ impl OrchestratorSettings {
trailing_day_grace: Duration,
producer: ProducerSettings,
person: Option<PersonSettings>,
+ newest_day_index_max_entries: usize,
) -> Result<Self, OrchestratorSettingsError> {
if run_poll_interval.is_zero() {
return Err(OrchestratorSettingsError::ZeroPollInterval);
@@ -106,6 +108,8 @@ impl OrchestratorSettings {
let bands_per_day = NonZeroU16::new(bands_per_day)
.filter(|bands| i16::try_from(bands.get()).is_ok())
.ok_or(OrchestratorSettingsError::BandsPerDayOutOfRange)?;
+ let newest_day_index_max_entries = NonZeroUsize::new(newest_day_index_max_entries)
+ .ok_or(OrchestratorSettingsError::ZeroNewestDayIndexEntries)?;
Ok(Self {
run_poll_interval,
max_concurrent_chunks,
@@ -121,6 +125,7 @@ impl OrchestratorSettings {
},
producer,
person,
+ newest_day_index_max_entries,
})
}
@@ -203,6 +208,7 @@ impl TryFrom<&Config> for OrchestratorSettings {
Duration::from_secs(config.seeder_trailing_day_grace_secs),
producer,
person,
+ config.seeder_newest_day_index_max_entries,
)?)
}
}
@@ -235,6 +241,8 @@ pub enum OrchestratorSettingsError {
ZeroPersonsPerChunk,
#[error("person max concurrent chunks must be greater than zero")]
ZeroPersonConcurrency,
+ #[error("newest-day index max entries must be greater than zero")]
+ ZeroNewestDayIndexEntries,
#[error(
"one person chunk needs ~{projected_secs}s at the configured rate and concurrency, over \
the {budget_secs}s scan budget (half of SEEDER_CH_MAX_EXECUTION_TIME_SECS); shrink \
@@ -344,6 +352,7 @@ mod tests {
Duration::ZERO,
producer_settings(),
None,
+ 1,
),
OrchestratorSettingsError::PollIntervalExceedsLivenessDeadline,
),
@@ -361,6 +370,7 @@ mod tests {
Duration::ZERO,
producer_settings(),
None,
+ 1,
),
OrchestratorSettingsError::ZeroPollInterval,
),
@@ -378,6 +388,7 @@ mod tests {
Duration::ZERO,
producer_settings(),
None,
+ 1,
),
OrchestratorSettingsError::ZeroConcurrency,
),
@@ -395,6 +406,7 @@ mod tests {
Duration::ZERO,
producer_settings(),
None,
+ 1,
),
OrchestratorSettingsError::LeaseTooShort,
),
@@ -412,6 +424,7 @@ mod tests {
Duration::ZERO,
producer_settings(),
None,
+ 1,
),
OrchestratorSettingsError::ZeroMaxAttempts,
),
@@ -429,6 +442,7 @@ mod tests {
Duration::ZERO,
producer_settings(),
None,
+ 1,
),
OrchestratorSettingsError::BandsPerDayOutOfRange,
),
@@ -446,6 +460,7 @@ mod tests {
Duration::ZERO,
producer_settings(),
None,
+ 1,
),
OrchestratorSettingsError::BandsPerDayOutOfRange,
),
diff --git a/rust/cohort-seeder/src/config.rs b/rust/cohort-seeder/src/config.rs
index 35b59c026ce6..7ef52da4e524 100644
--- a/rust/cohort-seeder/src/config.rs
+++ b/rust/cohort-seeder/src/config.rs
@@ -213,6 +213,14 @@ pub struct Config {
#[envconfig(default = "4000")]
pub seeder_max_inflight_tiles: usize,
+ /// Entries all newest-day indexes may hold together. A run skips a person's tile for a day
+ /// older than one it already scanned, on conditions where every leaf keeps only the newest day,
+ /// and its index holds one entry per (person, condition) pair, about 30-55 bytes, so the default
+ /// takes up to about 550 MB. Runs take entries first come, first served, and free them when they
+ /// end. Past the cap new pairs are not recorded, so their tiles are sent for every day.
+ #[envconfig(default = "10000000")]
+ pub seeder_newest_day_index_max_entries: usize,
+
#[envconfig(default = "400")]
pub seeder_max_lookback_days: u32,
diff --git a/rust/cohort-seeder/src/domain/chunk.rs b/rust/cohort-seeder/src/domain/chunk.rs
index 5e62d23c8915..c2394cffbbce 100644
--- a/rust/cohort-seeder/src/domain/chunk.rs
+++ b/rust/cohort-seeder/src/domain/chunk.rs
@@ -236,8 +236,9 @@ impl ClaimedChunk {
}
}
-/// A person chunk whose seeds were interleaved scan→enqueue — never buffered — awaiting the
-/// `scanning`→`produced` CAS.
+/// A chunk whose seeds are all enqueued, awaiting the `scanning`→`produced` CAS. Only the count and
+/// the scan volume remain: a person chunk streams its seeds as it scans, and a behavioral chunk
+/// frees its tiles once they are enqueued.
#[derive(Debug)]
pub struct StreamedChunk {
spec: ChunkSpec,
@@ -286,6 +287,17 @@ impl ScannedChunk {
pub const fn volume(&self) -> ScanVolume {
self.volume
}
+
+ /// Pure transition once every tile is enqueued or skipped, with `seeds_produced` counting the
+ /// enqueued ones. Dropping the vector here frees it before the mark and the ack wait. No DB
+ /// effect.
+ pub fn into_streamed(self, seeds_produced: u64) -> StreamedChunk {
+ StreamedChunk {
+ spec: self.spec,
+ seeds_produced,
+ volume: self.volume,
+ }
+ }
}
impl std::fmt::Debug for ScannedChunk {
diff --git a/rust/cohort-seeder/src/domain/mod.rs b/rust/cohort-seeder/src/domain/mod.rs
index b9f9e2fd0be3..b32175b7bc88 100644
--- a/rust/cohort-seeder/src/domain/mod.rs
+++ b/rust/cohort-seeder/src/domain/mod.rs
@@ -18,6 +18,7 @@ pub mod condition;
pub mod condition_analysis;
pub mod ids;
pub mod ledger;
+pub mod newest_day;
pub mod partition;
pub mod person;
pub mod person_analysis;
@@ -62,6 +63,7 @@ pub use ids::{
ScannedAtMs, UtcMillis, UtcMsRange, UtcRangeError,
};
pub use ledger::{MarkerFold, MarkerLedger, SettledVerdict, WithheldCohort};
+pub use newest_day::{ConditionSlot, NewestDayConditions};
pub use partition::{SeedPartition, SeedPartitionCountError, SeedPartitions};
pub use person::{
person_chunk_sentinel_day, tile_ranges, EvaluatedConditions, PersonChunkSpec,
diff --git a/rust/cohort-seeder/src/domain/newest_day.rs b/rust/cohort-seeder/src/domain/newest_day.rs
new file mode 100644
index 000000000000..037934202272
--- /dev/null
+++ b/rust/cohort-seeder/src/domain/newest_day.rs
@@ -0,0 +1,175 @@
+//! Domain layer: the conditions of a run on which only each person's newest scanned day changes
+//! stage-1 state, so an older day's tile can be skipped. Depends on `condition`, `ids`, and
+//! `cohort-core`.
+
+use std::collections::{BTreeSet, HashMap};
+
+use cohort_core::filters::{LeafStateMeta, TeamFilters};
+use cohort_core::{EvictionWindow, StateVariant};
+
+use super::condition::PinnedCondition;
+use super::ids::ConditionHash;
+
+/// A condition on which every leaf of the run keeps only its newest matching day. Only
+/// [`NewestDayConditions::slot`] hands one out, so an index can only key a qualifying condition.
+#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
+pub struct ConditionSlot(u32);
+
+/// The qualifying conditions of one run, each with a dense slot.
+#[derive(Debug)]
+pub struct NewestDayConditions(HashMap<ConditionHash, ConditionSlot>);
+
+impl NewestDayConditions {
+ /// The processor routes a tile to every leaf on its hash in the team's live catalog. A hash
+ /// qualifies only when every leaf on it in this run's frozen catalog does. A live leaf outside
+ /// the run gets every day from its own run.
+ pub fn of(conditions: &[PinnedCondition], filters: &TeamFilters) -> Self {
+ let hashes: BTreeSet<ConditionHash> =
+ conditions.iter().map(|condition| condition.hash).collect();
+ let qualifying = hashes
+ .into_iter()
+ .filter(|hash| every_leaf_keeps_only_newest_day(*hash, filters));
+ Self(qualifying.zip((0..).map(ConditionSlot)).collect())
+ }
+
+ pub fn slot(&self, hash: ConditionHash) -> Option<ConditionSlot> {
+ self.0.get(&hash).copied()
+ }
+}
+
+fn every_leaf_keeps_only_newest_day(hash: ConditionHash, filters: &TeamFilters) -> bool {
+ filters
+ .by_condition_to_lsk
+ .get(&hash.as_bytes())
+ .is_some_and(|lsks| {
+ !lsks.is_empty()
+ && lsks
+ .iter()
+ .all(|lsk| filters.by_lsk.get(lsk).is_some_and(keeps_only_newest_day))
+ })
+}
+
+/// Whether a leaf's merged state depends on the newest seeded day alone.
+///
+/// `merge_single` in `cohort-stream-processor/src/workers/seed_path.rs` is the source of truth, and
+/// this rule must change with it. It keeps `max(last_event_at, end of the tile's day)`, so an older
+/// day's tile leaves the state unchanged. Its drops are monotone in the day: when the newest tile
+/// falls before `from_day` or outside its window, every older tile does too. An explicit upper
+/// bound breaks that: the newest day can fall after `to_day` and drop, and the leaf then needs the
+/// older in-range day. Count variants sum every day.
+fn keeps_only_newest_day(meta: &LeafStateMeta) -> bool {
+ match (meta.variant, meta.window) {
+ (
+ StateVariant::BehavioralSingle,
+ Some(
+ EvictionWindow::RelativeDays { .. }
+ | EvictionWindow::RelativeSeconds { .. }
+ | EvictionWindow::Explicit { to_day: None, .. },
+ ),
+ ) => true,
+ (
+ StateVariant::BehavioralSingle,
+ Some(EvictionWindow::Explicit {
+ to_day: Some(_), ..
+ })
+ | None,
+ ) => false,
+ (
+ StateVariant::BehavioralDailyBuckets
+ | StateVariant::BehavioralCompressedHistory
+ | StateVariant::PersonProperty,
+ _,
+ ) => false,
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use chrono_tz::UTC;
+ use cohort_core::filters::{CohortId, TeamFiltersBuilder, TeamId};
+ use serde_json::{json, Value};
+
+ use super::*;
+ use crate::domain::Lookback;
+
+ const HASH: &str = "aaaaaaaaaaaaaaaa";
+ const EVENT: &str = "$pageview";
+
+ fn leaf(shape: Value) -> Value {
+ let mut leaf = json!({
+ "type": "behavioral",
+ "key": EVENT,
+ "conditionHash": HASH,
+ "bytecode": ["_H", 1, 32, EVENT, 32, "event", 1, 1, 11],
+ });
+ leaf.as_object_mut()
+ .unwrap()
+ .extend(shape.as_object().unwrap().clone());
+ leaf
+ }
+
+ fn sliding(days: u32) -> Value {
+ leaf(json!({ "value": "performed_event", "time_value": days, "time_interval": "day" }))
+ }
+
+ /// Each leaf is its own cohort, because the processor routes a tile to every cohort's leaves.
+ fn qualifies(leaves: &[Value]) -> bool {
+ let mut builder = TeamFiltersBuilder::default();
+ for (cohort, leaf) in (1..).zip(leaves) {
+ builder
+ .add_cohort(
+ CohortId(cohort),
+ TeamId(2),
+ &json!({ "properties": { "type": "AND", "values": [leaf] } }),
+ )
+ .unwrap();
+ }
+ let filters = builder.freeze(UTC);
+ let hash = ConditionHash::parse(HASH).unwrap();
+ let conditions = [PinnedCondition {
+ cohort_id: CohortId(1),
+ hash,
+ event_name: EVENT.to_owned(),
+ lookback: Lookback::SlidingDays(7),
+ }];
+ NewestDayConditions::of(&conditions, &filters)
+ .slot(hash)
+ .is_some()
+ }
+
+ #[test]
+ fn a_hash_qualifies_only_when_every_leaf_on_it_keeps_only_the_newest_day() {
+ let count = leaf(json!({
+ "value": "performed_event_multiple",
+ "time_value": 14,
+ "time_interval": "day",
+ "operator": "gte",
+ "operator_value": 2,
+ }));
+ let bounded = leaf(json!({
+ "value": "performed_event",
+ "explicit_datetime": "2026-01-01",
+ "explicit_datetime_to": "2026-02-01",
+ }));
+ let open_ended = leaf(json!({
+ "value": "performed_event",
+ "explicit_datetime": "2026-01-01",
+ }));
+ for (case, leaves, expected) in [
+ ("sliding leaves only", vec![sliding(7), sliding(30)], true),
+ ("plus a count leaf", vec![sliding(7), count], false),
+ (
+ "plus an explicit upper bound",
+ vec![sliding(7), bounded],
+ false,
+ ),
+ (
+ "plus an explicit range with no upper bound",
+ vec![sliding(7), open_ended],
+ true,
+ ),
+ ] {
+ assert_eq!(qualifies(&leaves), expected, "{case}");
+ }
+ }
+}
diff --git a/rust/cohort-seeder/src/observability/metrics.rs b/rust/cohort-seeder/src/observability/metrics.rs
index 613d9651eb03..1b724df3bd76 100644
--- a/rust/cohort-seeder/src/observability/metrics.rs
+++ b/rust/cohort-seeder/src/observability/metrics.rs
@@ -131,6 +131,13 @@ pub const CONDITIONS_EVALUATED: &str = "seeder_conditions_evaluated_total";
pub const HOGVM_ERRORS: &str = "seeder_hogvm_errors_total";
pub const AGGREGATE_ENTRIES: &str = "seeder_aggregate_entries";
pub const TILES_PRODUCED: &str = "seeder_tiles_produced_total";
+/// Behavioral tiles not sent because their run already recorded a newer day for the same person
+/// and condition (counter).
+pub const TILES_SKIPPED: &str = "seeder_tiles_skipped_total";
+/// (person, condition) pairs all newest-day indexes hold together (gauge). At
+/// `SEEDER_NEWEST_DAY_INDEX_MAX_ENTRIES` the budget is full, and tiles for new pairs are sent for
+/// every day.
+pub const NEWEST_DAY_INDEX_ENTRIES: &str = "seeder_newest_day_index_entries";
pub const TILE_PRODUCE_QUEUE_FULL: &str = "seeder_tile_produce_queue_full_total";
pub const TILE_PRODUCE_ERRORS: &str = "seeder_tile_produce_errors_total";
pub const PRODUCE_ACK_SECONDS: &str = "seeder_produce_ack_seconds";
diff --git a/rust/cohort-seeder/src/store/chunks.rs b/rust/cohort-seeder/src/store/chunks.rs
index f1f57d7a0d5c..b12153990315 100644
--- a/rust/cohort-seeder/src/store/chunks.rs
+++ b/rust/cohort-seeder/src/store/chunks.rs
@@ -25,7 +25,7 @@ use crate::domain::{
bands_for_day, person_chunk_sentinel_day, AttemptCount, Band, BandSpec, BandSpecError, ChunkId,
ChunkLease, ChunkSpec, ChunkStatus, ClaimEpoch, ClaimKind, ClaimedChunk, DaySchedule,
EnqueuedChunk, Halted, PersonRange, PersonRangeError, PlannedDay, ProduceHwms, ProducedChunk,
- RunId, SChunkMs, ScanVolume, ScannedChunk, StreamedChunk, UtcMillis,
+ RunId, SChunkMs, ScanVolume, StreamedChunk, UtcMillis,
};
use super::lease::LeaseHandle;
@@ -259,6 +259,10 @@ impl PgChunkStore {
-- A trailing day is scanned only once it has ended and its run is `trailing`.
-- Claimed while the run still seeds, it would lose its lease when the
-- reconciling CAS fires, because the heartbeat joins the same run statuses.
+ -- The newest-day index depends on this too. Readiness does not wait for a
+ -- trailing day. A trailing day scanned while a historical chunk still sends
+ -- would make that chunk skip its tiles, and readiness could finish before the
+ -- trailing day's tile lands.
AND (c.claimable_after IS NULL
OR (c.claimable_after <= now() AND r.status = 'trailing'))
AND ((c.status = $7 AND c.attempts < $4)
@@ -392,9 +396,9 @@ impl PgChunkStore {
Ok(rows.into_iter().map(ExhaustedRun::from).collect())
}
- /// The person path's mark: seeds were already streamed to Kafka during the scan, so only the
- /// count crosses the `scanning`→`produced` CAS.
- pub async fn mark_produced_streamed(
+ /// The mark for both paths: every seed is already enqueued, so only the count sent crosses the
+ /// `scanning`→`produced` CAS.
+ pub async fn mark_produced(
&self,
chunk: StreamedChunk,
) -> Result<EnqueuedChunk, Halted<StreamedChunk, ChunkStoreError>> {
@@ -410,30 +414,6 @@ impl PgChunkStore {
Ok(EnqueuedChunk::new(spec, seeds_produced))
}
- pub async fn mark_produced(
- &self,
- chunk: ScannedChunk,
- ) -> Result<EnqueuedChunk, Halted<ScannedChunk, ChunkStoreError>> {
- let spec = chunk.spec();
- let tiles_produced = match u64::try_from(chunk.tiles().len()) {
- Ok(count) => count,
- Err(_) => {
- return Err(Halted::failed(
- chunk,
- ChunkStoreError::TilesProducedOutOfRange(u64::MAX),
- ));
- }
- };
- let volume = chunk.volume();
- if let Err(error) = self
- .mark_produced_raw(spec.lease, tiles_produced, volume)
- .await
- {
- return Err(Halted::failed(chunk, error));
- }
- Ok(EnqueuedChunk::new(spec, tiles_produced))
- }
-
pub async fn confirm(
&self,
chunk: ProducedChunk,
diff --git a/rust/cohort-stream-processor/src/workers/seed_path.rs b/rust/cohort-stream-processor/src/workers/seed_path.rs
index 03e227dee8d3..2c75f1f6a424 100644
--- a/rust/cohort-stream-processor/src/workers/seed_path.rs
+++ b/rust/cohort-stream-processor/src/workers/seed_path.rs
@@ -1464,6 +1464,32 @@ mod tests {
prop_assert!(!replay_merged, "replay merged again");
}
}
+
+ /// The seeder sends a single leaf's tile only for each person's newest day
+ /// (`cohort-seeder/src/domain/newest_day.rs`). That is sound only while the newest tile
+ /// alone reaches the state of every tile in any order. If this fails, change that rule too.
+ #[test]
+ fn single_state_is_the_newest_tiles_alone(
+ tiles in prop::collection::vec((0i32..=12, 1u32..=5), 1..10),
+ window in prop_oneof![
+ (1u32..=14).prop_map(|days| EvictionWindow::RelativeDays { days }),
+ (1i64..=172_800).prop_map(|seconds| EvictionWindow::RelativeSeconds { seconds }),
+ prop::option::of(0i32..=12).prop_map(|back| EvictionWindow::Explicit {
+ from_day: back.map(|back| now_day() - back),
+ to_day: None,
+ }),
+ ],
+ ) {
+ let meta = single_meta(window);
+ let newest = *tiles
+ .iter()
+ .min_by_key(|(offset, _)| *offset)
+ .expect("the strategy draws at least one tile");
+ prop_assert_eq!(
+ apply_all(&meta, &tiles, now_day(), NOW_MS),
+ apply_all(&meta, &[newest], now_day(), NOW_MS),
+ );
+ }
}
// ---- live-equivalence: tile (d, N) on empty ≡ N same-day live events ----