:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(cohorts): share seeder claim slots fairly across teams and runs (#126603)
Matheus Batistadc25e8fe9776
diff --git a/docs/internal/realtime-cohorts/backfill-overview.md b/docs/internal/realtime-cohorts/backfill-overview.mdindex eb775d18c935..57fa6b620a0c 100644--- a/docs/internal/realtime-cohorts/backfill-overview.md+++ b/docs/internal/realtime-cohorts/backfill-overview.md@@ -306,5 +306,6 @@ A backlog, a failure or a slow catalog refresh makes it longer.So reconcile is dispatched long before most seeds apply, and the ordering on each partition is what keeps a reconcile behind its run's held seeds.- Reconcile requests are not fenced.A request that reaches a partition before the processor has loaded the cohort's current definition waits for the next catalog refresh, up to one refresh interval, and is withheld only if that refresh still disagrees.-- Behavioral chunks are claimed newest day first across every run, so the older days of a long-window run wait behind the newer days of every other run.- They are claimed only once no newer day waits, so under sustained load the wait has no bound.+- Behavioral claims share a replica's slots between teams first, then between a team's runs, and only then take the newest day.+ A claim never takes a slot back, so a run that arrives while a heavy run holds every slot waits at least until one of that run's chunks ends.+ When more teams wait than slots free, the newest day decides between them, so the older days of a long-window run can still wait behind the newer days of other runs.diff --git a/docs/internal/realtime-cohorts/seeder.md b/docs/internal/realtime-cohorts/seeder.mdindex d7be5a15646a..bd915703d14a 100644--- a/docs/internal/realtime-cohorts/seeder.md+++ b/docs/internal/realtime-cohorts/seeder.md@@ -82,7 +82,11 @@ It stays in `seeding`, holding the cohort's run slot, until an operator cancels### Claiming and fencingA worker claims the next chunk with one statement using `FOR UPDATE SKIP LOCKED`.-It picks behavioral chunks before person chunks, then the newest day first across every run.+It picks behavioral chunks before person chunks.+Within a kind, it serves the team that holds the fewest of this replica's claim slots, then the run that holds the fewest, and only then the newest day.+A heavy run's chunk can hold its slot for hours, so under day order alone that run ends up with every slot.+With this order, each slot that frees goes to the team and run that hold the fewest, so the split moves toward an even share.+A claim never takes a slot back, and a team that is alone gets every slot.It admits:- pending chunks and failed chunks past their backoff, both under the attempt cap,@@ -453,8 +457,11 @@ Set them only to go below the profile.## Things that surprise people- `B` is the moment the seeder first saw the run, not when the run was created, except for disaster-recovery runs, whose `B` Django pins.-- Behavioral claims take the newest day first across every run, so the older days of a long-window run wait behind the newer days of every other run.+- Claims share slots between teams first, then between a team's runs, and only then take the newest day.+ A run that holds no slot can still wait, because a team that holds fewer slots goes first.+ With one slot per kind, or when more teams wait than slots free, the newest day decides, so the older days of a long-window run wait behind the newer days of other runs.If none of its chunks moves for `--stalled-for-hours`, `manage_cohort_backfill_runs inventory` lists it as a `seeding-stalled` cancel candidate, even though it is only queued.+ Each replica counts only the slots it holds, so with more than one replica the split is fair on each replica, not exactly across them.- A chunk in `produced` has enqueued its tiles but may not have all acknowledgments yet.- The seeder writes the claim epoch into every tile, but the processor ignores it.Fencing is purely a Postgres concern, and duplicates are absorbed by max-merge.diff --git a/rust/cohort-seeder/src/app/orchestrator.rs b/rust/cohort-seeder/src/app/orchestrator.rsindex 164f9558a19f..fd7479e9cc64 100644--- a/rust/cohort-seeder/src/app/orchestrator.rs+++ b/rust/cohort-seeder/src/app/orchestrator.rs@@ -6,6 +6,7 @@use std::collections::{HashMap, HashSet};use std::time::{Duration, Instant};+use cohort_core::filters::TeamId;use common_types::cohort::TeamAllowlist;use lifecycle::Handle;use metrics::{counter, gauge};@@ -16,7 +17,7 @@ use tracing::{info, warn};use crate::clickhouse::person_scanner::PersonScanner;use crate::clickhouse::scanner::ChunkScanner;-use crate::domain::{ClaimKind, RunId};+use crate::domain::{ClaimCandidate, ClaimCandidates, ClaimKind, RunId, SlotLoad};use crate::kafka::pacing::TilePacer;use crate::kafka::producer::SeedTileProducer;use crate::observability::metrics::{@@ -55,14 +56,21 @@ pub struct PersonComponents {pub pacer: TilePacer,}+/// The run a spawned task works for. A chunk task's claim slot counts against `team_id` and `kind`.+struct InflightRun {+ run_id: RunId,+ team_id: TeamId,+ kind: RunKind,+}+/// Which run each spawned task works for. Keyed by task id, so a panicked task, whose result carries/// no run, still leaves the map, and un-tracks only itself.#[derive(Default)]-struct InflightTasks(HashMap<tokio::task::Id, RunId>);+struct InflightTasks(HashMap<tokio::task::Id, InflightRun>);impl InflightTasks {- fn insert(&mut self, task_id: tokio::task::Id, run_id: RunId) {- self.0.insert(task_id, run_id);+ fn insert(&mut self, task_id: tokio::task::Id, run: InflightRun) {+ self.0.insert(task_id, run);}fn remove(&mut self, task_id: tokio::task::Id) {@@ -70,7 +78,18 @@ impl InflightTasks {}fn has_run(&self, run_id: RunId) -> bool {- self.0.values().any(|inflight| *inflight == run_id)+ self.0.values().any(|inflight| inflight.run_id == run_id)+ }++ /// Counts only the tasks of `kind`, because each kind has its own slot budget.+ fn load_of(&self, kind: RunKind, team_id: TeamId, run_id: RunId) -> SlotLoad {+ self.0+ .values()+ .filter(|inflight| inflight.kind == kind && inflight.team_id == team_id)+ .fold(SlotLoad::default(), |load, inflight| SlotLoad {+ team: load.team + 1,+ run: load.run + u32::from(inflight.run_id == run_id),+ })}}@@ -370,8 +389,12 @@ impl SeederOrchestrator {if planning.len() >= MAX_CONCURRENT_PLANNING_SCANS || shutdown.is_cancelled() {return;}- let run_id = request.run.run_id;- if state.inflight.has_run(run_id) || state.in_backoff(run_id) {+ let run = InflightRun {+ run_id: request.run.run_id,+ team_id: request.run.team_id,+ kind: RunKind::PersonProperty,+ };+ if state.inflight.has_run(run.run_id) || state.in_backoff(run.run_id) {continue;}let handle = planning.spawn(plan_person_run(@@ -382,7 +405,7 @@ impl SeederOrchestrator {request,shutdown.clone(),));- state.inflight.insert(handle.id(), run_id);+ state.inflight.insert(handle.id(), run);}}@@ -452,9 +475,10 @@ impl SeederOrchestrator {}}- /// Claim work for whichever kind still has slot capacity. The claimable run-id set is- /// recomputed per claim from the remaining capacity, so a person chunk can never occupy a- /// behavioral slot (or vice versa) — the two budgets are independent by construction.+ /// Claim work for whichever kind still has slot capacity. The candidate set is recomputed per+ /// claim from the remaining capacity, so a person chunk can never occupy a behavioral slot (or+ /// vice versa) — the two budgets are independent by construction. Each candidate's slot load is+ /// recomputed per claim too, because every spawn adds a slot to its run and its team.async fn fill_claim_slots(&self,eligible_runs: &HashMap<RunId, PreparedRun>,@@ -472,27 +496,31 @@ impl SeederOrchestrator {person_tasks.len() < person_settings.max_concurrent_chunks.get()});let now = Instant::now();- let mut run_ids = Vec::with_capacity(eligible_runs.len());+ let mut candidates = Vec::with_capacity(eligible_runs.len());for (run_id, prepared) in eligible_runs {- let has_room = match prepared {- PreparedRun::Behavioral(_) => behavioral_room,- PreparedRun::Person(_) => person_room,+ let kind = prepared.kind();+ let has_room = match kind {+ RunKind::Behavioral => behavioral_room,+ RunKind::PersonProperty => person_room,};let admitted = has_room&& claims.breakers.admits(*run_id, now, claims.inflight.has_run(*run_id));if admitted {- run_ids.push(*run_id);+ candidates.push(ClaimCandidate {+ run_id: *run_id,+ load: claims.inflight.load_of(kind, prepared.team_id(), *run_id),+ });}}- if run_ids.is_empty() {+ let Some(candidates) = ClaimCandidates::new(candidates) else {return;- }+ };let claim = match self.store.claim_next(- &run_ids,+ &candidates,&self.claimant,self.settings.chunk_lease,self.settings.max_chunk_attempts,@@ -515,6 +543,11 @@ impl SeederOrchestrator {}continue;};+ let inflight = InflightRun {+ run_id: chunk_lease.run_id(),+ team_id: prepared.team_id(),+ kind: prepared.kind(),+ };match prepared {PreparedRun::Behavioral(prepared) => {record_claim(kind, RunKind::Behavioral);@@ -532,7 +565,7 @@ impl SeederOrchestrator {};let shutdown = shutdown.clone();let task = tasks.spawn(async move { execute_chunk(ctx, shutdown).await });- claims.inflight.insert(task.id(), chunk_lease.run_id());+ claims.inflight.insert(task.id(), inflight);}PreparedRun::Person(run) => {// Unreachable while the gate is off: discovery never yields person runs then.@@ -560,7 +593,7 @@ impl SeederOrchestrator {let shutdown = shutdown.clone();let task = person_tasks.spawn(async move { execute_person_chunk(ctx, shutdown).await });- claims.inflight.insert(task.id(), chunk_lease.run_id());+ claims.inflight.insert(task.id(), inflight);}}}@@ -658,4 +691,48 @@ mod tests {assert!(state.in_backoff(run_id));}++ /// Each kind has its own slot budget, so a team's person tasks must not push its behavioral+ /// runs behind another team's.+ #[tokio::test]+ async fn slot_loads_count_only_tasks_of_the_same_kind() {+ use RunKind::{Behavioral, PersonProperty};++ let (team, other_team) = (TeamId(2), TeamId(3));+ let [heavy, sibling, person, other] = [1, 2, 3, 4].map(|n| RunId(Uuid::from_u128(n)));+ let held = [+ (Behavioral, team, heavy),+ (Behavioral, team, heavy),+ (PersonProperty, team, person),+ (Behavioral, other_team, other),+ ];+ let mut tasks = JoinSet::new();+ let mut inflight = InflightTasks::default();+ for (kind, team_id, run_id) in held {+ let run = InflightRun {+ run_id,+ team_id,+ kind,+ };+ inflight.insert(tasks.spawn(async {}).id(), run);+ }++ let cases = [+ (Behavioral, team, heavy, (2, 2)),+ (Behavioral, team, sibling, (2, 0)),+ (PersonProperty, team, person, (1, 1)),+ (Behavioral, other_team, other, (1, 1)),+ ];+ for (kind, team_id, run_id, (team_load, run_load)) in cases {+ let expected = SlotLoad {+ team: team_load,+ run: run_load,+ };+ assert_eq!(+ inflight.load_of(kind, team_id, run_id),+ expected,+ "{kind:?} load of {run_id:?} in {team_id:?}"+ );+ }+ }}diff --git a/rust/cohort-seeder/src/app/prepare.rs b/rust/cohort-seeder/src/app/prepare.rsindex 317b12c64978..ee8615a5f42b 100644--- a/rust/cohort-seeder/src/app/prepare.rs+++ b/rust/cohort-seeder/src/app/prepare.rs@@ -11,6 +11,7 @@ use std::collections::{HashMap, HashSet};use std::sync::Arc;use cohort_core::bucket_tz::window_start_for_now;+use cohort_core::filters::TeamId;use common_types::cohort::TeamAllowlist;use metrics::{counter, gauge};use sqlx::PgPool;@@ -43,6 +44,24 @@ pub(super) enum PreparedRun {Person(Arc<PinnedPersonRun>),}+impl PreparedRun {+ /// The claim budget and [`run_ids_of_kind`] both resolve a run's kind through here, so the+ /// mapping lives in one place.+ pub(super) fn kind(&self) -> RunKind {+ match self {+ Self::Behavioral(_) => RunKind::Behavioral,+ Self::Person(_) => RunKind::PersonProperty,+ }+ }++ pub(super) fn team_id(&self) -> TeamId {+ match self {+ Self::Behavioral(prepared) => prepared.run.team_id,+ Self::Person(run) => run.team_id,+ }+ }+}+/// A behavioral run and the static analysis of its conditions, which every chunk of it shares.////// The analysis lives here rather than on the scan because it is a pure function of the pinned@@ -384,21 +403,13 @@ fn report_person_analysis(run_id: RunId, run: &PinnedPersonRun) {);}-/// The eligible run ids of one kind — the per-kind label on every shared chunk metric resolves-/// through here, so the mapping lives in one place.pub(super) fn run_ids_of_kind(eligible_runs: &HashMap<RunId, PreparedRun>,kind: RunKind,) -> Vec<RunId> {eligible_runs.iter()- .filter(|(_, prepared)| {- matches!(- (prepared, kind),- (PreparedRun::Behavioral(_), RunKind::Behavioral)- | (PreparedRun::Person(_), RunKind::PersonProperty)- )- })+ .filter(|(_, prepared)| prepared.kind() == kind).map(|(run_id, _)| *run_id).collect()}diff --git a/rust/cohort-seeder/src/domain/fair_share.rs b/rust/cohort-seeder/src/domain/fair_share.rsnew file mode 100644index 000000000000..c42c7d448a73--- /dev/null+++ b/rust/cohort-seeder/src/domain/fair_share.rs@@ -0,0 +1,33 @@+//! Domain layer (pure): the claim-slot loads that order the next claim across teams and runs.+//! Depends on `ids`.++use super::ids::RunId;++/// The claim slots a team and one of its runs hold on this replica, counted within one run kind.+/// The next claim serves the lowest team load, then the lowest run load inside that team.+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]+pub struct SlotLoad {+ pub team: u32,+ pub run: u32,+}++/// A run the next claim may take a chunk from, with the load its team and it already hold.+#[derive(Debug, Clone, Copy, PartialEq, Eq)]+pub struct ClaimCandidate {+ pub run_id: RunId,+ pub load: SlotLoad,+}++/// At least one candidate, so a claim over no run cannot be expressed.+#[derive(Debug)]+pub struct ClaimCandidates(Vec<ClaimCandidate>);++impl ClaimCandidates {+ pub fn new(candidates: Vec<ClaimCandidate>) -> Option<Self> {+ (!candidates.is_empty()).then_some(Self(candidates))+ }++ pub fn iter(&self) -> std::slice::Iter<'_, ClaimCandidate> {+ self.0.iter()+ }+}diff --git a/rust/cohort-seeder/src/domain/mod.rs b/rust/cohort-seeder/src/domain/mod.rsindex 3ccb385ff2aa..cdcfa1bd8368 100644--- a/rust/cohort-seeder/src/domain/mod.rs+++ b/rust/cohort-seeder/src/domain/mod.rs@@ -16,6 +16,7 @@ pub mod compare;pub mod completion;pub mod condition;pub mod condition_analysis;+pub mod fair_share;pub mod ids;pub mod ledger;pub mod newest_day;@@ -58,6 +59,7 @@ pub use condition::{EventNameSet, Lookback, PinnedCondition};pub use condition_analysis::{ConditionAnalyses, ConditionCensus, ConditionClass, MISSING_BYTECODE_REASON,};+pub use fair_share::{ClaimCandidate, ClaimCandidates, SlotLoad};pub use ids::{Band, ChunkId, ClaimEpoch, ConditionHash, ConditionHashError, DayIdx, RunId, SChunkMs,ScannedAtMs, UtcMillis, UtcMsRange, UtcRangeError,diff --git a/rust/cohort-seeder/src/store/chunks.rs b/rust/cohort-seeder/src/store/chunks.rsindex b12153990315..f45afc0d5e49 100644--- a/rust/cohort-seeder/src/store/chunks.rs+++ b/rust/cohort-seeder/src/store/chunks.rs@@ -23,9 +23,9 @@ use uuid::Uuid;use crate::domain::{bands_for_day, person_chunk_sentinel_day, AttemptCount, Band, BandSpec, BandSpecError, ChunkId,- ChunkLease, ChunkSpec, ChunkStatus, ClaimEpoch, ClaimKind, ClaimedChunk, DaySchedule,- EnqueuedChunk, Halted, PersonRange, PersonRangeError, PlannedDay, ProduceHwms, ProducedChunk,- RunId, SChunkMs, ScanVolume, StreamedChunk, UtcMillis,+ ChunkLease, ChunkSpec, ChunkStatus, ClaimCandidates, ClaimEpoch, ClaimKind, ClaimedChunk,+ DaySchedule, EnqueuedChunk, Halted, PersonRange, PersonRangeError, PlannedDay, ProduceHwms,+ ProducedChunk, RunId, SChunkMs, ScanVolume, StreamedChunk, UtcMillis,};use super::lease::LeaseHandle;@@ -240,21 +240,38 @@ impl PgChunkStore {pub async fn claim_next(&self,- run_ids: &[RunId],+ candidates: &ClaimCandidates,claimant: &Claimant,lease: LeaseDuration,max_attempts: MaxAttempts,) -> Result<Option<Claim>, ChunkStoreError> {- if run_ids.is_empty() {- return Ok(None);- }- let run_ids = run_ids.iter().map(|run_id| run_id.0).collect::<Vec<_>>();+ let run_ids = candidates+ .iter()+ .map(|candidate| candidate.run_id.0)+ .collect::<Vec<_>>();+ let team_loads = candidates+ .iter()+ .map(|candidate| i64::from(candidate.load.team))+ .collect::<Vec<_>>();+ let run_loads = candidates+ .iter()+ .map(|candidate| i64::from(candidate.load.run))+ .collect::<Vec<_>>();let sql = format!(r#"- WITH next_chunk AS (+ WITH candidate AS (+ SELECT * FROM unnest($1::uuid[], $10::bigint[], $11::bigint[])+ AS u(run_id, team_load, run_load)+ ), next_chunk AS (SELECT c.id, c.status IN {active} AS was_reclaimFROM cohort_backfill_chunks c+ JOIN candidate u ON u.run_id = c.run_idJOIN cohort_backfill_runs r ON r.id = c.run_id+ -- `ANY($1)` repeats the join so that the planner reads only the claimable chunks+ -- through `cohort_bfc_run_status_day_idx`. With the join alone, it probes each+ -- candidate through the `(run_id, day, band)` key and filters out every+ -- confirmed chunk row by row. Only a plan built for the bound arrays does this,+ -- so the statement is not cached.WHERE c.run_id = ANY($1) AND r.status IN {seedable}-- A trailing day is scanned only once it has ended and its run is `trailing`.-- Claimed while the run still seeds, it would lose its lease when the@@ -276,12 +293,19 @@ impl PgChunkStore {-- `r.backfill_kind = $9` is true only for person runs, and `false` sorts first,-- so behavioral chunks (readiness-gating, fence-sensitive) claim before person-- chunks. The kind key must lead: person chunks sit on a far-future sentinel day,- -- which `day DESC` alone would claim first. Newest day first lets a short-window+ -- which `day DESC` alone would claim first. Of the claimable chunks, only trailing+ -- days have a `claimable_after`, so the next key puts them ahead of every load,+ -- because their cohorts already serve as ready and under-count the boundary day+ -- until the trailing tiles apply. The load keys come next: the team holding+ -- the fewest of this replica's slots of the kind goes first, then the run holding+ -- the fewest. A heavy run's chunk holds its slot for hours while a cheap one frees+ -- its slot in seconds, so under day order alone the heavy run ends up holding+ -- every slot and every other run advances one day per freed slot. A team alone+ -- still gets every slot. Among equal loads, newest day first lets a short-window-- cohort, whose days are all recent, finish without waiting behind a long run's- -- older days. Claimable trailing days have no key of their own and sort among the- -- newest days, because their cohorts already serve as ready and under-count the- -- boundary day until the trailing tiles apply.- ORDER BY (r.backfill_kind = $9), c.day DESC, c.band+ -- older days.+ ORDER BY (r.backfill_kind = $9), (c.claimable_after IS NULL), u.team_load,+ u.run_load, c.day DESC, c.bandLIMIT 1FOR UPDATE OF c SKIP LOCKED)@@ -304,6 +328,9 @@ impl PgChunkStore {seedable = SEEDABLE_RUN_STATUSES_SQL,);let row = sqlx::query_as::<_, ClaimedRow>(&sql)+ // Postgres can switch a cached statement to a generic plan after five runs. That plan+ // loops over the candidates and reads every chunk of each, confirmed ones included.+ .persistent(false).bind(run_ids).bind(claimant.as_str()).bind(lease.as_secs())@@ -313,6 +340,8 @@ impl PgChunkStore {.bind(ChunkStatus::Pending.as_str()).bind(ChunkStatus::Failed.as_str()).bind(RunKind::PersonProperty.as_str())+ .bind(team_loads)+ .bind(run_loads).fetch_optional(&self.pool).await?;diff --git a/rust/cohort-seeder/tests/pg_chunks.rs b/rust/cohort-seeder/tests/pg_chunks.rsindex 3b3a5967c45d..266c7f9db62d 100644--- a/rust/cohort-seeder/tests/pg_chunks.rs+++ b/rust/cohort-seeder/tests/pg_chunks.rs@@ -19,8 +19,8 @@ use cohort_seeder::app::reconcile_dispatch::{};use cohort_seeder::app::{eligible_run_ids, fail_exhausted_runs_of_kind};use cohort_seeder::domain::{- tile_ranges, AttemptCount, ClaimEpoch, PersonRunValidation, PinnedWarning, ProduceHwms,- RetryBackoffPolicy, ScanVolume, ScopeKind,+ tile_ranges, AttemptCount, ClaimCandidate, ClaimCandidates, ClaimEpoch, PersonRunValidation,+ PinnedWarning, ProduceHwms, RetryBackoffPolicy, RunId, ScanVolume, ScopeKind, SlotLoad,};use cohort_seeder::store::chunks::{ChunkStoreError, PgChunkStore, PlanOutcome, NO_ERROR_RECORDED};use cohort_seeder::store::lease::LeaseFailure;@@ -47,6 +47,18 @@ const ONE_BAND: NonZeroU16 = NonZeroU16::MIN;/// something other than the backoff and want the chunk claimable on the next call.const NO_BACKOFF: Duration = Duration::ZERO;+/// Candidates that hold no claim slot, so they claim in kind, day and band order.+fn idle(run_ids: &[RunId]) -> ClaimCandidates {+ let candidates = run_ids+ .iter()+ .map(|&run_id| ClaimCandidate {+ run_id,+ load: SlotLoad::default(),+ })+ .collect();+ ClaimCandidates::new(candidates).expect("every scenario claims from at least one run")+}+/// Discovery honors an `Only` allowlist (self team only) and `All` admits every eligible run/// regardless of team, trigger, or already-seeding status.#[tokio::test]@@ -567,7 +579,7 @@ async fn planning_fans_out_bands_and_claims_carry_the_band_count() -> Result<()>let attempts5 = MaxAttempts::new(5)?;let claimed = store.claim_next(- &[seeding_run],+ &idle(&[seeding_run]),&Claimant::new("worker-a")?,lease60,attempts5,@@ -600,14 +612,14 @@ async fn concurrent_claims_take_disjoint_chunks() -> Result<()> {let lease60 = LeaseDuration::new(Duration::from_secs(60))?;let attempts5 = MaxAttempts::new(5)?;- let run_ids = [seeding_run];+ let candidates = idle(&[seeding_run]);let first_store = store.clone();let second_store = store.clone();let claimant_a = Claimant::new("worker-a")?;let claimant_b = Claimant::new("worker-b")?;let (first, second) = tokio::join!(- first_store.claim_next(&run_ids, &claimant_a, lease60, attempts5),- second_store.claim_next(&run_ids, &claimant_b, lease60, attempts5),+ first_store.claim_next(&candidates, &claimant_a, lease60, attempts5),+ second_store.claim_next(&candidates, &claimant_b, lease60, attempts5),);let first = first?.context("first claimant found no chunk")?;let second = second?.context("second claimant found no chunk")?;@@ -632,7 +644,7 @@ async fn expired_lease_reclaim_bumps_epoch_and_fences_the_stale_lease() -> Resullet run_ids = [seeding_run];let first = store- .claim_next(&run_ids, &Claimant::new("worker-a")?, lease60, attempts5)+ .claim_next(&idle(&run_ids), &Claimant::new("worker-a")?, lease60, attempts5).await?.context("first claimant found no chunk")?;let stale = first.chunk.spec().lease;@@ -645,7 +657,7 @@ async fn expired_lease_reclaim_bumps_epoch_and_fences_the_stale_lease() -> Resul.await?;let reclaimed = store- .claim_next(&run_ids, &Claimant::new("worker-c")?, lease60, attempts5)+ .claim_next(&idle(&run_ids), &Claimant::new("worker-c")?, lease60, attempts5).await?.context("expired chunk was not reclaimed")?;let reclaimed_lease = reclaimed.chunk.spec().lease;@@ -691,7 +703,12 @@ async fn unclaim_returns_chunk_to_pending_and_refunds_one_attempt() -> Result<()let run_ids = [seeding_run];let claimed = store- .claim_next(&run_ids, &Claimant::new("worker-a")?, lease60, attempts5)+ .claim_next(+ &idle(&run_ids),+ &Claimant::new("worker-a")?,+ lease60,+ attempts5,+ ).await?.context("claimant found no chunk")?;let lease = claimed.chunk.spec().lease;@@ -732,7 +749,7 @@ async fn a_failed_chunk_waits_out_its_backoff_before_it_is_claimable_again() ->let run_ids = [seeding_run];let claimed = store- .claim_next(&run_ids, &Claimant::new("worker-a")?, lease60, attempts5)+ .claim_next(&idle(&run_ids), &Claimant::new("worker-a")?, lease60, attempts5).await?.context("claimant found no chunk")?;let lease = claimed.chunk.spec().lease;@@ -761,7 +778,7 @@ async fn a_failed_chunk_waits_out_its_backoff_before_it_is_claimable_again() ->"expected a ~600s hold, got {remaining_secs}s");ensure!(store- .claim_next(&run_ids, &Claimant::new("worker-b")?, lease60, attempts5)+ .claim_next(&idle(&run_ids), &Claimant::new("worker-b")?, lease60, attempts5).await?.is_none());@@ -774,7 +791,7 @@ async fn a_failed_chunk_waits_out_its_backoff_before_it_is_claimable_again() ->.execute(&pool).await?;let retried = store- .claim_next(&run_ids, &Claimant::new("worker-c")?, lease60, attempts5)+ .claim_next(&idle(&run_ids), &Claimant::new("worker-c")?, lease60, attempts5).await?.context("chunk past its backoff was not claimable")?;ensure!(retried.chunk.spec().attempt.get() == 2);@@ -829,7 +846,12 @@ async fn the_recovery_path_draws_its_wait_from_the_chunks_attempt_count() -> Reslet mut longest_wait = Duration::ZERO;for day in days {let claimed = store- .claim_next(&run_ids, &Claimant::new("worker-a")?, lease60, attempts50)+ .claim_next(+ &idle(&run_ids),+ &Claimant::new("worker-a")?,+ lease60,+ attempts50,+ ).await?.context("a chunk under the cap was not claimable")?;let chunk = claimed.chunk;@@ -896,7 +918,7 @@ async fn the_backoff_gate_applies_only_to_the_failed_claim_arm() -> Result<()> {.execute(&pool).await?;let pending = store- .claim_next(&run_ids, &Claimant::new("worker-a")?, lease60, attempts5)+ .claim_next(&idle(&run_ids), &Claimant::new("worker-a")?, lease60, attempts5).await?.context("a pending chunk was gated by the backoff stamp")?;let pending_lease = pending.chunk.spec().lease;@@ -911,7 +933,7 @@ async fn the_backoff_gate_applies_only_to_the_failed_claim_arm() -> Result<()> {.execute(&pool).await?;let reclaimed = store- .claim_next(&run_ids, &Claimant::new("worker-b")?, lease60, attempts5)+ .claim_next(&idle(&run_ids), &Claimant::new("worker-b")?, lease60, attempts5).await?.context("an unclaimed chunk was gated by the backoff stamp")?;@@ -929,7 +951,7 @@ async fn the_backoff_gate_applies_only_to_the_failed_claim_arm() -> Result<()> {.execute(&pool).await?;let produced_reclaim = store- .claim_next(&run_ids, &Claimant::new("worker-c")?, lease60, attempts5)+ .claim_next(&idle(&run_ids), &Claimant::new("worker-c")?, lease60, attempts5).await?.context("an expired produced lease was gated by the backoff stamp")?;@@ -948,7 +970,7 @@ async fn the_backoff_gate_applies_only_to_the_failed_claim_arm() -> Result<()> {.execute(&pool).await?;ensure!(store- .claim_next(&run_ids, &Claimant::new("worker-d")?, lease60, attempts5)+ .claim_next(&idle(&run_ids), &Claimant::new("worker-d")?, lease60, attempts5).await?.is_some());Ok(())@@ -995,13 +1017,13 @@ async fn a_trailing_day_waits_out_its_hold_and_completes_its_trailing_run() -> R);let historical_claim = store- .claim_next(&run_ids, &claimant, lease60, attempts5)+ .claim_next(&idle(&run_ids), &claimant, lease60, attempts5).await?.context("the historical day was not claimable")?;ensure!(historical_claim.chunk.spec().day == 100);ensure!(store- .claim_next(&run_ids, &claimant, lease60, attempts5)+ .claim_next(&idle(&run_ids), &claimant, lease60, attempts5).await?.is_none(),"the trailing day was claimed before its hold lapsed"@@ -1050,7 +1072,7 @@ async fn a_trailing_day_waits_out_its_hold_and_completes_its_trailing_run() -> Rensure!(complete_trailing_runs(&pool, &run_ids).await? == 0);let trailing_claim = store- .claim_next(&run_ids, &claimant, lease60, attempts5)+ .claim_next(&idle(&run_ids), &claimant, lease60, attempts5).await?.context("the trailing day was not claimable after its hold lapsed")?;ensure!(trailing_claim.chunk.spec().day == 101);@@ -1101,7 +1123,12 @@ async fn a_seeding_run_neither_hands_out_a_lapsed_held_chunk_nor_completes_as_tr.await?;ensure!(store- .claim_next(&run_ids, &Claimant::new("worker-a")?, lease60, attempts5)+ .claim_next(+ &idle(&run_ids),+ &Claimant::new("worker-a")?,+ lease60,+ attempts5+ ).await?.is_none(),"a held chunk was claimed while its run was still seeding"@@ -1148,7 +1175,7 @@ async fn mark_produced_records_the_scan_byte_volume_that_planning_defaults_to_zelet claimed = store.claim_next(- &[seeding_run],+ &idle(&[seeding_run]),&Claimant::new("worker-a")?,LeaseDuration::new(Duration::from_secs(60))?,MaxAttempts::new(5)?,@@ -1199,7 +1226,7 @@ async fn attempt_cap_is_terminal_for_failed_but_reclaims_expired_produced() -> R.execute(&pool).await?;let retry = store- .claim_next(&run_ids, &Claimant::new("worker-d")?, lease60, attempts5)+ .claim_next(&idle(&run_ids), &Claimant::new("worker-d")?, lease60, attempts5).await?.context("chunk under the cap was not claimable")?;let retry_lease = retry.chunk.spec().lease;@@ -1214,14 +1241,14 @@ async fn attempt_cap_is_terminal_for_failed_but_reclaims_expired_produced() -> R.await?;drop(retry);ensure!(store- .claim_next(&run_ids, &Claimant::new("worker-terminal")?, lease60, attempts5)+ .claim_next(&idle(&run_ids), &Claimant::new("worker-terminal")?, lease60, attempts5).await?.is_none());// A second chunk, marked produced then expired at the cap, IS reclaimed.ensure!(planned_count(store.plan_chunks(seeding_run, historical([101]), ONE_BAND).await?)? == 1);let final_attempt = store- .claim_next(&run_ids, &Claimant::new("worker-e")?, lease60, attempts5)+ .claim_next(&idle(&run_ids), &Claimant::new("worker-e")?, lease60, attempts5).await?.context("second chunk was not claimable")?;let final_attempt_lease = final_attempt.chunk.spec().lease;@@ -1235,7 +1262,7 @@ async fn attempt_cap_is_terminal_for_failed_but_reclaims_expired_produced() -> R.await?;let observed = store- .claim_next(&run_ids, &Claimant::new("worker-f")?, lease60, attempts5)+ .claim_next(&idle(&run_ids), &Claimant::new("worker-f")?, lease60, attempts5).await?.context("expired produced chunk at the attempt cap was not reclaimed")?;let observed_lease = observed.chunk.spec().lease;@@ -1271,7 +1298,7 @@ async fn expired_scanning_chunk_at_the_cap_is_reaped_not_reclaimed() -> Result<(let attempts5 = MaxAttempts::new(5)?;let run_ids = [seeding_run];let claimed = store- .claim_next(&run_ids, &Claimant::new("worker-a")?, lease60, attempts5)+ .claim_next(&idle(&run_ids), &Claimant::new("worker-a")?, lease60, attempts5).await?.context("claimant found no chunk")?;let chunk_id = claimed.chunk.spec().lease.chunk_id();@@ -1292,7 +1319,7 @@ async fn expired_scanning_chunk_at_the_cap_is_reaped_not_reclaimed() -> Result<(.execute(&pool).await?;ensure!(store- .claim_next(&run_ids, &Claimant::new("worker-b")?, lease60, attempts5)+ .claim_next(&idle(&run_ids), &Claimant::new("worker-b")?, lease60, attempts5).await?.is_none());@@ -1306,7 +1333,7 @@ async fn expired_scanning_chunk_at_the_cap_is_reaped_not_reclaimed() -> Result<(ensure!(last_error.contains("attempt cap"));// Terminal: not claimable, and a second sweep is a no-op.ensure!(store- .claim_next(&run_ids, &Claimant::new("worker-c")?, lease60, attempts5)+ .claim_next(&idle(&run_ids), &Claimant::new("worker-c")?, lease60, attempts5).await?.is_none());ensure!(store.reap_poisoned_chunks(&run_ids, attempts5).await? == 0);@@ -1478,7 +1505,7 @@ async fn exhausted_chunk_fails_the_run_and_stops_further_claims() -> Result<()>// One chunk mid-scan with a live lease, one capped out, one left pending.let claimed = store- .claim_next(&run_ids, &claimant, lease60, attempts5)+ .claim_next(&idle(&run_ids), &claimant, lease60, attempts5).await?.context("claimant found no chunk")?;let live_lease = claimed.chunk.spec().lease;@@ -1524,7 +1551,7 @@ async fn exhausted_chunk_fails_the_run_and_stops_further_claims() -> Result<()>// The claim and heartbeat predicates both join `runs.status = 'seeding'`, so the pending// sibling stops being claimable and the in-flight one halts on its next beat.ensure!(store- .claim_next(&run_ids, &Claimant::new("worker-b")?, lease60, attempts5)+ .claim_next(&idle(&run_ids), &Claimant::new("worker-b")?, lease60, attempts5).await?.is_none());let still_pending: i64 = sqlx::query_scalar(@@ -1582,7 +1609,12 @@ async fn fail_truncates_chunk_and_run_error_columns() -> Result<()> {let attempts5 = MaxAttempts::new(5)?;let run_ids = [seeding_run];let claimed = store- .claim_next(&run_ids, &Claimant::new("worker-a")?, lease60, attempts5)+ .claim_next(+ &idle(&run_ids),+ &Claimant::new("worker-a")?,+ lease60,+ attempts5,+ ).await?.context("claimant found no chunk")?;let lease = claimed.chunk.spec().lease;@@ -1810,9 +1842,10 @@ async fn person_planning_is_all_or_nothing_and_idempotent() -> Result<()> {.await}-/// Behavioral days claim newest first across runs, and person chunks claim strictly after every-/// behavioral day on the shared claim loop even though their sentinel day is the newest. A person-/// claim decodes its range with `num_bands` = the run's chunk count.+/// Behavioral days claim newest first across runs with equal slot loads, and person chunks claim+/// strictly after every behavioral day on the shared claim loop, even though their sentinel day is+/// the newest and their run holds fewer slots. A person claim decodes its range with `num_bands` =+/// the run's chunk count.#[tokio::test]async fn behavioral_days_claim_newest_first_then_person_chunks_with_ranges() -> Result<()> {with_db(|pool| async move {@@ -1843,13 +1876,28 @@ async fn behavioral_days_claim_newest_first_then_person_chunks_with_ranges() ->let lease60 = LeaseDuration::new(Duration::from_secs(60))?;let attempts5 = MaxAttempts::new(5)?;- let run_ids = [behavioral_run, other_team_run, person_run];+ let busy = SlotLoad { team: 1, run: 1 };+ let candidates = ClaimCandidates::new(vec![+ ClaimCandidate {+ run_id: behavioral_run,+ load: busy,+ },+ ClaimCandidate {+ run_id: other_team_run,+ load: busy,+ },+ ClaimCandidate {+ run_id: person_run,+ load: SlotLoad::default(),+ },+ ])+ .context("no claim candidates")?;let claimant = Claimant::new("worker-a")?;let mut behavioral_claims = Vec::new();for _ in 0..3 {let claim = store- .claim_next(&run_ids, &claimant, lease60, attempts5)+ .claim_next(&candidates, &claimant, lease60, attempts5).await?.context("behavioral chunk was not claimed before person chunks")?;ensure!(claim.chunk.spec().person_range.is_none());@@ -1866,7 +1914,7 @@ async fn behavioral_days_claim_newest_first_then_person_chunks_with_ranges() ->);let first_person = store- .claim_next(&run_ids, &claimant, lease60, attempts5)+ .claim_next(&candidates, &claimant, lease60, attempts5).await?.context("first person chunk was not claimed")?;let first_person_spec = first_person.chunk.spec();@@ -1880,7 +1928,7 @@ async fn behavioral_days_claim_newest_first_then_person_chunks_with_ranges() ->ensure!(first_person_range.hi() == Some(boundary));let second_person = store- .claim_next(&run_ids, &claimant, lease60, attempts5)+ .claim_next(&candidates, &claimant, lease60, attempts5).await?.context("second person chunk was not claimed")?;let second_person_range = second_person@@ -1896,6 +1944,89 @@ async fn behavioral_days_claim_newest_first_then_person_chunks_with_ranges() ->.await}+/// A claim takes a lapsed trailing day first, then serves the team holding the fewest slots, then+/// the run holding the fewest, and only then the newest day.+#[tokio::test]+async fn claims_serve_the_least_loaded_team_then_run_before_the_newest_day() -> Result<()> {+ with_db(|pool| async move {+ let heavy_run =+ insert_run(&pool, 2, "team_enablement", "seeding", true, empty_pinned()).await?;+ // A team holds one active team-scoped run per kind, so the first run moves to cohort+ // scope before the second is inserted.+ sqlx::query(+ "UPDATE cohort_backfill_runs SET scope = 'cohort', cohort_id = 1 WHERE id = $1",+ )+ .bind(heavy_run)+ .execute(&pool)+ .await?;+ let sibling_run =+ insert_run(&pool, 2, "team_enablement", "seeding", true, empty_pinned()).await?;+ let other_team_run =+ insert_run(&pool, 3, "team_enablement", "seeding", true, empty_pinned()).await?;+ let trailing_run =+ insert_run(&pool, 4, "team_enablement", "seeding", true, empty_pinned()).await?;+ let store = PgChunkStore::new(pool.clone());+ for (run_id, day) in [(heavy_run, 100), (sibling_run, 98), (other_team_run, 97)] {+ ensure!(+ planned_count(+ store+ .plan_chunks(run_id, historical([day]), ONE_BAND)+ .await?+ )? == 1+ );+ }+ let lapsed = Utc::now() - chrono::Duration::hours(1);+ ensure!(+ planned_count(+ store+ .plan_chunks(trailing_run, [trailing(96, lapsed)], ONE_BAND)+ .await?+ )? == 1+ );+ sqlx::query("UPDATE cohort_backfill_runs SET status = 'trailing' WHERE id = $1")+ .bind(trailing_run)+ .execute(&pool)+ .await?;++ let lease60 = LeaseDuration::new(Duration::from_secs(60))?;+ let attempts5 = MaxAttempts::new(5)?;+ let claimant = Claimant::new("worker-a")?;+ let rounds = [+ // Team 4 holds the most slots and its trailing day is the oldest day, but the trailing+ // day still goes first.+ ([(2, 2), (2, 0), (1, 1), (3, 0)], (trailing_run, 96)),+ // Team 2 holds two slots and team 3 one, so team 3 goes first, ahead of team 2's newer+ // days and of its sibling run that holds none.+ ([(2, 2), (2, 0), (1, 1), (4, 1)], (other_team_run, 97)),+ // Both teams hold one slot, so inside team 2 the run holding none goes first, ahead of+ // the heavy run's newer day.+ ([(1, 1), (1, 0), (1, 1), (4, 1)], (sibling_run, 98)),+ ];+ for (loads, expected) in rounds {+ let candidates = [heavy_run, sibling_run, other_team_run, trailing_run]+ .into_iter()+ .zip(loads)+ .map(|(run_id, (team, run))| ClaimCandidate {+ run_id,+ load: SlotLoad { team, run },+ })+ .collect();+ let candidates = ClaimCandidates::new(candidates).context("no claim candidates")?;+ let claim = store+ .claim_next(&candidates, &claimant, lease60, attempts5)+ .await?+ .context("no chunk was claimed")?;+ let claimed = (claim.chunk.spec().lease.run_id(), claim.chunk.spec().day);+ ensure!(+ claimed == expected,+ "expected {expected:?}, got {claimed:?}"+ );+ }+ Ok(())+ })+ .await+}+/// Cancelling a run breaks its live leases: the claimed chunk's next heartbeat fences out, so the/// lease surfaces `Heartbeat(LeaseLost)` and its cancellation token trips.#[tokio::test]@@ -1910,7 +2041,7 @@ async fn cancelling_a_run_kills_its_live_lease_via_the_heartbeat() -> Result<()>let attempts5 = MaxAttempts::new(5)?;let run_ids = [seeding_run];let observed = store- .claim_next(&run_ids, &Claimant::new("worker-f")?, lease3, attempts5)+ .claim_next(&idle(&run_ids), &Claimant::new("worker-f")?, lease3, attempts5).await?.context("claimant found no chunk")?;let cancelled = observed.lease.cancellation_token();