Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(cohorts): share seeder claim slots fairly across teams and runs (#126603)

Matheus Batistadc25e8fe9776
diff --git a/docs/internal/realtime-cohorts/backfill-overview.md b/docs/internal/realtime-cohorts/backfill-overview.md
index eb775d18c935..57fa6b620a0c 100644
--- a/docs/internal/realtime-cohorts/backfill-overview.md
+++ b/docs/internal/realtime-cohorts/backfill-overview.md
@@ -306,5 +306,6 @@ A backlog, a failure or a slow catalog refresh makes it longer.
So reconcile is dispatched long before most seeds apply, and the ordering on each partition is what keeps a reconcile behind its run's held seeds.
- Reconcile requests are not fenced.
A request that reaches a partition before the processor has loaded the cohort's current definition waits for the next catalog refresh, up to one refresh interval, and is withheld only if that refresh still disagrees.
-- Behavioral chunks are claimed newest day first across every run, so the older days of a long-window run wait behind the newer days of every other run.
- They are claimed only once no newer day waits, so under sustained load the wait has no bound.
+- Behavioral claims share a replica's slots between teams first, then between a team's runs, and only then take the newest day.
+ A claim never takes a slot back, so a run that arrives while a heavy run holds every slot waits at least until one of that run's chunks ends.
+ When more teams wait than slots free, the newest day decides between them, so the older days of a long-window run can still wait behind the newer days of other runs.
diff --git a/docs/internal/realtime-cohorts/seeder.md b/docs/internal/realtime-cohorts/seeder.md
index d7be5a15646a..bd915703d14a 100644
--- a/docs/internal/realtime-cohorts/seeder.md
+++ b/docs/internal/realtime-cohorts/seeder.md
@@ -82,7 +82,11 @@ It stays in `seeding`, holding the cohort's run slot, until an operator cancels
### Claiming and fencing
A worker claims the next chunk with one statement using `FOR UPDATE SKIP LOCKED`.
-It picks behavioral chunks before person chunks, then the newest day first across every run.
+It picks behavioral chunks before person chunks.
+Within a kind, it serves the team that holds the fewest of this replica's claim slots, then the run that holds the fewest, and only then the newest day.
+A heavy run's chunk can hold its slot for hours, so under day order alone that run ends up with every slot.
+With this order, each slot that frees goes to the team and run that hold the fewest, so the split moves toward an even share.
+A claim never takes a slot back, and a team that is alone gets every slot.
It admits:
- pending chunks and failed chunks past their backoff, both under the attempt cap,
@@ -453,8 +457,11 @@ Set them only to go below the profile.
## Things that surprise people
- `B` is the moment the seeder first saw the run, not when the run was created, except for disaster-recovery runs, whose `B` Django pins.
-- Behavioral claims take the newest day first across every run, so the older days of a long-window run wait behind the newer days of every other run.
+- Claims share slots between teams first, then between a team's runs, and only then take the newest day.
+ A run that holds no slot can still wait, because a team that holds fewer slots goes first.
+ With one slot per kind, or when more teams wait than slots free, the newest day decides, so the older days of a long-window run wait behind the newer days of other runs.
If none of its chunks moves for `--stalled-for-hours`, `manage_cohort_backfill_runs inventory` lists it as a `seeding-stalled` cancel candidate, even though it is only queued.
+ Each replica counts only the slots it holds, so with more than one replica the split is fair on each replica, not exactly across them.
- A chunk in `produced` has enqueued its tiles but may not have all acknowledgments yet.
- The seeder writes the claim epoch into every tile, but the processor ignores it.
Fencing is purely a Postgres concern, and duplicates are absorbed by max-merge.
diff --git a/rust/cohort-seeder/src/app/orchestrator.rs b/rust/cohort-seeder/src/app/orchestrator.rs
index 164f9558a19f..fd7479e9cc64 100644
--- a/rust/cohort-seeder/src/app/orchestrator.rs
+++ b/rust/cohort-seeder/src/app/orchestrator.rs
@@ -6,6 +6,7 @@
use std::collections::{HashMap, HashSet};
use std::time::{Duration, Instant};
+use cohort_core::filters::TeamId;
use common_types::cohort::TeamAllowlist;
use lifecycle::Handle;
use metrics::{counter, gauge};
@@ -16,7 +17,7 @@ use tracing::{info, warn};
use crate::clickhouse::person_scanner::PersonScanner;
use crate::clickhouse::scanner::ChunkScanner;
-use crate::domain::{ClaimKind, RunId};
+use crate::domain::{ClaimCandidate, ClaimCandidates, ClaimKind, RunId, SlotLoad};
use crate::kafka::pacing::TilePacer;
use crate::kafka::producer::SeedTileProducer;
use crate::observability::metrics::{
@@ -55,14 +56,21 @@ pub struct PersonComponents {
pub pacer: TilePacer,
}
+/// The run a spawned task works for. A chunk task's claim slot counts against `team_id` and `kind`.
+struct InflightRun {
+ run_id: RunId,
+ team_id: TeamId,
+ kind: RunKind,
+}
+
/// Which run each spawned task works for. Keyed by task id, so a panicked task, whose result carries
/// no run, still leaves the map, and un-tracks only itself.
#[derive(Default)]
-struct InflightTasks(HashMap<tokio::task::Id, RunId>);
+struct InflightTasks(HashMap<tokio::task::Id, InflightRun>);
impl InflightTasks {
- fn insert(&mut self, task_id: tokio::task::Id, run_id: RunId) {
- self.0.insert(task_id, run_id);
+ fn insert(&mut self, task_id: tokio::task::Id, run: InflightRun) {
+ self.0.insert(task_id, run);
}
fn remove(&mut self, task_id: tokio::task::Id) {
@@ -70,7 +78,18 @@ impl InflightTasks {
}
fn has_run(&self, run_id: RunId) -> bool {
- self.0.values().any(|inflight| *inflight == run_id)
+ self.0.values().any(|inflight| inflight.run_id == run_id)
+ }
+
+ /// Counts only the tasks of `kind`, because each kind has its own slot budget.
+ fn load_of(&self, kind: RunKind, team_id: TeamId, run_id: RunId) -> SlotLoad {
+ self.0
+ .values()
+ .filter(|inflight| inflight.kind == kind && inflight.team_id == team_id)
+ .fold(SlotLoad::default(), |load, inflight| SlotLoad {
+ team: load.team + 1,
+ run: load.run + u32::from(inflight.run_id == run_id),
+ })
}
}
@@ -370,8 +389,12 @@ impl SeederOrchestrator {
if planning.len() >= MAX_CONCURRENT_PLANNING_SCANS || shutdown.is_cancelled() {
return;
}
- let run_id = request.run.run_id;
- if state.inflight.has_run(run_id) || state.in_backoff(run_id) {
+ let run = InflightRun {
+ run_id: request.run.run_id,
+ team_id: request.run.team_id,
+ kind: RunKind::PersonProperty,
+ };
+ if state.inflight.has_run(run.run_id) || state.in_backoff(run.run_id) {
continue;
}
let handle = planning.spawn(plan_person_run(
@@ -382,7 +405,7 @@ impl SeederOrchestrator {
request,
shutdown.clone(),
));
- state.inflight.insert(handle.id(), run_id);
+ state.inflight.insert(handle.id(), run);
}
}
@@ -452,9 +475,10 @@ impl SeederOrchestrator {
}
}
- /// Claim work for whichever kind still has slot capacity. The claimable run-id set is
- /// recomputed per claim from the remaining capacity, so a person chunk can never occupy a
- /// behavioral slot (or vice versa) — the two budgets are independent by construction.
+ /// Claim work for whichever kind still has slot capacity. The candidate set is recomputed per
+ /// claim from the remaining capacity, so a person chunk can never occupy a behavioral slot (or
+ /// vice versa) — the two budgets are independent by construction. Each candidate's slot load is
+ /// recomputed per claim too, because every spawn adds a slot to its run and its team.
async fn fill_claim_slots(
&self,
eligible_runs: &HashMap<RunId, PreparedRun>,
@@ -472,27 +496,31 @@ impl SeederOrchestrator {
person_tasks.len() < person_settings.max_concurrent_chunks.get()
});
let now = Instant::now();
- let mut run_ids = Vec::with_capacity(eligible_runs.len());
+ let mut candidates = Vec::with_capacity(eligible_runs.len());
for (run_id, prepared) in eligible_runs {
- let has_room = match prepared {
- PreparedRun::Behavioral(_) => behavioral_room,
- PreparedRun::Person(_) => person_room,
+ let kind = prepared.kind();
+ let has_room = match kind {
+ RunKind::Behavioral => behavioral_room,
+ RunKind::PersonProperty => person_room,
};
let admitted = has_room
&& claims
.breakers
.admits(*run_id, now, claims.inflight.has_run(*run_id));
if admitted {
- run_ids.push(*run_id);
+ candidates.push(ClaimCandidate {
+ run_id: *run_id,
+ load: claims.inflight.load_of(kind, prepared.team_id(), *run_id),
+ });
}
}
- if run_ids.is_empty() {
+ let Some(candidates) = ClaimCandidates::new(candidates) else {
return;
- }
+ };
let claim = match self
.store
.claim_next(
- &run_ids,
+ &candidates,
&self.claimant,
self.settings.chunk_lease,
self.settings.max_chunk_attempts,
@@ -515,6 +543,11 @@ impl SeederOrchestrator {
}
continue;
};
+ let inflight = InflightRun {
+ run_id: chunk_lease.run_id(),
+ team_id: prepared.team_id(),
+ kind: prepared.kind(),
+ };
match prepared {
PreparedRun::Behavioral(prepared) => {
record_claim(kind, RunKind::Behavioral);
@@ -532,7 +565,7 @@ impl SeederOrchestrator {
};
let shutdown = shutdown.clone();
let task = tasks.spawn(async move { execute_chunk(ctx, shutdown).await });
- claims.inflight.insert(task.id(), chunk_lease.run_id());
+ claims.inflight.insert(task.id(), inflight);
}
PreparedRun::Person(run) => {
// Unreachable while the gate is off: discovery never yields person runs then.
@@ -560,7 +593,7 @@ impl SeederOrchestrator {
let shutdown = shutdown.clone();
let task = person_tasks
.spawn(async move { execute_person_chunk(ctx, shutdown).await });
- claims.inflight.insert(task.id(), chunk_lease.run_id());
+ claims.inflight.insert(task.id(), inflight);
}
}
}
@@ -658,4 +691,48 @@ mod tests {
assert!(state.in_backoff(run_id));
}
+
+ /// Each kind has its own slot budget, so a team's person tasks must not push its behavioral
+ /// runs behind another team's.
+ #[tokio::test]
+ async fn slot_loads_count_only_tasks_of_the_same_kind() {
+ use RunKind::{Behavioral, PersonProperty};
+
+ let (team, other_team) = (TeamId(2), TeamId(3));
+ let [heavy, sibling, person, other] = [1, 2, 3, 4].map(|n| RunId(Uuid::from_u128(n)));
+ let held = [
+ (Behavioral, team, heavy),
+ (Behavioral, team, heavy),
+ (PersonProperty, team, person),
+ (Behavioral, other_team, other),
+ ];
+ let mut tasks = JoinSet::new();
+ let mut inflight = InflightTasks::default();
+ for (kind, team_id, run_id) in held {
+ let run = InflightRun {
+ run_id,
+ team_id,
+ kind,
+ };
+ inflight.insert(tasks.spawn(async {}).id(), run);
+ }
+
+ let cases = [
+ (Behavioral, team, heavy, (2, 2)),
+ (Behavioral, team, sibling, (2, 0)),
+ (PersonProperty, team, person, (1, 1)),
+ (Behavioral, other_team, other, (1, 1)),
+ ];
+ for (kind, team_id, run_id, (team_load, run_load)) in cases {
+ let expected = SlotLoad {
+ team: team_load,
+ run: run_load,
+ };
+ assert_eq!(
+ inflight.load_of(kind, team_id, run_id),
+ expected,
+ "{kind:?} load of {run_id:?} in {team_id:?}"
+ );
+ }
+ }
}
diff --git a/rust/cohort-seeder/src/app/prepare.rs b/rust/cohort-seeder/src/app/prepare.rs
index 317b12c64978..ee8615a5f42b 100644
--- a/rust/cohort-seeder/src/app/prepare.rs
+++ b/rust/cohort-seeder/src/app/prepare.rs
@@ -11,6 +11,7 @@ use std::collections::{HashMap, HashSet};
use std::sync::Arc;
use cohort_core::bucket_tz::window_start_for_now;
+use cohort_core::filters::TeamId;
use common_types::cohort::TeamAllowlist;
use metrics::{counter, gauge};
use sqlx::PgPool;
@@ -43,6 +44,24 @@ pub(super) enum PreparedRun {
Person(Arc<PinnedPersonRun>),
}
+impl PreparedRun {
+ /// The claim budget and [`run_ids_of_kind`] both resolve a run's kind through here, so the
+ /// mapping lives in one place.
+ pub(super) fn kind(&self) -> RunKind {
+ match self {
+ Self::Behavioral(_) => RunKind::Behavioral,
+ Self::Person(_) => RunKind::PersonProperty,
+ }
+ }
+
+ pub(super) fn team_id(&self) -> TeamId {
+ match self {
+ Self::Behavioral(prepared) => prepared.run.team_id,
+ Self::Person(run) => run.team_id,
+ }
+ }
+}
+
/// A behavioral run and the static analysis of its conditions, which every chunk of it shares.
///
/// The analysis lives here rather than on the scan because it is a pure function of the pinned
@@ -384,21 +403,13 @@ fn report_person_analysis(run_id: RunId, run: &PinnedPersonRun) {
);
}
-/// The eligible run ids of one kind — the per-kind label on every shared chunk metric resolves
-/// through here, so the mapping lives in one place.
pub(super) fn run_ids_of_kind(
eligible_runs: &HashMap<RunId, PreparedRun>,
kind: RunKind,
) -> Vec<RunId> {
eligible_runs
.iter()
- .filter(|(_, prepared)| {
- matches!(
- (prepared, kind),
- (PreparedRun::Behavioral(_), RunKind::Behavioral)
- | (PreparedRun::Person(_), RunKind::PersonProperty)
- )
- })
+ .filter(|(_, prepared)| prepared.kind() == kind)
.map(|(run_id, _)| *run_id)
.collect()
}
diff --git a/rust/cohort-seeder/src/domain/fair_share.rs b/rust/cohort-seeder/src/domain/fair_share.rs
new file mode 100644
index 000000000000..c42c7d448a73
--- /dev/null
+++ b/rust/cohort-seeder/src/domain/fair_share.rs
@@ -0,0 +1,33 @@
+//! Domain layer (pure): the claim-slot loads that order the next claim across teams and runs.
+//! Depends on `ids`.
+
+use super::ids::RunId;
+
+/// The claim slots a team and one of its runs hold on this replica, counted within one run kind.
+/// The next claim serves the lowest team load, then the lowest run load inside that team.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq)]
+pub struct SlotLoad {
+ pub team: u32,
+ pub run: u32,
+}
+
+/// A run the next claim may take a chunk from, with the load its team and it already hold.
+#[derive(Debug, Clone, Copy, PartialEq, Eq)]
+pub struct ClaimCandidate {
+ pub run_id: RunId,
+ pub load: SlotLoad,
+}
+
+/// At least one candidate, so a claim over no run cannot be expressed.
+#[derive(Debug)]
+pub struct ClaimCandidates(Vec<ClaimCandidate>);
+
+impl ClaimCandidates {
+ pub fn new(candidates: Vec<ClaimCandidate>) -> Option<Self> {
+ (!candidates.is_empty()).then_some(Self(candidates))
+ }
+
+ pub fn iter(&self) -> std::slice::Iter<'_, ClaimCandidate> {
+ self.0.iter()
+ }
+}
diff --git a/rust/cohort-seeder/src/domain/mod.rs b/rust/cohort-seeder/src/domain/mod.rs
index 3ccb385ff2aa..cdcfa1bd8368 100644
--- a/rust/cohort-seeder/src/domain/mod.rs
+++ b/rust/cohort-seeder/src/domain/mod.rs
@@ -16,6 +16,7 @@ pub mod compare;
pub mod completion;
pub mod condition;
pub mod condition_analysis;
+pub mod fair_share;
pub mod ids;
pub mod ledger;
pub mod newest_day;
@@ -58,6 +59,7 @@ pub use condition::{EventNameSet, Lookback, PinnedCondition};
pub use condition_analysis::{
ConditionAnalyses, ConditionCensus, ConditionClass, MISSING_BYTECODE_REASON,
};
+pub use fair_share::{ClaimCandidate, ClaimCandidates, SlotLoad};
pub use ids::{
Band, ChunkId, ClaimEpoch, ConditionHash, ConditionHashError, DayIdx, RunId, SChunkMs,
ScannedAtMs, UtcMillis, UtcMsRange, UtcRangeError,
diff --git a/rust/cohort-seeder/src/store/chunks.rs b/rust/cohort-seeder/src/store/chunks.rs
index b12153990315..f45afc0d5e49 100644
--- a/rust/cohort-seeder/src/store/chunks.rs
+++ b/rust/cohort-seeder/src/store/chunks.rs
@@ -23,9 +23,9 @@ use uuid::Uuid;
use crate::domain::{
bands_for_day, person_chunk_sentinel_day, AttemptCount, Band, BandSpec, BandSpecError, ChunkId,
- ChunkLease, ChunkSpec, ChunkStatus, ClaimEpoch, ClaimKind, ClaimedChunk, DaySchedule,
- EnqueuedChunk, Halted, PersonRange, PersonRangeError, PlannedDay, ProduceHwms, ProducedChunk,
- RunId, SChunkMs, ScanVolume, StreamedChunk, UtcMillis,
+ ChunkLease, ChunkSpec, ChunkStatus, ClaimCandidates, ClaimEpoch, ClaimKind, ClaimedChunk,
+ DaySchedule, EnqueuedChunk, Halted, PersonRange, PersonRangeError, PlannedDay, ProduceHwms,
+ ProducedChunk, RunId, SChunkMs, ScanVolume, StreamedChunk, UtcMillis,
};
use super::lease::LeaseHandle;
@@ -240,21 +240,38 @@ impl PgChunkStore {
pub async fn claim_next(
&self,
- run_ids: &[RunId],
+ candidates: &ClaimCandidates,
claimant: &Claimant,
lease: LeaseDuration,
max_attempts: MaxAttempts,
) -> Result<Option<Claim>, ChunkStoreError> {
- if run_ids.is_empty() {
- return Ok(None);
- }
- let run_ids = run_ids.iter().map(|run_id| run_id.0).collect::<Vec<_>>();
+ let run_ids = candidates
+ .iter()
+ .map(|candidate| candidate.run_id.0)
+ .collect::<Vec<_>>();
+ let team_loads = candidates
+ .iter()
+ .map(|candidate| i64::from(candidate.load.team))
+ .collect::<Vec<_>>();
+ let run_loads = candidates
+ .iter()
+ .map(|candidate| i64::from(candidate.load.run))
+ .collect::<Vec<_>>();
let sql = format!(
r#"
- WITH next_chunk AS (
+ WITH candidate AS (
+ SELECT * FROM unnest($1::uuid[], $10::bigint[], $11::bigint[])
+ AS u(run_id, team_load, run_load)
+ ), next_chunk AS (
SELECT c.id, c.status IN {active} AS was_reclaim
FROM cohort_backfill_chunks c
+ JOIN candidate u ON u.run_id = c.run_id
JOIN cohort_backfill_runs r ON r.id = c.run_id
+ -- `ANY($1)` repeats the join so that the planner reads only the claimable chunks
+ -- through `cohort_bfc_run_status_day_idx`. With the join alone, it probes each
+ -- candidate through the `(run_id, day, band)` key and filters out every
+ -- confirmed chunk row by row. Only a plan built for the bound arrays does this,
+ -- so the statement is not cached.
WHERE c.run_id = ANY($1) AND r.status IN {seedable}
-- A trailing day is scanned only once it has ended and its run is `trailing`.
-- Claimed while the run still seeds, it would lose its lease when the
@@ -276,12 +293,19 @@ impl PgChunkStore {
-- `r.backfill_kind = $9` is true only for person runs, and `false` sorts first,
-- so behavioral chunks (readiness-gating, fence-sensitive) claim before person
-- chunks. The kind key must lead: person chunks sit on a far-future sentinel day,
- -- which `day DESC` alone would claim first. Newest day first lets a short-window
+ -- which `day DESC` alone would claim first. Of the claimable chunks, only trailing
+ -- days have a `claimable_after`, so the next key puts them ahead of every load,
+ -- because their cohorts already serve as ready and under-count the boundary day
+ -- until the trailing tiles apply. The load keys come next: the team holding
+ -- the fewest of this replica's slots of the kind goes first, then the run holding
+ -- the fewest. A heavy run's chunk holds its slot for hours while a cheap one frees
+ -- its slot in seconds, so under day order alone the heavy run ends up holding
+ -- every slot and every other run advances one day per freed slot. A team alone
+ -- still gets every slot. Among equal loads, newest day first lets a short-window
-- cohort, whose days are all recent, finish without waiting behind a long run's
- -- older days. Claimable trailing days have no key of their own and sort among the
- -- newest days, because their cohorts already serve as ready and under-count the
- -- boundary day until the trailing tiles apply.
- ORDER BY (r.backfill_kind = $9), c.day DESC, c.band
+ -- older days.
+ ORDER BY (r.backfill_kind = $9), (c.claimable_after IS NULL), u.team_load,
+ u.run_load, c.day DESC, c.band
LIMIT 1
FOR UPDATE OF c SKIP LOCKED
)
@@ -304,6 +328,9 @@ impl PgChunkStore {
seedable = SEEDABLE_RUN_STATUSES_SQL,
);
let row = sqlx::query_as::<_, ClaimedRow>(&sql)
+ // Postgres can switch a cached statement to a generic plan after five runs. That plan
+ // loops over the candidates and reads every chunk of each, confirmed ones included.
+ .persistent(false)
.bind(run_ids)
.bind(claimant.as_str())
.bind(lease.as_secs())
@@ -313,6 +340,8 @@ impl PgChunkStore {
.bind(ChunkStatus::Pending.as_str())
.bind(ChunkStatus::Failed.as_str())
.bind(RunKind::PersonProperty.as_str())
+ .bind(team_loads)
+ .bind(run_loads)
.fetch_optional(&self.pool)
.await?;
diff --git a/rust/cohort-seeder/tests/pg_chunks.rs b/rust/cohort-seeder/tests/pg_chunks.rs
index 3b3a5967c45d..266c7f9db62d 100644
--- a/rust/cohort-seeder/tests/pg_chunks.rs
+++ b/rust/cohort-seeder/tests/pg_chunks.rs
@@ -19,8 +19,8 @@ use cohort_seeder::app::reconcile_dispatch::{
};
use cohort_seeder::app::{eligible_run_ids, fail_exhausted_runs_of_kind};
use cohort_seeder::domain::{
- tile_ranges, AttemptCount, ClaimEpoch, PersonRunValidation, PinnedWarning, ProduceHwms,
- RetryBackoffPolicy, ScanVolume, ScopeKind,
+ tile_ranges, AttemptCount, ClaimCandidate, ClaimCandidates, ClaimEpoch, PersonRunValidation,
+ PinnedWarning, ProduceHwms, RetryBackoffPolicy, RunId, ScanVolume, ScopeKind, SlotLoad,
};
use cohort_seeder::store::chunks::{ChunkStoreError, PgChunkStore, PlanOutcome, NO_ERROR_RECORDED};
use cohort_seeder::store::lease::LeaseFailure;
@@ -47,6 +47,18 @@ const ONE_BAND: NonZeroU16 = NonZeroU16::MIN;
/// something other than the backoff and want the chunk claimable on the next call.
const NO_BACKOFF: Duration = Duration::ZERO;
+/// Candidates that hold no claim slot, so they claim in kind, day and band order.
+fn idle(run_ids: &[RunId]) -> ClaimCandidates {
+ let candidates = run_ids
+ .iter()
+ .map(|&run_id| ClaimCandidate {
+ run_id,
+ load: SlotLoad::default(),
+ })
+ .collect();
+ ClaimCandidates::new(candidates).expect("every scenario claims from at least one run")
+}
+
/// Discovery honors an `Only` allowlist (self team only) and `All` admits every eligible run
/// regardless of team, trigger, or already-seeding status.
#[tokio::test]
@@ -567,7 +579,7 @@ async fn planning_fans_out_bands_and_claims_carry_the_band_count() -> Result<()>
let attempts5 = MaxAttempts::new(5)?;
let claimed = store
.claim_next(
- &[seeding_run],
+ &idle(&[seeding_run]),
&Claimant::new("worker-a")?,
lease60,
attempts5,
@@ -600,14 +612,14 @@ async fn concurrent_claims_take_disjoint_chunks() -> Result<()> {
let lease60 = LeaseDuration::new(Duration::from_secs(60))?;
let attempts5 = MaxAttempts::new(5)?;
- let run_ids = [seeding_run];
+ let candidates = idle(&[seeding_run]);
let first_store = store.clone();
let second_store = store.clone();
let claimant_a = Claimant::new("worker-a")?;
let claimant_b = Claimant::new("worker-b")?;
let (first, second) = tokio::join!(
- first_store.claim_next(&run_ids, &claimant_a, lease60, attempts5),
- second_store.claim_next(&run_ids, &claimant_b, lease60, attempts5),
+ first_store.claim_next(&candidates, &claimant_a, lease60, attempts5),
+ second_store.claim_next(&candidates, &claimant_b, lease60, attempts5),
);
let first = first?.context("first claimant found no chunk")?;
let second = second?.context("second claimant found no chunk")?;
@@ -632,7 +644,7 @@ async fn expired_lease_reclaim_bumps_epoch_and_fences_the_stale_lease() -> Resul
let run_ids = [seeding_run];
let first = store
- .claim_next(&run_ids, &Claimant::new("worker-a")?, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &Claimant::new("worker-a")?, lease60, attempts5)
.await?
.context("first claimant found no chunk")?;
let stale = first.chunk.spec().lease;
@@ -645,7 +657,7 @@ async fn expired_lease_reclaim_bumps_epoch_and_fences_the_stale_lease() -> Resul
.await?;
let reclaimed = store
- .claim_next(&run_ids, &Claimant::new("worker-c")?, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &Claimant::new("worker-c")?, lease60, attempts5)
.await?
.context("expired chunk was not reclaimed")?;
let reclaimed_lease = reclaimed.chunk.spec().lease;
@@ -691,7 +703,12 @@ async fn unclaim_returns_chunk_to_pending_and_refunds_one_attempt() -> Result<()
let run_ids = [seeding_run];
let claimed = store
- .claim_next(&run_ids, &Claimant::new("worker-a")?, lease60, attempts5)
+ .claim_next(
+ &idle(&run_ids),
+ &Claimant::new("worker-a")?,
+ lease60,
+ attempts5,
+ )
.await?
.context("claimant found no chunk")?;
let lease = claimed.chunk.spec().lease;
@@ -732,7 +749,7 @@ async fn a_failed_chunk_waits_out_its_backoff_before_it_is_claimable_again() ->
let run_ids = [seeding_run];
let claimed = store
- .claim_next(&run_ids, &Claimant::new("worker-a")?, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &Claimant::new("worker-a")?, lease60, attempts5)
.await?
.context("claimant found no chunk")?;
let lease = claimed.chunk.spec().lease;
@@ -761,7 +778,7 @@ async fn a_failed_chunk_waits_out_its_backoff_before_it_is_claimable_again() ->
"expected a ~600s hold, got {remaining_secs}s"
);
ensure!(store
- .claim_next(&run_ids, &Claimant::new("worker-b")?, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &Claimant::new("worker-b")?, lease60, attempts5)
.await?
.is_none());
@@ -774,7 +791,7 @@ async fn a_failed_chunk_waits_out_its_backoff_before_it_is_claimable_again() ->
.execute(&pool)
.await?;
let retried = store
- .claim_next(&run_ids, &Claimant::new("worker-c")?, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &Claimant::new("worker-c")?, lease60, attempts5)
.await?
.context("chunk past its backoff was not claimable")?;
ensure!(retried.chunk.spec().attempt.get() == 2);
@@ -829,7 +846,12 @@ async fn the_recovery_path_draws_its_wait_from_the_chunks_attempt_count() -> Res
let mut longest_wait = Duration::ZERO;
for day in days {
let claimed = store
- .claim_next(&run_ids, &Claimant::new("worker-a")?, lease60, attempts50)
+ .claim_next(
+ &idle(&run_ids),
+ &Claimant::new("worker-a")?,
+ lease60,
+ attempts50,
+ )
.await?
.context("a chunk under the cap was not claimable")?;
let chunk = claimed.chunk;
@@ -896,7 +918,7 @@ async fn the_backoff_gate_applies_only_to_the_failed_claim_arm() -> Result<()> {
.execute(&pool)
.await?;
let pending = store
- .claim_next(&run_ids, &Claimant::new("worker-a")?, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &Claimant::new("worker-a")?, lease60, attempts5)
.await?
.context("a pending chunk was gated by the backoff stamp")?;
let pending_lease = pending.chunk.spec().lease;
@@ -911,7 +933,7 @@ async fn the_backoff_gate_applies_only_to_the_failed_claim_arm() -> Result<()> {
.execute(&pool)
.await?;
let reclaimed = store
- .claim_next(&run_ids, &Claimant::new("worker-b")?, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &Claimant::new("worker-b")?, lease60, attempts5)
.await?
.context("an unclaimed chunk was gated by the backoff stamp")?;
@@ -929,7 +951,7 @@ async fn the_backoff_gate_applies_only_to_the_failed_claim_arm() -> Result<()> {
.execute(&pool)
.await?;
let produced_reclaim = store
- .claim_next(&run_ids, &Claimant::new("worker-c")?, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &Claimant::new("worker-c")?, lease60, attempts5)
.await?
.context("an expired produced lease was gated by the backoff stamp")?;
@@ -948,7 +970,7 @@ async fn the_backoff_gate_applies_only_to_the_failed_claim_arm() -> Result<()> {
.execute(&pool)
.await?;
ensure!(store
- .claim_next(&run_ids, &Claimant::new("worker-d")?, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &Claimant::new("worker-d")?, lease60, attempts5)
.await?
.is_some());
Ok(())
@@ -995,13 +1017,13 @@ async fn a_trailing_day_waits_out_its_hold_and_completes_its_trailing_run() -> R
);
let historical_claim = store
- .claim_next(&run_ids, &claimant, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &claimant, lease60, attempts5)
.await?
.context("the historical day was not claimable")?;
ensure!(historical_claim.chunk.spec().day == 100);
ensure!(
store
- .claim_next(&run_ids, &claimant, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &claimant, lease60, attempts5)
.await?
.is_none(),
"the trailing day was claimed before its hold lapsed"
@@ -1050,7 +1072,7 @@ async fn a_trailing_day_waits_out_its_hold_and_completes_its_trailing_run() -> R
ensure!(complete_trailing_runs(&pool, &run_ids).await? == 0);
let trailing_claim = store
- .claim_next(&run_ids, &claimant, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &claimant, lease60, attempts5)
.await?
.context("the trailing day was not claimable after its hold lapsed")?;
ensure!(trailing_claim.chunk.spec().day == 101);
@@ -1101,7 +1123,12 @@ async fn a_seeding_run_neither_hands_out_a_lapsed_held_chunk_nor_completes_as_tr
.await?;
ensure!(
store
- .claim_next(&run_ids, &Claimant::new("worker-a")?, lease60, attempts5)
+ .claim_next(
+ &idle(&run_ids),
+ &Claimant::new("worker-a")?,
+ lease60,
+ attempts5
+ )
.await?
.is_none(),
"a held chunk was claimed while its run was still seeding"
@@ -1148,7 +1175,7 @@ async fn mark_produced_records_the_scan_byte_volume_that_planning_defaults_to_ze
let claimed = store
.claim_next(
- &[seeding_run],
+ &idle(&[seeding_run]),
&Claimant::new("worker-a")?,
LeaseDuration::new(Duration::from_secs(60))?,
MaxAttempts::new(5)?,
@@ -1199,7 +1226,7 @@ async fn attempt_cap_is_terminal_for_failed_but_reclaims_expired_produced() -> R
.execute(&pool)
.await?;
let retry = store
- .claim_next(&run_ids, &Claimant::new("worker-d")?, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &Claimant::new("worker-d")?, lease60, attempts5)
.await?
.context("chunk under the cap was not claimable")?;
let retry_lease = retry.chunk.spec().lease;
@@ -1214,14 +1241,14 @@ async fn attempt_cap_is_terminal_for_failed_but_reclaims_expired_produced() -> R
.await?;
drop(retry);
ensure!(store
- .claim_next(&run_ids, &Claimant::new("worker-terminal")?, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &Claimant::new("worker-terminal")?, lease60, attempts5)
.await?
.is_none());
// A second chunk, marked produced then expired at the cap, IS reclaimed.
ensure!(planned_count(store.plan_chunks(seeding_run, historical([101]), ONE_BAND).await?)? == 1);
let final_attempt = store
- .claim_next(&run_ids, &Claimant::new("worker-e")?, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &Claimant::new("worker-e")?, lease60, attempts5)
.await?
.context("second chunk was not claimable")?;
let final_attempt_lease = final_attempt.chunk.spec().lease;
@@ -1235,7 +1262,7 @@ async fn attempt_cap_is_terminal_for_failed_but_reclaims_expired_produced() -> R
.await?;
let observed = store
- .claim_next(&run_ids, &Claimant::new("worker-f")?, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &Claimant::new("worker-f")?, lease60, attempts5)
.await?
.context("expired produced chunk at the attempt cap was not reclaimed")?;
let observed_lease = observed.chunk.spec().lease;
@@ -1271,7 +1298,7 @@ async fn expired_scanning_chunk_at_the_cap_is_reaped_not_reclaimed() -> Result<(
let attempts5 = MaxAttempts::new(5)?;
let run_ids = [seeding_run];
let claimed = store
- .claim_next(&run_ids, &Claimant::new("worker-a")?, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &Claimant::new("worker-a")?, lease60, attempts5)
.await?
.context("claimant found no chunk")?;
let chunk_id = claimed.chunk.spec().lease.chunk_id();
@@ -1292,7 +1319,7 @@ async fn expired_scanning_chunk_at_the_cap_is_reaped_not_reclaimed() -> Result<(
.execute(&pool)
.await?;
ensure!(store
- .claim_next(&run_ids, &Claimant::new("worker-b")?, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &Claimant::new("worker-b")?, lease60, attempts5)
.await?
.is_none());
@@ -1306,7 +1333,7 @@ async fn expired_scanning_chunk_at_the_cap_is_reaped_not_reclaimed() -> Result<(
ensure!(last_error.contains("attempt cap"));
// Terminal: not claimable, and a second sweep is a no-op.
ensure!(store
- .claim_next(&run_ids, &Claimant::new("worker-c")?, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &Claimant::new("worker-c")?, lease60, attempts5)
.await?
.is_none());
ensure!(store.reap_poisoned_chunks(&run_ids, attempts5).await? == 0);
@@ -1478,7 +1505,7 @@ async fn exhausted_chunk_fails_the_run_and_stops_further_claims() -> Result<()>
// One chunk mid-scan with a live lease, one capped out, one left pending.
let claimed = store
- .claim_next(&run_ids, &claimant, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &claimant, lease60, attempts5)
.await?
.context("claimant found no chunk")?;
let live_lease = claimed.chunk.spec().lease;
@@ -1524,7 +1551,7 @@ async fn exhausted_chunk_fails_the_run_and_stops_further_claims() -> Result<()>
// The claim and heartbeat predicates both join `runs.status = 'seeding'`, so the pending
// sibling stops being claimable and the in-flight one halts on its next beat.
ensure!(store
- .claim_next(&run_ids, &Claimant::new("worker-b")?, lease60, attempts5)
+ .claim_next(&idle(&run_ids), &Claimant::new("worker-b")?, lease60, attempts5)
.await?
.is_none());
let still_pending: i64 = sqlx::query_scalar(
@@ -1582,7 +1609,12 @@ async fn fail_truncates_chunk_and_run_error_columns() -> Result<()> {
let attempts5 = MaxAttempts::new(5)?;
let run_ids = [seeding_run];
let claimed = store
- .claim_next(&run_ids, &Claimant::new("worker-a")?, lease60, attempts5)
+ .claim_next(
+ &idle(&run_ids),
+ &Claimant::new("worker-a")?,
+ lease60,
+ attempts5,
+ )
.await?
.context("claimant found no chunk")?;
let lease = claimed.chunk.spec().lease;
@@ -1810,9 +1842,10 @@ async fn person_planning_is_all_or_nothing_and_idempotent() -> Result<()> {
.await
}
-/// Behavioral days claim newest first across runs, and person chunks claim strictly after every
-/// behavioral day on the shared claim loop even though their sentinel day is the newest. A person
-/// claim decodes its range with `num_bands` = the run's chunk count.
+/// Behavioral days claim newest first across runs with equal slot loads, and person chunks claim
+/// strictly after every behavioral day on the shared claim loop, even though their sentinel day is
+/// the newest and their run holds fewer slots. A person claim decodes its range with `num_bands` =
+/// the run's chunk count.
#[tokio::test]
async fn behavioral_days_claim_newest_first_then_person_chunks_with_ranges() -> Result<()> {
with_db(|pool| async move {
@@ -1843,13 +1876,28 @@ async fn behavioral_days_claim_newest_first_then_person_chunks_with_ranges() ->
let lease60 = LeaseDuration::new(Duration::from_secs(60))?;
let attempts5 = MaxAttempts::new(5)?;
- let run_ids = [behavioral_run, other_team_run, person_run];
+ let busy = SlotLoad { team: 1, run: 1 };
+ let candidates = ClaimCandidates::new(vec![
+ ClaimCandidate {
+ run_id: behavioral_run,
+ load: busy,
+ },
+ ClaimCandidate {
+ run_id: other_team_run,
+ load: busy,
+ },
+ ClaimCandidate {
+ run_id: person_run,
+ load: SlotLoad::default(),
+ },
+ ])
+ .context("no claim candidates")?;
let claimant = Claimant::new("worker-a")?;
let mut behavioral_claims = Vec::new();
for _ in 0..3 {
let claim = store
- .claim_next(&run_ids, &claimant, lease60, attempts5)
+ .claim_next(&candidates, &claimant, lease60, attempts5)
.await?
.context("behavioral chunk was not claimed before person chunks")?;
ensure!(claim.chunk.spec().person_range.is_none());
@@ -1866,7 +1914,7 @@ async fn behavioral_days_claim_newest_first_then_person_chunks_with_ranges() ->
);
let first_person = store
- .claim_next(&run_ids, &claimant, lease60, attempts5)
+ .claim_next(&candidates, &claimant, lease60, attempts5)
.await?
.context("first person chunk was not claimed")?;
let first_person_spec = first_person.chunk.spec();
@@ -1880,7 +1928,7 @@ async fn behavioral_days_claim_newest_first_then_person_chunks_with_ranges() ->
ensure!(first_person_range.hi() == Some(boundary));
let second_person = store
- .claim_next(&run_ids, &claimant, lease60, attempts5)
+ .claim_next(&candidates, &claimant, lease60, attempts5)
.await?
.context("second person chunk was not claimed")?;
let second_person_range = second_person
@@ -1896,6 +1944,89 @@ async fn behavioral_days_claim_newest_first_then_person_chunks_with_ranges() ->
.await
}
+/// A claim takes a lapsed trailing day first, then serves the team holding the fewest slots, then
+/// the run holding the fewest, and only then the newest day.
+#[tokio::test]
+async fn claims_serve_the_least_loaded_team_then_run_before_the_newest_day() -> Result<()> {
+ with_db(|pool| async move {
+ let heavy_run =
+ insert_run(&pool, 2, "team_enablement", "seeding", true, empty_pinned()).await?;
+ // A team holds one active team-scoped run per kind, so the first run moves to cohort
+ // scope before the second is inserted.
+ sqlx::query(
+ "UPDATE cohort_backfill_runs SET scope = 'cohort', cohort_id = 1 WHERE id = $1",
+ )
+ .bind(heavy_run)
+ .execute(&pool)
+ .await?;
+ let sibling_run =
+ insert_run(&pool, 2, "team_enablement", "seeding", true, empty_pinned()).await?;
+ let other_team_run =
+ insert_run(&pool, 3, "team_enablement", "seeding", true, empty_pinned()).await?;
+ let trailing_run =
+ insert_run(&pool, 4, "team_enablement", "seeding", true, empty_pinned()).await?;
+ let store = PgChunkStore::new(pool.clone());
+ for (run_id, day) in [(heavy_run, 100), (sibling_run, 98), (other_team_run, 97)] {
+ ensure!(
+ planned_count(
+ store
+ .plan_chunks(run_id, historical([day]), ONE_BAND)
+ .await?
+ )? == 1
+ );
+ }
+ let lapsed = Utc::now() - chrono::Duration::hours(1);
+ ensure!(
+ planned_count(
+ store
+ .plan_chunks(trailing_run, [trailing(96, lapsed)], ONE_BAND)
+ .await?
+ )? == 1
+ );
+ sqlx::query("UPDATE cohort_backfill_runs SET status = 'trailing' WHERE id = $1")
+ .bind(trailing_run)
+ .execute(&pool)
+ .await?;
+
+ let lease60 = LeaseDuration::new(Duration::from_secs(60))?;
+ let attempts5 = MaxAttempts::new(5)?;
+ let claimant = Claimant::new("worker-a")?;
+ let rounds = [
+ // Team 4 holds the most slots and its trailing day is the oldest day, but the trailing
+ // day still goes first.
+ ([(2, 2), (2, 0), (1, 1), (3, 0)], (trailing_run, 96)),
+ // Team 2 holds two slots and team 3 one, so team 3 goes first, ahead of team 2's newer
+ // days and of its sibling run that holds none.
+ ([(2, 2), (2, 0), (1, 1), (4, 1)], (other_team_run, 97)),
+ // Both teams hold one slot, so inside team 2 the run holding none goes first, ahead of
+ // the heavy run's newer day.
+ ([(1, 1), (1, 0), (1, 1), (4, 1)], (sibling_run, 98)),
+ ];
+ for (loads, expected) in rounds {
+ let candidates = [heavy_run, sibling_run, other_team_run, trailing_run]
+ .into_iter()
+ .zip(loads)
+ .map(|(run_id, (team, run))| ClaimCandidate {
+ run_id,
+ load: SlotLoad { team, run },
+ })
+ .collect();
+ let candidates = ClaimCandidates::new(candidates).context("no claim candidates")?;
+ let claim = store
+ .claim_next(&candidates, &claimant, lease60, attempts5)
+ .await?
+ .context("no chunk was claimed")?;
+ let claimed = (claim.chunk.spec().lease.run_id(), claim.chunk.spec().day);
+ ensure!(
+ claimed == expected,
+ "expected {expected:?}, got {claimed:?}"
+ );
+ }
+ Ok(())
+ })
+ .await
+}
+
/// Cancelling a run breaks its live leases: the claimed chunk's next heartbeat fences out, so the
/// lease surfaces `Heartbeat(LeaseLost)` and its cancellation token trips.
#[tokio::test]
@@ -1910,7 +2041,7 @@ async fn cancelling_a_run_kills_its_live_lease_via_the_heartbeat() -> Result<()>
let attempts5 = MaxAttempts::new(5)?;
let run_ids = [seeding_run];
let observed = store
- .claim_next(&run_ids, &Claimant::new("worker-f")?, lease3, attempts5)
+ .claim_next(&idle(&run_ids), &Claimant::new("worker-f")?, lease3, attempts5)
.await?
.context("claimant found no chunk")?;
let cancelled = observed.lease.cancellation_token();