:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(cohorts): pack the shuffler's input topic into source_partition (#121826)
Phil Haack09afae723a28
diff --git a/docs/internal/realtime-cohorts/README.md b/docs/internal/realtime-cohorts/README.mdindex 812dd7c7ec73..74f773320142 100644--- a/docs/internal/realtime-cohorts/README.md+++ b/docs/internal/realtime-cohorts/README.md@@ -147,8 +147,9 @@ Each page states the setup its example needs, and the details differ where a pag| Stage 1 | Folding an event into per-person leaf state and detecting flips || Stage 2 | Recomposing a cohort from its leaves when one of them flips || Person record | One row per person holding which person conditions match, plus fingerprints that skip unchanged work |-| Replay marks | Per row, the highest firehose offset applied from each firehose partition. Used to skip redelivered events |-| Firehose | The ingestion topic `clickhouse_events_json`, which the shuffler reads |+| Replay marks | Per row, the highest offset applied from each source partition. Used to skip redelivered events |+| Firehose | The ingestion topic `clickhouse_events_json`, which the shuffler reads by default |+| Source partition | The envelope's `source_partition`: one partition of one topic a shuffler reads, in that topic's range of numbers || Eviction deadline | The earliest time a leaf's answer can change without a new event || Sweep | The periodic pass that applies eviction deadlines || Tombstone | A forwarding address from a merged-away person to the survivor |diff --git a/docs/internal/realtime-cohorts/event-routing.md b/docs/internal/realtime-cohorts/event-routing.mdindex a8ed7255b1d7..782f95a72bb5 100644--- a/docs/internal/realtime-cohorts/event-routing.md+++ b/docs/internal/realtime-cohorts/event-routing.md@@ -78,15 +78,17 @@ Each forwarded event becomes a slim JSON record on `cohort_stream_events`.A change made any other way, such as an API edit or a person merge, is seen on the person's next event.- Events ingested **without person processing** still carry a real `person_id` when the distinct id has a person, but their `person_properties` is an empty object.The envelope does not say which kind of event it is, so the processor evaluates those events as a person with no properties.-- `source_partition` and `source_offset` are the event's coordinates in the firehose.+- `source_partition` and `source_offset` are the event's coordinates in the topic the shuffler read.The processor uses them to recognize an event it already applied.+- `source_partition` is the real Kafka partition only for `clickhouse_events_json`.+ [Running a shuffler on another topic](#running-a-shuffler-on-another-topic) explains how other topics number it.- Events that the processor re-keys after a person merge also carry `redirected_from` and `redirect_hops`.The shuffler never sets them.### Delivery and commitsThe shuffler sends forwards without waiting for each acknowledgment, and keeps a per-partition ledger of forwards still in flight.-The offset it commits for a firehose partition is the oldest offset still in flight, or one past the highest offset seen when nothing is in flight.+The offset it commits for an input partition is the oldest offset still in flight, or one past the highest offset seen when nothing is in flight.It commits every few seconds.Across crashes and rebalances this is at least once: a restart replays the uncommitted tail, and the processor discards what it already applied.@@ -103,6 +105,33 @@ The consumer group starts at the tail of the firehose when it has no committed oA shuffler outage longer than the retention silently skips the gap.History is the backfill's job, not the shuffler's.+### Running a shuffler on another topic++A second shuffler deployment can read another ingestion topic, such as `realtime_only_events_json`, into the same `cohort_stream_events`.+Kafka numbers every topic's partitions from 0, and the processor keys its replay marks by `source_partition` alone.+If two topics wrote the same numbers, the processor would drop most of the second topic's events as replays, because the firehose's offsets are far higher.++So the shuffler gives each input topic a slot in a fixed table and writes partition `p` of slot `s` as `s * 1_000_000 + p`.+`clickhouse_events_json` has slot 0 and writes the real partition.+`realtime_only_events_json` has slot 1.+The processor and its stored state do not change, so any processor build handles the second topic, and a processor rollback needs no wipe.+To find the Kafka partition of an envelope, take `source_partition` modulo 1,000,000.+The shuffler stops with a failure on a partition at or above 1,000,000, because that number falls in the next slot's range.++Starting one commits the pipeline to these conditions:++- The topic has a slot in the shuffler's table.+ The shuffler refuses to start on a topic that does not.+- A slot never passes to another topic.+ Stored replay marks keep the numbers that the first topic wrote, and they would mask the new topic's events.+- The deployment has its own `KAFKA_CONSUMER_GROUP`.+ Commits stay correct in a shared group, because Kafka keeps offsets per topic.+ But a pod roll in either deployment would then rebalance both, and each would replay its uncommitted tail.+- Each event reaches the pipeline on only one input topic.+ The two copies of such an event have different source partitions, so the processor folds it twice.+- The topic's retention is no longer than the firehose's.+ The processor sizes its tombstone retention from the firehose's retention, and a straggler that outlives the tombstone is not redirected.+## Partition affinityPartition affinity is the invariant everything else rests on.@@ -205,8 +234,8 @@ The processor sees firehose offset 9870001 from partition 3 for p-1 a second tim- The shuffler and the processor each read their own copy of `REALTIME_COHORT_TEAM_ALLOWLIST` at startup, and each polls Postgres on its own schedule.Only teams in both are processed, and a newly enabled team can lose a few minutes of events until both have refreshed.Merges also need the team in Node's merge allowlist, and backfills need it in the seeder's copy.-- The processor recognizes applied events with a high-water mark per firehose partition, not a set of offsets.- The shuffler's producer does not guarantee ordering on retries, so an event that is overtaken by a later event from the same firehose partition is skipped as if it were a replay.+- The processor recognizes applied events with a high-water mark per source partition, not a set of offsets.+ The shuffler's producer does not guarantee ordering on retries, so an event that is overtaken by a later event from the same source partition is skipped as if it were a replay.- The broker timestamp on `cohort_stream_events` is the shuffler's send time, not the event's time.The processor's live watermark, which paces backfill, is built from these timestamps.[Seed apply and reconcile](seed-apply-and-reconcile.md) explains why that matters.diff --git a/docs/internal/realtime-cohorts/invariants.md b/docs/internal/realtime-cohorts/invariants.mdindex b020d1c0d28b..f160af507d13 100644--- a/docs/internal/realtime-cohorts/invariants.md+++ b/docs/internal/realtime-cohorts/invariants.md@@ -95,9 +95,12 @@ Most of these are not enforced by a single test, and a violation usually shows u**12. A redelivered event changes nothing.**- Why: input is at least once.-- Kept by: per-row replay marks, the highest firehose offset applied from each firehose partition.-- Assumes: one person's events from one firehose partition arrive in offset order.- An event overtaken by a later one from the same firehose partition is skipped as a replay.+- Kept by: per-row replay marks, the highest offset applied from each source partition, and the shuffler's partition numbering, which gives each input topic its own source partitions.+- Assumes: one person's events from one source partition arrive in offset order.+ An event overtaken by a later one from the same source partition is skipped as a replay.+- Assumes: each event reaches the pipeline on only one input topic.+ An event that arrives on two topics has a different source partition on each, so it is folded twice.+- See [live evaluation](live-evaluation.md#replay-safety).- Limit: the marks live inside their row.Once the sweep deletes a row, or the person record time-to-live expires a record, a redelivered old event is folded again.diff --git a/docs/internal/realtime-cohorts/live-evaluation.md b/docs/internal/realtime-cohorts/live-evaluation.mdindex 1a56ed531ef8..95b964eda9d8 100644--- a/docs/internal/realtime-cohorts/live-evaluation.md+++ b/docs/internal/realtime-cohorts/live-evaluation.md@@ -132,16 +132,16 @@ It stores the set of person condition hashes that currently match, plus:- a **properties fingerprint**, a digest of the raw `person_properties` string last evaluated,- a **catalog fingerprint**, a digest of the team's distinct person condition hashes when last evaluated,-- a **stamp**, the event time and firehose offset of the newest event applied.+- a **stamp**, the event time and source offset of the newest event applied.For each event with person properties, the worker makes one of four decisions.-| Decision | When | What happens |-| --------------- | --------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |-| Replay | The record's replay mark for the event's firehose partition is at or above the event's offset | Nothing |-| Stale | The event is not newer than the stamp, comparing event time and then firehose offset | Record the event as seen. Membership does not change |-| Skip evaluation | Both fingerprints match the record | Adopt the new stamp. No HogVM runs |-| Evaluate | Either fingerprint changed, or there is no record | Parse the properties, run every person condition of the team, diff against the stored set, and emit a transition for each hash that entered or left |+| Decision | When | What happens |+| --------------- | ------------------------------------------------------------------------------------------- | --------------------------------------------------------------------------------------------------------------------------------------------------- |+| Replay | The record's replay mark for the event's source partition is at or above the event's offset | Nothing |+| Stale | The event is not newer than the stamp, comparing event time and then source offset | Record the event as seen. Membership does not change |+| Skip evaluation | Both fingerprints match the record | Adopt the new stamp. No HogVM runs |+| Evaluate | Either fingerprint changed, or there is no record | Parse the properties, run every person condition of the team, diff against the stored set, and emit a transition for each hash that entered or left |Most events run no person condition, because they are either stale or have unchanged fingerprints.When something does change, the worker evaluates the team's whole set of person conditions, not only the ones touching the changed key.@@ -161,15 +161,17 @@ Reconcile and backfill clean up membership for cohorts whose definitions changedEvents arrive at least once.A crash, a rebalance or a shuffler retry can deliver the same event twice.-Counting it twice would corrupt a daily counter, so every behavioral row and every person record keeps **replay marks**: for each partition of the ingestion topic the shuffler reads, called the firehose in [event routing](event-routing.md), the highest offset folded into this row.-An event whose offset is at or below the mark for its firehose partition is a replay and changes nothing.+Counting it twice would corrupt a daily counter, so every behavioral row and every person record keeps **replay marks**: for each source partition, the highest offset folded into this row.+A source partition is one partition of a topic a shuffler reads, named by the envelope's `source_partition`.+Each topic writes its partitions into its own range of numbers ([event routing](event-routing.md#running-a-shuffler-on-another-topic)), so a mark from one topic never decides a replay on another.+An event whose offset is at or below the mark for its source partition is a replay and changes nothing.The marks are per row, not global.Two leaves of the same person each know independently which events they already counted.Rows that absorbed a merged person also keep that person's marks under the merged-away person's id, so a late event for the merged-away person is still recognized.-This check relies on one ordering property: events from one firehose partition for one person reach the processor in offset order.-If that breaks, an event that arrives after a higher offset from its firehose partition is dropped as a replay.+This check relies on one ordering property: events from one source partition for one person reach the processor in offset order.+If that breaks, an event that arrives after a higher offset from its source partition is dropped as a replay.The marks also live inside their row, so when the sweep deletes a row, a redelivered old event is folded again.## Stage 2: composing cohortsdiff --git a/docs/internal/realtime-cohorts/merges-and-cascades.md b/docs/internal/realtime-cohorts/merges-and-cascades.mdindex b82159844b53..70df9c7988dc 100644--- a/docs/internal/realtime-cohorts/merges-and-cascades.md+++ b/docs/internal/realtime-cohorts/merges-and-cascades.md@@ -105,7 +105,7 @@ Adding counts is right because each live event was folded into exactly one of thOne narrow race breaks that: a backfill tile scanned after the merge, which already counts the old person's events under the survivor, can land on the survivor before the transfer does.The replay-marks rule matters for late events.-An event for the old person that is re-addressed to the survivor keeps its own firehose coordinates, the event's partition and offset on the ingestion topic.+An event for the old person that is re-addressed to the survivor keeps its own source coordinates, the event's `source_partition` and `source_offset`.The survivor checks them against the marks kept under the old person's id.An event the old person already counted is skipped, and a new one is counted.diff --git a/docs/internal/realtime-cohorts/state-store-and-durability.md b/docs/internal/realtime-cohorts/state-store-and-durability.mdindex 17cd74490b5b..78c4ad527c5d 100644--- a/docs/internal/realtime-cohorts/state-store-and-durability.md+++ b/docs/internal/realtime-cohorts/state-store-and-durability.md@@ -22,7 +22,7 @@ A column family is a separate keyspace inside the database, with its own setting| Column family | One row per | Holds || ------------------------- | ------------------------------------ | -------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |-| `cf_behavioral` | person and behavioral leaf state key | Behavioral leaf state, its eviction deadline, and replay marks: the firehose offsets already applied |+| `cf_behavioral` | person and behavioral leaf state key | Behavioral leaf state, its eviction deadline, and replay marks: the source offsets already applied || `cf_person_records` | person | The person record: matched person conditions, fingerprints, stamp, replay marks || `cf_stage2` | cohort and person | The membership bit the processor registered for the pair. On the live, sweep, merge, cascade and reconcile paths it is the recomputed truth. On the seed path it is what downstream was last told. Also reconcile dirty markers, and an inventory of membership rows received through merges || `cf_merge_drains_applied` | merge message | Marker: this merge was drained |diff --git a/rust/cohort-core/src/events.rs b/rust/cohort-core/src/events.rsindex 10247f66b433..45990275086a 100644--- a/rust/cohort-core/src/events.rs+++ b/rust/cohort-core/src/events.rs@@ -19,6 +19,9 @@ pub struct CohortStreamEvent {pub person_properties: Option<String>,pub elements_chain: Option<String>,pub source_offset: i64,+ /// Not always a Kafka partition. Each shuffler input topic writes its partitions into its own+ /// range (`SourcePartitionRange` in the shuffler), so a reader must treat the value as an opaque+ /// `i32` key and must not bound or narrow it.pub source_partition: i32,/// The merge origin person, set when a post-merge straggler is redirected to the merged-into/// person. `None` for a normal event. Stage 1 routes replay-dedup throughdiff --git a/rust/cohort-event-shuffler/src/config.rs b/rust/cohort-event-shuffler/src/config.rsindex c73d7ecc9dfc..63e1559164a3 100644--- a/rust/cohort-event-shuffler/src/config.rs+++ b/rust/cohort-event-shuffler/src/config.rs@@ -2,12 +2,14 @@use std::time::Duration;+use anyhow::Context;use common_database::PoolConfig;use common_kafka::config::{ConsumerConfig, KafkaConfig};use common_types::cohort::TeamAllowlist;use envconfig::Envconfig;use crate::consumer::ShufflerSettings;+use crate::event::SourcePartitionRange;const POOL_NAME: &str = "posthog_cohort";@@ -119,12 +121,20 @@ impl Config {format!("{}:{}", self.bind_host, self.bind_port)}- pub fn shuffler_settings(&self) -> ShufflerSettings {- ShufflerSettings {+ pub fn shuffler_settings(&self) -> anyhow::Result<ShufflerSettings> {+ let source_partitions = SourcePartitionRange::for_input_topic(&self.input_topic)+ .with_context(|| {+ format!(+ "INPUT_TOPIC {:?} has no slot in the shuffler's input topic table",+ self.input_topic+ )+ })?;+ Ok(ShufflerSettings {+ source_partitions,max_inflight_forwards: self.max_inflight_forwards,commit_interval: Duration::from_millis(self.commit_interval_ms),queue_full_backoff: Duration::from_millis(self.queue_full_backoff_ms),- }+ })}pub fn team_index_refresh_interval(&self) -> Duration {@@ -262,6 +272,13 @@ mod tests {assert_eq!(consumer.kafka_consumer_topic, "clickhouse_events_json");}+ #[test]+ fn shuffler_settings_refuse_an_input_topic_without_a_slot() {+ let mut config = test_config();+ config.input_topic = "clickhouse_events_json_v2".to_string();+ assert!(config.shuffler_settings().is_err());+ }+#[test]fn statement_timeout_zero_means_database_default() {let mut config = test_config();diff --git a/rust/cohort-event-shuffler/src/consumer.rs b/rust/cohort-event-shuffler/src/consumer.rsindex 1ec6c99852bf..61f4dc7c751a 100644--- a/rust/cohort-event-shuffler/src/consumer.rs+++ b/rust/cohort-event-shuffler/src/consumer.rs@@ -1,4 +1,4 @@-//! Pipelined consume → filter → re-key → produce loop for `clickhouse_events_json`.+//! Pipelined consume → filter → re-key → produce loop for the input topic.//!//! Drops events with no `person_id`, skips teams with no realtime cohorts, then re-keys the//! survivors by `(team_id, person_id)`, gating on the realtime-cohort filter catalog.@@ -24,7 +24,7 @@ use serde::{de::IgnoredAny, Deserialize};use tokio::task::JoinError;use tracing::{debug, error, info, warn};-use crate::event::{personless_mode, CohortStreamEvent};+use crate::event::{personless_mode, CohortStreamEvent, SourcePartitionRange};use crate::filter_team_index::TeamIndex;use crate::ledger::{DeliveryOutcome, Ledger, NextOffset, Observation, Resolution, SourceOffset, SourcePartition,@@ -151,6 +151,7 @@ type CommitFuture =#[derive(Debug, Clone, Copy)]pub struct ShufflerSettings {+ pub source_partitions: SourcePartitionRange,pub max_inflight_forwards: usize,pub commit_interval: Duration,pub queue_full_backoff: Duration,@@ -339,6 +340,19 @@ impl EventShuffler {counter!(EVENTS_CONSUMED).increment(1);let partition = SourcePartition(offset.partition());let source_offset = SourceOffset(offset.get_value());+ // Returning before the ledger observes the message leaves its partition uncommitted, so a+ // fixed build reads the message again.+ let Some(envelope_partition) = self+ .settings+ .source_partitions+ .envelope_partition(partition)+ else {+ self.handle.signal_failure(format!(+ "input partition {} has no source_partition in this topic's range",+ partition.0+ ));+ return Intake::Open;+ };let (event, person_id) = match decoded {Decoded::Forward { event, person_id } => (event, person_id),@@ -366,7 +380,7 @@ impl EventShuffler {event: CohortStreamEvent::from_clickhouse(*event,person_id,- partition.0,+ envelope_partition,source_offset.0,),partition,diff --git a/rust/cohort-event-shuffler/src/event.rs b/rust/cohort-event-shuffler/src/event.rsindex c850719abecf..f81cafd7f31f 100644--- a/rust/cohort-event-shuffler/src/event.rs+++ b/rust/cohort-event-shuffler/src/event.rs@@ -7,6 +7,8 @@use common_types::{ClickHouseEvent, PersonMode};use serde::{Deserialize, Serialize};+use crate::ledger::SourcePartition;+/// Single source of truth for the re-key string. Every producer targeting a topic co-partitioned/// with `cohort_stream_events` must use this so a given `(team_id, person_id)` lands on the same/// partition — and downstream worker — across topics and runtimes (Rust here, Node for@@ -16,8 +18,42 @@ pub fn partition_key(team_id: i32, person_id: &str) -> String {format!("{team_id}:{person_id}")}-/// `source_partition` / `source_offset` carry the upstream `clickhouse_events_json` coordinates so-/// the downstream processor can make counter increments replay-safe via per-key offset tracking.+/// Each input topic's slot in the `source_partition` numbers of `cohort_stream_events`. Kafka+/// numbers every topic's partitions from 0. The processor keys its replay marks by+/// `source_partition` alone, so two topics that shared numbers would mask each other's lower offsets+/// as replays. A slot must never pass to another topic, because stored replay marks keep the numbers+/// that the first topic wrote.+const INPUT_TOPIC_SLOTS: [(&str, u8); 2] = [+ ("clickhouse_events_json", 0),+ ("realtime_only_events_json", 1),+];++#[derive(Debug, Clone, Copy)]+pub struct SourcePartitionRange {+ slot: u8,+}++impl SourcePartitionRange {+ const STRIDE: i32 = 1_000_000;++ pub fn for_input_topic(input_topic: &str) -> Option<Self> {+ INPUT_TOPIC_SLOTS+ .iter()+ .find(|(topic, _)| *topic == input_topic)+ .map(|&(_, slot)| Self { slot })+ }++ /// The envelope's `source_partition` for `input_partition`. `None` for a partition outside+ /// `0..1_000_000`, whose number would fall in another slot's range.+ pub fn envelope_partition(self, input_partition: SourcePartition) -> Option<i32> {+ (0..Self::STRIDE)+ .contains(&input_partition.0)+ .then(|| i32::from(self.slot) * Self::STRIDE + input_partition.0)+ }+}++/// `source_partition` / `source_offset` carry the upstream coordinates, numbered by+/// [`SourcePartitionRange`], so the downstream processor can make counter increments replay-safe.#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]pub struct CohortStreamEvent {pub team_id: i32,@@ -132,6 +168,41 @@ mod tests {assert_eq!(a, b);}+ #[test]+ fn each_input_topic_writes_its_partitions_into_its_own_range() {+ for (topic, partition, expected) in [+ ("clickhouse_events_json", 5, Some(5)),+ ("clickhouse_events_json", 1_000_005, None),+ ("realtime_only_events_json", 5, Some(1_000_005)),+ ("realtime_only_events_json", 1_000_000, None),+ ] {+ let range = SourcePartitionRange::for_input_topic(topic).unwrap();+ assert_eq!(+ range.envelope_partition(SourcePartition(partition)),+ expected,+ "{topic} partition {partition}"+ );+ }+ }++ #[test]+ fn no_two_input_topics_share_a_slot_and_no_topic_has_two() {+ let slots: std::collections::BTreeSet<u8> =+ INPUT_TOPIC_SLOTS.iter().map(|&(_, slot)| slot).collect();+ assert_eq!(+ slots.len(),+ INPUT_TOPIC_SLOTS.len(),+ "two topics share a slot, so the processor would drop one topic's events as replays"+ );+ let topics: std::collections::BTreeSet<&str> =+ INPUT_TOPIC_SLOTS.iter().map(|&(topic, _)| topic).collect();+ assert_eq!(+ topics.len(),+ INPUT_TOPIC_SLOTS.len(),+ "a topic has two rows, and the shuffler uses only the first"+ );+ }+#[test]fn from_clickhouse_maps_every_field() {let event = sample_event(42, Some("01928aaa-bbbb-cccc-dddd-eeeeeeeeeeee"));diff --git a/rust/cohort-event-shuffler/src/main.rs b/rust/cohort-event-shuffler/src/main.rsindex 085fe825153a..0a4b85bb6599 100644--- a/rust/cohort-event-shuffler/src/main.rs+++ b/rust/cohort-event-shuffler/src/main.rs@@ -37,6 +37,7 @@ fn main() -> Result<()> {async fn async_main(config: Config) -> Result<()> {init_tracing();log_startup(&config);+ let settings = config.shuffler_settings()?;let mut manager = Manager::builder(SERVICE_NAME).with_global_shutdown_timeout(Duration::from_secs(30))@@ -108,13 +109,7 @@ async fn async_main(config: Config) -> Result<()> {.await;});- let shuffler = EventShuffler::new(- consumer,- producer,- team_index,- consumer_handle,- config.shuffler_settings(),- );+ let shuffler = EventShuffler::new(consumer, producer, team_index, consumer_handle, settings);tokio::spawn(async move {shuffler.process().await;});diff --git a/rust/cohort-event-shuffler/tests/integration.rs b/rust/cohort-event-shuffler/tests/integration.rsindex aa538b9a55aa..3d7fd3f79135 100644--- a/rust/cohort-event-shuffler/tests/integration.rs+++ b/rust/cohort-event-shuffler/tests/integration.rs@@ -22,6 +22,7 @@ use tokio_util::sync::CancellationToken;use uuid::Uuid;const INPUT_TOPIC: &str = "clickhouse_events_json";+const SECOND_INPUT_TOPIC: &str = "realtime_only_events_json";const OUTPUT_TOPIC: &str = "cohort_stream_events";const OUTPUT_PARTITIONS: i32 = 8;@@ -140,24 +141,24 @@ async fn collect_output(collected}-/// Polls the shuffler group's committed offset for input partition 0 until it reaches+/// Polls the shuffler group's committed offset for partition 0 of the input topic until it reaches/// `expected` (the next-offset past everything produced), proving the async ledger commit/// settled forwards, drops, skips, and the poison pill. The prober never subscribes, so it/// fetches offsets without joining (and rebalancing) the shuffler's group.-async fn wait_for_committed_offset(bootstrap: &str, group: &str, expected: i64) {+async fn wait_for_committed_offset(config: &Config, expected: i64) {let prober: StreamConsumer = ClientConfig::new()- .set("bootstrap.servers", bootstrap)- .set("group.id", group)+ .set("bootstrap.servers", &config.kafka_hosts)+ .set("group.id", &config.kafka_consumer_group).set("enable.auto.commit", "false").create().expect("failed to create committed-offset prober");let mut tpl = TopicPartitionList::new();- tpl.add_partition(INPUT_TOPIC, 0);+ tpl.add_partition(&config.input_topic, 0);let deadline = Instant::now() + Duration::from_secs(30);loop {if let Ok(committed) = prober.committed_offsets(tpl.clone(), Duration::from_secs(5)) {- if let Some(elem) = committed.elements_for_topic(INPUT_TOPIC).first() {+ if let Some(elem) = committed.elements_for_topic(&config.input_topic).first() {if let rdkafka::Offset::Offset(value) = elem.offset() {assert!(value <= expected,@@ -182,13 +183,18 @@ async fn shuffler_forwards_only_gated_events_with_stable_partitioning() {let (cluster, input_producer): (_, FutureProducer<_>) =common_kafka::test::create_mock_kafka().await;cluster- .create_topic(INPUT_TOPIC, 1, 1)+ .create_topic(SECOND_INPUT_TOPIC, 1, 1).expect("create input topic");cluster.create_topic(OUTPUT_TOPIC, OUTPUT_PARTITIONS, 1).expect("create output topic");let bootstrap = cluster.bootstrap_servers();- let config = test_config(&bootstrap);+ // The second input topic makes the envelope's source_partition differ from the Kafka+ // partition that the shuffler commits.+ let config = Config {+ input_topic: SECOND_INPUT_TOPIC.to_string(),+ ..test_config(&bootstrap)+ };let team_index = Arc::new(TeamIndex::from_teams([2, 7])); // team 99 is absent (no realtime cohorts)@@ -253,7 +259,11 @@ async fn shuffler_forwards_only_gated_events_with_stable_partitioning() {let payload = serde_json::to_string(&clickhouse_event(uuid, spec)).unwrap();let key = uuid.to_string();input_producer- .send_result(FutureRecord::to(INPUT_TOPIC).key(&key).payload(&payload))+ .send_result(+ FutureRecord::to(SECOND_INPUT_TOPIC)+ .key(&key)+ .payload(&payload),+ ).expect("enqueue input").await.expect("input produce canceled")@@ -263,7 +273,7 @@ async fn shuffler_forwards_only_gated_events_with_stable_partitioning() {if index == GARBAGE_AFTER_INDEX {input_producer.send_result(- FutureRecord::to(INPUT_TOPIC)+ FutureRecord::to(SECOND_INPUT_TOPIC).key("garbage").payload("this is not json"),)@@ -298,7 +308,7 @@ async fn shuffler_forwards_only_gated_events_with_stable_partitioning() {producer,team_index,handle,- config.shuffler_settings(),+ config.shuffler_settings().unwrap(),);tokio::spawn(async move { shuffler.process().await });@@ -366,7 +376,7 @@ async fn shuffler_forwards_only_gated_events_with_stable_partitioning() {first.properties.as_deref(),Some(r#"{"$current_url":"/pricing"}"#));- assert_eq!(first.source_partition, 0);+ assert_eq!(first.source_partition, 1_000_000);assert_eq!(first.source_offset, 0,"first produced event has input offset 0"@@ -374,7 +384,7 @@ async fn shuffler_forwards_only_gated_events_with_stable_partitioning() {// Commit progression: every input offset — settled drops/skips, acked forwards, and the// poison pill — ends up covered by an explicit ledger commit.- wait_for_committed_offset(&bootstrap, "shuffler-itest", total_input_messages).await;+ wait_for_committed_offset(&config, total_input_messages).await;}/// Graceful shutdown must commit through the drain path, not the periodic tick. With@@ -484,7 +494,7 @@ async fn shuffler_commits_via_drain_on_graceful_shutdown() {producer,team_index,handle,- config.shuffler_settings(),+ config.shuffler_settings().unwrap(),);let task = tokio::spawn(async move { shuffler.process().await });@@ -494,10 +504,14 @@ async fn shuffler_commits_via_drain_on_graceful_shutdown() {expected_forwardable,"forwards must flow before shutdown so the ledger has observed every offset",);+ assert!(+ output.iter().all(|(_, event)| event.source_partition == 0),+ "clickhouse_events_json has slot 0, so its envelopes must carry the real input partition",+ );// Nothing has committed the input yet (periodic tick is an hour out); the drain must.shutdown.cancel();- wait_for_committed_offset(&bootstrap, "shuffler-itest", total_input_messages).await;+ wait_for_committed_offset(&config, total_input_messages).await;task.await.expect("shuffler task panicked");}@@ -593,7 +607,7 @@ async fn shuffler_retries_forwards_parked_on_queue_full() {producer,team_index,handle,- config.shuffler_settings(),+ config.shuffler_settings().unwrap(),);tokio::spawn(async move { shuffler.process().await });@@ -606,5 +620,5 @@ async fn shuffler_retries_forwards_parked_on_queue_full() {);// Commit must still advance past the whole burst (settled drop/skip + acked forwards).- wait_for_committed_offset(&bootstrap, "shuffler-itest", total_input_messages).await;+ wait_for_committed_offset(&config, total_input_messages).await;}diff --git a/rust/cohort-stream-processor/tests/stage1_worker.rs b/rust/cohort-stream-processor/tests/stage1_worker.rsindex 765dc96ae57d..57dd4df2b789 100644--- a/rust/cohort-stream-processor/tests/stage1_worker.rs+++ b/rust/cohort-stream-processor/tests/stage1_worker.rs@@ -51,6 +51,7 @@ const PARTITION_ID: u16 = 0;const BEHAVIORAL_HASH: [u8; 16] = *b"0123456789abcdef";const PERSON_HASH: [u8; 16] = *b"fedcba9876543210";const BASE_TS: &str = "2026-05-26 12:34:56.789000";+const SECOND_TOPIC_PARTITION_10: i32 = 1_000_010;/// A window over 180 days routes `performed_event_multiple` to the compressed history variant.const COMPRESSED_WINDOW_DAYS: i64 = 365;@@ -789,8 +790,14 @@ fn cross_partition_low_offset_is_not_masked_by_a_high_offset_elsewhere() {let out = process_event(PARTITION_ID, &store, &filters, &enter).unwrap();assert_eq!(out.transitions[0].kind, TransitionKind::Entered);- // Leave from source partition 20 at a low offset, with a newer event time so argMax applies it.- let leave = person_event(p, "[email protected]", "2026-05-26 13:00:00.000000", 20, 3);+ // Leave from the second topic at a low offset, with a newer event time so argMax applies it.+ let leave = person_event(+ p,+ "[email protected]",+ "2026-05-26 13:00:00.000000",+ SECOND_TOPIC_PARTITION_10,+ 3,+ );let out = process_event(PARTITION_ID, &store, &filters, &leave).unwrap();assert_eq!(out.transitions.len(),@@ -801,7 +808,7 @@ fn cross_partition_low_offset_is_not_masked_by_a_high_offset_elsewhere() {let applied = person_record_at(&store, p).unwrap().applied_offsets;assert_high_water(&applied, 10, 100);- assert_high_water(&applied, 20, 3);+ assert_high_water(&applied, SECOND_TOPIC_PARTITION_10, 3);}#[test]