:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
perf(cohorts): copy property blobs verbatim and presize the lz4 buffer (#128843)
Matheus Batistacb4b6dab4b07
diff --git a/docs/internal/realtime-cohorts/event-routing.md b/docs/internal/realtime-cohorts/event-routing.mdindex d96d47af1339..cf28033a38c3 100644--- a/docs/internal/realtime-cohorts/event-routing.md+++ b/docs/internal/realtime-cohorts/event-routing.md@@ -36,8 +36,9 @@ It parses every message in two phases.1. **A cheap gate.**The first parse reads only `team_id` and whether `person_id` is present.Large fields such as `properties` and `person_properties` are skipped without being allocated.-2. **A full parse for survivors.**- Only events that pass the gate are parsed completely and turned into the output envelope.+2. **A forward parse for survivors.**+ Only events that pass the gate are parsed again, and this parse reads only the fields it needs to build the output envelope.+ The blobs it forwards, `properties`, `person_properties` and `elements_chain`, are copied as they arrived, without being unescaped.An event passes the gate when it has a person id and its team passes two checks.@@ -107,6 +108,7 @@ A broker or network outage longer than that timeout abandons every forward stillOther losses:- a message that cannot be parsed,+- an event whose forwarded blob holds a lone surrogate escape: the shuffler copies blobs without checking them, and the processor then refuses the whole envelope,- a forward the producer rejects outright, for example because it is too large,- an event for a team that has just gained its first realtime cohort and is not yet in the team index.These are counted together with every other out-of-scope team, so they are not separately visible.@@ -211,7 +213,7 @@ Person p-1 is `0192f3a0-7c1e-7d2a-9b3c-000000000001`.1. Ingestion writes a `$pageview` from p-1 to firehose partition 3, at offset 9870001.2. The shuffler's cheap gate reads `team_id: 7` and sees a `person_id`.Team 7 is in the team index, so the event passes.-3. The full parse builds the envelope with `source_partition: 3, source_offset: 9870001`.+3. The forward parse builds the envelope with `source_partition: 3, source_offset: 9870001`.4. The key is `"7:0192f3a0-7c1e-7d2a-9b3c-000000000001"`.`murmur2` of it, masked and taken modulo 64, is 26.5. The shuffler sends to `cohort_stream_events` partition 26 and records offset 9870001 as in flight for firehose partition 3.@@ -228,7 +230,8 @@ The processor sees firehose offset 9870001 from partition 3 for p-1 a second tim## Optimizations- **Two-phase parsing.**- The gate parses two fields, and only survivors pay for a full parse.+ The gate parses two fields, and only survivors pay for the forward parse.+ The forward parse copies the property blobs as they arrived instead of unescaping and escaping them again.Most firehose events are discarded at the gate.- **Pipelined forwarding.**Consuming, producing and committing run independently.diff --git a/rust/Cargo.lock b/rust/Cargo.lockindex 450e8c200271..e24c58f4de69 100644--- a/rust/Cargo.lock+++ b/rust/Cargo.lock@@ -2774,6 +2774,7 @@ dependencies = ["anyhow","arc-swap","axum 0.7.9",+ "cohort-core","common-alloc","common-cohort-team-access","common-database",diff --git a/rust/cohort-event-shuffler/Cargo.toml b/rust/cohort-event-shuffler/Cargo.tomlindex 3d89549e4f67..47f0fe2b9d6d 100644--- a/rust/cohort-event-shuffler/Cargo.toml+++ b/rust/cohort-event-shuffler/Cargo.toml@@ -24,7 +24,7 @@ metrics-exporter-prometheus = { workspace = true }rand = { workspace = true }rdkafka = { workspace = true }serde = { workspace = true }-serde_json = { workspace = true }+serde_json = { workspace = true, features = ["raw_value"] }sqlx = { workspace = true }thiserror = { workspace = true }tokio = { workspace = true }@@ -33,6 +33,7 @@ tracing-subscriber = { workspace = true, features = ["env-filter", "json"] }uuid = { workspace = true }[dev-dependencies]+cohort-core = { path = "../cohort-core" }common-cohort-team-access = { path = "../common/cohort-team-access", features = ["test-util"] }proptest = "1"tokio-util = { workspace = true }diff --git a/rust/cohort-event-shuffler/src/consumer.rs b/rust/cohort-event-shuffler/src/consumer.rsindex b057146dac19..34a9c168e4d0 100644--- a/rust/cohort-event-shuffler/src/consumer.rs+++ b/rust/cohort-event-shuffler/src/consumer.rs@@ -13,7 +13,6 @@ use std::pin::Pin;use std::time::{Duration, Instant};use common_kafka::kafka_consumer::{Offset, RecvErr, SingleTopicConsumer};-use common_types::ClickHouseEvent;use futures::stream::FuturesUnordered;use futures::StreamExt;use lifecycle::Handle;@@ -23,7 +22,7 @@ use serde::{de::IgnoredAny, Deserialize};use tokio::task::JoinError;use tracing::{debug, error, info, warn};-use crate::event::{personless_mode, CohortStreamEvent, SourcePartitionRange};+use crate::event::{personless_mode, CohortStreamEvent, ForwardFields, SourcePartitionRange};use crate::filter_team_index::TeamGate;use crate::ledger::{DeliveryOutcome, Ledger, NextOffset, Observation, Resolution, SourceOffset, SourcePartition,@@ -88,14 +87,11 @@ fn classify_gate(gate: &GateFields, team_gate: &TeamGate) -> GateOutcome {GateOutcome::Forward}-/// `event` is boxed to keep the enum small: `Forward` is the only large variant and is built only-/// for survivors.+/// `Forward` is boxed to keep the enum small: it is the only large variant and is built only for+/// survivors.#[derive(Debug)]enum Decoded {- Forward {- event: Box<ClickHouseEvent>,- person_id: String,- },+ Forward(Box<ForwardFields>),DropNoPersonId,SkipTeamGate,/// Serde failure at either phase — counted and settled, never an `Err`. Infallibility matters:@@ -107,8 +103,8 @@ enum Decoded {Unparseable,}-/// Cheap gate parse for every event; a full `ClickHouseEvent` parse only for survivors, over the-/// same bytes so forwarded envelopes are byte-identical to a single-parse path.+/// Cheap gate parse for every event; survivors get a second parse, over the same bytes, of only the+/// fields the envelope forwards.fn decode_gated(payload: &[u8], team_gate: &TeamGate) -> Decoded {let Ok(gate) = serde_json::from_slice::<GateFields>(payload) else {return Decoded::Unparseable;@@ -116,21 +112,8 @@ fn decode_gated(payload: &[u8], team_gate: &TeamGate) -> Decoded {match classify_gate(&gate, team_gate) {GateOutcome::DropNoPersonId => Decoded::DropNoPersonId,GateOutcome::SkipTeamGate => Decoded::SkipTeamGate,- GateOutcome::Forward => {- let Ok(mut event) = serde_json::from_slice::<ClickHouseEvent>(payload) else {- return Decoded::Unparseable;- };- let Some(person_id) = event.person_id.take() else {- // Unreachable: the gate saw a present non-null person_id, and a non-string one would- // have failed the full parse above. Fail safe instead of panicking.- warn!("gate-forwarded event had no person_id after full parse; dropping");- return Decoded::DropNoPersonId;- };- Decoded::Forward {- event: Box::new(event),- person_id,- }- }+ GateOutcome::Forward => serde_json::from_slice::<Box<ForwardFields>>(payload)+ .map_or(Decoded::Unparseable, Decoded::Forward),}}@@ -353,8 +336,8 @@ impl EventShuffler {return Intake::Open;};- let (event, person_id) = match decoded {- Decoded::Forward { event, person_id } => (event, person_id),+ let fields = match decoded {+ Decoded::Forward(fields) => fields,Decoded::DropNoPersonId => {counter!(EVENTS_DROPPED_NO_PERSON_ID).increment(1);ledger.observe(partition, source_offset, Observation::Settled);@@ -372,16 +355,11 @@ impl EventShuffler {}};- if let Some(person_mode) = personless_mode(event.person_mode) {+ if let Some(person_mode) = personless_mode(fields.person_mode) {counter!(EVENTS_PERSON_PAYLOAD_DROPPED, "person_mode" => person_mode).increment(1);}let forward = Box::new(PendingForward {- event: CohortStreamEvent::from_clickhouse(- *event,- person_id,- envelope_partition,- source_offset.0,- ),+ event: CohortStreamEvent::from_forward(*fields, envelope_partition, source_offset.0),partition,offset: source_offset,});@@ -633,7 +611,7 @@ mod tests {use super::*;use std::sync::Arc;- use crate::event::sample_clickhouse_event;+ use crate::event::{sample_clickhouse_event, RawJsonString};use crate::filter_team_index::TeamIndex;use common_cohort_team_access::TeamAccess;use common_types::cohort::TeamAllowlist;@@ -721,9 +699,9 @@ mod tests {#[test]fn dropped_event_never_materializes_blobs() {- // A JSON-object `properties` fails the full parse (the field is a stringified blob): outside- // the index the gate ignores the blob and skips; inside it the survivor parse rejects it as- // unparseable.+ // A JSON-object `properties` fails the survivor parse (the field is a stringified blob):+ // outside the index the gate ignores the blob and skips; inside it the survivor parse rejects+ // it as unparseable.let mut value: Value =serde_json::to_value(sample_clickhouse_event(99, Some("p"))).unwrap();value["properties"] = json!({ "a": [1, 2, 3] });@@ -749,7 +727,7 @@ mod tests {#[test]fn missing_person_mode_skips_mismatched_team_instead_of_erroring() {- // `person_mode` is required by the full parse but ignored by the gate.+ // `person_mode` is required by the survivor parse but ignored by the gate.let mut value: Value =serde_json::to_value(sample_clickhouse_event(99, Some("p"))).unwrap();value.as_object_mut().unwrap().remove("person_mode");@@ -782,13 +760,7 @@ mod tests {let mut empty = base.clone();empty["person_id"] = json!("");match decode(empty) {- Decoded::Forward { event, person_id } => {- assert_eq!(person_id, "");- assert!(- event.person_id.is_none(),- "person_id must be taken from the event"- );- }+ Decoded::Forward(fields) => assert_eq!(fields.person_id, ""),other => panic!("expected Forward, got {other:?}"),}@@ -796,30 +768,26 @@ mod tests {let mut valid = base.clone();valid["person_id"] = json!("p");match decode(valid) {- Decoded::Forward { event, person_id } => {- assert_eq!(person_id, "p");- assert!(event.person_id.is_none());- }+ Decoded::Forward(fields) => assert_eq!(fields.person_id, "p"),other => panic!("expected Forward, got {other:?}"),}}#[test]- fn survivor_fully_materializes_the_forward_envelope() {+ fn survivor_parses_the_forwarded_fields() {let payload = serde_json::to_vec(&sample_clickhouse_event(2, Some("p"))).unwrap();match decode_gated(&payload, &gate_with(&[2])) {- Decoded::Forward { event, person_id } => {- assert_eq!(person_id, "p");- assert_eq!(event.team_id, 2);- assert!(event.person_id.is_none(), "person_id should be taken");- // Blobs skipped on the drop path are fully parsed for survivors.+ Decoded::Forward(fields) => {+ assert_eq!(fields.person_id, "p");+ assert_eq!(fields.team_id, 2);+ // Blobs skipped on the drop path are captured, still escaped, for survivors.assert_eq!(- event.properties.as_deref(),- Some(r#"{"$current_url":"/pricing"}"#)+ fields.properties.as_ref().map(RawJsonString::get),+ Some(r#""{\"$current_url\":\"/pricing\"}""#));assert_eq!(- event.person_properties.as_deref(),- Some(r#"{"email":"[email protected]"}"#)+ fields.person_properties.as_ref().map(RawJsonString::get),+ Some(r#""{\"email\":\"[email protected]\"}""#));}other => panic!("expected Forward, got {other:?}"),diff --git a/rust/cohort-event-shuffler/src/event.rs b/rust/cohort-event-shuffler/src/event.rsindex f81cafd7f31f..9351cc2854bf 100644--- a/rust/cohort-event-shuffler/src/event.rs+++ b/rust/cohort-event-shuffler/src/event.rs@@ -4,8 +4,12 @@//! `created_at`, `project_id` and similar are intentionally dropped. `person_mode` is consumed, not//! forwarded: it decides whether `person_properties` is the person's properties or a placeholder.-use common_types::{ClickHouseEvent, PersonMode};-use serde::{Deserialize, Serialize};+#[cfg(test)]+use common_types::ClickHouseEvent;+use common_types::PersonMode;+use serde::{de, Deserialize, Deserializer, Serialize};+use serde_json::value::RawValue;+use uuid::Uuid;use crate::ledger::SourcePartition;@@ -52,9 +56,63 @@ impl SourcePartitionRange {}}+/// A JSON string token, quotes and escapes included, exactly as the firehose spelled it. The+/// shuffler never reads the blobs it forwards, so it copies them instead of unescaping and+/// re-escaping them. The processor's `Option<String>` decode reads the same string from either+/// spelling. serde_json's capture does not check that `\u` surrogate escapes pair up, so a+/// forwarded blob with a lone surrogate reaches the processor, which then rejects the whole+/// envelope as an undeserializable message.+#[derive(Debug, Clone, Serialize)]+#[serde(transparent)]+pub struct RawJsonString(Box<RawValue>);++impl RawJsonString {+ pub fn get(&self) -> &str {+ self.0.get()+ }+}++impl<'de> Deserialize<'de> for RawJsonString {+ fn deserialize<D: Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> {+ let raw = Box::<RawValue>::deserialize(deserializer)?;+ // serde_json skips the whitespace before a value it captures, so only a string token+ // starts with a quote.+ if raw.get().starts_with('"') {+ Ok(Self(raw))+ } else {+ Err(de::Error::custom("expected a JSON string"))+ }+ }+}++impl PartialEq for RawJsonString {+ fn eq(&self, other: &Self) -> bool {+ self.get() == other.get()+ }+}++impl Eq for RawJsonString {}++/// The fields of a firehose event that the envelope forwards, parsed only for gate survivors. The+/// other fields stay undeclared, so serde skips them without allocating, and a missing or mistyped+/// one does not make the event unparseable.+#[derive(Debug, Deserialize)]+pub(crate) struct ForwardFields {+ pub(crate) uuid: Uuid,+ pub(crate) team_id: i32,+ pub(crate) event: String,+ pub(crate) distinct_id: String,+ pub(crate) person_id: String,+ pub(crate) timestamp: String,+ pub(crate) properties: Option<RawJsonString>,+ pub(crate) person_properties: Option<RawJsonString>,+ pub(crate) elements_chain: Option<RawJsonString>,+ pub(crate) person_mode: PersonMode,+}+/// `source_partition` / `source_offset` carry the upstream coordinates, numbered by/// [`SourcePartitionRange`], so the downstream processor can make counter increments replay-safe.-#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]+#[derive(Debug, Clone, PartialEq, Eq, Serialize)]pub struct CohortStreamEvent {pub team_id: i32,pub person_id: String,@@ -62,9 +120,9 @@ pub struct CohortStreamEvent {pub uuid: String,pub event: String,pub timestamp: String,- pub properties: Option<String>,- pub person_properties: Option<String>,- pub elements_chain: Option<String>,+ pub properties: Option<RawJsonString>,+ pub person_properties: Option<RawJsonString>,+ pub elements_chain: Option<RawJsonString>,pub source_offset: i64,pub source_partition: i32,}@@ -82,28 +140,35 @@ pub(crate) fn personless_mode(mode: PersonMode) -> Option<&'static str> {}impl CohortStreamEvent {- /// Moves every owned field out of `event` rather than cloning; `person_id` is extracted- /// upstream and passed in separately. A personless event forwards `person_properties` as null,- /// so the processor leaves the person's record alone and still counts the event for behavioral- /// conditions.- pub fn from_clickhouse(- event: ClickHouseEvent,- person_id: String,+ /// A personless event forwards `person_properties` as null, so the processor leaves the+ /// person's record alone and still counts the event for behavioral conditions.+ pub(crate) fn from_forward(+ fields: ForwardFields,source_partition: i32,source_offset: i64,) -> Self {+ let ForwardFields {+ uuid,+ team_id,+ event,+ distinct_id,+ person_id,+ timestamp,+ properties,+ person_properties,+ elements_chain,+ person_mode,+ } = fields;Self {- team_id: event.team_id,+ team_id,person_id,- distinct_id: event.distinct_id,- uuid: event.uuid.to_string(),- event: event.event,- timestamp: event.timestamp,- properties: event.properties,- person_properties: event- .person_properties- .filter(|_| personless_mode(event.person_mode).is_none()),- elements_chain: event.elements_chain,+ distinct_id,+ uuid: uuid.to_string(),+ event,+ timestamp,+ properties,+ person_properties: person_properties.filter(|_| personless_mode(person_mode).is_none()),+ elements_chain,source_offset,source_partition,}@@ -116,8 +181,6 @@ impl CohortStreamEvent {#[cfg(test)]pub(crate) fn sample_clickhouse_event(team_id: i32, person_id: Option<&str>) -> ClickHouseEvent {- use uuid::Uuid;-ClickHouseEvent {uuid: Uuid::from_u128(0x0192_8aaa_bbbb_cccc_dddd_eeee_eeee_eeee),team_id,@@ -151,10 +214,22 @@ pub(crate) fn sample_clickhouse_event(team_id: i32, person_id: Option<&str>) ->mod tests {use super::*;+ use cohort_core::events::CohortStreamEvent as ProcessorEvent;+fn sample_event(team_id: i32, person_id: Option<&str>) -> ClickHouseEvent {sample_clickhouse_event(team_id, person_id)}+ /// The firehose message for `event`, parsed as the survivor path parses it.+ fn forward_fields(event: &ClickHouseEvent) -> ForwardFields {+ serde_json::from_slice(&serde_json::to_vec(event).unwrap()).unwrap()+ }++ /// The envelope as the processor decodes it from the wire.+ fn processor_view(envelope: &CohortStreamEvent) -> ProcessorEvent {+ serde_json::from_str(&serde_json::to_string(envelope).unwrap()).unwrap()+ }+#[test]fn partition_key_is_team_colon_person() {assert_eq!(partition_key(42, "abc"), "42:abc");@@ -204,21 +279,18 @@ mod tests {}#[test]- fn from_clickhouse_maps_every_field() {+ fn from_forward_maps_every_field() {let event = sample_event(42, Some("01928aaa-bbbb-cccc-dddd-eeeeeeeeeeee"));- // from_clickhouse consumes the event, so capture the uuid first.- let expected_uuid = event.uuid.to_string();- let envelope = CohortStreamEvent::from_clickhouse(- event,- "01928aaa-bbbb-cccc-dddd-eeeeeeeeeeee".to_string(),+ let envelope = processor_view(&CohortStreamEvent::from_forward(+ forward_fields(&event),17,12345,- );+ ));assert_eq!(envelope.team_id, 42);assert_eq!(envelope.person_id, "01928aaa-bbbb-cccc-dddd-eeeeeeeeeeee");assert_eq!(envelope.distinct_id, "[email protected]");- assert_eq!(envelope.uuid, expected_uuid);+ assert_eq!(envelope.uuid, event.uuid.to_string());assert_eq!(envelope.event, "$pageview");assert_eq!(envelope.timestamp, "2026-05-26 12:34:56.789000");assert_eq!(@@ -235,32 +307,32 @@ mod tests {}#[test]- fn from_clickhouse_preserves_null_optionals() {+ fn from_forward_preserves_null_optionals() {let mut event = sample_event(1, Some("p"));event.properties = None;event.person_properties = None;event.elements_chain = None;- let envelope = CohortStreamEvent::from_clickhouse(event, "p".to_string(), 0, 0);+ let envelope = CohortStreamEvent::from_forward(forward_fields(&event), 0, 0);assert!(envelope.properties.is_none());assert!(envelope.person_properties.is_none());assert!(envelope.elements_chain.is_none());}#[test]- fn from_clickhouse_forwards_a_person_payload_only_for_a_full_event() {+ fn from_forward_forwards_a_person_payload_only_for_a_full_event() {let envelope = |person_mode| {let event = ClickHouseEvent {person_mode,person_properties: Some("{}".to_string()),..sample_event(2, Some("p"))};- CohortStreamEvent::from_clickhouse(event, "p".to_string(), 0, 0)+ CohortStreamEvent::from_forward(forward_fields(&event), 0, 0)};let full = envelope(PersonMode::Full);assert_eq!(- full.person_properties.as_deref(),+ processor_view(&full).person_properties.as_deref(),Some("{}"),"a full event with no person properties still describes the person");@@ -279,15 +351,53 @@ mod tests {#[test]fn envelope_partition_key_matches_free_function() {let event = sample_event(99, Some("xyz"));- let envelope = CohortStreamEvent::from_clickhouse(event, "xyz".to_string(), 3, 9);+ let envelope = CohortStreamEvent::from_forward(forward_fields(&event), 3, 9);assert_eq!(envelope.partition_key(), "99:xyz");assert_eq!(envelope.partition_key(), partition_key(99, "xyz"));}+ #[test]+ fn forwarded_blobs_keep_the_firehose_spelling() {+ // A decode and re-encode would respell each of these escapes: `\u00e3` as `ã`, `\/` as+ // `/`, `\u000a` as `\n`.+ let properties = r#""{\"city\":\"S\u00e3o Paulo\"}""#;+ let person_properties = r#""{\"home\":\"https:\/\/example.com\"}""#;+ let elements_chain = r#""a:href=\"\/x\"\u000a""#;+ let payload = format!(+ r#"{{"uuid": "01928aaa-bbbb-cccc-dddd-eeeeeeeeeeee", "team_id": 2,+ "event": "$pageview", "distinct_id": "d", "person_id": "p",+ "timestamp": "2026-05-26 12:34:56.789000", "person_mode": "full",+ "properties": {properties}, "person_properties": {person_properties},+ "elements_chain": {elements_chain}}}"#+ );++ let fields: ForwardFields = serde_json::from_str(&payload).unwrap();+ let envelope =+ serde_json::to_string(&CohortStreamEvent::from_forward(fields, 0, 0)).unwrap();++ for (key, raw) in [+ ("properties", properties),+ ("person_properties", person_properties),+ ("elements_chain", elements_chain),+ ] {+ assert!(+ envelope.contains(&format!(r#""{key}":{raw}"#)),+ "`{key}` was respelled in {envelope}"+ );+ }+ let view: ProcessorEvent = serde_json::from_str(&envelope).unwrap();+ assert_eq!(view.properties.as_deref(), Some(r#"{"city":"São Paulo"}"#));+ assert_eq!(+ view.person_properties.as_deref(),+ Some(r#"{"home":"https://example.com"}"#)+ );+ assert_eq!(view.elements_chain.as_deref(), Some("a:href=\"/x\"\n"));+ }+#[test]fn serialized_envelope_has_tdd_4_3_keys() {let event = sample_event(42, Some("p1"));- let envelope = CohortStreamEvent::from_clickhouse(event, "p1".to_string(), 17, 12345);+ let envelope = CohortStreamEvent::from_forward(forward_fields(&event), 17, 12345);let value: serde_json::Value = serde_json::to_value(&envelope).unwrap();let obj = value.as_object().unwrap();@@ -310,13 +420,4 @@ mod tests {assert_eq!(obj["source_offset"], serde_json::json!(12345));assert_eq!(obj["source_partition"], serde_json::json!(17));}-- #[test]- fn envelope_round_trips_through_json() {- let event = sample_event(42, Some("p1"));- let envelope = CohortStreamEvent::from_clickhouse(event, "p1".to_string(), 17, 12345);- let json = serde_json::to_string(&envelope).unwrap();- let decoded: CohortStreamEvent = serde_json::from_str(&json).unwrap();- assert_eq!(envelope, decoded);- }}diff --git a/rust/cohort-event-shuffler/tests/integration.rs b/rust/cohort-event-shuffler/tests/integration.rsindex b85b2b17a60c..65453a98393e 100644--- a/rust/cohort-event-shuffler/tests/integration.rs+++ b/rust/cohort-event-shuffler/tests/integration.rs@@ -6,9 +6,9 @@use std::sync::Arc;use std::time::{Duration, Instant};+use cohort_core::events::CohortStreamEvent;use cohort_event_shuffler::config::Config;use cohort_event_shuffler::consumer::EventShuffler;-use cohort_event_shuffler::event::CohortStreamEvent;use cohort_event_shuffler::filter_team_index::{TeamGate, TeamIndex};use cohort_event_shuffler::producer::CohortStreamProducer;use common_cohort_team_access::TeamAccess;@@ -128,7 +128,7 @@ async fn collect_output(Ok(Ok(msg)) => {if let Some(payload) = msg.payload() {let envelope: CohortStreamEvent = serde_json::from_slice(payload)- .expect("output payload must be an envelope");+ .expect("the processor must decode every envelope");collected.push((msg.partition(), envelope));if collected.len() >= expected && grace_until.is_none() {grace_until = Some(Instant::now() + Duration::from_secs(1));diff --git a/rust/cohort-stream-processor/src/config.rs b/rust/cohort-stream-processor/src/config.rsindex 8dc67ab4e154..353f9af79089 100644--- a/rust/cohort-stream-processor/src/config.rs+++ b/rust/cohort-stream-processor/src/config.rs@@ -27,6 +27,15 @@ const POOL_NAME: &str = "posthog_cohort";/// loses its replay dedup, so a non-zero TTL below this floor warns at startup.const MIN_SAFE_PERSON_RECORD_TTL_DAYS: u32 = 30;+/// `message.max.bytes` on every consumer, although the key reads as a producer setting. librdkafka+/// also uses it to cap the first buffer it allocates for decompressing an LZ4 record batch, whose+/// frame usually carries no content size, and then grows that buffer in steps until the batch fits+/// (`rd_kafka_lz4_decompress` in librdkafka's `rdkafka_lz4.c`). The 1 MB default starts below a+/// large decompressed batch. The key also floors `fetch.max.bytes`: an explicit value below it+/// fails client creation, and a smaller `queued.max.messages.kbytes` does not bound the fetch+/// size.+const CONSUMER_MESSAGE_MAX_BYTES: u32 = 8 * 1024 * 1024;+#[derive(Envconfig, Clone, Debug)]pub struct Config {/// Host for the observability HTTP server (`/_health`, `/_ready`, `/metrics`).@@ -1193,7 +1202,8 @@ impl Config {.set("enable.auto.commit", "false").set("enable.auto.offset.store", "false").set("auto.offset.reset", "earliest")- .set("socket.timeout.ms", "10000");+ .set("socket.timeout.ms", "10000")+ .set("message.max.bytes", CONSUMER_MESSAGE_MAX_BYTES.to_string());if let Some(id) = self.pod_identity() {config.set("client.id", id);@@ -1645,6 +1655,13 @@ mod tests {let client = config.consumer_client_config();assert_eq!(client.get("queued.max.messages.kbytes"), Some("131072"));assert_eq!(client.get("queued.min.messages"), Some("2000"));+ assert_eq!(client.get("message.max.bytes"), Some("8388608"));+ assert_eq!(+ config+ .follower_client_config("cohort-stream-seeds")+ .get("message.max.bytes"),+ Some("8388608"),+ );}#[test]