Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

perf(cohorts): copy property blobs verbatim and presize the lz4 buffer (#128843)

Matheus Batistacb4b6dab4b07
diff --git a/docs/internal/realtime-cohorts/event-routing.md b/docs/internal/realtime-cohorts/event-routing.md
index d96d47af1339..cf28033a38c3 100644
--- a/docs/internal/realtime-cohorts/event-routing.md
+++ b/docs/internal/realtime-cohorts/event-routing.md
@@ -36,8 +36,9 @@ It parses every message in two phases.
1. **A cheap gate.**
The first parse reads only `team_id` and whether `person_id` is present.
Large fields such as `properties` and `person_properties` are skipped without being allocated.
-2. **A full parse for survivors.**
- Only events that pass the gate are parsed completely and turned into the output envelope.
+2. **A forward parse for survivors.**
+ Only events that pass the gate are parsed again, and this parse reads only the fields it needs to build the output envelope.
+ The blobs it forwards, `properties`, `person_properties` and `elements_chain`, are copied as they arrived, without being unescaped.
An event passes the gate when it has a person id and its team passes two checks.
@@ -107,6 +108,7 @@ A broker or network outage longer than that timeout abandons every forward still
Other losses:
- a message that cannot be parsed,
+- an event whose forwarded blob holds a lone surrogate escape: the shuffler copies blobs without checking them, and the processor then refuses the whole envelope,
- a forward the producer rejects outright, for example because it is too large,
- an event for a team that has just gained its first realtime cohort and is not yet in the team index.
These are counted together with every other out-of-scope team, so they are not separately visible.
@@ -211,7 +213,7 @@ Person p-1 is `0192f3a0-7c1e-7d2a-9b3c-000000000001`.
1. Ingestion writes a `$pageview` from p-1 to firehose partition 3, at offset 9870001.
2. The shuffler's cheap gate reads `team_id: 7` and sees a `person_id`.
Team 7 is in the team index, so the event passes.
-3. The full parse builds the envelope with `source_partition: 3, source_offset: 9870001`.
+3. The forward parse builds the envelope with `source_partition: 3, source_offset: 9870001`.
4. The key is `"7:0192f3a0-7c1e-7d2a-9b3c-000000000001"`.
`murmur2` of it, masked and taken modulo 64, is 26.
5. The shuffler sends to `cohort_stream_events` partition 26 and records offset 9870001 as in flight for firehose partition 3.
@@ -228,7 +230,8 @@ The processor sees firehose offset 9870001 from partition 3 for p-1 a second tim
## Optimizations
- **Two-phase parsing.**
- The gate parses two fields, and only survivors pay for a full parse.
+ The gate parses two fields, and only survivors pay for the forward parse.
+ The forward parse copies the property blobs as they arrived instead of unescaping and escaping them again.
Most firehose events are discarded at the gate.
- **Pipelined forwarding.**
Consuming, producing and committing run independently.
diff --git a/rust/Cargo.lock b/rust/Cargo.lock
index 450e8c200271..e24c58f4de69 100644
--- a/rust/Cargo.lock
+++ b/rust/Cargo.lock
@@ -2774,6 +2774,7 @@ dependencies = [
"anyhow",
"arc-swap",
"axum 0.7.9",
+ "cohort-core",
"common-alloc",
"common-cohort-team-access",
"common-database",
diff --git a/rust/cohort-event-shuffler/Cargo.toml b/rust/cohort-event-shuffler/Cargo.toml
index 3d89549e4f67..47f0fe2b9d6d 100644
--- a/rust/cohort-event-shuffler/Cargo.toml
+++ b/rust/cohort-event-shuffler/Cargo.toml
@@ -24,7 +24,7 @@ metrics-exporter-prometheus = { workspace = true }
rand = { workspace = true }
rdkafka = { workspace = true }
serde = { workspace = true }
-serde_json = { workspace = true }
+serde_json = { workspace = true, features = ["raw_value"] }
sqlx = { workspace = true }
thiserror = { workspace = true }
tokio = { workspace = true }
@@ -33,6 +33,7 @@ tracing-subscriber = { workspace = true, features = ["env-filter", "json"] }
uuid = { workspace = true }
[dev-dependencies]
+cohort-core = { path = "../cohort-core" }
common-cohort-team-access = { path = "../common/cohort-team-access", features = ["test-util"] }
proptest = "1"
tokio-util = { workspace = true }
diff --git a/rust/cohort-event-shuffler/src/consumer.rs b/rust/cohort-event-shuffler/src/consumer.rs
index b057146dac19..34a9c168e4d0 100644
--- a/rust/cohort-event-shuffler/src/consumer.rs
+++ b/rust/cohort-event-shuffler/src/consumer.rs
@@ -13,7 +13,6 @@ use std::pin::Pin;
use std::time::{Duration, Instant};
use common_kafka::kafka_consumer::{Offset, RecvErr, SingleTopicConsumer};
-use common_types::ClickHouseEvent;
use futures::stream::FuturesUnordered;
use futures::StreamExt;
use lifecycle::Handle;
@@ -23,7 +22,7 @@ use serde::{de::IgnoredAny, Deserialize};
use tokio::task::JoinError;
use tracing::{debug, error, info, warn};
-use crate::event::{personless_mode, CohortStreamEvent, SourcePartitionRange};
+use crate::event::{personless_mode, CohortStreamEvent, ForwardFields, SourcePartitionRange};
use crate::filter_team_index::TeamGate;
use crate::ledger::{
DeliveryOutcome, Ledger, NextOffset, Observation, Resolution, SourceOffset, SourcePartition,
@@ -88,14 +87,11 @@ fn classify_gate(gate: &GateFields, team_gate: &TeamGate) -> GateOutcome {
GateOutcome::Forward
}
-/// `event` is boxed to keep the enum small: `Forward` is the only large variant and is built only
-/// for survivors.
+/// `Forward` is boxed to keep the enum small: it is the only large variant and is built only for
+/// survivors.
#[derive(Debug)]
enum Decoded {
- Forward {
- event: Box<ClickHouseEvent>,
- person_id: String,
- },
+ Forward(Box<ForwardFields>),
DropNoPersonId,
SkipTeamGate,
/// Serde failure at either phase — counted and settled, never an `Err`. Infallibility matters:
@@ -107,8 +103,8 @@ enum Decoded {
Unparseable,
}
-/// Cheap gate parse for every event; a full `ClickHouseEvent` parse only for survivors, over the
-/// same bytes so forwarded envelopes are byte-identical to a single-parse path.
+/// Cheap gate parse for every event; survivors get a second parse, over the same bytes, of only the
+/// fields the envelope forwards.
fn decode_gated(payload: &[u8], team_gate: &TeamGate) -> Decoded {
let Ok(gate) = serde_json::from_slice::<GateFields>(payload) else {
return Decoded::Unparseable;
@@ -116,21 +112,8 @@ fn decode_gated(payload: &[u8], team_gate: &TeamGate) -> Decoded {
match classify_gate(&gate, team_gate) {
GateOutcome::DropNoPersonId => Decoded::DropNoPersonId,
GateOutcome::SkipTeamGate => Decoded::SkipTeamGate,
- GateOutcome::Forward => {
- let Ok(mut event) = serde_json::from_slice::<ClickHouseEvent>(payload) else {
- return Decoded::Unparseable;
- };
- let Some(person_id) = event.person_id.take() else {
- // Unreachable: the gate saw a present non-null person_id, and a non-string one would
- // have failed the full parse above. Fail safe instead of panicking.
- warn!("gate-forwarded event had no person_id after full parse; dropping");
- return Decoded::DropNoPersonId;
- };
- Decoded::Forward {
- event: Box::new(event),
- person_id,
- }
- }
+ GateOutcome::Forward => serde_json::from_slice::<Box<ForwardFields>>(payload)
+ .map_or(Decoded::Unparseable, Decoded::Forward),
}
}
@@ -353,8 +336,8 @@ impl EventShuffler {
return Intake::Open;
};
- let (event, person_id) = match decoded {
- Decoded::Forward { event, person_id } => (event, person_id),
+ let fields = match decoded {
+ Decoded::Forward(fields) => fields,
Decoded::DropNoPersonId => {
counter!(EVENTS_DROPPED_NO_PERSON_ID).increment(1);
ledger.observe(partition, source_offset, Observation::Settled);
@@ -372,16 +355,11 @@ impl EventShuffler {
}
};
- if let Some(person_mode) = personless_mode(event.person_mode) {
+ if let Some(person_mode) = personless_mode(fields.person_mode) {
counter!(EVENTS_PERSON_PAYLOAD_DROPPED, "person_mode" => person_mode).increment(1);
}
let forward = Box::new(PendingForward {
- event: CohortStreamEvent::from_clickhouse(
- *event,
- person_id,
- envelope_partition,
- source_offset.0,
- ),
+ event: CohortStreamEvent::from_forward(*fields, envelope_partition, source_offset.0),
partition,
offset: source_offset,
});
@@ -633,7 +611,7 @@ mod tests {
use super::*;
use std::sync::Arc;
- use crate::event::sample_clickhouse_event;
+ use crate::event::{sample_clickhouse_event, RawJsonString};
use crate::filter_team_index::TeamIndex;
use common_cohort_team_access::TeamAccess;
use common_types::cohort::TeamAllowlist;
@@ -721,9 +699,9 @@ mod tests {
#[test]
fn dropped_event_never_materializes_blobs() {
- // A JSON-object `properties` fails the full parse (the field is a stringified blob): outside
- // the index the gate ignores the blob and skips; inside it the survivor parse rejects it as
- // unparseable.
+ // A JSON-object `properties` fails the survivor parse (the field is a stringified blob):
+ // outside the index the gate ignores the blob and skips; inside it the survivor parse rejects
+ // it as unparseable.
let mut value: Value =
serde_json::to_value(sample_clickhouse_event(99, Some("p"))).unwrap();
value["properties"] = json!({ "a": [1, 2, 3] });
@@ -749,7 +727,7 @@ mod tests {
#[test]
fn missing_person_mode_skips_mismatched_team_instead_of_erroring() {
- // `person_mode` is required by the full parse but ignored by the gate.
+ // `person_mode` is required by the survivor parse but ignored by the gate.
let mut value: Value =
serde_json::to_value(sample_clickhouse_event(99, Some("p"))).unwrap();
value.as_object_mut().unwrap().remove("person_mode");
@@ -782,13 +760,7 @@ mod tests {
let mut empty = base.clone();
empty["person_id"] = json!("");
match decode(empty) {
- Decoded::Forward { event, person_id } => {
- assert_eq!(person_id, "");
- assert!(
- event.person_id.is_none(),
- "person_id must be taken from the event"
- );
- }
+ Decoded::Forward(fields) => assert_eq!(fields.person_id, ""),
other => panic!("expected Forward, got {other:?}"),
}
@@ -796,30 +768,26 @@ mod tests {
let mut valid = base.clone();
valid["person_id"] = json!("p");
match decode(valid) {
- Decoded::Forward { event, person_id } => {
- assert_eq!(person_id, "p");
- assert!(event.person_id.is_none());
- }
+ Decoded::Forward(fields) => assert_eq!(fields.person_id, "p"),
other => panic!("expected Forward, got {other:?}"),
}
}
#[test]
- fn survivor_fully_materializes_the_forward_envelope() {
+ fn survivor_parses_the_forwarded_fields() {
let payload = serde_json::to_vec(&sample_clickhouse_event(2, Some("p"))).unwrap();
match decode_gated(&payload, &gate_with(&[2])) {
- Decoded::Forward { event, person_id } => {
- assert_eq!(person_id, "p");
- assert_eq!(event.team_id, 2);
- assert!(event.person_id.is_none(), "person_id should be taken");
- // Blobs skipped on the drop path are fully parsed for survivors.
+ Decoded::Forward(fields) => {
+ assert_eq!(fields.person_id, "p");
+ assert_eq!(fields.team_id, 2);
+ // Blobs skipped on the drop path are captured, still escaped, for survivors.
assert_eq!(
- event.properties.as_deref(),
- Some(r#"{"$current_url":"/pricing"}"#)
+ fields.properties.as_ref().map(RawJsonString::get),
+ Some(r#""{\"$current_url\":\"/pricing\"}""#)
);
assert_eq!(
- event.person_properties.as_deref(),
- Some(r#"{"email":"[email protected]"}"#)
+ fields.person_properties.as_ref().map(RawJsonString::get),
+ Some(r#""{\"email\":\"[email protected]\"}""#)
);
}
other => panic!("expected Forward, got {other:?}"),
diff --git a/rust/cohort-event-shuffler/src/event.rs b/rust/cohort-event-shuffler/src/event.rs
index f81cafd7f31f..9351cc2854bf 100644
--- a/rust/cohort-event-shuffler/src/event.rs
+++ b/rust/cohort-event-shuffler/src/event.rs
@@ -4,8 +4,12 @@
//! `created_at`, `project_id` and similar are intentionally dropped. `person_mode` is consumed, not
//! forwarded: it decides whether `person_properties` is the person's properties or a placeholder.
-use common_types::{ClickHouseEvent, PersonMode};
-use serde::{Deserialize, Serialize};
+#[cfg(test)]
+use common_types::ClickHouseEvent;
+use common_types::PersonMode;
+use serde::{de, Deserialize, Deserializer, Serialize};
+use serde_json::value::RawValue;
+use uuid::Uuid;
use crate::ledger::SourcePartition;
@@ -52,9 +56,63 @@ impl SourcePartitionRange {
}
}
+/// A JSON string token, quotes and escapes included, exactly as the firehose spelled it. The
+/// shuffler never reads the blobs it forwards, so it copies them instead of unescaping and
+/// re-escaping them. The processor's `Option<String>` decode reads the same string from either
+/// spelling. serde_json's capture does not check that `\u` surrogate escapes pair up, so a
+/// forwarded blob with a lone surrogate reaches the processor, which then rejects the whole
+/// envelope as an undeserializable message.
+#[derive(Debug, Clone, Serialize)]
+#[serde(transparent)]
+pub struct RawJsonString(Box<RawValue>);
+
+impl RawJsonString {
+ pub fn get(&self) -> &str {
+ self.0.get()
+ }
+}
+
+impl<'de> Deserialize<'de> for RawJsonString {
+ fn deserialize<D: Deserializer<'de>>(deserializer: D) -> Result<Self, D::Error> {
+ let raw = Box::<RawValue>::deserialize(deserializer)?;
+ // serde_json skips the whitespace before a value it captures, so only a string token
+ // starts with a quote.
+ if raw.get().starts_with('"') {
+ Ok(Self(raw))
+ } else {
+ Err(de::Error::custom("expected a JSON string"))
+ }
+ }
+}
+
+impl PartialEq for RawJsonString {
+ fn eq(&self, other: &Self) -> bool {
+ self.get() == other.get()
+ }
+}
+
+impl Eq for RawJsonString {}
+
+/// The fields of a firehose event that the envelope forwards, parsed only for gate survivors. The
+/// other fields stay undeclared, so serde skips them without allocating, and a missing or mistyped
+/// one does not make the event unparseable.
+#[derive(Debug, Deserialize)]
+pub(crate) struct ForwardFields {
+ pub(crate) uuid: Uuid,
+ pub(crate) team_id: i32,
+ pub(crate) event: String,
+ pub(crate) distinct_id: String,
+ pub(crate) person_id: String,
+ pub(crate) timestamp: String,
+ pub(crate) properties: Option<RawJsonString>,
+ pub(crate) person_properties: Option<RawJsonString>,
+ pub(crate) elements_chain: Option<RawJsonString>,
+ pub(crate) person_mode: PersonMode,
+}
+
/// `source_partition` / `source_offset` carry the upstream coordinates, numbered by
/// [`SourcePartitionRange`], so the downstream processor can make counter increments replay-safe.
-#[derive(Debug, Clone, PartialEq, Eq, Serialize, Deserialize)]
+#[derive(Debug, Clone, PartialEq, Eq, Serialize)]
pub struct CohortStreamEvent {
pub team_id: i32,
pub person_id: String,
@@ -62,9 +120,9 @@ pub struct CohortStreamEvent {
pub uuid: String,
pub event: String,
pub timestamp: String,
- pub properties: Option<String>,
- pub person_properties: Option<String>,
- pub elements_chain: Option<String>,
+ pub properties: Option<RawJsonString>,
+ pub person_properties: Option<RawJsonString>,
+ pub elements_chain: Option<RawJsonString>,
pub source_offset: i64,
pub source_partition: i32,
}
@@ -82,28 +140,35 @@ pub(crate) fn personless_mode(mode: PersonMode) -> Option<&'static str> {
}
impl CohortStreamEvent {
- /// Moves every owned field out of `event` rather than cloning; `person_id` is extracted
- /// upstream and passed in separately. A personless event forwards `person_properties` as null,
- /// so the processor leaves the person's record alone and still counts the event for behavioral
- /// conditions.
- pub fn from_clickhouse(
- event: ClickHouseEvent,
- person_id: String,
+ /// A personless event forwards `person_properties` as null, so the processor leaves the
+ /// person's record alone and still counts the event for behavioral conditions.
+ pub(crate) fn from_forward(
+ fields: ForwardFields,
source_partition: i32,
source_offset: i64,
) -> Self {
+ let ForwardFields {
+ uuid,
+ team_id,
+ event,
+ distinct_id,
+ person_id,
+ timestamp,
+ properties,
+ person_properties,
+ elements_chain,
+ person_mode,
+ } = fields;
Self {
- team_id: event.team_id,
+ team_id,
person_id,
- distinct_id: event.distinct_id,
- uuid: event.uuid.to_string(),
- event: event.event,
- timestamp: event.timestamp,
- properties: event.properties,
- person_properties: event
- .person_properties
- .filter(|_| personless_mode(event.person_mode).is_none()),
- elements_chain: event.elements_chain,
+ distinct_id,
+ uuid: uuid.to_string(),
+ event,
+ timestamp,
+ properties,
+ person_properties: person_properties.filter(|_| personless_mode(person_mode).is_none()),
+ elements_chain,
source_offset,
source_partition,
}
@@ -116,8 +181,6 @@ impl CohortStreamEvent {
#[cfg(test)]
pub(crate) fn sample_clickhouse_event(team_id: i32, person_id: Option<&str>) -> ClickHouseEvent {
- use uuid::Uuid;
-
ClickHouseEvent {
uuid: Uuid::from_u128(0x0192_8aaa_bbbb_cccc_dddd_eeee_eeee_eeee),
team_id,
@@ -151,10 +214,22 @@ pub(crate) fn sample_clickhouse_event(team_id: i32, person_id: Option<&str>) ->
mod tests {
use super::*;
+ use cohort_core::events::CohortStreamEvent as ProcessorEvent;
+
fn sample_event(team_id: i32, person_id: Option<&str>) -> ClickHouseEvent {
sample_clickhouse_event(team_id, person_id)
}
+ /// The firehose message for `event`, parsed as the survivor path parses it.
+ fn forward_fields(event: &ClickHouseEvent) -> ForwardFields {
+ serde_json::from_slice(&serde_json::to_vec(event).unwrap()).unwrap()
+ }
+
+ /// The envelope as the processor decodes it from the wire.
+ fn processor_view(envelope: &CohortStreamEvent) -> ProcessorEvent {
+ serde_json::from_str(&serde_json::to_string(envelope).unwrap()).unwrap()
+ }
+
#[test]
fn partition_key_is_team_colon_person() {
assert_eq!(partition_key(42, "abc"), "42:abc");
@@ -204,21 +279,18 @@ mod tests {
}
#[test]
- fn from_clickhouse_maps_every_field() {
+ fn from_forward_maps_every_field() {
let event = sample_event(42, Some("01928aaa-bbbb-cccc-dddd-eeeeeeeeeeee"));
- // from_clickhouse consumes the event, so capture the uuid first.
- let expected_uuid = event.uuid.to_string();
- let envelope = CohortStreamEvent::from_clickhouse(
- event,
- "01928aaa-bbbb-cccc-dddd-eeeeeeeeeeee".to_string(),
+ let envelope = processor_view(&CohortStreamEvent::from_forward(
+ forward_fields(&event),
17,
12345,
- );
+ ));
assert_eq!(envelope.team_id, 42);
assert_eq!(envelope.person_id, "01928aaa-bbbb-cccc-dddd-eeeeeeeeeeee");
assert_eq!(envelope.distinct_id, "[email protected]");
- assert_eq!(envelope.uuid, expected_uuid);
+ assert_eq!(envelope.uuid, event.uuid.to_string());
assert_eq!(envelope.event, "$pageview");
assert_eq!(envelope.timestamp, "2026-05-26 12:34:56.789000");
assert_eq!(
@@ -235,32 +307,32 @@ mod tests {
}
#[test]
- fn from_clickhouse_preserves_null_optionals() {
+ fn from_forward_preserves_null_optionals() {
let mut event = sample_event(1, Some("p"));
event.properties = None;
event.person_properties = None;
event.elements_chain = None;
- let envelope = CohortStreamEvent::from_clickhouse(event, "p".to_string(), 0, 0);
+ let envelope = CohortStreamEvent::from_forward(forward_fields(&event), 0, 0);
assert!(envelope.properties.is_none());
assert!(envelope.person_properties.is_none());
assert!(envelope.elements_chain.is_none());
}
#[test]
- fn from_clickhouse_forwards_a_person_payload_only_for_a_full_event() {
+ fn from_forward_forwards_a_person_payload_only_for_a_full_event() {
let envelope = |person_mode| {
let event = ClickHouseEvent {
person_mode,
person_properties: Some("{}".to_string()),
..sample_event(2, Some("p"))
};
- CohortStreamEvent::from_clickhouse(event, "p".to_string(), 0, 0)
+ CohortStreamEvent::from_forward(forward_fields(&event), 0, 0)
};
let full = envelope(PersonMode::Full);
assert_eq!(
- full.person_properties.as_deref(),
+ processor_view(&full).person_properties.as_deref(),
Some("{}"),
"a full event with no person properties still describes the person"
);
@@ -279,15 +351,53 @@ mod tests {
#[test]
fn envelope_partition_key_matches_free_function() {
let event = sample_event(99, Some("xyz"));
- let envelope = CohortStreamEvent::from_clickhouse(event, "xyz".to_string(), 3, 9);
+ let envelope = CohortStreamEvent::from_forward(forward_fields(&event), 3, 9);
assert_eq!(envelope.partition_key(), "99:xyz");
assert_eq!(envelope.partition_key(), partition_key(99, "xyz"));
}
+ #[test]
+ fn forwarded_blobs_keep_the_firehose_spelling() {
+ // A decode and re-encode would respell each of these escapes: `\u00e3` as `ã`, `\/` as
+ // `/`, `\u000a` as `\n`.
+ let properties = r#""{\"city\":\"S\u00e3o Paulo\"}""#;
+ let person_properties = r#""{\"home\":\"https:\/\/example.com\"}""#;
+ let elements_chain = r#""a:href=\"\/x\"\u000a""#;
+ let payload = format!(
+ r#"{{"uuid": "01928aaa-bbbb-cccc-dddd-eeeeeeeeeeee", "team_id": 2,
+ "event": "$pageview", "distinct_id": "d", "person_id": "p",
+ "timestamp": "2026-05-26 12:34:56.789000", "person_mode": "full",
+ "properties": {properties}, "person_properties": {person_properties},
+ "elements_chain": {elements_chain}}}"#
+ );
+
+ let fields: ForwardFields = serde_json::from_str(&payload).unwrap();
+ let envelope =
+ serde_json::to_string(&CohortStreamEvent::from_forward(fields, 0, 0)).unwrap();
+
+ for (key, raw) in [
+ ("properties", properties),
+ ("person_properties", person_properties),
+ ("elements_chain", elements_chain),
+ ] {
+ assert!(
+ envelope.contains(&format!(r#""{key}":{raw}"#)),
+ "`{key}` was respelled in {envelope}"
+ );
+ }
+ let view: ProcessorEvent = serde_json::from_str(&envelope).unwrap();
+ assert_eq!(view.properties.as_deref(), Some(r#"{"city":"São Paulo"}"#));
+ assert_eq!(
+ view.person_properties.as_deref(),
+ Some(r#"{"home":"https://example.com"}"#)
+ );
+ assert_eq!(view.elements_chain.as_deref(), Some("a:href=\"/x\"\n"));
+ }
+
#[test]
fn serialized_envelope_has_tdd_4_3_keys() {
let event = sample_event(42, Some("p1"));
- let envelope = CohortStreamEvent::from_clickhouse(event, "p1".to_string(), 17, 12345);
+ let envelope = CohortStreamEvent::from_forward(forward_fields(&event), 17, 12345);
let value: serde_json::Value = serde_json::to_value(&envelope).unwrap();
let obj = value.as_object().unwrap();
@@ -310,13 +420,4 @@ mod tests {
assert_eq!(obj["source_offset"], serde_json::json!(12345));
assert_eq!(obj["source_partition"], serde_json::json!(17));
}
-
- #[test]
- fn envelope_round_trips_through_json() {
- let event = sample_event(42, Some("p1"));
- let envelope = CohortStreamEvent::from_clickhouse(event, "p1".to_string(), 17, 12345);
- let json = serde_json::to_string(&envelope).unwrap();
- let decoded: CohortStreamEvent = serde_json::from_str(&json).unwrap();
- assert_eq!(envelope, decoded);
- }
}
diff --git a/rust/cohort-event-shuffler/tests/integration.rs b/rust/cohort-event-shuffler/tests/integration.rs
index b85b2b17a60c..65453a98393e 100644
--- a/rust/cohort-event-shuffler/tests/integration.rs
+++ b/rust/cohort-event-shuffler/tests/integration.rs
@@ -6,9 +6,9 @@
use std::sync::Arc;
use std::time::{Duration, Instant};
+use cohort_core::events::CohortStreamEvent;
use cohort_event_shuffler::config::Config;
use cohort_event_shuffler::consumer::EventShuffler;
-use cohort_event_shuffler::event::CohortStreamEvent;
use cohort_event_shuffler::filter_team_index::{TeamGate, TeamIndex};
use cohort_event_shuffler::producer::CohortStreamProducer;
use common_cohort_team_access::TeamAccess;
@@ -128,7 +128,7 @@ async fn collect_output(
Ok(Ok(msg)) => {
if let Some(payload) = msg.payload() {
let envelope: CohortStreamEvent = serde_json::from_slice(payload)
- .expect("output payload must be an envelope");
+ .expect("the processor must decode every envelope");
collected.push((msg.partition(), envelope));
if collected.len() >= expected && grace_until.is_none() {
grace_until = Some(Instant::now() + Duration::from_secs(1));
diff --git a/rust/cohort-stream-processor/src/config.rs b/rust/cohort-stream-processor/src/config.rs
index 8dc67ab4e154..353f9af79089 100644
--- a/rust/cohort-stream-processor/src/config.rs
+++ b/rust/cohort-stream-processor/src/config.rs
@@ -27,6 +27,15 @@ const POOL_NAME: &str = "posthog_cohort";
/// loses its replay dedup, so a non-zero TTL below this floor warns at startup.
const MIN_SAFE_PERSON_RECORD_TTL_DAYS: u32 = 30;
+/// `message.max.bytes` on every consumer, although the key reads as a producer setting. librdkafka
+/// also uses it to cap the first buffer it allocates for decompressing an LZ4 record batch, whose
+/// frame usually carries no content size, and then grows that buffer in steps until the batch fits
+/// (`rd_kafka_lz4_decompress` in librdkafka's `rdkafka_lz4.c`). The 1 MB default starts below a
+/// large decompressed batch. The key also floors `fetch.max.bytes`: an explicit value below it
+/// fails client creation, and a smaller `queued.max.messages.kbytes` does not bound the fetch
+/// size.
+const CONSUMER_MESSAGE_MAX_BYTES: u32 = 8 * 1024 * 1024;
+
#[derive(Envconfig, Clone, Debug)]
pub struct Config {
/// Host for the observability HTTP server (`/_health`, `/_ready`, `/metrics`).
@@ -1193,7 +1202,8 @@ impl Config {
.set("enable.auto.commit", "false")
.set("enable.auto.offset.store", "false")
.set("auto.offset.reset", "earliest")
- .set("socket.timeout.ms", "10000");
+ .set("socket.timeout.ms", "10000")
+ .set("message.max.bytes", CONSUMER_MESSAGE_MAX_BYTES.to_string());
if let Some(id) = self.pod_identity() {
config.set("client.id", id);
@@ -1645,6 +1655,13 @@ mod tests {
let client = config.consumer_client_config();
assert_eq!(client.get("queued.max.messages.kbytes"), Some("131072"));
assert_eq!(client.get("queued.min.messages"), Some("2000"));
+ assert_eq!(client.get("message.max.bytes"), Some("8388608"));
+ assert_eq!(
+ config
+ .follower_client_config("cohort-stream-seeds")
+ .get("message.max.bytes"),
+ Some("8388608"),
+ );
}
#[test]