:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
use std::collections::HashMap;
use std::sync::Arc;
use common_kafka_consumer::{Offset, PolledMessage};
use serde::{Deserialize, Serialize};
/// Matches `SerializedKafkaMessage` in `nodejs/src/ingestion/api/types.ts`.
/// Values are raw UTF-8 strings (PostHog Kafka messages are always JSON text).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SerializedKafkaMessage {
pub topic: Arc<str>,
pub partition: i32,
pub offset: i64,
pub timestamp: i64,
pub key: Option<String>,
pub value: Option<String>,
pub headers: HashMap<String, String>,
/// The ledger generation the message was charged under. Workers never
/// see it.
#[serde(skip)]
pub generation: u64,
}
impl SerializedKafkaMessage {
/// Approximate payload size for queue accounting: key plus value bytes,
/// ignoring headers and framing.
pub fn payload_bytes(&self) -> usize {
self.key.as_ref().map_or(0, String::len) + self.value.as_ref().map_or(0, String::len)
}
}
/// The demux's view of a message: the Kafka key is the routing key.
impl From<SerializedKafkaMessage> for PolledMessage<String, SerializedKafkaMessage> {
fn from(message: SerializedKafkaMessage) -> Self {
PolledMessage {
offset: Offset(message.offset),
key: message.key.clone(),
inner: message,
}
}
}
/// One poll's messages for one routing key on one partition, in offset order.
pub type Group = common_kafka_consumer::Group<String, SerializedKafkaMessage>;
/// The demux that builds one poll's groups.
pub type Accumulator = common_kafka_consumer::Accumulator<String, SerializedKafkaMessage>;