Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

1.6 KB
Raw Blame
use std::collections::HashMap;
use std::sync::Arc;

use common_kafka_consumer::{Offset, PolledMessage};
use serde::{Deserialize, Serialize};

/// Matches `SerializedKafkaMessage` in `nodejs/src/ingestion/api/types.ts`.
/// Values are raw UTF-8 strings (PostHog Kafka messages are always JSON text).
#[derive(Debug, Clone, Serialize, Deserialize)]
pub struct SerializedKafkaMessage {
    pub topic: Arc<str>,
    pub partition: i32,
    pub offset: i64,
    pub timestamp: i64,
    pub key: Option<String>,
    pub value: Option<String>,
    pub headers: HashMap<String, String>,
    /// The ledger generation the message was charged under. Workers never
    /// see it.
    #[serde(skip)]
    pub generation: u64,
}

impl SerializedKafkaMessage {
    /// Approximate payload size for queue accounting: key plus value bytes,
    /// ignoring headers and framing.
    pub fn payload_bytes(&self) -> usize {
        self.key.as_ref().map_or(0, String::len) + self.value.as_ref().map_or(0, String::len)
    }
}

/// The demux's view of a message: the Kafka key is the routing key.
impl From<SerializedKafkaMessage> for PolledMessage<String, SerializedKafkaMessage> {
    fn from(message: SerializedKafkaMessage) -> Self {
        PolledMessage {
            offset: Offset(message.offset),
            key: message.key.clone(),
            inner: message,
        }
    }
}

/// One poll's messages for one routing key on one partition, in offset order.
pub type Group = common_kafka_consumer::Group<String, SerializedKafkaMessage>;

/// The demux that builds one poll's groups.
pub type Accumulator = common_kafka_consumer::Accumulator<String, SerializedKafkaMessage>;