:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
//! How the consumer loop turns what it receives into what it does next. The
//! consumer owns the Kafka client and the batcher outputs, hands each event to
//! the polling strategy, and carries out the [`Step`]s the strategy returns.
mod collector;
mod continuous;
#[cfg(test)]
mod harness;
mod poll;
use std::sync::Arc;
use common_kafka_consumer::{GroupCompletion, TopicPartition};
use metrics::counter;
use rdkafka::message::Message;
use tokio::time::Instant;
use tracing::info;
use crate::settlement::{Commit, GroupSettlement};
use collector::Collector;
pub(crate) use collector::PollLimits;
pub(crate) use continuous::ContinuousPolling;
pub(crate) use poll::{
debug_partition_offsets, debug_partition_spans, BatchStats, CollectedBatch, InFlightPoll,
PartitionDeliveries,
};
pub(crate) enum Step {
/// Charge the poll, submit it to the batcher, and hand it back through
/// [`PollingStrategy::submitted`].
Submit(CollectedBatch),
/// Submit the offsets and resume the partitions.
Commit(Commit),
/// Shutdown drained the in-flight work.
Stop,
}
pub(crate) struct Parts {
pub(crate) limits: PollLimits,
pub(crate) settlement: Arc<GroupSettlement>,
pub(crate) group_id: String,
}
pub(crate) trait PollingStrategy {
fn reading(&mut self) -> bool;
fn deadline(&self) -> Option<Instant>;
fn on_received<M: Message>(&mut self, message: &M, now: Instant);
fn on_completed(&mut self, completion: &GroupCompletion) -> anyhow::Result<()>;
fn on_ticked(&mut self);
fn on_assigned(&mut self, topic_partitions: &[TopicPartition]);
fn on_drain_started(&mut self, topic_partitions: &[TopicPartition]);
/// The commit holds every partition's frontier, not only the drained ones.
fn on_drain_finished(&mut self, topic_partitions: &[TopicPartition]) -> Option<Commit>;
/// Every message received before the revoke belongs to the old
/// assignment: the batcher purges it, or Kafka delivers it again from the
/// last commit.
fn on_revoked(&mut self, topic_partitions: &[TopicPartition]);
/// The poll in progress is not charged yet, so it is dropped and replays
/// from the last commit.
fn on_shutdown(&mut self);
fn on_submitted(&mut self, poll: InFlightPoll);
fn next_step(&mut self, now: Instant) -> Option<Step>;
}
fn report_revoked(stripped: u64) {
if stripped > 0 {
counter!("ingestion_consumer_polls_dropped_on_revoke_total").increment(stripped);
info!(stripped, "Removed revoked partitions' messages");
}
}