Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(personhog): send leader ingestion warnings through a dedicated producer (#122878)

Zach Brown0c0ff9f36061
diff --git a/.agents/skills/adding-ingestion-warnings/SKILL.md b/.agents/skills/adding-ingestion-warnings/SKILL.md
index 39051cb5629e..df2f6203558d 100644
--- a/.agents/skills/adding-ingestion-warnings/SKILL.md
+++ b/.agents/skills/adding-ingestion-warnings/SKILL.md
@@ -93,7 +93,7 @@ Warning::new(WarningType::MyNewType)
`into_row` takes a `WarningSource` (declare one const per emit site, as capture does in `src/lib.rs`) and injects `teamId` and the registry's `category`/`severity` over the caller's details (they cannot be spoofed or forgotten), and stamps the ClickHouse-format timestamp. A direct-row type needs **no** `captureProduced` flag and no `from_tag` arm — register it, regenerate the artifact, and emit. See the module doc in `rust/common/ingestion_warnings/src/serializer.rs` for the envelope-vs-row correspondence table and the trust rationale (the envelope lane is attacker-writable, so the consumer stamps classification; the row topic is ACL-guarded, so the producer does).
-The crate's `RowWarningEmitter` (`src/row.rs`) is the producer for this transport: `KafkaRowWarningEmitter` wraps a dedicated fire-and-forget `ThreadedProducer`, throttles per `(team_id, scope, type)` where `scope` is the caller's debounce entity (a distinct id, say), keys rows by team, and reports through the same `ingestion_warnings_total` outcomes as the envelope emitter. `test_support::CollectingRowEmitter` records rows for assertions. The emitter's throttle keeps the crate's default tracked-key cap unless the caller raises it, and a per-entity scope needs a cap sized for that cardinality (see `MAX_TRACKED_SCOPES` in the identity service). personhog-identity is the reference caller (`rust/personhog-identity/src/warnings.rs`: a `WarningSource` per emit site, a pure verdict-to-warning mapping, and `install` building the producer from env with the emitter-enabled gauge); the leader still carries its own older `WarningsProducer`.
+The crate's `RowWarningEmitter` (`src/row.rs`) is the producer for this transport: `KafkaRowWarningEmitter` wraps a dedicated fire-and-forget `ThreadedProducer`, throttles per `(team_id, scope, type)` where `scope` is the caller's debounce entity (a distinct id, say), keys rows by team, and reports through the same `ingestion_warnings_total` outcomes as the envelope emitter. `test_support::CollectingRowEmitter` records rows for assertions. The emitter's throttle keeps the crate's default tracked-key cap unless the caller raises it, and a per-entity scope needs a cap sized for that cardinality. `install_row_emitter` (`src/row_install.rs`) builds the emitter from `RowWarningsSettings`: its own producer, a per-type budget per minute like capture's, a tracked-scope cap (`DEFAULT_MAX_TRACKED_SCOPES`), and the emitter-enabled gauge. Both personhog services call it from env-backed `INGESTION_WARNINGS_*` settings. personhog-identity is the reference caller (`rust/personhog-identity/src/warnings.rs`: a `WarningSource` per emit site and a pure verdict-to-warning mapping); personhog-leader (`rust/personhog-leader/src/warnings.rs`) emits admission size violations scoped per team.
Emitting from Rust:
diff --git a/rust/Cargo.lock b/rust/Cargo.lock
index 22b923751bc7..5bc2258f4946 100644
--- a/rust/Cargo.lock
+++ b/rust/Cargo.lock
@@ -3048,11 +3048,13 @@ dependencies = [
"common-liveness",
"common-types",
"governor",
+ "lifecycle",
"metrics",
"metrics-util",
"rdkafka",
"rstest",
"serde_json",
+ "tokio",
"tracing",
"uuid",
]
@@ -9106,7 +9108,6 @@ dependencies = [
"lifecycle",
"metrics",
"metrics-exporter-prometheus",
- "metrics-util",
"personhog-common",
"personhog-coordination",
"personhog-proto",
diff --git a/rust/common/ingestion_warnings/Cargo.toml b/rust/common/ingestion_warnings/Cargo.toml
index 349ef35d7d83..b4aea76aff0e 100644
--- a/rust/common/ingestion_warnings/Cargo.toml
+++ b/rust/common/ingestion_warnings/Cargo.toml
@@ -16,9 +16,11 @@ chrono = { workspace = true }
common-kafka = { path = "../kafka" }
common-types = { path = "../types" }
governor = { workspace = true }
+lifecycle = { path = "../lifecycle" }
metrics = { workspace = true }
rdkafka = { workspace = true }
serde_json = { workspace = true }
+tokio = { workspace = true }
tracing = { workspace = true }
uuid = { workspace = true }
diff --git a/rust/common/ingestion_warnings/src/lib.rs b/rust/common/ingestion_warnings/src/lib.rs
index 82247a4e4687..fcc2358aface 100644
--- a/rust/common/ingestion_warnings/src/lib.rs
+++ b/rust/common/ingestion_warnings/src/lib.rs
@@ -39,6 +39,7 @@
pub mod registry;
pub mod request;
pub mod row;
+pub mod row_install;
pub mod serializer;
pub mod test_support;
pub mod throttle;
@@ -58,6 +59,7 @@ use tracing::warn;
pub use registry::WarningType;
pub use request::{emit_request_warning, WarningRequestContext, UNKNOWN_ATTRIBUTION};
pub use row::{KafkaRowWarningEmitter, RowWarningEmitter};
+pub use row_install::{install_row_emitter, RowWarningsSettings};
pub use throttle::{ThrottleDecision, WarningThrottle};
/// Counter of emission attempts: labels `type` (warning type), `source`
diff --git a/rust/common/ingestion_warnings/src/row_install.rs b/rust/common/ingestion_warnings/src/row_install.rs
new file mode 100644
index 000000000000..bc1f43a44b12
--- /dev/null
+++ b/rust/common/ingestion_warnings/src/row_install.rs
@@ -0,0 +1,339 @@
+use std::num::NonZeroU32;
+use std::sync::Arc;
+use std::time::Duration;
+
+use common_kafka::config::KafkaConfig;
+use common_kafka::kafka_producer::create_threaded_kafka_producer_no_ping;
+use metrics::gauge;
+
+use crate::row::{KafkaRowWarningEmitter, RowWarningEmitter};
+use crate::throttle::{WarningThrottle, DEFAULT_MAX_TRACKED_KEYS};
+use crate::{observe_delivery, INGESTION_WARNINGS_EMITTER_ENABLED};
+
+pub const DEFAULT_TYPE_BUDGET_PER_MINUTE: NonZeroU32 = NonZeroU32::new(60).unwrap();
+/// Default for `INGESTION_WARNINGS_MAX_TRACKED_SCOPES`. Row scopes can be per
+/// distinct id, so the throttle default (sized for capture's per-token keys)
+/// would fill within the hour a key lives and silence every team on the pod.
+/// The type budgets admit well under this many keys an hour.
+pub const DEFAULT_MAX_TRACKED_SCOPES: usize = 1_000_000;
+const _: () = assert!(DEFAULT_MAX_TRACKED_SCOPES > DEFAULT_MAX_TRACKED_KEYS);
+const THROTTLE_SWEEP_INTERVAL: Duration = Duration::from_secs(60);
+
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct RowWarningsSettings {
+ pub enabled: bool,
+ pub kafka_hosts: String,
+ pub kafka_tls: bool,
+ pub topic: String,
+ pub kafka_queue_mib: u32,
+ pub kafka_message_max_bytes: u32,
+ pub kafka_acks: String,
+ pub kafka_retries: u32,
+ pub kafka_linger_ms: u32,
+ pub kafka_queue_messages: u32,
+ pub kafka_message_timeout_ms: u32,
+ pub type_budget_per_minute: NonZeroU32,
+ pub max_tracked_scopes: usize,
+}
+
+impl RowWarningsSettings {
+ pub fn kafka_config(&self, client_id: &str) -> KafkaConfig {
+ KafkaConfig {
+ kafka_hosts: self.kafka_hosts.clone(),
+ kafka_tls: self.kafka_tls,
+ kafka_client_id: client_id.to_string(),
+ kafka_producer_acks: Some(self.kafka_acks.clone()),
+ kafka_producer_retries: Some(self.kafka_retries),
+ kafka_producer_linger_ms: self.kafka_linger_ms,
+ kafka_producer_queue_messages: self.kafka_queue_messages,
+ kafka_message_timeout_ms: self.kafka_message_timeout_ms,
+ kafka_producer_queue_mib: self.kafka_queue_mib,
+ kafka_producer_message_max_bytes: Some(self.kafka_message_max_bytes),
+ ..Default::default()
+ }
+ }
+
+ pub fn disabled_reason(&self) -> Option<&'static str> {
+ if !self.enabled {
+ return Some("disabled");
+ }
+ if self.kafka_hosts.is_empty() {
+ return Some("hosts_unset");
+ }
+ if self.topic.is_empty() {
+ return Some("topic_unset");
+ }
+ None
+ }
+
+ fn throttle(&self) -> WarningThrottle {
+ WarningThrottle::default()
+ .with_max_tracked_keys(self.max_tracked_scopes)
+ .with_type_budget(
+ Duration::from_secs(60) / self.type_budget_per_minute.get(),
+ self.type_budget_per_minute,
+ )
+ }
+}
+
+fn report_emitter_enabled(reason: &'static str, enabled: bool) {
+ gauge!(INGESTION_WARNINGS_EMITTER_ENABLED, "reason" => reason).set(if enabled {
+ 1.0
+ } else {
+ 0.0
+ });
+}
+
+/// Register `handle` as advisory: the producer's stats callback is its only liveness signal.
+pub fn install_row_emitter(
+ settings: &RowWarningsSettings,
+ client_id: &str,
+ handle: Option<lifecycle::Handle>,
+) -> Option<Arc<dyn RowWarningEmitter>> {
+ if !settings.enabled {
+ return None;
+ }
+ if let Some(reason) = settings.disabled_reason() {
+ tracing::warn!(
+ reason,
+ "ingestion warnings enabled but unusable; emitter disabled"
+ );
+ report_emitter_enabled(reason, false);
+ return None;
+ }
+ let Some(handle) = handle else {
+ tracing::warn!(
+ "ingestion warnings enabled but no lifecycle handle was registered; emitter disabled"
+ );
+ report_emitter_enabled("no_handle", false);
+ return None;
+ };
+ let producer = match create_threaded_kafka_producer_no_ping(
+ &settings.kafka_config(client_id),
+ handle.clone(),
+ observe_delivery,
+ ) {
+ Ok(producer) => producer,
+ Err(e) => {
+ tracing::error!(error = %e, "failed to create ingestion warnings producer; emitter disabled");
+ report_emitter_enabled("producer_create_failed", false);
+ return None;
+ }
+ };
+ if settings.max_tracked_scopes <= DEFAULT_MAX_TRACKED_KEYS {
+ tracing::warn!(
+ max_tracked_scopes = settings.max_tracked_scopes,
+ "ingestion warnings scope cap is at or below the per-token default; a busy pod can fill it within the hour and drop every team's warnings"
+ );
+ }
+ let emitter = Arc::new(KafkaRowWarningEmitter::with_throttle(
+ producer,
+ settings.topic.clone(),
+ settings.throttle(),
+ ));
+
+ // Liveness comes from the producer's stats callback alone, so a wedged
+ // poll thread trips the deadline; a heartbeat here would mask it.
+ let background = emitter.clone();
+ tokio::spawn(async move {
+ let mut sweep = tokio::time::interval(THROTTLE_SWEEP_INTERVAL);
+ loop {
+ tokio::select! {
+ _ = sweep.tick() => background.sweep_throttle(),
+ _ = handle.shutdown_recv() => break,
+ }
+ }
+ });
+
+ report_emitter_enabled("ok", true);
+ tracing::info!(
+ topic = settings.topic.as_str(),
+ "ingestion warnings emitter enabled"
+ );
+ Some(emitter)
+}
+
+#[cfg(test)]
+mod tests {
+ use std::time::Instant;
+
+ use metrics_util::debugging::{DebugValue, DebuggingRecorder};
+ use rstest::rstest;
+
+ use super::*;
+ use crate::row::throttle_key;
+ use crate::throttle::ThrottleDecision;
+ use crate::WarningType;
+
+ const CLIENT_ID: &str = "row-install-test";
+
+ fn settings(enabled: bool, hosts: &str, topic: &str) -> RowWarningsSettings {
+ RowWarningsSettings {
+ enabled,
+ kafka_hosts: hosts.to_string(),
+ kafka_tls: false,
+ topic: topic.to_string(),
+ kafka_queue_mib: 16,
+ kafka_message_max_bytes: 1_048_576,
+ kafka_acks: "1".to_string(),
+ kafka_retries: 0,
+ kafka_linger_ms: 100,
+ kafka_queue_messages: 10_000,
+ kafka_message_timeout_ms: 5_000,
+ type_budget_per_minute: DEFAULT_TYPE_BUDGET_PER_MINUTE,
+ max_tracked_scopes: DEFAULT_MAX_TRACKED_SCOPES,
+ }
+ }
+
+ #[rstest]
+ #[case::off(false, "broker:9092", "topic", Some("disabled"))]
+ #[case::no_hosts(true, "", "topic", Some("hosts_unset"))]
+ #[case::no_topic(true, "broker:9092", "", Some("topic_unset"))]
+ #[case::usable(true, "broker:9092", "topic", None)]
+ fn unusable_settings_name_their_reason(
+ #[case] enabled: bool,
+ #[case] hosts: &str,
+ #[case] topic: &str,
+ #[case] expected: Option<&'static str>,
+ ) {
+ assert_eq!(settings(enabled, hosts, topic).disabled_reason(), expected);
+ }
+
+ #[test]
+ fn the_producer_config_carries_every_setting_and_stays_isolated() {
+ let config = RowWarningsSettings {
+ kafka_acks: "all".to_string(),
+ kafka_retries: 2,
+ kafka_linger_ms: 250,
+ kafka_queue_messages: 500,
+ kafka_message_timeout_ms: 9_000,
+ ..settings(true, "broker:9092", "topic")
+ }
+ .kafka_config(CLIENT_ID);
+ assert_eq!(config.kafka_hosts, "broker:9092");
+ assert_eq!(config.kafka_client_id, CLIENT_ID);
+ assert_eq!(config.kafka_producer_acks.as_deref(), Some("all"));
+ assert_eq!(config.kafka_producer_retries, Some(2));
+ assert_eq!(config.kafka_producer_linger_ms, 250);
+ assert_eq!(config.kafka_producer_queue_messages, 500);
+ assert_eq!(config.kafka_message_timeout_ms, 9_000);
+ assert_eq!(config.kafka_producer_queue_mib, 16);
+ assert_eq!(config.kafka_producer_message_max_bytes, Some(1_048_576));
+ assert!(config.kafka_producer_enable_idempotence.is_none());
+ assert!(config.kafka_producer_batch_size.is_none());
+ }
+
+ #[rstest]
+ #[case::hosts_unset("", "hosts_unset")]
+ #[case::no_handle("broker:9092", "no_handle")]
+ fn an_unusable_install_reports_its_reason_on_the_gauge(
+ #[case] hosts: &str,
+ #[case] reason: &str,
+ ) {
+ let recorder = DebuggingRecorder::new();
+ let snapshotter = recorder.snapshotter();
+ let emitter = metrics::with_local_recorder(&recorder, || {
+ install_row_emitter(&settings(true, hosts, "topic"), CLIENT_ID, None)
+ });
+ assert!(emitter.is_none());
+ let gauges: Vec<(String, f64)> = snapshotter
+ .snapshot()
+ .into_vec()
+ .into_iter()
+ .filter(|(key, _, _, _)| key.key().name() == INGESTION_WARNINGS_EMITTER_ENABLED)
+ .map(|(key, _, _, value)| {
+ let reason = key
+ .key()
+ .labels()
+ .find(|label| label.key() == "reason")
+ .expect("reason label")
+ .value()
+ .to_string();
+ let DebugValue::Gauge(value) = value else {
+ panic!("emitter enabled is a gauge");
+ };
+ (reason, value.into_inner())
+ })
+ .collect();
+ assert_eq!(gauges, vec![(reason.to_string(), 0.0)]);
+ }
+
+ #[test]
+ fn the_throttle_tracks_only_admitted_scopes_and_caps_above_the_shared_default() {
+ let throttle = settings(true, "broker:9092", "topic").throttle();
+ let warning = WarningType::CannotMergeAlreadyIdentified;
+ let mut admitted = 0;
+ for n in 0..=DEFAULT_MAX_TRACKED_KEYS {
+ match throttle.check(&throttle_key(7, &n.to_string()), warning) {
+ ThrottleDecision::Emit => admitted += 1,
+ ThrottleDecision::TypeBudgetExhausted => {}
+ other => panic!("scope {n}: {other:?}"),
+ }
+ }
+ assert!(admitted >= DEFAULT_TYPE_BUDGET_PER_MINUTE.get() as usize);
+ assert_eq!(
+ throttle.tracked_keys(),
+ admitted,
+ "a scope the budget refused must not hold a key"
+ );
+ }
+
+ #[test]
+ fn the_throttle_takes_its_budget_and_cap_from_settings() {
+ let throttle = RowWarningsSettings {
+ type_budget_per_minute: NonZeroU32::new(2).unwrap(),
+ max_tracked_scopes: 3,
+ ..settings(true, "broker:9092", "topic")
+ }
+ .throttle();
+ let warning = WarningType::CannotMergeAlreadyIdentified;
+ assert_eq!(
+ throttle.check(&throttle_key(7, "a"), warning),
+ ThrottleDecision::Emit
+ );
+ assert_eq!(
+ throttle.check(&throttle_key(7, "b"), warning),
+ ThrottleDecision::Emit
+ );
+ assert_eq!(
+ throttle.check(&throttle_key(7, "c"), warning),
+ ThrottleDecision::TypeBudgetExhausted
+ );
+ let later = Instant::now() + Duration::from_secs(31);
+ assert_eq!(
+ throttle.check_at(&throttle_key(7, "c"), warning, later),
+ ThrottleDecision::Emit
+ );
+ assert_eq!(
+ throttle.check(&throttle_key(7, "d"), warning),
+ ThrottleDecision::CardinalityCapped
+ );
+ }
+
+ #[test]
+ fn the_type_budget_refills_at_the_stated_rate() {
+ let throttle = settings(true, "broker:9092", "topic").throttle();
+ let warning = WarningType::MergeRaceCondition;
+ for n in 0..DEFAULT_TYPE_BUDGET_PER_MINUTE.get() {
+ assert_eq!(
+ throttle.check(&throttle_key(7, &n.to_string()), warning),
+ ThrottleDecision::Emit,
+ "scope {n}"
+ );
+ }
+ assert_eq!(
+ throttle.check(&throttle_key(7, "over"), warning),
+ ThrottleDecision::TypeBudgetExhausted
+ );
+ let soon = Instant::now() + Duration::from_millis(500);
+ assert_eq!(
+ throttle.check_at(&throttle_key(7, "soon"), warning, soon),
+ ThrottleDecision::TypeBudgetExhausted
+ );
+ let later = Instant::now() + Duration::from_millis(1100);
+ assert_eq!(
+ throttle.check_at(&throttle_key(7, "later"), warning, later),
+ ThrottleDecision::Emit
+ );
+ }
+}
diff --git a/rust/personhog-identity/Cargo.toml b/rust/personhog-identity/Cargo.toml
index 97e456fb9790..a58245044bd6 100644
--- a/rust/personhog-identity/Cargo.toml
+++ b/rust/personhog-identity/Cargo.toml
@@ -38,7 +38,6 @@ tracing-subscriber = { workspace = true }
uuid = { workspace = true }
[dev-dependencies]
-metrics-util = { workspace = true }
rstest = "0.26"
tokio = { workspace = true, features = ["test-util"] }
diff --git a/rust/personhog-identity/src/config.rs b/rust/personhog-identity/src/config.rs
index b453cadeeae1..0098b4499d1e 100644
--- a/rust/personhog-identity/src/config.rs
+++ b/rust/personhog-identity/src/config.rs
@@ -234,10 +234,10 @@ pub struct Config {
#[envconfig(default = "5000")]
pub ingestion_warnings_kafka_message_timeout_ms: u32,
- /// Warnings a second per type that one pod sends, shared by every team
+ /// Warnings of one type a minute that one pod sends, shared by every team
/// on it; the rest are dropped as `type_budget_exhausted`.
#[envconfig(default = "60")]
- pub ingestion_warnings_type_budget_per_second: NonZeroU32,
+ pub ingestion_warnings_type_budget_per_minute: NonZeroU32,
/// Throttle keys (team, distinct id, type) a pod remembers for the hour a
/// key lives, at roughly 135 bytes each.
@@ -518,8 +518,8 @@ impl Config {
Duration::from_secs(self.lifecycle_op_retention_hours * 3600)
}
- pub fn ingestion_warnings_settings(&self) -> crate::warnings::WarningsSettings {
- crate::warnings::WarningsSettings {
+ pub fn ingestion_warnings_settings(&self) -> common_ingestion_warnings::RowWarningsSettings {
+ common_ingestion_warnings::RowWarningsSettings {
enabled: self.ingestion_warnings_enabled,
kafka_hosts: self.ingestion_warnings_kafka_hosts.clone(),
kafka_tls: self.ingestion_warnings_kafka_tls,
@@ -531,7 +531,7 @@ impl Config {
kafka_linger_ms: self.ingestion_warnings_kafka_linger_ms,
kafka_queue_messages: self.ingestion_warnings_kafka_queue_messages,
kafka_message_timeout_ms: self.ingestion_warnings_kafka_message_timeout_ms,
- type_budget_per_second: self.ingestion_warnings_type_budget_per_second,
+ type_budget_per_minute: self.ingestion_warnings_type_budget_per_minute,
max_tracked_scopes: self.ingestion_warnings_max_tracked_scopes,
}
}
diff --git a/rust/personhog-identity/src/warnings.rs b/rust/personhog-identity/src/warnings.rs
index 46ca521d0025..461d87b764a8 100644
--- a/rust/personhog-identity/src/warnings.rs
+++ b/rust/personhog-identity/src/warnings.rs
@@ -4,18 +4,13 @@
//! call failures stay with the caller, which alone knows its retry budget
//! and merge-mode policy.
-use std::num::NonZeroU32;
use std::sync::Arc;
use std::time::Duration;
use common_ingestion_warnings::serializer::Warning;
-use common_ingestion_warnings::throttle::{WarningThrottle, DEFAULT_MAX_TRACKED_KEYS};
use common_ingestion_warnings::{
- observe_delivery, KafkaRowWarningEmitter, RowWarningEmitter, WarningSource, WarningType,
- INGESTION_WARNINGS_EMITTER_ENABLED,
+ install_row_emitter, RowWarningEmitter, RowWarningsSettings, WarningSource, WarningType,
};
-use common_kafka::config::KafkaConfig;
-use common_kafka::kafka_producer::create_threaded_kafka_producer_no_ping;
use uuid::Uuid;
use crate::lifecycle::merge::{
@@ -39,15 +34,6 @@ pub const MERGE_SAGA: WarningSource = WarningSource {
};
const KAFKA_CLIENT_ID: &str = "personhog-identity-ingestion-warnings";
-/// Default for `INGESTION_WARNINGS_TYPE_BUDGET_PER_SECOND`.
-pub const DEFAULT_TYPE_BUDGET_PER_SECOND: NonZeroU32 = NonZeroU32::new(60).unwrap();
-/// Default for `INGESTION_WARNINGS_MAX_TRACKED_SCOPES`. Keys are per distinct
-/// id, so the crate default (sized for capture's per-token keys) would fill
-/// within the hour a key lives and silence every team on the pod. The type
-/// budgets admit well under this many keys an hour.
-pub const DEFAULT_MAX_TRACKED_SCOPES: usize = 1_000_000;
-const _: () = assert!(DEFAULT_MAX_TRACKED_SCOPES > DEFAULT_MAX_TRACKED_KEYS);
-const THROTTLE_SWEEP_INTERVAL: Duration = Duration::from_secs(60);
/// The distinct id column's own limit: a legal id is never cut, and an
/// oversized one is cut before it is copied into details.
const MAX_DETAIL_ID_CHARS: usize = 400;
@@ -196,143 +182,16 @@ impl MergeWarnings {
}
}
-#[derive(Debug, Clone, PartialEq, Eq)]
-pub struct WarningsSettings {
- pub enabled: bool,
- pub kafka_hosts: String,
- pub kafka_tls: bool,
- pub topic: String,
- pub kafka_queue_mib: u32,
- pub kafka_message_max_bytes: u32,
- pub kafka_acks: String,
- pub kafka_retries: u32,
- pub kafka_linger_ms: u32,
- pub kafka_queue_messages: u32,
- pub kafka_message_timeout_ms: u32,
- pub type_budget_per_second: NonZeroU32,
- pub max_tracked_scopes: usize,
-}
-
-impl WarningsSettings {
- pub fn kafka_config(&self) -> KafkaConfig {
- KafkaConfig {
- kafka_hosts: self.kafka_hosts.clone(),
- kafka_tls: self.kafka_tls,
- kafka_client_id: KAFKA_CLIENT_ID.to_string(),
- kafka_producer_acks: Some(self.kafka_acks.clone()),
- kafka_producer_retries: Some(self.kafka_retries),
- kafka_producer_linger_ms: self.kafka_linger_ms,
- kafka_producer_queue_messages: self.kafka_queue_messages,
- kafka_message_timeout_ms: self.kafka_message_timeout_ms,
- kafka_producer_queue_mib: self.kafka_queue_mib,
- kafka_producer_message_max_bytes: Some(self.kafka_message_max_bytes),
- ..Default::default()
- }
- }
-
- pub fn disabled_reason(&self) -> Option<&'static str> {
- if !self.enabled {
- return Some("disabled");
- }
- if self.kafka_hosts.is_empty() {
- return Some("hosts_unset");
- }
- if self.topic.is_empty() {
- return Some("topic_unset");
- }
- None
+pub fn install(settings: &RowWarningsSettings, handle: Option<lifecycle::Handle>) -> MergeWarnings {
+ match install_row_emitter(settings, KAFKA_CLIENT_ID, handle) {
+ Some(emitter) => MergeWarnings::new(emitter),
+ None => MergeWarnings::disabled(),
}
}
-fn report_emitter_enabled(reason: &'static str, enabled: bool) {
- common_metrics::gauge(
- INGESTION_WARNINGS_EMITTER_ENABLED,
- &[("reason".to_string(), reason.to_string())],
- if enabled { 1.0 } else { 0.0 },
- );
-}
-
-fn throttle(settings: &WarningsSettings) -> WarningThrottle {
- WarningThrottle::default()
- .with_max_tracked_keys(settings.max_tracked_scopes)
- .with_type_budget(
- Duration::from_secs(1) / settings.type_budget_per_second.get(),
- settings.type_budget_per_second,
- )
-}
-
-pub fn install(settings: &WarningsSettings, handle: Option<lifecycle::Handle>) -> MergeWarnings {
- if !settings.enabled {
- return MergeWarnings::disabled();
- }
- if let Some(reason) = settings.disabled_reason() {
- tracing::warn!(
- reason,
- "ingestion warnings enabled but unusable; emitter disabled"
- );
- report_emitter_enabled(reason, false);
- return MergeWarnings::disabled();
- }
- let Some(handle) = handle else {
- tracing::warn!(
- "ingestion warnings enabled but no lifecycle handle was registered; emitter disabled"
- );
- report_emitter_enabled("no_handle", false);
- return MergeWarnings::disabled();
- };
- let producer = match create_threaded_kafka_producer_no_ping(
- &settings.kafka_config(),
- handle.clone(),
- observe_delivery,
- ) {
- Ok(producer) => producer,
- Err(e) => {
- tracing::error!(error = %e, "failed to create ingestion warnings producer; emitter disabled");
- report_emitter_enabled("producer_create_failed", false);
- return MergeWarnings::disabled();
- }
- };
- if settings.max_tracked_scopes <= DEFAULT_MAX_TRACKED_KEYS {
- tracing::warn!(
- max_tracked_scopes = settings.max_tracked_scopes,
- "ingestion warnings scope cap is at or below the per-token default; a busy pod can fill it within the hour and drop every team's warnings"
- );
- }
- let emitter = Arc::new(KafkaRowWarningEmitter::with_throttle(
- producer,
- settings.topic.clone(),
- throttle(settings),
- ));
-
- // Liveness comes from the producer's stats callback alone, so a wedged
- // poll thread trips the deadline; a heartbeat here would mask it.
- let background = emitter.clone();
- tokio::spawn(async move {
- let mut sweep = tokio::time::interval(THROTTLE_SWEEP_INTERVAL);
- loop {
- tokio::select! {
- _ = sweep.tick() => background.sweep_throttle(),
- _ = handle.shutdown_recv() => break,
- }
- }
- });
-
- report_emitter_enabled("ok", true);
- tracing::info!(
- topic = settings.topic.as_str(),
- "ingestion warnings emitter enabled"
- );
- MergeWarnings::new(emitter)
-}
-
#[cfg(test)]
mod tests {
- use std::time::Instant;
-
- use common_ingestion_warnings::row::throttle_key;
use common_ingestion_warnings::test_support::CollectingRowEmitter;
- use common_ingestion_warnings::throttle::ThrottleDecision;
- use metrics_util::debugging::{DebugValue, DebuggingRecorder};
use rstest::rstest;
use serde_json::Value;
@@ -522,169 +381,4 @@ mod tests {
"an illegal target is about the target"
);
}
-
- fn settings(enabled: bool, hosts: &str, topic: &str) -> WarningsSettings {
- WarningsSettings {
- enabled,
- kafka_hosts: hosts.to_string(),
- kafka_tls: false,
- topic: topic.to_string(),
- kafka_queue_mib: 16,
- kafka_message_max_bytes: 1_048_576,
- kafka_acks: "1".to_string(),
- kafka_retries: 0,
- kafka_linger_ms: 100,
- kafka_queue_messages: 10_000,
- kafka_message_timeout_ms: 5_000,
- type_budget_per_second: DEFAULT_TYPE_BUDGET_PER_SECOND,
- max_tracked_scopes: DEFAULT_MAX_TRACKED_SCOPES,
- }
- }
-
- #[rstest]
- #[case::off(false, "broker:9092", "topic", Some("disabled"))]
- #[case::no_hosts(true, "", "topic", Some("hosts_unset"))]
- #[case::no_topic(true, "broker:9092", "", Some("topic_unset"))]
- #[case::usable(true, "broker:9092", "topic", None)]
- fn unusable_settings_name_their_reason(
- #[case] enabled: bool,
- #[case] hosts: &str,
- #[case] topic: &str,
- #[case] expected: Option<&'static str>,
- ) {
- assert_eq!(settings(enabled, hosts, topic).disabled_reason(), expected);
- }
-
- #[test]
- fn the_producer_config_carries_every_setting_and_stays_isolated() {
- let config = WarningsSettings {
- kafka_acks: "all".to_string(),
- kafka_retries: 2,
- kafka_linger_ms: 250,
- kafka_queue_messages: 500,
- kafka_message_timeout_ms: 9_000,
- ..settings(true, "broker:9092", "topic")
- }
- .kafka_config();
- assert_eq!(config.kafka_hosts, "broker:9092");
- assert_eq!(config.kafka_client_id, KAFKA_CLIENT_ID);
- assert_eq!(config.kafka_producer_acks.as_deref(), Some("all"));
- assert_eq!(config.kafka_producer_retries, Some(2));
- assert_eq!(config.kafka_producer_linger_ms, 250);
- assert_eq!(config.kafka_producer_queue_messages, 500);
- assert_eq!(config.kafka_message_timeout_ms, 9_000);
- assert_eq!(config.kafka_producer_queue_mib, 16);
- assert_eq!(config.kafka_producer_message_max_bytes, Some(1_048_576));
- assert!(config.kafka_producer_enable_idempotence.is_none());
- assert!(config.kafka_producer_batch_size.is_none());
- }
-
- #[rstest]
- #[case::hosts_unset("", "hosts_unset")]
- #[case::no_handle("broker:9092", "no_handle")]
- fn an_unusable_install_reports_its_reason_on_the_gauge(
- #[case] hosts: &str,
- #[case] reason: &str,
- ) {
- let recorder = DebuggingRecorder::new();
- let snapshotter = recorder.snapshotter();
- let warnings = metrics::with_local_recorder(&recorder, || {
- install(&settings(true, hosts, "topic"), None)
- });
- assert!(!warnings.is_enabled());
- let gauges: Vec<(String, f64)> = snapshotter
- .snapshot()
- .into_vec()
- .into_iter()
- .filter(|(key, _, _, _)| key.key().name() == INGESTION_WARNINGS_EMITTER_ENABLED)
- .map(|(key, _, _, value)| {
- let reason = key
- .key()
- .labels()
- .find(|label| label.key() == "reason")
- .expect("reason label")
- .value()
- .to_string();
- let DebugValue::Gauge(value) = value else {
- panic!("emitter enabled is a gauge");
- };
- (reason, value.into_inner())
- })
- .collect();
- assert_eq!(gauges, vec![(reason.to_string(), 0.0)]);
- }
-
- #[test]
- fn the_throttle_tracks_only_admitted_scopes_and_caps_above_the_shared_default() {
- let throttle = throttle(&settings(true, "broker:9092", "topic"));
- let warning = WarningType::CannotMergeAlreadyIdentified;
- let mut admitted = 0;
- for n in 0..=DEFAULT_MAX_TRACKED_KEYS {
- match throttle.check(&throttle_key(7, &n.to_string()), warning) {
- ThrottleDecision::Emit => admitted += 1,
- ThrottleDecision::TypeBudgetExhausted => {}
- other => panic!("scope {n}: {other:?}"),
- }
- }
- assert!(admitted >= DEFAULT_TYPE_BUDGET_PER_SECOND.get() as usize);
- assert_eq!(
- throttle.tracked_keys(),
- admitted,
- "a scope the budget refused must not hold a key"
- );
- }
-
- #[test]
- fn the_throttle_takes_its_budget_and_cap_from_settings() {
- let throttle = throttle(&WarningsSettings {
- type_budget_per_second: NonZeroU32::new(2).unwrap(),
- max_tracked_scopes: 3,
- ..settings(true, "broker:9092", "topic")
- });
- let warning = WarningType::CannotMergeAlreadyIdentified;
- assert_eq!(
- throttle.check(&throttle_key(7, "a"), warning),
- ThrottleDecision::Emit
- );
- assert_eq!(
- throttle.check(&throttle_key(7, "b"), warning),
- ThrottleDecision::Emit
- );
- assert_eq!(
- throttle.check(&throttle_key(7, "c"), warning),
- ThrottleDecision::TypeBudgetExhausted
- );
- let later = Instant::now() + Duration::from_millis(600);
- assert_eq!(
- throttle.check_at(&throttle_key(7, "c"), warning, later),
- ThrottleDecision::Emit
- );
- assert_eq!(
- throttle.check(&throttle_key(7, "d"), warning),
- ThrottleDecision::CardinalityCapped
- );
- }
-
- #[test]
- fn the_type_budget_refills_at_the_stated_rate() {
- let throttle = throttle(&settings(true, "broker:9092", "topic"));
- let warning = WarningType::MergeRaceCondition;
- for n in 0..DEFAULT_TYPE_BUDGET_PER_SECOND.get() {
- assert_eq!(
- throttle.check(&throttle_key(7, &n.to_string()), warning),
- ThrottleDecision::Emit,
- "scope {n}"
- );
- }
- assert_eq!(
- throttle.check(&throttle_key(7, "over"), warning),
- ThrottleDecision::TypeBudgetExhausted
- );
- // A budget that refilled one permit a second would still refuse.
- let later = Instant::now() + Duration::from_millis(100);
- assert_eq!(
- throttle.check_at(&throttle_key(7, "later"), warning, later),
- ThrottleDecision::Emit
- );
- }
}
diff --git a/rust/personhog-leader/README.md b/rust/personhog-leader/README.md
index 0f65bd797646..c325f5e511b1 100644
--- a/rust/personhog-leader/README.md
+++ b/rust/personhog-leader/README.md
@@ -133,7 +133,15 @@ the writer (produce an unapplyable record).
Trims and rejections emit `person_properties_size_violation` ingestion
warnings (`src/warnings.rs`), throttled per (team, type) to match the Node
-pipeline's limiter. The writer-side weld
+pipeline's limiter. They go through a dedicated fire-and-forget producer,
+not the changelog producer, because the warnings topic can live on a
+different cluster from the changelog. `INGESTION_WARNINGS_ENABLED`
+(default `false`) turns it on, and `INGESTION_WARNINGS_KAFKA_HOSTS`,
+`INGESTION_WARNINGS_KAFKA_TLS` and `INGESTION_WARNINGS_TOPIC` (default
+`clickhouse_ingestion_warnings`) name the destination. Enabled with no
+hosts set, the leader logs a warning and emits nothing. Health reads from
+`ingestion_warnings_emitter_enabled{reason}` and
+`ingestion_warnings_total{source="personhog-leader",outcome}`. The writer-side weld
(`personhog-writer/tests/admission_weld.rs`) runs hostile property fixtures
through these exact functions and the writer's real statement against live
Postgres, asserting byte-exact round-trips.
diff --git a/rust/personhog-leader/src/config.rs b/rust/personhog-leader/src/config.rs
index d1cf1d9d0bb5..4f4e2a69baa9 100644
--- a/rust/personhog-leader/src/config.rs
+++ b/rust/personhog-leader/src/config.rs
@@ -1,7 +1,9 @@
use std::fs;
use std::net::SocketAddr;
+use std::num::NonZeroU32;
use std::time::Duration;
+use common_ingestion_warnings::RowWarningsSettings;
use common_kafka::config::KafkaConfig;
use envconfig::Envconfig;
use personhog_common::h2_window::{Http2Windows, WindowSize};
@@ -197,11 +199,49 @@ pub struct Config {
#[envconfig(default = "524288")]
pub properties_trim_target: usize,
- /// Topic for in-product ingestion warnings emitted when admission
- /// trims or rejects an update.
+ #[envconfig(default = "false")]
+ pub ingestion_warnings_enabled: bool,
+
+ #[envconfig(default = "")]
+ pub ingestion_warnings_kafka_hosts: String,
+
+ #[envconfig(default = "false")]
+ pub ingestion_warnings_kafka_tls: bool,
+
#[envconfig(default = "clickhouse_ingestion_warnings")]
pub ingestion_warnings_topic: String,
+ #[envconfig(default = "16")]
+ pub ingestion_warnings_kafka_queue_mib: u32,
+
+ #[envconfig(default = "1048576")]
+ pub ingestion_warnings_kafka_message_max_bytes: u32,
+
+ /// The defaults below are fire-and-forget: a warning is worth less than
+ /// the cost of retrying it, and a slow warnings topic must not hold an update.
+ #[envconfig(default = "1")]
+ pub ingestion_warnings_kafka_acks: String,
+
+ #[envconfig(default = "0")]
+ pub ingestion_warnings_kafka_retries: u32,
+
+ #[envconfig(default = "100")]
+ pub ingestion_warnings_kafka_linger_ms: u32,
+
+ #[envconfig(default = "10000")]
+ pub ingestion_warnings_kafka_queue_messages: u32,
+
+ #[envconfig(default = "5000")]
+ pub ingestion_warnings_kafka_message_timeout_ms: u32,
+
+ /// Warnings of one type a minute that one pod sends, shared by every team
+ /// on it; the rest are dropped as `type_budget_exhausted`.
+ #[envconfig(default = "60")]
+ pub ingestion_warnings_type_budget_per_minute: NonZeroU32,
+
+ #[envconfig(default = "1000000")]
+ pub ingestion_warnings_max_tracked_scopes: usize,
+
// ── Dirty index / changelog recovery ─────────────────────────
/// How often to poll the writer's committed offsets and prune dirty
/// index marks the writer has applied to PG. A tick costs one batched
@@ -757,6 +797,24 @@ impl Config {
Duration::from_secs(self.heartbeat_interval_secs)
}
+ pub fn ingestion_warnings_settings(&self) -> RowWarningsSettings {
+ RowWarningsSettings {
+ enabled: self.ingestion_warnings_enabled,
+ kafka_hosts: self.ingestion_warnings_kafka_hosts.clone(),
+ kafka_tls: self.ingestion_warnings_kafka_tls,
+ topic: self.ingestion_warnings_topic.clone(),
+ kafka_queue_mib: self.ingestion_warnings_kafka_queue_mib,
+ kafka_message_max_bytes: self.ingestion_warnings_kafka_message_max_bytes,
+ kafka_acks: self.ingestion_warnings_kafka_acks.clone(),
+ kafka_retries: self.ingestion_warnings_kafka_retries,
+ kafka_linger_ms: self.ingestion_warnings_kafka_linger_ms,
+ kafka_queue_messages: self.ingestion_warnings_kafka_queue_messages,
+ kafka_message_timeout_ms: self.ingestion_warnings_kafka_message_timeout_ms,
+ type_budget_per_minute: self.ingestion_warnings_type_budget_per_minute,
+ max_tracked_scopes: self.ingestion_warnings_max_tracked_scopes,
+ }
+ }
+
/// Resolve the K8s namespace from config or the service account mount.
pub fn resolve_k8s_namespace(&self) -> Result<String, String> {
if !self.k8s_namespace.is_empty() {
diff --git a/rust/personhog-leader/src/main.rs b/rust/personhog-leader/src/main.rs
index 9e16b4a34082..ba06cb9c5e3b 100644
--- a/rust/personhog-leader/src/main.rs
+++ b/rust/personhog-leader/src/main.rs
@@ -48,7 +48,7 @@ use personhog_leader::settle::{prune_and_settle_tick, retired_topic};
use personhog_leader::warming::{
changelog_partition_count, WarmClientPools, WarmingConfig, WarmingRetryPolicy,
};
-use personhog_leader::warnings::WarningsProducer;
+use personhog_leader::warnings;
common_alloc::used!();
@@ -149,6 +149,15 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
);
let changelog_handle = manager.register("changelog-config", ComponentOptions::new());
let migration_handle = manager.register("migration-duty", ComponentOptions::new());
+ let warnings_settings = config.ingestion_warnings_settings();
+ let warnings_handle = warnings_settings.disabled_reason().is_none().then(|| {
+ manager.register(
+ "ingestion-warnings",
+ ComponentOptions::new()
+ .with_liveness_deadline(Duration::from_secs(30))
+ .is_advisory(true),
+ )
+ });
let readiness = manager.readiness_handler();
let liveness = manager.liveness_handler();
@@ -418,10 +427,8 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
})
.expect("Failed to build changelog recovery consumer pool"),
);
- let warnings = WarningsProducer::new(
- kafka_producer.clone(),
- config.ingestion_warnings_topic.clone(),
- );
+ let warnings = warnings::install(&warnings_settings, warnings_handle);
+ let warnings_to_flush = warnings.clone();
let fence_scan = fallback.clone();
// All derived from LEASE_TTL, so an operator debugging a timeout
// cannot recover them without redoing the derivation.
@@ -677,18 +684,16 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
}
});
- // Periodic sweep of idle per-key locks, refilled warning-throttle
- // keys, and parked fence connections nothing consumed (a cancelled
- // inbound handoff leaves no convergence behind to discard them).
+ // Periodic sweep of idle per-key locks and parked fence connections
+ // nothing consumed (a cancelled inbound handoff leaves no convergence
+ // behind to discard them).
let sweep_locks = Arc::clone(&locks);
- let sweep_warnings = warnings.clone();
let sweep_fenced = Arc::clone(&fenced);
tokio::spawn(async move {
let mut interval = tokio::time::interval(Duration::from_secs(60));
loop {
interval.tick().await;
sweep_idle_locks(&sweep_locks);
- sweep_warnings.sweep_throttle();
sweep_fenced.sweep_prepared();
}
});
@@ -784,7 +789,15 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {
}
});
- monitor_guard.wait().await?;
+ let waited = monitor_guard.wait().await;
+
+ // Flush before the result propagates: a failed component or a timed-out
+ // drain is exactly the shutdown that leaves warnings queued.
+ if let Err(e) = tokio::task::spawn_blocking(move || warnings_to_flush.flush()).await {
+ tracing::warn!(error = %e, "ingestion warnings flush task panicked");
+ }
+
+ waited?;
Ok(())
}
diff --git a/rust/personhog-leader/src/service.rs b/rust/personhog-leader/src/service.rs
index 5a75fbd03539..962097fb2562 100644
--- a/rust/personhog-leader/src/service.rs
+++ b/rust/personhog-leader/src/service.rs
@@ -1683,9 +1683,7 @@ mod tests {
use common_kafka::config::KafkaConfig;
use envconfig::Envconfig;
- use health::HealthRegistry;
use personhog_proto::personhog::types::v1::{LifecycleOpType, ReleaseOutcome};
- use rdkafka::ClientConfig;
use tonic::Code;
use super::*;
@@ -1807,19 +1805,10 @@ mod tests {
assert_eq!(stats.clamped_numbers, 1);
}
- /// A service with no PG pool and a producer that never connects —
- /// enough to exercise the miss path, where no test reaches Kafka or PG.
+ /// A service with no PG pool and no reachable Kafka — enough to
+ /// exercise the miss path, where no test reaches Kafka or PG.
async fn make_test_service() -> PersonHogLeaderService {
let kafka = KafkaConfig::init_from_hashmap(&HashMap::new()).unwrap();
- let liveness = HealthRegistry::new("test")
- .register("kafka".to_string(), Duration::from_secs(60))
- .await;
- let producer: rdkafka::producer::FutureProducer<
- common_kafka::kafka_producer::KafkaContext,
- > = ClientConfig::new()
- .set("bootstrap.servers", "127.0.0.1:1")
- .create_with_context(common_kafka::kafka_producer::KafkaContext::from(liveness))
- .unwrap();
PersonHogLeaderService::new(
Arc::new(PartitionedCache::new(16)),
None,
@@ -1847,7 +1836,7 @@ mod tests {
.expect("build recovery pool"),
),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
// Lazily built: nothing here connects until a write acquires
// a partition, which these tests never do.
diff --git a/rust/personhog-leader/src/warnings.rs b/rust/personhog-leader/src/warnings.rs
index 64116ddbc537..2abf5d5e24e8 100644
--- a/rust/personhog-leader/src/warnings.rs
+++ b/rust/personhog-leader/src/warnings.rs
@@ -1,14 +1,11 @@
use std::sync::Arc;
-
-use metrics::counter;
-use rdkafka::producer::{FutureProducer, FutureRecord};
-use tracing::warn;
+use std::time::Duration;
use common_ingestion_warnings::registry::WarningType;
-use common_ingestion_warnings::serializer::Warning as WarningPayload;
-use common_ingestion_warnings::throttle::WarningThrottle;
-use common_ingestion_warnings::WarningSource;
-use common_kafka::kafka_producer::KafkaContext;
+use common_ingestion_warnings::serializer::Warning;
+use common_ingestion_warnings::{
+ install_row_emitter, RowWarningEmitter, RowWarningsSettings, WarningSource,
+};
/// Warnings emitted from the leader's admission check, ahead of the write
/// path proper. `pipeline_step` lands in the v2 table's `pipeline_step`
@@ -19,6 +16,14 @@ const LEADER_ADMISSION: WarningSource = WarningSource {
pipeline_step: "personhog_admission",
};
+const KAFKA_CLIENT_ID: &str = "personhog-leader-ingestion-warnings";
+const SHUTDOWN_FLUSH: Duration = Duration::from_secs(2);
+
+/// The throttle scope within a team. Admission warnings are budgeted per
+/// team and type, so one person hammered with oversized updates and many
+/// such persons on one team both yield one warning an hour.
+const TEAM_SCOPE: &str = "";
+
/// An in-product ingestion warning about a person's property size —
/// emitted when admission trims an update to fit the Postgres constraint
/// or rejects one that cannot fit. Carries no property values, only
@@ -31,86 +36,53 @@ pub struct SizeViolationWarning {
pub message: String,
}
-/// Produces ingestion warnings to Kafka so users see property size
-/// violations in-product, with the payload built by the shared
-/// ingestion-warnings crate so type and classification come from the
-/// pipeline registry. Shares the leader's changelog producer (rdkafka
-/// producers are topic-agnostic), so buffered warnings flush with it on
-/// shutdown; emission is fire-and-forget and never blocks or fails the
-/// update that triggered it.
-///
-/// Emission is throttled per `(team, type)` with the pipeline's default
-/// budget (one per hour), matching the Node consumers' warning limiter:
-/// one oversized person being hammered with updates yields one warning an
-/// hour, not one per update. Cloning shares the throttle, so every handle
-/// draws from the same budget.
+/// Sends size violation warnings so users see them in-product, with the
+/// payload built by the shared ingestion-warnings crate so type and
+/// classification come from the pipeline registry. Emission is
+/// fire-and-forget and never blocks or fails the update that triggered it.
#[derive(Clone)]
pub struct WarningsProducer {
- producer: FutureProducer<KafkaContext>,
- topic: String,
- throttle: Arc<WarningThrottle>,
+ emitter: Option<Arc<dyn RowWarningEmitter>>,
}
impl WarningsProducer {
- pub fn new(producer: FutureProducer<KafkaContext>, topic: String) -> Self {
- Self::with_throttle(producer, topic, WarningThrottle::default())
- }
-
- /// Construct with an explicit throttle — tests use a larger burst so
- /// consecutive enforcement actions for one team all surface.
- pub fn with_throttle(
- producer: FutureProducer<KafkaContext>,
- topic: String,
- throttle: WarningThrottle,
- ) -> Self {
+ pub fn new(emitter: Arc<dyn RowWarningEmitter>) -> Self {
Self {
- producer,
- topic,
- throttle: Arc::new(throttle),
+ emitter: Some(emitter),
}
}
+ pub fn disabled() -> Self {
+ Self { emitter: None }
+ }
+
pub fn emit(&self, warning: &SizeViolationWarning) {
- let warning_type = WarningType::PersonPropertiesSizeViolation;
- if let Some(outcome) = self
- .throttle
- .check(&warning.team_id.to_string(), warning_type)
- .drop_outcome()
- {
- counter!(
- "personhog_leader_ingestion_warnings_suppressed_total",
- "outcome" => outcome
- )
- .increment(1);
+ let Some(emitter) = &self.emitter else {
return;
- }
-
+ };
// teamId, category, severity, and pipelineStep are injected by the
// terminal.
- let payload = WarningPayload::new(warning_type)
+ let payload = Warning::new(WarningType::PersonPropertiesSizeViolation)
.with_detail("personId", warning.person_uuid.clone())
- .with_detail("message", warning.message.clone())
- .into_row(warning.team_id, LEADER_ADMISSION);
-
- let payload_bytes = serde_json::to_vec(&payload).unwrap_or_default();
- let key = warning.team_id.to_string();
- let record = FutureRecord::to(&self.topic)
- .payload(&payload_bytes)
- .key(&key);
+ .with_detail("message", warning.message.clone());
+ emitter.emit(warning.team_id, TEAM_SCOPE, LEADER_ADMISSION, payload);
+ }
- match self.producer.send_result(record) {
- Ok(_future) => {
- counter!("personhog_leader_ingestion_warnings_emitted_total").increment(1);
- }
- Err((e, _)) => {
- warn!(error = %e, "failed to enqueue ingestion warning");
- }
+ /// Blocking; call once the gRPC server, the only caller of `emit`, has
+ /// stopped. The emitter's own component is advisory, so it cannot flush last.
+ pub fn flush(&self) {
+ if let Some(emitter) = &self.emitter {
+ emitter.flush(SHUTDOWN_FLUSH);
}
}
+}
- /// Evict fully refilled throttle keys, bounding memory. Called from the
- /// leader's periodic maintenance task.
- pub fn sweep_throttle(&self) {
- self.throttle.sweep();
+pub fn install(
+ settings: &RowWarningsSettings,
+ handle: Option<lifecycle::Handle>,
+) -> WarningsProducer {
+ match install_row_emitter(settings, KAFKA_CLIENT_ID, handle) {
+ Some(emitter) => WarningsProducer::new(emitter),
+ None => WarningsProducer::disabled(),
}
}
diff --git a/rust/personhog-leader/tests/commit_integration.rs b/rust/personhog-leader/tests/commit_integration.rs
index 9a20f319c479..ca37389b35ba 100644
--- a/rust/personhog-leader/tests/commit_integration.rs
+++ b/rust/personhog-leader/tests/commit_integration.rs
@@ -6,8 +6,8 @@ use std::sync::Arc;
use std::time::{Duration, Instant};
use common::{
- create_local_kafka_producer, seed_person, served_captures, target, test_cached_person,
- test_kafka_config, test_recovery, unique_team_id, KAFKA_BOOTSTRAP,
+ seed_person, served_captures, target, test_cached_person, test_kafka_config, test_recovery,
+ unique_team_id, KAFKA_BOOTSTRAP,
};
use dashmap::DashMap;
use personhog_common::partitioning::partition_for_person;
@@ -80,7 +80,6 @@ async fn start_harness() -> Harness {
cache.create_partition(partition);
seed_person(&cache, partition, seed);
let locks = Arc::new(DashMap::new());
- let producer = create_local_kafka_producer().await;
let service = Arc::new(PersonHogLeaderService::new(
Arc::clone(&cache),
None,
@@ -90,7 +89,7 @@ async fn start_harness() -> Harness {
Arc::new(DirtyIndex::new(1_000_000)),
test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
Arc::clone(&fenced),
common::live_authority(),
diff --git a/rust/personhog-leader/tests/common/mod.rs b/rust/personhog-leader/tests/common/mod.rs
index 8cbff5d90a08..8b79c4b3096a 100644
--- a/rust/personhog-leader/tests/common/mod.rs
+++ b/rust/personhog-leader/tests/common/mod.rs
@@ -11,8 +11,12 @@ use std::time::Duration;
use dashmap::DashMap;
use async_trait::async_trait;
+use common_ingestion_warnings::throttle::WarningThrottle;
+use common_ingestion_warnings::{observe_delivery, KafkaRowWarningEmitter};
use common_kafka::config::KafkaConfig;
-use common_kafka::kafka_producer::{create_kafka_producer, KafkaContext};
+use common_kafka::kafka_producer::{
+ create_kafka_producer, create_threaded_kafka_producer_no_ping, KafkaContext,
+};
use health::HealthRegistry;
use personhog_coordination::authority::AuthorityClock;
use personhog_coordination::changelog::LiveChangelog;
@@ -318,6 +322,24 @@ pub async fn create_local_kafka_producer() -> FutureProducer<KafkaContext> {
.expect("failed to connect to local Kafka")
}
+/// A warnings producer against local Kafka, for tests that read warnings back.
+pub async fn local_warnings_producer(topic: &str, throttle: WarningThrottle) -> WarningsProducer {
+ let registry = HealthRegistry::new("test");
+ let handle = registry
+ .register("ingestion-warnings".to_string(), Duration::from_secs(30))
+ .await;
+ let config = KafkaConfig {
+ kafka_hosts: KAFKA_BOOTSTRAP.to_string(),
+ kafka_producer_linger_ms: 0,
+ ..Default::default()
+ };
+ let producer = create_threaded_kafka_producer_no_ping(&config, handle, observe_delivery)
+ .expect("failed to create the warnings producer");
+ WarningsProducer::new(Arc::new(KafkaRowWarningEmitter::with_throttle(
+ producer, topic, throttle,
+ )))
+}
+
/// One test's changelog topic: two tests acquiring the same partition
/// of a shared one would fence each other.
pub struct TestTopic {
@@ -385,18 +407,15 @@ pub async fn create_mock_kafka_with_partitions(
/// A changelog topic and a producer for it, with `NUM_PARTITIONS`
/// partitions so warming can fetch every partition's watermarks.
-pub async fn create_test_kafka() -> (TestTopic, FutureProducer<KafkaContext>) {
+pub async fn create_test_kafka() -> TestTopic {
create_test_kafka_with_partitions(NUM_PARTITIONS as i32).await
}
/// `create_test_kafka` with a partition count the caller pins.
-pub async fn create_test_kafka_with_partitions(
- partitions: i32,
-) -> (TestTopic, FutureProducer<KafkaContext>) {
+pub async fn create_test_kafka_with_partitions(partitions: i32) -> TestTopic {
let topic = unique_topic();
create_broker_topic(&topic, partitions).await;
- let producer = create_local_kafka_producer().await;
- (TestTopic { topic }, producer)
+ TestTopic { topic }
}
/// A topic of this test's own, deleted when the handle drops.
@@ -472,7 +491,6 @@ pub async fn start_leader_pod(
.unwrap()
.expect("seed config/changelog before starting a pod")
.topic;
- let kafka_producer = create_local_kafka_producer().await;
// Pod with real handoff handler. Warming consumer reads from the same
// mock broker the producer is publishing to so the topic actually
@@ -542,7 +560,7 @@ pub async fn start_leader_pod(
Arc::clone(&dirty_index),
recovery,
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(dashmap::DashMap::new()),
Arc::clone(&fenced),
Arc::clone(&authority),
@@ -582,7 +600,6 @@ pub async fn start_leader_pod_with_lease_ttl(
.unwrap()
.expect("seed config/changelog before starting a pod")
.topic;
- let kafka_producer = create_local_kafka_producer().await;
let heartbeat_secs = (lease_ttl as u64 / 3).max(1);
let inflight = Arc::new(InflightTracker::new());
@@ -648,7 +665,7 @@ pub async fn start_leader_pod_with_lease_ttl(
Arc::clone(&dirty_index),
recovery,
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(dashmap::DashMap::new()),
Arc::clone(&fenced),
Arc::clone(&authority),
@@ -729,7 +746,7 @@ pub async fn start_leader_with_pg_fallback(
cancel: CancellationToken,
) -> (SocketAddr, Arc<PartitionedCache>, TestTopic) {
let cache = Arc::new(PartitionedCache::new(1 << 20));
- let (mock_cluster, kafka_producer) = create_test_kafka().await;
+ let mock_cluster = create_test_kafka().await;
let pool = create_persons_pool().await;
let service = PersonHogLeaderService::new(
@@ -745,7 +762,7 @@ pub async fn start_leader_with_pg_fallback(
Arc::new(DirtyIndex::new(1_000_000)),
test_recovery(&mock_cluster.bootstrap_servers()),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(dashmap::DashMap::new()),
fenced_producers_acquired(&mock_cluster.topic, NUM_PARTITIONS as i32).await,
live_authority(),
diff --git a/rust/personhog-leader/tests/fence_integration.rs b/rust/personhog-leader/tests/fence_integration.rs
index 53cdcd130d9a..e22f91374648 100644
--- a/rust/personhog-leader/tests/fence_integration.rs
+++ b/rust/personhog-leader/tests/fence_integration.rs
@@ -96,7 +96,7 @@ struct FenceHarness {
}
async fn start_fence_harness(mut seed: CachedPerson, fallback: Option<PgFallback>) -> FenceHarness {
- let (mock_cluster, kafka_producer) = create_test_kafka().await;
+ let mock_cluster = create_test_kafka().await;
let bootstrap = mock_cluster.bootstrap_servers();
// Every harness gets its own team, so concurrent tests — and leftover
@@ -125,7 +125,7 @@ async fn start_fence_harness(mut seed: CachedPerson, fallback: Option<PgFallback
Arc::clone(&dirty_index),
test_recovery(&bootstrap),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::clone(&fences),
common::fenced_producers_acquired(&mock_cluster.topic, NUM_PARTITIONS as i32).await,
common::live_authority(),
@@ -1446,7 +1446,7 @@ async fn at_capacity_new_fences_shed_but_reseals_succeed() {
let team_id = unique_team_id();
let cache = Arc::new(PartitionedCache::new(1 << 20));
- let (_mock_cluster, kafka_producer) = create_test_kafka().await;
+ let _mock_cluster = create_test_kafka().await;
let service = PersonHogLeaderService::new(
Arc::clone(&cache),
None,
@@ -1456,7 +1456,7 @@ async fn at_capacity_new_fences_shed_but_reseals_succeed() {
Arc::new(DirtyIndex::new(1_000_000)),
test_recovery(&_mock_cluster.bootstrap_servers()),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
common::fenced_producers_acquired(&_mock_cluster.topic, NUM_PARTITIONS as i32).await,
common::live_authority(),
diff --git a/rust/personhog-leader/tests/integration.rs b/rust/personhog-leader/tests/integration.rs
index 235280ecb36d..7209764fcc34 100644
--- a/rust/personhog-leader/tests/integration.rs
+++ b/rust/personhog-leader/tests/integration.rs
@@ -177,7 +177,7 @@ async fn service_accepts_requests_after_coordination_warmup() {
async fn unowned_partition_returns_failed_precondition() {
// Create service + cache directly, no coordination
let cache = Arc::new(PartitionedCache::new(1 << 20));
- let (_mock_cluster, kafka_producer) = create_test_kafka().await;
+ let _mock_cluster = create_test_kafka().await;
// One set for both: a second would fence the first at the broker.
let fenced =
common::fenced_producers_acquired(&_mock_cluster.topic, NUM_PARTITIONS as i32).await;
@@ -190,7 +190,7 @@ async fn unowned_partition_returns_failed_precondition() {
Arc::new(DirtyIndex::new(1_000_000)),
test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
Arc::clone(&fenced),
common::live_authority(),
@@ -263,7 +263,7 @@ async fn unowned_partition_returns_failed_precondition() {
#[tokio::test]
async fn missing_partition_metadata_returns_invalid_argument() {
let cache = Arc::new(PartitionedCache::new(1 << 20));
- let (_mock_cluster, kafka_producer) = create_test_kafka().await;
+ let _mock_cluster = create_test_kafka().await;
// One set for both: a second would fence the first at the broker.
let fenced =
common::fenced_producers_acquired(&_mock_cluster.topic, NUM_PARTITIONS as i32).await;
@@ -276,7 +276,7 @@ async fn missing_partition_metadata_returns_invalid_argument() {
Arc::new(DirtyIndex::new(1_000_000)),
test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
Arc::clone(&fenced),
common::live_authority(),
@@ -352,7 +352,7 @@ async fn missing_partition_metadata_returns_invalid_argument() {
#[tokio::test]
async fn mismatched_partition_metadata_returns_failed_precondition() {
let cache = Arc::new(PartitionedCache::new(1 << 20));
- let (_mock_cluster, kafka_producer) = create_test_kafka().await;
+ let _mock_cluster = create_test_kafka().await;
// One set for both: a second would fence the first at the broker.
let fenced =
common::fenced_producers_acquired(&_mock_cluster.topic, NUM_PARTITIONS as i32).await;
@@ -365,7 +365,7 @@ async fn mismatched_partition_metadata_returns_failed_precondition() {
Arc::new(DirtyIndex::new(1_000_000)),
test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
Arc::clone(&fenced),
common::live_authority(),
@@ -444,7 +444,7 @@ async fn mismatched_partition_metadata_returns_failed_precondition() {
#[tokio::test]
async fn writes_fenced_after_drain_reads_still_served() {
let cache = Arc::new(PartitionedCache::new(1 << 20));
- let (_mock_cluster, kafka_producer) = create_test_kafka().await;
+ let _mock_cluster = create_test_kafka().await;
let inflight = Arc::new(InflightTracker::new());
let dirty_index = Arc::new(DirtyIndex::new(1_000_000));
let recovery = test_recovery(KAFKA_BOOTSTRAP);
@@ -461,7 +461,7 @@ async fn writes_fenced_after_drain_reads_still_served() {
Arc::clone(&dirty_index),
Arc::clone(&recovery),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
Arc::clone(&fenced),
common::live_authority(),
@@ -590,7 +590,7 @@ async fn writes_fenced_after_drain_reads_still_served() {
#[tokio::test]
async fn drain_fences_before_waiting_on_inflight() {
let cache = Arc::new(PartitionedCache::new(1 << 20));
- let (_mock_cluster, kafka_producer) = create_test_kafka().await;
+ let _mock_cluster = create_test_kafka().await;
let inflight = Arc::new(InflightTracker::new());
let dirty_index = Arc::new(DirtyIndex::new(1_000_000));
let recovery = test_recovery(KAFKA_BOOTSTRAP);
@@ -607,7 +607,7 @@ async fn drain_fences_before_waiting_on_inflight() {
Arc::clone(&dirty_index),
Arc::clone(&recovery),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
Arc::clone(&fenced),
common::live_authority(),
@@ -924,7 +924,7 @@ async fn update_produces_person_state_to_kafka() {
const PERSON_ID: i64 = 2;
let routing_partition: u32 = partition_for_person(1, PERSON_ID, NUM_PARTITIONS);
assert_eq!(routing_partition, 2, "test key must map to partition 2");
- let (mock_cluster, kafka_producer) = create_test_kafka_with_partitions(4).await;
+ let mock_cluster = create_test_kafka_with_partitions(4).await;
let cache = Arc::new(PartitionedCache::new(1 << 20));
let service = PersonHogLeaderService::new(
@@ -936,7 +936,7 @@ async fn update_produces_person_state_to_kafka() {
Arc::new(DirtyIndex::new(1_000_000)),
test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
common::fenced_producers_acquired(&mock_cluster.topic, 4).await,
common::live_authority(),
@@ -1040,7 +1040,7 @@ async fn update_produces_person_state_to_kafka() {
#[tokio::test]
async fn kafka_produce_failure_leaves_cache_unchanged() {
let cache = Arc::new(PartitionedCache::new(1 << 20));
- let (mock_cluster, kafka_producer) = create_test_kafka().await;
+ let mock_cluster = create_test_kafka().await;
let fenced =
common::fenced_producers_acquired(&mock_cluster.topic, NUM_PARTITIONS as i32).await;
@@ -1053,7 +1053,7 @@ async fn kafka_produce_failure_leaves_cache_unchanged() {
Arc::new(DirtyIndex::new(1_000_000)),
test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
Arc::clone(&fenced),
common::live_authority(),
@@ -1179,7 +1179,6 @@ async fn e2e_update_produces_to_local_kafka() {
let owned = common::owned_broker_topic(NUM_PARTITIONS as i32).await;
let topic = owned.topic.clone();
let cache = Arc::new(PartitionedCache::new(1 << 20));
- let kafka_producer = create_local_kafka_producer().await;
let service = PersonHogLeaderService::new(
Arc::clone(&cache),
@@ -1190,7 +1189,7 @@ async fn e2e_update_produces_to_local_kafka() {
Arc::new(DirtyIndex::new(1_000_000)),
test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
common::fenced_producers_acquired(&topic, NUM_PARTITIONS as i32).await,
common::live_authority(),
@@ -1510,7 +1509,7 @@ async fn update_triggers_pg_fallback_then_applies_changes() {
async fn evicted_dirty_person_recovers_from_changelog() {
const PERSON_ID: i64 = 2;
let routing_partition: u32 = partition_for_person(1, PERSON_ID, NUM_PARTITIONS);
- let (mock_cluster, kafka_producer) = create_test_kafka_with_partitions(4).await;
+ let mock_cluster = create_test_kafka_with_partitions(4).await;
let cache = Arc::new(PartitionedCache::new(1 << 20));
let dirty_index = Arc::new(DirtyIndex::new(1_000_000));
@@ -1527,7 +1526,7 @@ async fn evicted_dirty_person_recovers_from_changelog() {
Arc::clone(&dirty_index),
Arc::clone(&recovery),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
common::fenced_producers_acquired(&mock_cluster.topic, 4).await,
common::live_authority(),
@@ -1616,7 +1615,7 @@ async fn evicted_dirty_person_recovers_from_changelog() {
async fn dirty_person_with_failed_recovery_is_unavailable_not_stale() {
const PERSON_ID: i64 = 2;
let routing_partition: u32 = partition_for_person(1, PERSON_ID, NUM_PARTITIONS);
- let (mock_cluster, kafka_producer) = create_test_kafka_with_partitions(4).await;
+ let mock_cluster = create_test_kafka_with_partitions(4).await;
let cache = Arc::new(PartitionedCache::new(1 << 20));
let dirty_index = Arc::new(DirtyIndex::new(1_000_000));
@@ -1630,7 +1629,7 @@ async fn dirty_person_with_failed_recovery_is_unavailable_not_stale() {
Arc::clone(&dirty_index),
Arc::clone(&recovery),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
common::fenced_producers_acquired(&mock_cluster.topic, 4).await,
common::live_authority(),
@@ -1728,7 +1727,7 @@ async fn writes_shed_when_dirty_index_is_full() {
.find(|pid| partition_for_person(1, *pid, NUM_PARTITIONS) == partition)
.expect("a second person maps to the same partition");
- let (mock_cluster, kafka_producer) = create_test_kafka_with_partitions(4).await;
+ let mock_cluster = create_test_kafka_with_partitions(4).await;
let cache = Arc::new(PartitionedCache::new(1 << 20));
// Capacity 1: the first person's mark fills the index.
let dirty_index = Arc::new(DirtyIndex::new(1));
@@ -1741,7 +1740,7 @@ async fn writes_shed_when_dirty_index_is_full() {
Arc::clone(&dirty_index),
test_recovery(&mock_cluster.bootstrap_servers()),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
common::fenced_producers_acquired(&mock_cluster.topic, 4).await,
common::live_authority(),
@@ -1827,7 +1826,7 @@ async fn writes_shed_when_dirty_index_is_full() {
async fn recovery_fails_when_record_version_disagrees_with_the_mark() {
const PERSON_ID: i64 = 2;
let routing_partition: u32 = partition_for_person(1, PERSON_ID, NUM_PARTITIONS);
- let (mock_cluster, kafka_producer) = create_test_kafka_with_partitions(4).await;
+ let mock_cluster = create_test_kafka_with_partitions(4).await;
let cache = Arc::new(PartitionedCache::new(1 << 20));
let dirty_index = Arc::new(DirtyIndex::new(1_000_000));
@@ -1841,7 +1840,7 @@ async fn recovery_fails_when_record_version_disagrees_with_the_mark() {
Arc::clone(&dirty_index),
Arc::clone(&recovery),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
common::fenced_producers_acquired(&mock_cluster.topic, 4).await,
common::live_authority(),
@@ -1938,7 +1937,7 @@ async fn recovery_reuses_the_partition_consumer_across_fetches() {
.find(|pid| partition_for_person(1, *pid, NUM_PARTITIONS) == partition)
.expect("a second person maps to the same partition");
- let (mock_cluster, kafka_producer) = create_test_kafka_with_partitions(4).await;
+ let mock_cluster = create_test_kafka_with_partitions(4).await;
let cache = Arc::new(PartitionedCache::new(1 << 20));
let dirty_index = Arc::new(DirtyIndex::new(1_000_000));
let recovery = test_recovery(&mock_cluster.bootstrap_servers());
@@ -1951,7 +1950,7 @@ async fn recovery_reuses_the_partition_consumer_across_fetches() {
Arc::clone(&dirty_index),
Arc::clone(&recovery),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
common::fenced_producers_acquired(&mock_cluster.topic, 4).await,
common::live_authority(),
@@ -2063,7 +2062,8 @@ async fn cancelled_recovery_returns_its_consumer_to_the_pool() {
use personhog_leader::recovery::{ChangelogRecovery, RecoveryConfig};
use rdkafka::producer::FutureRecord;
- let (mock_cluster, producer) = create_test_kafka().await;
+ let mock_cluster = create_test_kafka().await;
+ let producer = create_local_kafka_producer().await;
let mut kafka = test_kafka_config();
kafka.kafka_hosts = mock_cluster.bootstrap_servers();
// pool_size 1: a single leaked consumer exhausts the pool.
@@ -2151,7 +2151,7 @@ async fn cancelled_recovery_returns_its_consumer_to_the_pool() {
async fn oversize_updates_are_rejected_and_oversized_rows_remediated() {
const PERSON_ID: i64 = 9;
let routing_partition: u32 = partition_for_person(1, PERSON_ID, NUM_PARTITIONS);
- let (mock_cluster, kafka_producer) = create_test_kafka_with_partitions(4).await;
+ let mock_cluster = create_test_kafka_with_partitions(4).await;
// This test reads its warnings back, so they must be its own: on a
// shared topic it would collect whichever two arrived first.
let warnings_topic = common::owned_broker_topic(1).await;
@@ -2171,11 +2171,11 @@ async fn oversize_updates_are_rejected_and_oversized_rows_remediated() {
// Burst 2 so the trim and reject below — same team, same type —
// both clear the throttle; the third enforcement action then
// exercises suppression.
- WarningsProducer::with_throttle(
- kafka_producer,
- warnings_topic.topic.clone(),
+ common::local_warnings_producer(
+ &warnings_topic.topic,
WarningThrottle::new(DEFAULT_THROTTLE_PERIOD, NonZeroU32::new(2).unwrap()),
- ),
+ )
+ .await,
Arc::new(DashMap::new()),
common::fenced_producers_acquired(&mock_cluster.topic, 4).await,
common::live_authority(),
@@ -2619,7 +2619,7 @@ async fn oversize_updates_are_rejected_and_oversized_rows_remediated() {
#[tokio::test]
async fn an_unresolved_version_is_never_reused() {
let cache = Arc::new(PartitionedCache::new(1 << 20));
- let (_mock_cluster, kafka_producer) = create_test_kafka().await;
+ let _mock_cluster = create_test_kafka().await;
// One set for both: a second would fence the first at the broker.
let fenced =
common::fenced_producers_acquired(&_mock_cluster.topic, NUM_PARTITIONS as i32).await;
@@ -2633,7 +2633,7 @@ async fn an_unresolved_version_is_never_reused() {
Arc::new(DirtyIndex::new(1_000_000)),
test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
Arc::clone(&fenced),
common::live_authority(),
@@ -2702,7 +2702,7 @@ async fn an_unresolved_version_is_never_reused() {
async fn a_fenced_write_that_bounces_does_not_hand_its_version_back() {
let topic = format!("fenced_svc_{}", uuid::Uuid::new_v4().simple());
let cache = Arc::new(PartitionedCache::new(1 << 20));
- let (_mock_cluster, kafka_producer) = create_test_kafka().await;
+ let _mock_cluster = create_test_kafka().await;
let fenced = Arc::new(common::test_fenced_producers());
let service = PersonHogLeaderService::new(
@@ -2714,7 +2714,7 @@ async fn a_fenced_write_that_bounces_does_not_hand_its_version_back() {
Arc::new(DirtyIndex::new(1_000_000)),
test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
Arc::clone(&fenced),
common::live_authority(),
@@ -2826,7 +2826,7 @@ async fn an_unknown_outcome_keeps_its_version(
) {
let topic = format!("fenced_uncertain_{}", uuid::Uuid::new_v4().simple());
let cache = Arc::new(PartitionedCache::new(1 << 20));
- let (_mock_cluster, kafka_producer) = create_test_kafka().await;
+ let _mock_cluster = create_test_kafka().await;
let fenced = Arc::new(common::test_fenced_producers());
let emitted_versions = Arc::new(personhog_leader::emitted::EmittedVersions::new(1_000_000));
@@ -2839,7 +2839,7 @@ async fn an_unknown_outcome_keeps_its_version(
Arc::new(DirtyIndex::new(1_000_000)),
test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
Arc::clone(&fenced),
common::live_authority(),
@@ -2918,7 +2918,7 @@ async fn scalar_fields_merge_rather_than_assign() {
const HOUR_MS: i64 = 3_600_000;
let topic = format!("scalar_merge_{}", uuid::Uuid::new_v4().simple());
let cache = Arc::new(PartitionedCache::new(1 << 20));
- let (_mock_cluster, kafka_producer) = create_test_kafka().await;
+ let _mock_cluster = create_test_kafka().await;
let emitted_versions = Arc::new(personhog_leader::emitted::EmittedVersions::new(1_000_000));
let service = PersonHogLeaderService::new(
@@ -2930,7 +2930,7 @@ async fn scalar_fields_merge_rather_than_assign() {
Arc::new(DirtyIndex::new(1_000_000)),
test_recovery(KAFKA_BOOTSTRAP),
PropertySizeLimits::new(655360, 524288),
- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),
+ WarningsProducer::disabled(),
Arc::new(DashMap::new()),
common::fenced_producers_acquired(&topic, NUM_PARTITIONS as i32).await,
common::live_authority(),
diff --git a/rust/personhog-leader/tests/warming_integration.rs b/rust/personhog-leader/tests/warming_integration.rs
index 28f19fd15174..5dddb1e264e8 100644
--- a/rust/personhog-leader/tests/warming_integration.rs
+++ b/rust/personhog-leader/tests/warming_integration.rs
@@ -1064,7 +1064,8 @@ async fn a_restart_warms_the_old_generations_congruent_tail_for_its_keys() {
use personhog_coordination::changelog::KeyFilter;
use personhog_leader::cache::Stream;
- let (new_topic, producer) = common::create_test_kafka_with_partitions(4).await;
+ let new_topic = common::create_test_kafka_with_partitions(4).await;
+ let producer = common::create_local_kafka_producer().await;
let old_topic = common::owned_broker_topic(2).await;
let bootstrap = new_topic.bootstrap_servers();
@@ -1185,7 +1186,8 @@ async fn an_applied_old_record_never_shadows_a_newer_applied_live_one() {
use personhog_common::partitioning::partition_for_person;
use personhog_coordination::changelog::KeyFilter;
- let (new_topic, producer) = common::create_test_kafka_with_partitions(2).await;
+ let new_topic = common::create_test_kafka_with_partitions(2).await;
+ let producer = common::create_local_kafka_producer().await;
let old_topic = common::owned_broker_topic(2).await;
let bootstrap = new_topic.bootstrap_servers();