:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(personhog): send leader ingestion warnings through a dedicated producer (#122878)
Zach Brown0c0ff9f36061
diff --git a/.agents/skills/adding-ingestion-warnings/SKILL.md b/.agents/skills/adding-ingestion-warnings/SKILL.mdindex 39051cb5629e..df2f6203558d 100644--- a/.agents/skills/adding-ingestion-warnings/SKILL.md+++ b/.agents/skills/adding-ingestion-warnings/SKILL.md@@ -93,7 +93,7 @@ Warning::new(WarningType::MyNewType)`into_row` takes a `WarningSource` (declare one const per emit site, as capture does in `src/lib.rs`) and injects `teamId` and the registry's `category`/`severity` over the caller's details (they cannot be spoofed or forgotten), and stamps the ClickHouse-format timestamp. A direct-row type needs **no** `captureProduced` flag and no `from_tag` arm — register it, regenerate the artifact, and emit. See the module doc in `rust/common/ingestion_warnings/src/serializer.rs` for the envelope-vs-row correspondence table and the trust rationale (the envelope lane is attacker-writable, so the consumer stamps classification; the row topic is ACL-guarded, so the producer does).-The crate's `RowWarningEmitter` (`src/row.rs`) is the producer for this transport: `KafkaRowWarningEmitter` wraps a dedicated fire-and-forget `ThreadedProducer`, throttles per `(team_id, scope, type)` where `scope` is the caller's debounce entity (a distinct id, say), keys rows by team, and reports through the same `ingestion_warnings_total` outcomes as the envelope emitter. `test_support::CollectingRowEmitter` records rows for assertions. The emitter's throttle keeps the crate's default tracked-key cap unless the caller raises it, and a per-entity scope needs a cap sized for that cardinality (see `MAX_TRACKED_SCOPES` in the identity service). personhog-identity is the reference caller (`rust/personhog-identity/src/warnings.rs`: a `WarningSource` per emit site, a pure verdict-to-warning mapping, and `install` building the producer from env with the emitter-enabled gauge); the leader still carries its own older `WarningsProducer`.+The crate's `RowWarningEmitter` (`src/row.rs`) is the producer for this transport: `KafkaRowWarningEmitter` wraps a dedicated fire-and-forget `ThreadedProducer`, throttles per `(team_id, scope, type)` where `scope` is the caller's debounce entity (a distinct id, say), keys rows by team, and reports through the same `ingestion_warnings_total` outcomes as the envelope emitter. `test_support::CollectingRowEmitter` records rows for assertions. The emitter's throttle keeps the crate's default tracked-key cap unless the caller raises it, and a per-entity scope needs a cap sized for that cardinality. `install_row_emitter` (`src/row_install.rs`) builds the emitter from `RowWarningsSettings`: its own producer, a per-type budget per minute like capture's, a tracked-scope cap (`DEFAULT_MAX_TRACKED_SCOPES`), and the emitter-enabled gauge. Both personhog services call it from env-backed `INGESTION_WARNINGS_*` settings. personhog-identity is the reference caller (`rust/personhog-identity/src/warnings.rs`: a `WarningSource` per emit site and a pure verdict-to-warning mapping); personhog-leader (`rust/personhog-leader/src/warnings.rs`) emits admission size violations scoped per team.Emitting from Rust:diff --git a/rust/Cargo.lock b/rust/Cargo.lockindex 22b923751bc7..5bc2258f4946 100644--- a/rust/Cargo.lock+++ b/rust/Cargo.lock@@ -3048,11 +3048,13 @@ dependencies = ["common-liveness","common-types","governor",+ "lifecycle","metrics","metrics-util","rdkafka","rstest","serde_json",+ "tokio","tracing","uuid",]@@ -9106,7 +9108,6 @@ dependencies = ["lifecycle","metrics","metrics-exporter-prometheus",- "metrics-util","personhog-common","personhog-coordination","personhog-proto",diff --git a/rust/common/ingestion_warnings/Cargo.toml b/rust/common/ingestion_warnings/Cargo.tomlindex 349ef35d7d83..b4aea76aff0e 100644--- a/rust/common/ingestion_warnings/Cargo.toml+++ b/rust/common/ingestion_warnings/Cargo.toml@@ -16,9 +16,11 @@ chrono = { workspace = true }common-kafka = { path = "../kafka" }common-types = { path = "../types" }governor = { workspace = true }+lifecycle = { path = "../lifecycle" }metrics = { workspace = true }rdkafka = { workspace = true }serde_json = { workspace = true }+tokio = { workspace = true }tracing = { workspace = true }uuid = { workspace = true }diff --git a/rust/common/ingestion_warnings/src/lib.rs b/rust/common/ingestion_warnings/src/lib.rsindex 82247a4e4687..fcc2358aface 100644--- a/rust/common/ingestion_warnings/src/lib.rs+++ b/rust/common/ingestion_warnings/src/lib.rs@@ -39,6 +39,7 @@pub mod registry;pub mod request;pub mod row;+pub mod row_install;pub mod serializer;pub mod test_support;pub mod throttle;@@ -58,6 +59,7 @@ use tracing::warn;pub use registry::WarningType;pub use request::{emit_request_warning, WarningRequestContext, UNKNOWN_ATTRIBUTION};pub use row::{KafkaRowWarningEmitter, RowWarningEmitter};+pub use row_install::{install_row_emitter, RowWarningsSettings};pub use throttle::{ThrottleDecision, WarningThrottle};/// Counter of emission attempts: labels `type` (warning type), `source`diff --git a/rust/common/ingestion_warnings/src/row_install.rs b/rust/common/ingestion_warnings/src/row_install.rsnew file mode 100644index 000000000000..bc1f43a44b12--- /dev/null+++ b/rust/common/ingestion_warnings/src/row_install.rs@@ -0,0 +1,339 @@+use std::num::NonZeroU32;+use std::sync::Arc;+use std::time::Duration;++use common_kafka::config::KafkaConfig;+use common_kafka::kafka_producer::create_threaded_kafka_producer_no_ping;+use metrics::gauge;++use crate::row::{KafkaRowWarningEmitter, RowWarningEmitter};+use crate::throttle::{WarningThrottle, DEFAULT_MAX_TRACKED_KEYS};+use crate::{observe_delivery, INGESTION_WARNINGS_EMITTER_ENABLED};++pub const DEFAULT_TYPE_BUDGET_PER_MINUTE: NonZeroU32 = NonZeroU32::new(60).unwrap();+/// Default for `INGESTION_WARNINGS_MAX_TRACKED_SCOPES`. Row scopes can be per+/// distinct id, so the throttle default (sized for capture's per-token keys)+/// would fill within the hour a key lives and silence every team on the pod.+/// The type budgets admit well under this many keys an hour.+pub const DEFAULT_MAX_TRACKED_SCOPES: usize = 1_000_000;+const _: () = assert!(DEFAULT_MAX_TRACKED_SCOPES > DEFAULT_MAX_TRACKED_KEYS);+const THROTTLE_SWEEP_INTERVAL: Duration = Duration::from_secs(60);++#[derive(Debug, Clone, PartialEq, Eq)]+pub struct RowWarningsSettings {+ pub enabled: bool,+ pub kafka_hosts: String,+ pub kafka_tls: bool,+ pub topic: String,+ pub kafka_queue_mib: u32,+ pub kafka_message_max_bytes: u32,+ pub kafka_acks: String,+ pub kafka_retries: u32,+ pub kafka_linger_ms: u32,+ pub kafka_queue_messages: u32,+ pub kafka_message_timeout_ms: u32,+ pub type_budget_per_minute: NonZeroU32,+ pub max_tracked_scopes: usize,+}++impl RowWarningsSettings {+ pub fn kafka_config(&self, client_id: &str) -> KafkaConfig {+ KafkaConfig {+ kafka_hosts: self.kafka_hosts.clone(),+ kafka_tls: self.kafka_tls,+ kafka_client_id: client_id.to_string(),+ kafka_producer_acks: Some(self.kafka_acks.clone()),+ kafka_producer_retries: Some(self.kafka_retries),+ kafka_producer_linger_ms: self.kafka_linger_ms,+ kafka_producer_queue_messages: self.kafka_queue_messages,+ kafka_message_timeout_ms: self.kafka_message_timeout_ms,+ kafka_producer_queue_mib: self.kafka_queue_mib,+ kafka_producer_message_max_bytes: Some(self.kafka_message_max_bytes),+ ..Default::default()+ }+ }++ pub fn disabled_reason(&self) -> Option<&'static str> {+ if !self.enabled {+ return Some("disabled");+ }+ if self.kafka_hosts.is_empty() {+ return Some("hosts_unset");+ }+ if self.topic.is_empty() {+ return Some("topic_unset");+ }+ None+ }++ fn throttle(&self) -> WarningThrottle {+ WarningThrottle::default()+ .with_max_tracked_keys(self.max_tracked_scopes)+ .with_type_budget(+ Duration::from_secs(60) / self.type_budget_per_minute.get(),+ self.type_budget_per_minute,+ )+ }+}++fn report_emitter_enabled(reason: &'static str, enabled: bool) {+ gauge!(INGESTION_WARNINGS_EMITTER_ENABLED, "reason" => reason).set(if enabled {+ 1.0+ } else {+ 0.0+ });+}++/// Register `handle` as advisory: the producer's stats callback is its only liveness signal.+pub fn install_row_emitter(+ settings: &RowWarningsSettings,+ client_id: &str,+ handle: Option<lifecycle::Handle>,+) -> Option<Arc<dyn RowWarningEmitter>> {+ if !settings.enabled {+ return None;+ }+ if let Some(reason) = settings.disabled_reason() {+ tracing::warn!(+ reason,+ "ingestion warnings enabled but unusable; emitter disabled"+ );+ report_emitter_enabled(reason, false);+ return None;+ }+ let Some(handle) = handle else {+ tracing::warn!(+ "ingestion warnings enabled but no lifecycle handle was registered; emitter disabled"+ );+ report_emitter_enabled("no_handle", false);+ return None;+ };+ let producer = match create_threaded_kafka_producer_no_ping(+ &settings.kafka_config(client_id),+ handle.clone(),+ observe_delivery,+ ) {+ Ok(producer) => producer,+ Err(e) => {+ tracing::error!(error = %e, "failed to create ingestion warnings producer; emitter disabled");+ report_emitter_enabled("producer_create_failed", false);+ return None;+ }+ };+ if settings.max_tracked_scopes <= DEFAULT_MAX_TRACKED_KEYS {+ tracing::warn!(+ max_tracked_scopes = settings.max_tracked_scopes,+ "ingestion warnings scope cap is at or below the per-token default; a busy pod can fill it within the hour and drop every team's warnings"+ );+ }+ let emitter = Arc::new(KafkaRowWarningEmitter::with_throttle(+ producer,+ settings.topic.clone(),+ settings.throttle(),+ ));++ // Liveness comes from the producer's stats callback alone, so a wedged+ // poll thread trips the deadline; a heartbeat here would mask it.+ let background = emitter.clone();+ tokio::spawn(async move {+ let mut sweep = tokio::time::interval(THROTTLE_SWEEP_INTERVAL);+ loop {+ tokio::select! {+ _ = sweep.tick() => background.sweep_throttle(),+ _ = handle.shutdown_recv() => break,+ }+ }+ });++ report_emitter_enabled("ok", true);+ tracing::info!(+ topic = settings.topic.as_str(),+ "ingestion warnings emitter enabled"+ );+ Some(emitter)+}++#[cfg(test)]+mod tests {+ use std::time::Instant;++ use metrics_util::debugging::{DebugValue, DebuggingRecorder};+ use rstest::rstest;++ use super::*;+ use crate::row::throttle_key;+ use crate::throttle::ThrottleDecision;+ use crate::WarningType;++ const CLIENT_ID: &str = "row-install-test";++ fn settings(enabled: bool, hosts: &str, topic: &str) -> RowWarningsSettings {+ RowWarningsSettings {+ enabled,+ kafka_hosts: hosts.to_string(),+ kafka_tls: false,+ topic: topic.to_string(),+ kafka_queue_mib: 16,+ kafka_message_max_bytes: 1_048_576,+ kafka_acks: "1".to_string(),+ kafka_retries: 0,+ kafka_linger_ms: 100,+ kafka_queue_messages: 10_000,+ kafka_message_timeout_ms: 5_000,+ type_budget_per_minute: DEFAULT_TYPE_BUDGET_PER_MINUTE,+ max_tracked_scopes: DEFAULT_MAX_TRACKED_SCOPES,+ }+ }++ #[rstest]+ #[case::off(false, "broker:9092", "topic", Some("disabled"))]+ #[case::no_hosts(true, "", "topic", Some("hosts_unset"))]+ #[case::no_topic(true, "broker:9092", "", Some("topic_unset"))]+ #[case::usable(true, "broker:9092", "topic", None)]+ fn unusable_settings_name_their_reason(+ #[case] enabled: bool,+ #[case] hosts: &str,+ #[case] topic: &str,+ #[case] expected: Option<&'static str>,+ ) {+ assert_eq!(settings(enabled, hosts, topic).disabled_reason(), expected);+ }++ #[test]+ fn the_producer_config_carries_every_setting_and_stays_isolated() {+ let config = RowWarningsSettings {+ kafka_acks: "all".to_string(),+ kafka_retries: 2,+ kafka_linger_ms: 250,+ kafka_queue_messages: 500,+ kafka_message_timeout_ms: 9_000,+ ..settings(true, "broker:9092", "topic")+ }+ .kafka_config(CLIENT_ID);+ assert_eq!(config.kafka_hosts, "broker:9092");+ assert_eq!(config.kafka_client_id, CLIENT_ID);+ assert_eq!(config.kafka_producer_acks.as_deref(), Some("all"));+ assert_eq!(config.kafka_producer_retries, Some(2));+ assert_eq!(config.kafka_producer_linger_ms, 250);+ assert_eq!(config.kafka_producer_queue_messages, 500);+ assert_eq!(config.kafka_message_timeout_ms, 9_000);+ assert_eq!(config.kafka_producer_queue_mib, 16);+ assert_eq!(config.kafka_producer_message_max_bytes, Some(1_048_576));+ assert!(config.kafka_producer_enable_idempotence.is_none());+ assert!(config.kafka_producer_batch_size.is_none());+ }++ #[rstest]+ #[case::hosts_unset("", "hosts_unset")]+ #[case::no_handle("broker:9092", "no_handle")]+ fn an_unusable_install_reports_its_reason_on_the_gauge(+ #[case] hosts: &str,+ #[case] reason: &str,+ ) {+ let recorder = DebuggingRecorder::new();+ let snapshotter = recorder.snapshotter();+ let emitter = metrics::with_local_recorder(&recorder, || {+ install_row_emitter(&settings(true, hosts, "topic"), CLIENT_ID, None)+ });+ assert!(emitter.is_none());+ let gauges: Vec<(String, f64)> = snapshotter+ .snapshot()+ .into_vec()+ .into_iter()+ .filter(|(key, _, _, _)| key.key().name() == INGESTION_WARNINGS_EMITTER_ENABLED)+ .map(|(key, _, _, value)| {+ let reason = key+ .key()+ .labels()+ .find(|label| label.key() == "reason")+ .expect("reason label")+ .value()+ .to_string();+ let DebugValue::Gauge(value) = value else {+ panic!("emitter enabled is a gauge");+ };+ (reason, value.into_inner())+ })+ .collect();+ assert_eq!(gauges, vec![(reason.to_string(), 0.0)]);+ }++ #[test]+ fn the_throttle_tracks_only_admitted_scopes_and_caps_above_the_shared_default() {+ let throttle = settings(true, "broker:9092", "topic").throttle();+ let warning = WarningType::CannotMergeAlreadyIdentified;+ let mut admitted = 0;+ for n in 0..=DEFAULT_MAX_TRACKED_KEYS {+ match throttle.check(&throttle_key(7, &n.to_string()), warning) {+ ThrottleDecision::Emit => admitted += 1,+ ThrottleDecision::TypeBudgetExhausted => {}+ other => panic!("scope {n}: {other:?}"),+ }+ }+ assert!(admitted >= DEFAULT_TYPE_BUDGET_PER_MINUTE.get() as usize);+ assert_eq!(+ throttle.tracked_keys(),+ admitted,+ "a scope the budget refused must not hold a key"+ );+ }++ #[test]+ fn the_throttle_takes_its_budget_and_cap_from_settings() {+ let throttle = RowWarningsSettings {+ type_budget_per_minute: NonZeroU32::new(2).unwrap(),+ max_tracked_scopes: 3,+ ..settings(true, "broker:9092", "topic")+ }+ .throttle();+ let warning = WarningType::CannotMergeAlreadyIdentified;+ assert_eq!(+ throttle.check(&throttle_key(7, "a"), warning),+ ThrottleDecision::Emit+ );+ assert_eq!(+ throttle.check(&throttle_key(7, "b"), warning),+ ThrottleDecision::Emit+ );+ assert_eq!(+ throttle.check(&throttle_key(7, "c"), warning),+ ThrottleDecision::TypeBudgetExhausted+ );+ let later = Instant::now() + Duration::from_secs(31);+ assert_eq!(+ throttle.check_at(&throttle_key(7, "c"), warning, later),+ ThrottleDecision::Emit+ );+ assert_eq!(+ throttle.check(&throttle_key(7, "d"), warning),+ ThrottleDecision::CardinalityCapped+ );+ }++ #[test]+ fn the_type_budget_refills_at_the_stated_rate() {+ let throttle = settings(true, "broker:9092", "topic").throttle();+ let warning = WarningType::MergeRaceCondition;+ for n in 0..DEFAULT_TYPE_BUDGET_PER_MINUTE.get() {+ assert_eq!(+ throttle.check(&throttle_key(7, &n.to_string()), warning),+ ThrottleDecision::Emit,+ "scope {n}"+ );+ }+ assert_eq!(+ throttle.check(&throttle_key(7, "over"), warning),+ ThrottleDecision::TypeBudgetExhausted+ );+ let soon = Instant::now() + Duration::from_millis(500);+ assert_eq!(+ throttle.check_at(&throttle_key(7, "soon"), warning, soon),+ ThrottleDecision::TypeBudgetExhausted+ );+ let later = Instant::now() + Duration::from_millis(1100);+ assert_eq!(+ throttle.check_at(&throttle_key(7, "later"), warning, later),+ ThrottleDecision::Emit+ );+ }+}diff --git a/rust/personhog-identity/Cargo.toml b/rust/personhog-identity/Cargo.tomlindex 97e456fb9790..a58245044bd6 100644--- a/rust/personhog-identity/Cargo.toml+++ b/rust/personhog-identity/Cargo.toml@@ -38,7 +38,6 @@ tracing-subscriber = { workspace = true }uuid = { workspace = true }[dev-dependencies]-metrics-util = { workspace = true }rstest = "0.26"tokio = { workspace = true, features = ["test-util"] }diff --git a/rust/personhog-identity/src/config.rs b/rust/personhog-identity/src/config.rsindex b453cadeeae1..0098b4499d1e 100644--- a/rust/personhog-identity/src/config.rs+++ b/rust/personhog-identity/src/config.rs@@ -234,10 +234,10 @@ pub struct Config {#[envconfig(default = "5000")]pub ingestion_warnings_kafka_message_timeout_ms: u32,- /// Warnings a second per type that one pod sends, shared by every team+ /// Warnings of one type a minute that one pod sends, shared by every team/// on it; the rest are dropped as `type_budget_exhausted`.#[envconfig(default = "60")]- pub ingestion_warnings_type_budget_per_second: NonZeroU32,+ pub ingestion_warnings_type_budget_per_minute: NonZeroU32,/// Throttle keys (team, distinct id, type) a pod remembers for the hour a/// key lives, at roughly 135 bytes each.@@ -518,8 +518,8 @@ impl Config {Duration::from_secs(self.lifecycle_op_retention_hours * 3600)}- pub fn ingestion_warnings_settings(&self) -> crate::warnings::WarningsSettings {- crate::warnings::WarningsSettings {+ pub fn ingestion_warnings_settings(&self) -> common_ingestion_warnings::RowWarningsSettings {+ common_ingestion_warnings::RowWarningsSettings {enabled: self.ingestion_warnings_enabled,kafka_hosts: self.ingestion_warnings_kafka_hosts.clone(),kafka_tls: self.ingestion_warnings_kafka_tls,@@ -531,7 +531,7 @@ impl Config {kafka_linger_ms: self.ingestion_warnings_kafka_linger_ms,kafka_queue_messages: self.ingestion_warnings_kafka_queue_messages,kafka_message_timeout_ms: self.ingestion_warnings_kafka_message_timeout_ms,- type_budget_per_second: self.ingestion_warnings_type_budget_per_second,+ type_budget_per_minute: self.ingestion_warnings_type_budget_per_minute,max_tracked_scopes: self.ingestion_warnings_max_tracked_scopes,}}diff --git a/rust/personhog-identity/src/warnings.rs b/rust/personhog-identity/src/warnings.rsindex 46ca521d0025..461d87b764a8 100644--- a/rust/personhog-identity/src/warnings.rs+++ b/rust/personhog-identity/src/warnings.rs@@ -4,18 +4,13 @@//! call failures stay with the caller, which alone knows its retry budget//! and merge-mode policy.-use std::num::NonZeroU32;use std::sync::Arc;use std::time::Duration;use common_ingestion_warnings::serializer::Warning;-use common_ingestion_warnings::throttle::{WarningThrottle, DEFAULT_MAX_TRACKED_KEYS};use common_ingestion_warnings::{- observe_delivery, KafkaRowWarningEmitter, RowWarningEmitter, WarningSource, WarningType,- INGESTION_WARNINGS_EMITTER_ENABLED,+ install_row_emitter, RowWarningEmitter, RowWarningsSettings, WarningSource, WarningType,};-use common_kafka::config::KafkaConfig;-use common_kafka::kafka_producer::create_threaded_kafka_producer_no_ping;use uuid::Uuid;use crate::lifecycle::merge::{@@ -39,15 +34,6 @@ pub const MERGE_SAGA: WarningSource = WarningSource {};const KAFKA_CLIENT_ID: &str = "personhog-identity-ingestion-warnings";-/// Default for `INGESTION_WARNINGS_TYPE_BUDGET_PER_SECOND`.-pub const DEFAULT_TYPE_BUDGET_PER_SECOND: NonZeroU32 = NonZeroU32::new(60).unwrap();-/// Default for `INGESTION_WARNINGS_MAX_TRACKED_SCOPES`. Keys are per distinct-/// id, so the crate default (sized for capture's per-token keys) would fill-/// within the hour a key lives and silence every team on the pod. The type-/// budgets admit well under this many keys an hour.-pub const DEFAULT_MAX_TRACKED_SCOPES: usize = 1_000_000;-const _: () = assert!(DEFAULT_MAX_TRACKED_SCOPES > DEFAULT_MAX_TRACKED_KEYS);-const THROTTLE_SWEEP_INTERVAL: Duration = Duration::from_secs(60);/// The distinct id column's own limit: a legal id is never cut, and an/// oversized one is cut before it is copied into details.const MAX_DETAIL_ID_CHARS: usize = 400;@@ -196,143 +182,16 @@ impl MergeWarnings {}}-#[derive(Debug, Clone, PartialEq, Eq)]-pub struct WarningsSettings {- pub enabled: bool,- pub kafka_hosts: String,- pub kafka_tls: bool,- pub topic: String,- pub kafka_queue_mib: u32,- pub kafka_message_max_bytes: u32,- pub kafka_acks: String,- pub kafka_retries: u32,- pub kafka_linger_ms: u32,- pub kafka_queue_messages: u32,- pub kafka_message_timeout_ms: u32,- pub type_budget_per_second: NonZeroU32,- pub max_tracked_scopes: usize,-}--impl WarningsSettings {- pub fn kafka_config(&self) -> KafkaConfig {- KafkaConfig {- kafka_hosts: self.kafka_hosts.clone(),- kafka_tls: self.kafka_tls,- kafka_client_id: KAFKA_CLIENT_ID.to_string(),- kafka_producer_acks: Some(self.kafka_acks.clone()),- kafka_producer_retries: Some(self.kafka_retries),- kafka_producer_linger_ms: self.kafka_linger_ms,- kafka_producer_queue_messages: self.kafka_queue_messages,- kafka_message_timeout_ms: self.kafka_message_timeout_ms,- kafka_producer_queue_mib: self.kafka_queue_mib,- kafka_producer_message_max_bytes: Some(self.kafka_message_max_bytes),- ..Default::default()- }- }-- pub fn disabled_reason(&self) -> Option<&'static str> {- if !self.enabled {- return Some("disabled");- }- if self.kafka_hosts.is_empty() {- return Some("hosts_unset");- }- if self.topic.is_empty() {- return Some("topic_unset");- }- None+pub fn install(settings: &RowWarningsSettings, handle: Option<lifecycle::Handle>) -> MergeWarnings {+ match install_row_emitter(settings, KAFKA_CLIENT_ID, handle) {+ Some(emitter) => MergeWarnings::new(emitter),+ None => MergeWarnings::disabled(),}}-fn report_emitter_enabled(reason: &'static str, enabled: bool) {- common_metrics::gauge(- INGESTION_WARNINGS_EMITTER_ENABLED,- &[("reason".to_string(), reason.to_string())],- if enabled { 1.0 } else { 0.0 },- );-}--fn throttle(settings: &WarningsSettings) -> WarningThrottle {- WarningThrottle::default()- .with_max_tracked_keys(settings.max_tracked_scopes)- .with_type_budget(- Duration::from_secs(1) / settings.type_budget_per_second.get(),- settings.type_budget_per_second,- )-}--pub fn install(settings: &WarningsSettings, handle: Option<lifecycle::Handle>) -> MergeWarnings {- if !settings.enabled {- return MergeWarnings::disabled();- }- if let Some(reason) = settings.disabled_reason() {- tracing::warn!(- reason,- "ingestion warnings enabled but unusable; emitter disabled"- );- report_emitter_enabled(reason, false);- return MergeWarnings::disabled();- }- let Some(handle) = handle else {- tracing::warn!(- "ingestion warnings enabled but no lifecycle handle was registered; emitter disabled"- );- report_emitter_enabled("no_handle", false);- return MergeWarnings::disabled();- };- let producer = match create_threaded_kafka_producer_no_ping(- &settings.kafka_config(),- handle.clone(),- observe_delivery,- ) {- Ok(producer) => producer,- Err(e) => {- tracing::error!(error = %e, "failed to create ingestion warnings producer; emitter disabled");- report_emitter_enabled("producer_create_failed", false);- return MergeWarnings::disabled();- }- };- if settings.max_tracked_scopes <= DEFAULT_MAX_TRACKED_KEYS {- tracing::warn!(- max_tracked_scopes = settings.max_tracked_scopes,- "ingestion warnings scope cap is at or below the per-token default; a busy pod can fill it within the hour and drop every team's warnings"- );- }- let emitter = Arc::new(KafkaRowWarningEmitter::with_throttle(- producer,- settings.topic.clone(),- throttle(settings),- ));-- // Liveness comes from the producer's stats callback alone, so a wedged- // poll thread trips the deadline; a heartbeat here would mask it.- let background = emitter.clone();- tokio::spawn(async move {- let mut sweep = tokio::time::interval(THROTTLE_SWEEP_INTERVAL);- loop {- tokio::select! {- _ = sweep.tick() => background.sweep_throttle(),- _ = handle.shutdown_recv() => break,- }- }- });-- report_emitter_enabled("ok", true);- tracing::info!(- topic = settings.topic.as_str(),- "ingestion warnings emitter enabled"- );- MergeWarnings::new(emitter)-}-#[cfg(test)]mod tests {- use std::time::Instant;-- use common_ingestion_warnings::row::throttle_key;use common_ingestion_warnings::test_support::CollectingRowEmitter;- use common_ingestion_warnings::throttle::ThrottleDecision;- use metrics_util::debugging::{DebugValue, DebuggingRecorder};use rstest::rstest;use serde_json::Value;@@ -522,169 +381,4 @@ mod tests {"an illegal target is about the target");}-- fn settings(enabled: bool, hosts: &str, topic: &str) -> WarningsSettings {- WarningsSettings {- enabled,- kafka_hosts: hosts.to_string(),- kafka_tls: false,- topic: topic.to_string(),- kafka_queue_mib: 16,- kafka_message_max_bytes: 1_048_576,- kafka_acks: "1".to_string(),- kafka_retries: 0,- kafka_linger_ms: 100,- kafka_queue_messages: 10_000,- kafka_message_timeout_ms: 5_000,- type_budget_per_second: DEFAULT_TYPE_BUDGET_PER_SECOND,- max_tracked_scopes: DEFAULT_MAX_TRACKED_SCOPES,- }- }-- #[rstest]- #[case::off(false, "broker:9092", "topic", Some("disabled"))]- #[case::no_hosts(true, "", "topic", Some("hosts_unset"))]- #[case::no_topic(true, "broker:9092", "", Some("topic_unset"))]- #[case::usable(true, "broker:9092", "topic", None)]- fn unusable_settings_name_their_reason(- #[case] enabled: bool,- #[case] hosts: &str,- #[case] topic: &str,- #[case] expected: Option<&'static str>,- ) {- assert_eq!(settings(enabled, hosts, topic).disabled_reason(), expected);- }-- #[test]- fn the_producer_config_carries_every_setting_and_stays_isolated() {- let config = WarningsSettings {- kafka_acks: "all".to_string(),- kafka_retries: 2,- kafka_linger_ms: 250,- kafka_queue_messages: 500,- kafka_message_timeout_ms: 9_000,- ..settings(true, "broker:9092", "topic")- }- .kafka_config();- assert_eq!(config.kafka_hosts, "broker:9092");- assert_eq!(config.kafka_client_id, KAFKA_CLIENT_ID);- assert_eq!(config.kafka_producer_acks.as_deref(), Some("all"));- assert_eq!(config.kafka_producer_retries, Some(2));- assert_eq!(config.kafka_producer_linger_ms, 250);- assert_eq!(config.kafka_producer_queue_messages, 500);- assert_eq!(config.kafka_message_timeout_ms, 9_000);- assert_eq!(config.kafka_producer_queue_mib, 16);- assert_eq!(config.kafka_producer_message_max_bytes, Some(1_048_576));- assert!(config.kafka_producer_enable_idempotence.is_none());- assert!(config.kafka_producer_batch_size.is_none());- }-- #[rstest]- #[case::hosts_unset("", "hosts_unset")]- #[case::no_handle("broker:9092", "no_handle")]- fn an_unusable_install_reports_its_reason_on_the_gauge(- #[case] hosts: &str,- #[case] reason: &str,- ) {- let recorder = DebuggingRecorder::new();- let snapshotter = recorder.snapshotter();- let warnings = metrics::with_local_recorder(&recorder, || {- install(&settings(true, hosts, "topic"), None)- });- assert!(!warnings.is_enabled());- let gauges: Vec<(String, f64)> = snapshotter- .snapshot()- .into_vec()- .into_iter()- .filter(|(key, _, _, _)| key.key().name() == INGESTION_WARNINGS_EMITTER_ENABLED)- .map(|(key, _, _, value)| {- let reason = key- .key()- .labels()- .find(|label| label.key() == "reason")- .expect("reason label")- .value()- .to_string();- let DebugValue::Gauge(value) = value else {- panic!("emitter enabled is a gauge");- };- (reason, value.into_inner())- })- .collect();- assert_eq!(gauges, vec![(reason.to_string(), 0.0)]);- }-- #[test]- fn the_throttle_tracks_only_admitted_scopes_and_caps_above_the_shared_default() {- let throttle = throttle(&settings(true, "broker:9092", "topic"));- let warning = WarningType::CannotMergeAlreadyIdentified;- let mut admitted = 0;- for n in 0..=DEFAULT_MAX_TRACKED_KEYS {- match throttle.check(&throttle_key(7, &n.to_string()), warning) {- ThrottleDecision::Emit => admitted += 1,- ThrottleDecision::TypeBudgetExhausted => {}- other => panic!("scope {n}: {other:?}"),- }- }- assert!(admitted >= DEFAULT_TYPE_BUDGET_PER_SECOND.get() as usize);- assert_eq!(- throttle.tracked_keys(),- admitted,- "a scope the budget refused must not hold a key"- );- }-- #[test]- fn the_throttle_takes_its_budget_and_cap_from_settings() {- let throttle = throttle(&WarningsSettings {- type_budget_per_second: NonZeroU32::new(2).unwrap(),- max_tracked_scopes: 3,- ..settings(true, "broker:9092", "topic")- });- let warning = WarningType::CannotMergeAlreadyIdentified;- assert_eq!(- throttle.check(&throttle_key(7, "a"), warning),- ThrottleDecision::Emit- );- assert_eq!(- throttle.check(&throttle_key(7, "b"), warning),- ThrottleDecision::Emit- );- assert_eq!(- throttle.check(&throttle_key(7, "c"), warning),- ThrottleDecision::TypeBudgetExhausted- );- let later = Instant::now() + Duration::from_millis(600);- assert_eq!(- throttle.check_at(&throttle_key(7, "c"), warning, later),- ThrottleDecision::Emit- );- assert_eq!(- throttle.check(&throttle_key(7, "d"), warning),- ThrottleDecision::CardinalityCapped- );- }-- #[test]- fn the_type_budget_refills_at_the_stated_rate() {- let throttle = throttle(&settings(true, "broker:9092", "topic"));- let warning = WarningType::MergeRaceCondition;- for n in 0..DEFAULT_TYPE_BUDGET_PER_SECOND.get() {- assert_eq!(- throttle.check(&throttle_key(7, &n.to_string()), warning),- ThrottleDecision::Emit,- "scope {n}"- );- }- assert_eq!(- throttle.check(&throttle_key(7, "over"), warning),- ThrottleDecision::TypeBudgetExhausted- );- // A budget that refilled one permit a second would still refuse.- let later = Instant::now() + Duration::from_millis(100);- assert_eq!(- throttle.check_at(&throttle_key(7, "later"), warning, later),- ThrottleDecision::Emit- );- }}diff --git a/rust/personhog-leader/README.md b/rust/personhog-leader/README.mdindex 0f65bd797646..c325f5e511b1 100644--- a/rust/personhog-leader/README.md+++ b/rust/personhog-leader/README.md@@ -133,7 +133,15 @@ the writer (produce an unapplyable record).Trims and rejections emit `person_properties_size_violation` ingestionwarnings (`src/warnings.rs`), throttled per (team, type) to match the Node-pipeline's limiter. The writer-side weld+pipeline's limiter. They go through a dedicated fire-and-forget producer,+not the changelog producer, because the warnings topic can live on a+different cluster from the changelog. `INGESTION_WARNINGS_ENABLED`+(default `false`) turns it on, and `INGESTION_WARNINGS_KAFKA_HOSTS`,+`INGESTION_WARNINGS_KAFKA_TLS` and `INGESTION_WARNINGS_TOPIC` (default+`clickhouse_ingestion_warnings`) name the destination. Enabled with no+hosts set, the leader logs a warning and emits nothing. Health reads from+`ingestion_warnings_emitter_enabled{reason}` and+`ingestion_warnings_total{source="personhog-leader",outcome}`. The writer-side weld(`personhog-writer/tests/admission_weld.rs`) runs hostile property fixturesthrough these exact functions and the writer's real statement against livePostgres, asserting byte-exact round-trips.diff --git a/rust/personhog-leader/src/config.rs b/rust/personhog-leader/src/config.rsindex d1cf1d9d0bb5..4f4e2a69baa9 100644--- a/rust/personhog-leader/src/config.rs+++ b/rust/personhog-leader/src/config.rs@@ -1,7 +1,9 @@use std::fs;use std::net::SocketAddr;+use std::num::NonZeroU32;use std::time::Duration;+use common_ingestion_warnings::RowWarningsSettings;use common_kafka::config::KafkaConfig;use envconfig::Envconfig;use personhog_common::h2_window::{Http2Windows, WindowSize};@@ -197,11 +199,49 @@ pub struct Config {#[envconfig(default = "524288")]pub properties_trim_target: usize,- /// Topic for in-product ingestion warnings emitted when admission- /// trims or rejects an update.+ #[envconfig(default = "false")]+ pub ingestion_warnings_enabled: bool,++ #[envconfig(default = "")]+ pub ingestion_warnings_kafka_hosts: String,++ #[envconfig(default = "false")]+ pub ingestion_warnings_kafka_tls: bool,+#[envconfig(default = "clickhouse_ingestion_warnings")]pub ingestion_warnings_topic: String,+ #[envconfig(default = "16")]+ pub ingestion_warnings_kafka_queue_mib: u32,++ #[envconfig(default = "1048576")]+ pub ingestion_warnings_kafka_message_max_bytes: u32,++ /// The defaults below are fire-and-forget: a warning is worth less than+ /// the cost of retrying it, and a slow warnings topic must not hold an update.+ #[envconfig(default = "1")]+ pub ingestion_warnings_kafka_acks: String,++ #[envconfig(default = "0")]+ pub ingestion_warnings_kafka_retries: u32,++ #[envconfig(default = "100")]+ pub ingestion_warnings_kafka_linger_ms: u32,++ #[envconfig(default = "10000")]+ pub ingestion_warnings_kafka_queue_messages: u32,++ #[envconfig(default = "5000")]+ pub ingestion_warnings_kafka_message_timeout_ms: u32,++ /// Warnings of one type a minute that one pod sends, shared by every team+ /// on it; the rest are dropped as `type_budget_exhausted`.+ #[envconfig(default = "60")]+ pub ingestion_warnings_type_budget_per_minute: NonZeroU32,++ #[envconfig(default = "1000000")]+ pub ingestion_warnings_max_tracked_scopes: usize,+// ── Dirty index / changelog recovery ─────────────────────────/// How often to poll the writer's committed offsets and prune dirty/// index marks the writer has applied to PG. A tick costs one batched@@ -757,6 +797,24 @@ impl Config {Duration::from_secs(self.heartbeat_interval_secs)}+ pub fn ingestion_warnings_settings(&self) -> RowWarningsSettings {+ RowWarningsSettings {+ enabled: self.ingestion_warnings_enabled,+ kafka_hosts: self.ingestion_warnings_kafka_hosts.clone(),+ kafka_tls: self.ingestion_warnings_kafka_tls,+ topic: self.ingestion_warnings_topic.clone(),+ kafka_queue_mib: self.ingestion_warnings_kafka_queue_mib,+ kafka_message_max_bytes: self.ingestion_warnings_kafka_message_max_bytes,+ kafka_acks: self.ingestion_warnings_kafka_acks.clone(),+ kafka_retries: self.ingestion_warnings_kafka_retries,+ kafka_linger_ms: self.ingestion_warnings_kafka_linger_ms,+ kafka_queue_messages: self.ingestion_warnings_kafka_queue_messages,+ kafka_message_timeout_ms: self.ingestion_warnings_kafka_message_timeout_ms,+ type_budget_per_minute: self.ingestion_warnings_type_budget_per_minute,+ max_tracked_scopes: self.ingestion_warnings_max_tracked_scopes,+ }+ }+/// Resolve the K8s namespace from config or the service account mount.pub fn resolve_k8s_namespace(&self) -> Result<String, String> {if !self.k8s_namespace.is_empty() {diff --git a/rust/personhog-leader/src/main.rs b/rust/personhog-leader/src/main.rsindex 9e16b4a34082..ba06cb9c5e3b 100644--- a/rust/personhog-leader/src/main.rs+++ b/rust/personhog-leader/src/main.rs@@ -48,7 +48,7 @@ use personhog_leader::settle::{prune_and_settle_tick, retired_topic};use personhog_leader::warming::{changelog_partition_count, WarmClientPools, WarmingConfig, WarmingRetryPolicy,};-use personhog_leader::warnings::WarningsProducer;+use personhog_leader::warnings;common_alloc::used!();@@ -149,6 +149,15 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {);let changelog_handle = manager.register("changelog-config", ComponentOptions::new());let migration_handle = manager.register("migration-duty", ComponentOptions::new());+ let warnings_settings = config.ingestion_warnings_settings();+ let warnings_handle = warnings_settings.disabled_reason().is_none().then(|| {+ manager.register(+ "ingestion-warnings",+ ComponentOptions::new()+ .with_liveness_deadline(Duration::from_secs(30))+ .is_advisory(true),+ )+ });let readiness = manager.readiness_handler();let liveness = manager.liveness_handler();@@ -418,10 +427,8 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {}).expect("Failed to build changelog recovery consumer pool"),);- let warnings = WarningsProducer::new(- kafka_producer.clone(),- config.ingestion_warnings_topic.clone(),- );+ let warnings = warnings::install(&warnings_settings, warnings_handle);+ let warnings_to_flush = warnings.clone();let fence_scan = fallback.clone();// All derived from LEASE_TTL, so an operator debugging a timeout// cannot recover them without redoing the derivation.@@ -677,18 +684,16 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {}});- // Periodic sweep of idle per-key locks, refilled warning-throttle- // keys, and parked fence connections nothing consumed (a cancelled- // inbound handoff leaves no convergence behind to discard them).+ // Periodic sweep of idle per-key locks and parked fence connections+ // nothing consumed (a cancelled inbound handoff leaves no convergence+ // behind to discard them).let sweep_locks = Arc::clone(&locks);- let sweep_warnings = warnings.clone();let sweep_fenced = Arc::clone(&fenced);tokio::spawn(async move {let mut interval = tokio::time::interval(Duration::from_secs(60));loop {interval.tick().await;sweep_idle_locks(&sweep_locks);- sweep_warnings.sweep_throttle();sweep_fenced.sweep_prepared();}});@@ -784,7 +789,15 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {}});- monitor_guard.wait().await?;+ let waited = monitor_guard.wait().await;++ // Flush before the result propagates: a failed component or a timed-out+ // drain is exactly the shutdown that leaves warnings queued.+ if let Err(e) = tokio::task::spawn_blocking(move || warnings_to_flush.flush()).await {+ tracing::warn!(error = %e, "ingestion warnings flush task panicked");+ }++ waited?;Ok(())}diff --git a/rust/personhog-leader/src/service.rs b/rust/personhog-leader/src/service.rsindex 5a75fbd03539..962097fb2562 100644--- a/rust/personhog-leader/src/service.rs+++ b/rust/personhog-leader/src/service.rs@@ -1683,9 +1683,7 @@ mod tests {use common_kafka::config::KafkaConfig;use envconfig::Envconfig;- use health::HealthRegistry;use personhog_proto::personhog::types::v1::{LifecycleOpType, ReleaseOutcome};- use rdkafka::ClientConfig;use tonic::Code;use super::*;@@ -1807,19 +1805,10 @@ mod tests {assert_eq!(stats.clamped_numbers, 1);}- /// A service with no PG pool and a producer that never connects —- /// enough to exercise the miss path, where no test reaches Kafka or PG.+ /// A service with no PG pool and no reachable Kafka — enough to+ /// exercise the miss path, where no test reaches Kafka or PG.async fn make_test_service() -> PersonHogLeaderService {let kafka = KafkaConfig::init_from_hashmap(&HashMap::new()).unwrap();- let liveness = HealthRegistry::new("test")- .register("kafka".to_string(), Duration::from_secs(60))- .await;- let producer: rdkafka::producer::FutureProducer<- common_kafka::kafka_producer::KafkaContext,- > = ClientConfig::new()- .set("bootstrap.servers", "127.0.0.1:1")- .create_with_context(common_kafka::kafka_producer::KafkaContext::from(liveness))- .unwrap();PersonHogLeaderService::new(Arc::new(PartitionedCache::new(16)),None,@@ -1847,7 +1836,7 @@ mod tests {.expect("build recovery pool"),),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),// Lazily built: nothing here connects until a write acquires// a partition, which these tests never do.diff --git a/rust/personhog-leader/src/warnings.rs b/rust/personhog-leader/src/warnings.rsindex 64116ddbc537..2abf5d5e24e8 100644--- a/rust/personhog-leader/src/warnings.rs+++ b/rust/personhog-leader/src/warnings.rs@@ -1,14 +1,11 @@use std::sync::Arc;--use metrics::counter;-use rdkafka::producer::{FutureProducer, FutureRecord};-use tracing::warn;+use std::time::Duration;use common_ingestion_warnings::registry::WarningType;-use common_ingestion_warnings::serializer::Warning as WarningPayload;-use common_ingestion_warnings::throttle::WarningThrottle;-use common_ingestion_warnings::WarningSource;-use common_kafka::kafka_producer::KafkaContext;+use common_ingestion_warnings::serializer::Warning;+use common_ingestion_warnings::{+ install_row_emitter, RowWarningEmitter, RowWarningsSettings, WarningSource,+};/// Warnings emitted from the leader's admission check, ahead of the write/// path proper. `pipeline_step` lands in the v2 table's `pipeline_step`@@ -19,6 +16,14 @@ const LEADER_ADMISSION: WarningSource = WarningSource {pipeline_step: "personhog_admission",};+const KAFKA_CLIENT_ID: &str = "personhog-leader-ingestion-warnings";+const SHUTDOWN_FLUSH: Duration = Duration::from_secs(2);++/// The throttle scope within a team. Admission warnings are budgeted per+/// team and type, so one person hammered with oversized updates and many+/// such persons on one team both yield one warning an hour.+const TEAM_SCOPE: &str = "";+/// An in-product ingestion warning about a person's property size —/// emitted when admission trims an update to fit the Postgres constraint/// or rejects one that cannot fit. Carries no property values, only@@ -31,86 +36,53 @@ pub struct SizeViolationWarning {pub message: String,}-/// Produces ingestion warnings to Kafka so users see property size-/// violations in-product, with the payload built by the shared-/// ingestion-warnings crate so type and classification come from the-/// pipeline registry. Shares the leader's changelog producer (rdkafka-/// producers are topic-agnostic), so buffered warnings flush with it on-/// shutdown; emission is fire-and-forget and never blocks or fails the-/// update that triggered it.-///-/// Emission is throttled per `(team, type)` with the pipeline's default-/// budget (one per hour), matching the Node consumers' warning limiter:-/// one oversized person being hammered with updates yields one warning an-/// hour, not one per update. Cloning shares the throttle, so every handle-/// draws from the same budget.+/// Sends size violation warnings so users see them in-product, with the+/// payload built by the shared ingestion-warnings crate so type and+/// classification come from the pipeline registry. Emission is+/// fire-and-forget and never blocks or fails the update that triggered it.#[derive(Clone)]pub struct WarningsProducer {- producer: FutureProducer<KafkaContext>,- topic: String,- throttle: Arc<WarningThrottle>,+ emitter: Option<Arc<dyn RowWarningEmitter>>,}impl WarningsProducer {- pub fn new(producer: FutureProducer<KafkaContext>, topic: String) -> Self {- Self::with_throttle(producer, topic, WarningThrottle::default())- }-- /// Construct with an explicit throttle — tests use a larger burst so- /// consecutive enforcement actions for one team all surface.- pub fn with_throttle(- producer: FutureProducer<KafkaContext>,- topic: String,- throttle: WarningThrottle,- ) -> Self {+ pub fn new(emitter: Arc<dyn RowWarningEmitter>) -> Self {Self {- producer,- topic,- throttle: Arc::new(throttle),+ emitter: Some(emitter),}}+ pub fn disabled() -> Self {+ Self { emitter: None }+ }+pub fn emit(&self, warning: &SizeViolationWarning) {- let warning_type = WarningType::PersonPropertiesSizeViolation;- if let Some(outcome) = self- .throttle- .check(&warning.team_id.to_string(), warning_type)- .drop_outcome()- {- counter!(- "personhog_leader_ingestion_warnings_suppressed_total",- "outcome" => outcome- )- .increment(1);+ let Some(emitter) = &self.emitter else {return;- }-+ };// teamId, category, severity, and pipelineStep are injected by the// terminal.- let payload = WarningPayload::new(warning_type)+ let payload = Warning::new(WarningType::PersonPropertiesSizeViolation).with_detail("personId", warning.person_uuid.clone())- .with_detail("message", warning.message.clone())- .into_row(warning.team_id, LEADER_ADMISSION);-- let payload_bytes = serde_json::to_vec(&payload).unwrap_or_default();- let key = warning.team_id.to_string();- let record = FutureRecord::to(&self.topic)- .payload(&payload_bytes)- .key(&key);+ .with_detail("message", warning.message.clone());+ emitter.emit(warning.team_id, TEAM_SCOPE, LEADER_ADMISSION, payload);+ }- match self.producer.send_result(record) {- Ok(_future) => {- counter!("personhog_leader_ingestion_warnings_emitted_total").increment(1);- }- Err((e, _)) => {- warn!(error = %e, "failed to enqueue ingestion warning");- }+ /// Blocking; call once the gRPC server, the only caller of `emit`, has+ /// stopped. The emitter's own component is advisory, so it cannot flush last.+ pub fn flush(&self) {+ if let Some(emitter) = &self.emitter {+ emitter.flush(SHUTDOWN_FLUSH);}}+}- /// Evict fully refilled throttle keys, bounding memory. Called from the- /// leader's periodic maintenance task.- pub fn sweep_throttle(&self) {- self.throttle.sweep();+pub fn install(+ settings: &RowWarningsSettings,+ handle: Option<lifecycle::Handle>,+) -> WarningsProducer {+ match install_row_emitter(settings, KAFKA_CLIENT_ID, handle) {+ Some(emitter) => WarningsProducer::new(emitter),+ None => WarningsProducer::disabled(),}}diff --git a/rust/personhog-leader/tests/commit_integration.rs b/rust/personhog-leader/tests/commit_integration.rsindex 9a20f319c479..ca37389b35ba 100644--- a/rust/personhog-leader/tests/commit_integration.rs+++ b/rust/personhog-leader/tests/commit_integration.rs@@ -6,8 +6,8 @@ use std::sync::Arc;use std::time::{Duration, Instant};use common::{- create_local_kafka_producer, seed_person, served_captures, target, test_cached_person,- test_kafka_config, test_recovery, unique_team_id, KAFKA_BOOTSTRAP,+ seed_person, served_captures, target, test_cached_person, test_kafka_config, test_recovery,+ unique_team_id, KAFKA_BOOTSTRAP,};use dashmap::DashMap;use personhog_common::partitioning::partition_for_person;@@ -80,7 +80,6 @@ async fn start_harness() -> Harness {cache.create_partition(partition);seed_person(&cache, partition, seed);let locks = Arc::new(DashMap::new());- let producer = create_local_kafka_producer().await;let service = Arc::new(PersonHogLeaderService::new(Arc::clone(&cache),None,@@ -90,7 +89,7 @@ async fn start_harness() -> Harness {Arc::new(DirtyIndex::new(1_000_000)),test_recovery(KAFKA_BOOTSTRAP),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),Arc::clone(&fenced),common::live_authority(),diff --git a/rust/personhog-leader/tests/common/mod.rs b/rust/personhog-leader/tests/common/mod.rsindex 8cbff5d90a08..8b79c4b3096a 100644--- a/rust/personhog-leader/tests/common/mod.rs+++ b/rust/personhog-leader/tests/common/mod.rs@@ -11,8 +11,12 @@ use std::time::Duration;use dashmap::DashMap;use async_trait::async_trait;+use common_ingestion_warnings::throttle::WarningThrottle;+use common_ingestion_warnings::{observe_delivery, KafkaRowWarningEmitter};use common_kafka::config::KafkaConfig;-use common_kafka::kafka_producer::{create_kafka_producer, KafkaContext};+use common_kafka::kafka_producer::{+ create_kafka_producer, create_threaded_kafka_producer_no_ping, KafkaContext,+};use health::HealthRegistry;use personhog_coordination::authority::AuthorityClock;use personhog_coordination::changelog::LiveChangelog;@@ -318,6 +322,24 @@ pub async fn create_local_kafka_producer() -> FutureProducer<KafkaContext> {.expect("failed to connect to local Kafka")}+/// A warnings producer against local Kafka, for tests that read warnings back.+pub async fn local_warnings_producer(topic: &str, throttle: WarningThrottle) -> WarningsProducer {+ let registry = HealthRegistry::new("test");+ let handle = registry+ .register("ingestion-warnings".to_string(), Duration::from_secs(30))+ .await;+ let config = KafkaConfig {+ kafka_hosts: KAFKA_BOOTSTRAP.to_string(),+ kafka_producer_linger_ms: 0,+ ..Default::default()+ };+ let producer = create_threaded_kafka_producer_no_ping(&config, handle, observe_delivery)+ .expect("failed to create the warnings producer");+ WarningsProducer::new(Arc::new(KafkaRowWarningEmitter::with_throttle(+ producer, topic, throttle,+ )))+}+/// One test's changelog topic: two tests acquiring the same partition/// of a shared one would fence each other.pub struct TestTopic {@@ -385,18 +407,15 @@ pub async fn create_mock_kafka_with_partitions(/// A changelog topic and a producer for it, with `NUM_PARTITIONS`/// partitions so warming can fetch every partition's watermarks.-pub async fn create_test_kafka() -> (TestTopic, FutureProducer<KafkaContext>) {+pub async fn create_test_kafka() -> TestTopic {create_test_kafka_with_partitions(NUM_PARTITIONS as i32).await}/// `create_test_kafka` with a partition count the caller pins.-pub async fn create_test_kafka_with_partitions(- partitions: i32,-) -> (TestTopic, FutureProducer<KafkaContext>) {+pub async fn create_test_kafka_with_partitions(partitions: i32) -> TestTopic {let topic = unique_topic();create_broker_topic(&topic, partitions).await;- let producer = create_local_kafka_producer().await;- (TestTopic { topic }, producer)+ TestTopic { topic }}/// A topic of this test's own, deleted when the handle drops.@@ -472,7 +491,6 @@ pub async fn start_leader_pod(.unwrap().expect("seed config/changelog before starting a pod").topic;- let kafka_producer = create_local_kafka_producer().await;// Pod with real handoff handler. Warming consumer reads from the same// mock broker the producer is publishing to so the topic actually@@ -542,7 +560,7 @@ pub async fn start_leader_pod(Arc::clone(&dirty_index),recovery,PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(dashmap::DashMap::new()),Arc::clone(&fenced),Arc::clone(&authority),@@ -582,7 +600,6 @@ pub async fn start_leader_pod_with_lease_ttl(.unwrap().expect("seed config/changelog before starting a pod").topic;- let kafka_producer = create_local_kafka_producer().await;let heartbeat_secs = (lease_ttl as u64 / 3).max(1);let inflight = Arc::new(InflightTracker::new());@@ -648,7 +665,7 @@ pub async fn start_leader_pod_with_lease_ttl(Arc::clone(&dirty_index),recovery,PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(dashmap::DashMap::new()),Arc::clone(&fenced),Arc::clone(&authority),@@ -729,7 +746,7 @@ pub async fn start_leader_with_pg_fallback(cancel: CancellationToken,) -> (SocketAddr, Arc<PartitionedCache>, TestTopic) {let cache = Arc::new(PartitionedCache::new(1 << 20));- let (mock_cluster, kafka_producer) = create_test_kafka().await;+ let mock_cluster = create_test_kafka().await;let pool = create_persons_pool().await;let service = PersonHogLeaderService::new(@@ -745,7 +762,7 @@ pub async fn start_leader_with_pg_fallback(Arc::new(DirtyIndex::new(1_000_000)),test_recovery(&mock_cluster.bootstrap_servers()),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(dashmap::DashMap::new()),fenced_producers_acquired(&mock_cluster.topic, NUM_PARTITIONS as i32).await,live_authority(),diff --git a/rust/personhog-leader/tests/fence_integration.rs b/rust/personhog-leader/tests/fence_integration.rsindex 53cdcd130d9a..e22f91374648 100644--- a/rust/personhog-leader/tests/fence_integration.rs+++ b/rust/personhog-leader/tests/fence_integration.rs@@ -96,7 +96,7 @@ struct FenceHarness {}async fn start_fence_harness(mut seed: CachedPerson, fallback: Option<PgFallback>) -> FenceHarness {- let (mock_cluster, kafka_producer) = create_test_kafka().await;+ let mock_cluster = create_test_kafka().await;let bootstrap = mock_cluster.bootstrap_servers();// Every harness gets its own team, so concurrent tests — and leftover@@ -125,7 +125,7 @@ async fn start_fence_harness(mut seed: CachedPerson, fallback: Option<PgFallbackArc::clone(&dirty_index),test_recovery(&bootstrap),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::clone(&fences),common::fenced_producers_acquired(&mock_cluster.topic, NUM_PARTITIONS as i32).await,common::live_authority(),@@ -1446,7 +1446,7 @@ async fn at_capacity_new_fences_shed_but_reseals_succeed() {let team_id = unique_team_id();let cache = Arc::new(PartitionedCache::new(1 << 20));- let (_mock_cluster, kafka_producer) = create_test_kafka().await;+ let _mock_cluster = create_test_kafka().await;let service = PersonHogLeaderService::new(Arc::clone(&cache),None,@@ -1456,7 +1456,7 @@ async fn at_capacity_new_fences_shed_but_reseals_succeed() {Arc::new(DirtyIndex::new(1_000_000)),test_recovery(&_mock_cluster.bootstrap_servers()),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),common::fenced_producers_acquired(&_mock_cluster.topic, NUM_PARTITIONS as i32).await,common::live_authority(),diff --git a/rust/personhog-leader/tests/integration.rs b/rust/personhog-leader/tests/integration.rsindex 235280ecb36d..7209764fcc34 100644--- a/rust/personhog-leader/tests/integration.rs+++ b/rust/personhog-leader/tests/integration.rs@@ -177,7 +177,7 @@ async fn service_accepts_requests_after_coordination_warmup() {async fn unowned_partition_returns_failed_precondition() {// Create service + cache directly, no coordinationlet cache = Arc::new(PartitionedCache::new(1 << 20));- let (_mock_cluster, kafka_producer) = create_test_kafka().await;+ let _mock_cluster = create_test_kafka().await;// One set for both: a second would fence the first at the broker.let fenced =common::fenced_producers_acquired(&_mock_cluster.topic, NUM_PARTITIONS as i32).await;@@ -190,7 +190,7 @@ async fn unowned_partition_returns_failed_precondition() {Arc::new(DirtyIndex::new(1_000_000)),test_recovery(KAFKA_BOOTSTRAP),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),Arc::clone(&fenced),common::live_authority(),@@ -263,7 +263,7 @@ async fn unowned_partition_returns_failed_precondition() {#[tokio::test]async fn missing_partition_metadata_returns_invalid_argument() {let cache = Arc::new(PartitionedCache::new(1 << 20));- let (_mock_cluster, kafka_producer) = create_test_kafka().await;+ let _mock_cluster = create_test_kafka().await;// One set for both: a second would fence the first at the broker.let fenced =common::fenced_producers_acquired(&_mock_cluster.topic, NUM_PARTITIONS as i32).await;@@ -276,7 +276,7 @@ async fn missing_partition_metadata_returns_invalid_argument() {Arc::new(DirtyIndex::new(1_000_000)),test_recovery(KAFKA_BOOTSTRAP),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),Arc::clone(&fenced),common::live_authority(),@@ -352,7 +352,7 @@ async fn missing_partition_metadata_returns_invalid_argument() {#[tokio::test]async fn mismatched_partition_metadata_returns_failed_precondition() {let cache = Arc::new(PartitionedCache::new(1 << 20));- let (_mock_cluster, kafka_producer) = create_test_kafka().await;+ let _mock_cluster = create_test_kafka().await;// One set for both: a second would fence the first at the broker.let fenced =common::fenced_producers_acquired(&_mock_cluster.topic, NUM_PARTITIONS as i32).await;@@ -365,7 +365,7 @@ async fn mismatched_partition_metadata_returns_failed_precondition() {Arc::new(DirtyIndex::new(1_000_000)),test_recovery(KAFKA_BOOTSTRAP),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),Arc::clone(&fenced),common::live_authority(),@@ -444,7 +444,7 @@ async fn mismatched_partition_metadata_returns_failed_precondition() {#[tokio::test]async fn writes_fenced_after_drain_reads_still_served() {let cache = Arc::new(PartitionedCache::new(1 << 20));- let (_mock_cluster, kafka_producer) = create_test_kafka().await;+ let _mock_cluster = create_test_kafka().await;let inflight = Arc::new(InflightTracker::new());let dirty_index = Arc::new(DirtyIndex::new(1_000_000));let recovery = test_recovery(KAFKA_BOOTSTRAP);@@ -461,7 +461,7 @@ async fn writes_fenced_after_drain_reads_still_served() {Arc::clone(&dirty_index),Arc::clone(&recovery),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),Arc::clone(&fenced),common::live_authority(),@@ -590,7 +590,7 @@ async fn writes_fenced_after_drain_reads_still_served() {#[tokio::test]async fn drain_fences_before_waiting_on_inflight() {let cache = Arc::new(PartitionedCache::new(1 << 20));- let (_mock_cluster, kafka_producer) = create_test_kafka().await;+ let _mock_cluster = create_test_kafka().await;let inflight = Arc::new(InflightTracker::new());let dirty_index = Arc::new(DirtyIndex::new(1_000_000));let recovery = test_recovery(KAFKA_BOOTSTRAP);@@ -607,7 +607,7 @@ async fn drain_fences_before_waiting_on_inflight() {Arc::clone(&dirty_index),Arc::clone(&recovery),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),Arc::clone(&fenced),common::live_authority(),@@ -924,7 +924,7 @@ async fn update_produces_person_state_to_kafka() {const PERSON_ID: i64 = 2;let routing_partition: u32 = partition_for_person(1, PERSON_ID, NUM_PARTITIONS);assert_eq!(routing_partition, 2, "test key must map to partition 2");- let (mock_cluster, kafka_producer) = create_test_kafka_with_partitions(4).await;+ let mock_cluster = create_test_kafka_with_partitions(4).await;let cache = Arc::new(PartitionedCache::new(1 << 20));let service = PersonHogLeaderService::new(@@ -936,7 +936,7 @@ async fn update_produces_person_state_to_kafka() {Arc::new(DirtyIndex::new(1_000_000)),test_recovery(KAFKA_BOOTSTRAP),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),common::fenced_producers_acquired(&mock_cluster.topic, 4).await,common::live_authority(),@@ -1040,7 +1040,7 @@ async fn update_produces_person_state_to_kafka() {#[tokio::test]async fn kafka_produce_failure_leaves_cache_unchanged() {let cache = Arc::new(PartitionedCache::new(1 << 20));- let (mock_cluster, kafka_producer) = create_test_kafka().await;+ let mock_cluster = create_test_kafka().await;let fenced =common::fenced_producers_acquired(&mock_cluster.topic, NUM_PARTITIONS as i32).await;@@ -1053,7 +1053,7 @@ async fn kafka_produce_failure_leaves_cache_unchanged() {Arc::new(DirtyIndex::new(1_000_000)),test_recovery(KAFKA_BOOTSTRAP),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),Arc::clone(&fenced),common::live_authority(),@@ -1179,7 +1179,6 @@ async fn e2e_update_produces_to_local_kafka() {let owned = common::owned_broker_topic(NUM_PARTITIONS as i32).await;let topic = owned.topic.clone();let cache = Arc::new(PartitionedCache::new(1 << 20));- let kafka_producer = create_local_kafka_producer().await;let service = PersonHogLeaderService::new(Arc::clone(&cache),@@ -1190,7 +1189,7 @@ async fn e2e_update_produces_to_local_kafka() {Arc::new(DirtyIndex::new(1_000_000)),test_recovery(KAFKA_BOOTSTRAP),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),common::fenced_producers_acquired(&topic, NUM_PARTITIONS as i32).await,common::live_authority(),@@ -1510,7 +1509,7 @@ async fn update_triggers_pg_fallback_then_applies_changes() {async fn evicted_dirty_person_recovers_from_changelog() {const PERSON_ID: i64 = 2;let routing_partition: u32 = partition_for_person(1, PERSON_ID, NUM_PARTITIONS);- let (mock_cluster, kafka_producer) = create_test_kafka_with_partitions(4).await;+ let mock_cluster = create_test_kafka_with_partitions(4).await;let cache = Arc::new(PartitionedCache::new(1 << 20));let dirty_index = Arc::new(DirtyIndex::new(1_000_000));@@ -1527,7 +1526,7 @@ async fn evicted_dirty_person_recovers_from_changelog() {Arc::clone(&dirty_index),Arc::clone(&recovery),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),common::fenced_producers_acquired(&mock_cluster.topic, 4).await,common::live_authority(),@@ -1616,7 +1615,7 @@ async fn evicted_dirty_person_recovers_from_changelog() {async fn dirty_person_with_failed_recovery_is_unavailable_not_stale() {const PERSON_ID: i64 = 2;let routing_partition: u32 = partition_for_person(1, PERSON_ID, NUM_PARTITIONS);- let (mock_cluster, kafka_producer) = create_test_kafka_with_partitions(4).await;+ let mock_cluster = create_test_kafka_with_partitions(4).await;let cache = Arc::new(PartitionedCache::new(1 << 20));let dirty_index = Arc::new(DirtyIndex::new(1_000_000));@@ -1630,7 +1629,7 @@ async fn dirty_person_with_failed_recovery_is_unavailable_not_stale() {Arc::clone(&dirty_index),Arc::clone(&recovery),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),common::fenced_producers_acquired(&mock_cluster.topic, 4).await,common::live_authority(),@@ -1728,7 +1727,7 @@ async fn writes_shed_when_dirty_index_is_full() {.find(|pid| partition_for_person(1, *pid, NUM_PARTITIONS) == partition).expect("a second person maps to the same partition");- let (mock_cluster, kafka_producer) = create_test_kafka_with_partitions(4).await;+ let mock_cluster = create_test_kafka_with_partitions(4).await;let cache = Arc::new(PartitionedCache::new(1 << 20));// Capacity 1: the first person's mark fills the index.let dirty_index = Arc::new(DirtyIndex::new(1));@@ -1741,7 +1740,7 @@ async fn writes_shed_when_dirty_index_is_full() {Arc::clone(&dirty_index),test_recovery(&mock_cluster.bootstrap_servers()),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),common::fenced_producers_acquired(&mock_cluster.topic, 4).await,common::live_authority(),@@ -1827,7 +1826,7 @@ async fn writes_shed_when_dirty_index_is_full() {async fn recovery_fails_when_record_version_disagrees_with_the_mark() {const PERSON_ID: i64 = 2;let routing_partition: u32 = partition_for_person(1, PERSON_ID, NUM_PARTITIONS);- let (mock_cluster, kafka_producer) = create_test_kafka_with_partitions(4).await;+ let mock_cluster = create_test_kafka_with_partitions(4).await;let cache = Arc::new(PartitionedCache::new(1 << 20));let dirty_index = Arc::new(DirtyIndex::new(1_000_000));@@ -1841,7 +1840,7 @@ async fn recovery_fails_when_record_version_disagrees_with_the_mark() {Arc::clone(&dirty_index),Arc::clone(&recovery),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),common::fenced_producers_acquired(&mock_cluster.topic, 4).await,common::live_authority(),@@ -1938,7 +1937,7 @@ async fn recovery_reuses_the_partition_consumer_across_fetches() {.find(|pid| partition_for_person(1, *pid, NUM_PARTITIONS) == partition).expect("a second person maps to the same partition");- let (mock_cluster, kafka_producer) = create_test_kafka_with_partitions(4).await;+ let mock_cluster = create_test_kafka_with_partitions(4).await;let cache = Arc::new(PartitionedCache::new(1 << 20));let dirty_index = Arc::new(DirtyIndex::new(1_000_000));let recovery = test_recovery(&mock_cluster.bootstrap_servers());@@ -1951,7 +1950,7 @@ async fn recovery_reuses_the_partition_consumer_across_fetches() {Arc::clone(&dirty_index),Arc::clone(&recovery),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),common::fenced_producers_acquired(&mock_cluster.topic, 4).await,common::live_authority(),@@ -2063,7 +2062,8 @@ async fn cancelled_recovery_returns_its_consumer_to_the_pool() {use personhog_leader::recovery::{ChangelogRecovery, RecoveryConfig};use rdkafka::producer::FutureRecord;- let (mock_cluster, producer) = create_test_kafka().await;+ let mock_cluster = create_test_kafka().await;+ let producer = create_local_kafka_producer().await;let mut kafka = test_kafka_config();kafka.kafka_hosts = mock_cluster.bootstrap_servers();// pool_size 1: a single leaked consumer exhausts the pool.@@ -2151,7 +2151,7 @@ async fn cancelled_recovery_returns_its_consumer_to_the_pool() {async fn oversize_updates_are_rejected_and_oversized_rows_remediated() {const PERSON_ID: i64 = 9;let routing_partition: u32 = partition_for_person(1, PERSON_ID, NUM_PARTITIONS);- let (mock_cluster, kafka_producer) = create_test_kafka_with_partitions(4).await;+ let mock_cluster = create_test_kafka_with_partitions(4).await;// This test reads its warnings back, so they must be its own: on a// shared topic it would collect whichever two arrived first.let warnings_topic = common::owned_broker_topic(1).await;@@ -2171,11 +2171,11 @@ async fn oversize_updates_are_rejected_and_oversized_rows_remediated() {// Burst 2 so the trim and reject below — same team, same type —// both clear the throttle; the third enforcement action then// exercises suppression.- WarningsProducer::with_throttle(- kafka_producer,- warnings_topic.topic.clone(),+ common::local_warnings_producer(+ &warnings_topic.topic,WarningThrottle::new(DEFAULT_THROTTLE_PERIOD, NonZeroU32::new(2).unwrap()),- ),+ )+ .await,Arc::new(DashMap::new()),common::fenced_producers_acquired(&mock_cluster.topic, 4).await,common::live_authority(),@@ -2619,7 +2619,7 @@ async fn oversize_updates_are_rejected_and_oversized_rows_remediated() {#[tokio::test]async fn an_unresolved_version_is_never_reused() {let cache = Arc::new(PartitionedCache::new(1 << 20));- let (_mock_cluster, kafka_producer) = create_test_kafka().await;+ let _mock_cluster = create_test_kafka().await;// One set for both: a second would fence the first at the broker.let fenced =common::fenced_producers_acquired(&_mock_cluster.topic, NUM_PARTITIONS as i32).await;@@ -2633,7 +2633,7 @@ async fn an_unresolved_version_is_never_reused() {Arc::new(DirtyIndex::new(1_000_000)),test_recovery(KAFKA_BOOTSTRAP),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),Arc::clone(&fenced),common::live_authority(),@@ -2702,7 +2702,7 @@ async fn an_unresolved_version_is_never_reused() {async fn a_fenced_write_that_bounces_does_not_hand_its_version_back() {let topic = format!("fenced_svc_{}", uuid::Uuid::new_v4().simple());let cache = Arc::new(PartitionedCache::new(1 << 20));- let (_mock_cluster, kafka_producer) = create_test_kafka().await;+ let _mock_cluster = create_test_kafka().await;let fenced = Arc::new(common::test_fenced_producers());let service = PersonHogLeaderService::new(@@ -2714,7 +2714,7 @@ async fn a_fenced_write_that_bounces_does_not_hand_its_version_back() {Arc::new(DirtyIndex::new(1_000_000)),test_recovery(KAFKA_BOOTSTRAP),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),Arc::clone(&fenced),common::live_authority(),@@ -2826,7 +2826,7 @@ async fn an_unknown_outcome_keeps_its_version() {let topic = format!("fenced_uncertain_{}", uuid::Uuid::new_v4().simple());let cache = Arc::new(PartitionedCache::new(1 << 20));- let (_mock_cluster, kafka_producer) = create_test_kafka().await;+ let _mock_cluster = create_test_kafka().await;let fenced = Arc::new(common::test_fenced_producers());let emitted_versions = Arc::new(personhog_leader::emitted::EmittedVersions::new(1_000_000));@@ -2839,7 +2839,7 @@ async fn an_unknown_outcome_keeps_its_version(Arc::new(DirtyIndex::new(1_000_000)),test_recovery(KAFKA_BOOTSTRAP),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),Arc::clone(&fenced),common::live_authority(),@@ -2918,7 +2918,7 @@ async fn scalar_fields_merge_rather_than_assign() {const HOUR_MS: i64 = 3_600_000;let topic = format!("scalar_merge_{}", uuid::Uuid::new_v4().simple());let cache = Arc::new(PartitionedCache::new(1 << 20));- let (_mock_cluster, kafka_producer) = create_test_kafka().await;+ let _mock_cluster = create_test_kafka().await;let emitted_versions = Arc::new(personhog_leader::emitted::EmittedVersions::new(1_000_000));let service = PersonHogLeaderService::new(@@ -2930,7 +2930,7 @@ async fn scalar_fields_merge_rather_than_assign() {Arc::new(DirtyIndex::new(1_000_000)),test_recovery(KAFKA_BOOTSTRAP),PropertySizeLimits::new(655360, 524288),- WarningsProducer::new(kafka_producer, "clickhouse_ingestion_warnings".to_string()),+ WarningsProducer::disabled(),Arc::new(DashMap::new()),common::fenced_producers_acquired(&topic, NUM_PARTITIONS as i32).await,common::live_authority(),diff --git a/rust/personhog-leader/tests/warming_integration.rs b/rust/personhog-leader/tests/warming_integration.rsindex 28f19fd15174..5dddb1e264e8 100644--- a/rust/personhog-leader/tests/warming_integration.rs+++ b/rust/personhog-leader/tests/warming_integration.rs@@ -1064,7 +1064,8 @@ async fn a_restart_warms_the_old_generations_congruent_tail_for_its_keys() {use personhog_coordination::changelog::KeyFilter;use personhog_leader::cache::Stream;- let (new_topic, producer) = common::create_test_kafka_with_partitions(4).await;+ let new_topic = common::create_test_kafka_with_partitions(4).await;+ let producer = common::create_local_kafka_producer().await;let old_topic = common::owned_broker_topic(2).await;let bootstrap = new_topic.bootstrap_servers();@@ -1185,7 +1186,8 @@ async fn an_applied_old_record_never_shadows_a_newer_applied_live_one() {use personhog_common::partitioning::partition_for_person;use personhog_coordination::changelog::KeyFilter;- let (new_topic, producer) = common::create_test_kafka_with_partitions(2).await;+ let new_topic = common::create_test_kafka_with_partitions(2).await;+ let producer = common::create_local_kafka_producer().await;let old_topic = common::owned_broker_topic(2).await;let bootstrap = new_topic.bootstrap_servers();