:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
feat(personhog): gate changelog retire on a list of consumer groups (#121248)
Zach Brownc927f647131e
diff --git a/rust/ingestion-control-plane/README.md b/rust/ingestion-control-plane/README.mdindex 6f075a1bd3fd..c066eb96901f 100644--- a/rust/ingestion-control-plane/README.md+++ b/rust/ingestion-control-plane/README.md@@ -23,7 +23,7 @@ Lists live ingestion-consumer pods and serves the consumer's routing debug UI pe### Personhog topology-Protocol-aware view of personhog's etcd coordination state (`PERSONHOG_ETCD_PREFIX`, default `/personhog/`): the whole prefix is read in one consistent snapshot and interpreted with the `personhog-coordination` types and the coordinator's own pure predicates, so the derived diagnostics cannot drift from the protocol. Shows the elected coordinator, registered pods and routers (with remaining lease TTLs), per-partition ownership, and in-flight handoffs with their phase age and what each is waiting on — distinguishing a stuck participant ("waiting on freeze acks from router-2") from a stuck coordinator ("quorum met; waiting on the coordinator to advance"). Derived issues cover stuck handoffs past their per-phase deadline, assignments owned by unregistered pods with no replanning handoff, lingering `Complete` records the coordinator hasn't cleaned up, dead new owners awaiting cancellation, unassigned partitions, stale acks from previous handoff attempts, and unparseable records. The deadline knobs (`PERSONHOG_HANDOFF_DEADLINE_SECS`, `PERSONHOG_WARMING_DEADLINE_SECS`) must match the coordinator's to agree with its cancellation decisions. The tool writes no coordination state except from the changelog panel (seeding `config/changelog`, requesting or cancelling a migration, and retiring a drained generation), and a force-reassignment must go through a handoff record, never a raw assignment write.+Protocol-aware view of personhog's etcd coordination state (`PERSONHOG_ETCD_PREFIX`, default `/personhog/`): the whole prefix is read in one consistent snapshot and interpreted with the `personhog-coordination` types and the coordinator's own pure predicates, so the derived diagnostics cannot drift from the protocol. Shows the elected coordinator, registered pods and routers (with remaining lease TTLs), per-partition ownership, and in-flight handoffs with their phase age and what each is waiting on — distinguishing a stuck participant ("waiting on freeze acks from router-2") from a stuck coordinator ("quorum met; waiting on the coordinator to advance"). Derived issues cover stuck handoffs past their per-phase deadline, assignments owned by unregistered pods with no replanning handoff, lingering `Complete` records the coordinator hasn't cleaned up, dead new owners awaiting cancellation, unassigned partitions, stale acks from previous handoff attempts, and unparseable records. The deadline knobs (`PERSONHOG_HANDOFF_DEADLINE_SECS`, `PERSONHOG_WARMING_DEADLINE_SECS`) must match the coordinator's to agree with its cancellation decisions. The tool writes no coordination state except from the changelog panel (seeding `config/changelog`, adding or removing a consumer group that retire waits on, requesting or cancelling a migration, and retiring a drained generation), and a force-reassignment must go through a handoff record, never a raw assignment write.### Personhog paritydiff --git a/rust/ingestion-control-plane/src/api.rs b/rust/ingestion-control-plane/src/api.rsindex d87307b91c15..0cb24a05d5fb 100644--- a/rust/ingestion-control-plane/src/api.rs+++ b/rust/ingestion-control-plane/src/api.rs@@ -511,6 +511,42 @@ async fn personhog_retire_topic(Ok(Json(json!({ "retired": request.topic.trim() })))}+#[derive(Deserialize)]+struct RetireGroupRequest {+ group: String,+}++async fn personhog_add_retire_group(+ State(state): State<AppState>,+ Json(request): Json<RetireGroupRequest>,+) -> Result<Json<serde_json::Value>, ApiError> {+ let prefix = state.config.personhog_etcd_prefix.clone();+ validated_etcd_key(&state, &prefix)?;+ let group = request.group.trim();+ if group.is_empty() {+ return Err(ApiError::bad_request("group must not be empty"));+ }+ let client = etcd_client(&state).await?;+ changelog::add_retire_group(&client, &prefix, group)+ .await+ .map_err(|e| ApiError::bad_request(format!("add retire group refused: {e:#}")))?;+ Ok(Json(json!({ "group": group })))+}++async fn personhog_remove_retire_group(+ State(state): State<AppState>,+ Json(request): Json<RetireGroupRequest>,+) -> Result<Json<serde_json::Value>, ApiError> {+ let prefix = state.config.personhog_etcd_prefix.clone();+ validated_etcd_key(&state, &prefix)?;+ let group = request.group.trim();+ let client = etcd_client(&state).await?;+ changelog::remove_retire_group(&client, &prefix, group)+ .await+ .map_err(|e| ApiError::bad_request(format!("remove retire group refused: {e:#}")))?;+ Ok(Json(json!({ "removed": group })))+}+#[derive(Deserialize)]struct StopMigrationRequest {/// The request or saga the operator saw; anything else is left alone.@@ -577,6 +613,10 @@ pub fn router(state: AppState) -> Router {"/api/personhog/changelog/retire",post(personhog_retire_topic),)+ .route(+ "/api/personhog/changelog/retire-groups",+ post(personhog_add_retire_group).delete(personhog_remove_retire_group),+ )// The consumer debug UI is served per pod; its relative `debug/...`// fetches resolve to the proxy route below. Pods are addressed by// namespace + name so identical names across lanes cannot collide.diff --git a/rust/ingestion-control-plane/src/changelog.rs b/rust/ingestion-control-plane/src/changelog.rsindex 994ad7c6dc18..7965cd6134bf 100644--- a/rust/ingestion-control-plane/src/changelog.rs+++ b/rust/ingestion-control-plane/src/changelog.rs@@ -87,6 +87,7 @@ pub struct GroupLag {pub struct ChangelogView {pub config: Option<ChangelogConfig>,pub writer: Option<GroupLag>,+ pub retire_groups: Vec<GroupLag>,pub migration: Option<MigrationState>,pub request: Option<MigrationRequest>,pub history: Vec<MigrationState>,@@ -101,13 +102,14 @@ pub struct GroupDrain {pub error: Option<String>,}-/// A draining generation and how far the writer group has read it.+/// A draining generation and how far each gating group has read it.#[derive(Serialize)]pub struct DrainingView {pub topic: String,pub partitions: u32,pub epoch: u64,pub writer: GroupDrain,+ pub retire_groups: Vec<GroupDrain>,}async fn group_drain(config: &Config, group: &str, topic: &str) -> GroupDrain {@@ -140,6 +142,28 @@ async fn group_drain(config: &Config, group: &str, topic: &str) -> GroupDrain {}}+/// Add a consumer group, such as the relay's, that retire then waits on as well.+pub async fn add_retire_group(client: &Client, prefix: &str, group: &str) -> anyhow::Result<()> {+ if !store(client, prefix)+ .add_changelog_retire_group(group)+ .await?+ {+ bail!("config/changelog is not seeded or moved; refresh and retry");+ }+ Ok(())+}++/// Stop retire waiting on a group, for a consumer that no longer runs.+pub async fn remove_retire_group(client: &Client, prefix: &str, group: &str) -> anyhow::Result<()> {+ if !store(client, prefix)+ .remove_changelog_retire_group(group)+ .await?+ {+ bail!("config/changelog is not seeded or moved; refresh and retry");+ }+ Ok(())+}+/// Drop a drained generation from the config; deleting the topic stays a Terraform change.pub async fn retire(config: &Config,@@ -149,26 +173,30 @@ pub async fn retire() -> anyhow::Result<()> {let config = &config.personhog_kafka();let store = store(client, prefix);- let live = store- .get_changelog_config()+ let (live, mod_revision) = store+ .get_changelog_config_with_mod_revision().await?.context("config/changelog is not seeded")?;if !live.draining.iter().any(|g| g.topic == topic) {bail!("{topic} is not a draining generation");}- let group = &live.writer_group;- let drain = group_drain(config, group, topic).await;- if let Some(error) = drain.error {- bail!("group {group}: {error}");- }- if drain.partitions == 0 || drain.drained_partitions < drain.partitions {- bail!(- "group {group} has read {} of {} partitions of {topic} to the end",- drain.drained_partitions,- drain.partitions- );+ for group in live.gated_groups() {+ let drain = group_drain(config, group, topic).await;+ if let Some(error) = drain.error {+ bail!("group {group}: {error}");+ }+ if drain.partitions == 0 || drain.drained_partitions < drain.partitions {+ bail!(+ "group {group} has read {} of {} partitions of {topic} to the end",+ drain.drained_partitions,+ drain.partitions+ );+ }}- if !store.retire_changelog_topic(topic).await? {+ if !store+ .retire_changelog_topic(&live, mod_revision, topic)+ .await?+ {bail!("the config moved; refresh and retry");}Ok(())@@ -294,6 +322,13 @@ pub async fn view(config: &Config, client: &Client, prefix: &str) -> anyhow::Respartitions: generation.partitions,epoch: generation.epoch,writer: group_drain(config, &changelog.writer_group, &generation.topic).await,+ retire_groups: {+ let mut drains = Vec::new();+ for group in &changelog.retire_groups {+ drains.push(group_drain(config, group, &generation.topic).await);+ }+ drains+ },});}}@@ -301,9 +336,16 @@ pub async fn view(config: &Config, client: &Client, prefix: &str) -> anyhow::ResSome(changelog) => Some(group_lag(config, &changelog.writer_group, &changelog.topic).await),None => None,};+ let mut retire_groups = Vec::new();+ if let Some(changelog) = &changelog {+ for group in &changelog.retire_groups {+ retire_groups.push(group_lag(config, group, &changelog.topic).await);+ }+ }Ok(ChangelogView {config: changelog,writer,+ retire_groups,migration,request,history,diff --git a/rust/ingestion-control-plane/src/ui/index.html b/rust/ingestion-control-plane/src/ui/index.htmlindex d76d6885beee..7eb1a381d705 100644--- a/rust/ingestion-control-plane/src/ui/index.html+++ b/rust/ingestion-control-plane/src/ui/index.html@@ -1387,9 +1387,28 @@if (seq !== this.refreshSeq) return;this.render(data);// The lag scan talks to Kafka, so it never blocks the topology render.- fetchJSON("api/personhog/changelog").then(- (view) => { if (seq === this.refreshSeq) this.renderChangelog(view); },- (e) => { if (seq === this.refreshSeq) this.ui.changelogStatus.replaceChildren(el("div", { class: "error-box" }, String(e))); });+ this.refreshChangelog();+ },++ // One changelog view in flight at a time, always rendered: a scan slower+ // than the refresh interval would otherwise be discarded every time.+ refreshChangelog() {+ if (this.changelogInFlight) {+ this.changelogPending = true;+ return;+ }+ this.changelogInFlight = true;+ fetchJSON("api/personhog/changelog")+ .then(+ (view) => this.renderChangelog(view),+ (e) => this.ui.changelogStatus.replaceChildren(el("div", { class: "error-box" }, String(e))))+ .finally(() => {+ this.changelogInFlight = false;+ if (this.changelogPending) {+ this.changelogPending = false;+ this.refreshChangelog();+ }+ });},renderChangelog(view) {@@ -1397,7 +1416,9 @@if (this.ui.changelogFormKey !== formKey) {this.ui.changelogFormKey = formKey;this.ui.changelogError.replaceChildren();- this.ui.changelogForm.replaceChildren(view.config ? this.migrationForm(view) : this.seedForm());+ this.ui.changelogForm.replaceChildren(...(view.config+ ? [this.migrationForm(view), this.retireGroupForm()]+ : [this.seedForm()]));}if (!view.config) {this.ui.changelogStatus.replaceChildren(@@ -1417,11 +1438,11 @@? el("div", { class: "empty" }, "No writer group."): el("table", {},el("thead", {}, el("tr", {},- el("th", {}, "writer group"), el("th", {}, "partitions"), el("th", {}, "lagging"), el("th", {}, "total lag"),+ el("th", {}, "consumer group"), el("th", {}, "partitions"), el("th", {}, "lagging"), el("th", {}, "total lag"),el("th", { title: "A grow's restart replays about this many records per partition; a shrink by k replays about k partitions' worth. Larger replays lengthen the pause." }, "worst partition"))),- el("tbody", {}, [view.writer].map((g) =>+ el("tbody", {}, [view.writer, ...view.retire_groups].map((g, i) =>el("tr", {},- el("td", { class: "mono" }, g.group),+ el("td", { class: "mono" }, g.group, i === 0 ? null : this.removeRetireGroupButton(g.group)),el("td", { class: "mono" }, g.error ? el("span", { class: "pill bad", title: g.error }, "scan failed") : fmtInt(g.partitions)),el("td", { class: "mono" }, g.error ? "–" : fmtInt(g.lagging_partitions)),el("td", { class: "mono" }, g.error ? "–" : el("span", { class: `pill ${g.total_lag === 0 ? "good" : "warn"}` }, fmtInt(g.total_lag))),@@ -1430,9 +1451,36 @@changelogFormKeyOf(view) {if (!view.config) return "unseeded";- if (view.migration) return `migration:${view.migration.id}:${view.migration.phase}`;- if (view.request) return `request:${view.request.id}`;- return "idle";+ const groups = (view.config.retire_groups || []).join(",");+ if (view.migration) return `migration:${view.migration.id}:${view.migration.phase}:${groups}`;+ if (view.request) return `request:${view.request.id}:${groups}`;+ return `idle:${groups}`;+ },++ removeRetireGroupButton(group) {+ const button = el("button", { style: "margin-left: 8px" }, "Remove");+ button.onclick = () => this.postChangelogAction("api/personhog/changelog/retire-groups", {+ method: "DELETE",+ headers: { "content-type": "application/json" },+ body: JSON.stringify({ group }),+ }, `Stop retire waiting on ${group}?\n\nOnly do this for a consumer that no longer runs. Retire stops waiting for it, so a generation it has not read to the end can retire and be deleted. If it still runs, it refuses to start on its next restart.`);+ return button;+ },++ retireGroupForm() {+ const groupInput = el("input", { type: "text", placeholder: "consumer group, e.g. personhog-changelog-relay", style: "width: 360px" });+ const button = el("button", {}, "Add retire group");+ button.onclick = () => {+ const group = groupInput.value.trim();+ if (!group) return;+ this.postChangelogAction("api/personhog/changelog/retire-groups", {+ method: "POST",+ headers: { "content-type": "application/json" },+ body: JSON.stringify({ group }),+ }, `Add ${group} as a retire group?\n\nRetire then also waits for this group to read each generation to the end. A changelog consumer such as the relay refuses to start until its group is listed. Remove it from its row once that consumer no longer runs.`);+ };+ return el("div", { class: "row", style: "padding: 6px 14px; border-top: 1px solid var(--border)" },+ el("span", { class: "muted" }, "Other consumers retire waits on:"), groupInput, button);},async postChangelogAction(path, options, confirmText) {@@ -1538,18 +1586,19 @@: el("ul", { class: "mono", style: "margin: 4px 0" }, r.waiting.slice(0, 12).map((w) => el("li", {}, w))));}for (const g of view.draining) {- const w = g.writer;- const drained = !w.error && w.partitions > 0 && w.drained_partitions === w.partitions;+ const groups = [g.writer, ...g.retire_groups];+ const isDrained = (d) => !d.error && d.partitions > 0 && d.drained_partitions === d.partitions;+ const drained = groups.every(isDrained);const retire = el("button", { disabled: !drained }, `Retire ${g.topic}`);retire.onclick = () => this.postChangelogAction("api/personhog/changelog/retire", {method: "POST",headers: { "content-type": "application/json" },body: JSON.stringify({ topic: g.topic }),- }, `Retire ${g.topic}?\n\nThe writer group has read it to the end. It leaves the config; delete the topic in Terraform afterwards.`);+ }, `Retire ${g.topic}?\n\nEvery gating group has read it to the end. It leaves the config; delete the topic in Terraform afterwards.`);body.append(el("div", { class: "row", style: "margin-top: 6px" },el("span", { class: "pill muted mono" }, `${g.topic} (${fmtInt(g.partitions)}, epoch ${fmtInt(g.epoch)}) draining`),- el("span", { class: `pill ${w.error ? "bad" : drained ? "good" : "warn"}`, title: w.error || "" },- `${w.group}: ${fmtInt(w.drained_partitions)}/${fmtInt(w.partitions)}`),+ groups.map((d) => el("span", { class: `pill ${d.error ? "bad" : isDrained(d) ? "good" : "warn"}`, title: d.error || "" },+ `${d.group}: ${fmtInt(d.drained_partitions)}/${fmtInt(d.partitions)}`)),retire));}if (view.history.length > 0) {diff --git a/rust/personhog-coordination/README.md b/rust/personhog-coordination/README.mdindex a948781289e3..9daadeee7650 100644--- a/rust/personhog-coordination/README.md+++ b/rust/personhog-coordination/README.md@@ -125,7 +125,7 @@ Several failure modes are handled explicitly:## Changelog migrations-A changelog resize is a new topic and a cutover, never in-place growth. `config/changelog` (`{topic, partitions, epoch, writer_group, draining}`) is the one live object every participant captures by a strong read at acquisition, and `migrations/current` is the saga only the coordinator creates and advances.+A changelog resize is a new topic and a cutover, never in-place growth. `config/changelog` (`{topic, partitions, epoch, writer_group, retire_groups, draining}`) is the one live object every participant captures by a strong read at acquisition, and `migrations/current` is the saga only the coordinator creates and advances.The control plane only writes a request, `migrations/requested`. The coordinator starts the saga from it only while the fleet is quiet: no handoff record, and every old partition assigned to a registered, Ready pod. It waits, holding nothing, until then, and creates `migrations/current` in a transaction that requires no handoff record to exist at commit. Creating the saga record moves the planner's plan basis, so no handoff can begin after the start either. The owner half of quiet is checked when the request is evaluated, so an owner that begins shutting down in between is held like any other and reaches the freeze's `holder_died` cancel if it deregisters. The freeze also cancels the saga if it ever finds an ordinary handoff, before anything that handoff touches could be stranded; the quiet start is what keeps that from happening, so a start does not end in a cancel. Admission (the live config, the target topic, no generation still draining, no rollout) runs at the same step, and a request still waiting five minutes after the coordinator first saw it is rejected, with what it was waiting on, so a saga only starts while whoever asked is plausibly watching.@@ -133,7 +133,7 @@ Freezing holds every partition in place (a held self-handoff that never passes DThe ordinary planner places only when no saga holds placement: no migration, or a Resuming one whose holds are all restarted. Every coordinator plan carries the record it was decided on, and its transactions require that record unchanged, so a stalled pass that wakes after a successor acted lands nothing. The planner's record is its plan basis: the changelog config and the migration record, read in one transaction and compared by the config's mod revision (a flip or retire moves it) and the migration's create revision (a new or archived saga moves it; the driver's own writes do not). A migration pass's record is the saga at the mod revision it read, which any write to it moves: a cancel, a phase change, a blocking-list write, the archive. A saga cancelled before its flip and archived leaves the planner's basis as it was, and a plan landing after it meets the world it was planned for.-A leader serving a partition holds in its cache every acked write for that partition's keys that Postgres does not have yet, so a restart warms from every source that can hold one: for each draining generation, the old partitions congruent to the new one modulo `gcd(old, new)`, from the writer's committed offset to their sealed end and filtered to the keys that hash to the new partition, then the live partition itself. Marks from an old source keep their own stream (`topic`, source partition), so recovery fetches them from the old topic and the prune settles them against the writer's offsets there. A mark also names the cache partition it belongs to, and a release or prune of one partition only removes the marks and lifecycle fences homed there: after a flip, an old partition and a new one on the same pod share keys, and either may be released or pruned first. Nothing in the saga waits for the writer: it drains the oldest undrained generation first, flushes a sealed partition's tail as soon as it reaches the end, and re-subscribes in place once the group has drained the topic; its offsets carry their topic, since a resubscribe reuses the partition numbers. The control plane's retire op removes a generation from `draining` once the writer group has read each partition to its end; the config watch delivers a retire as it delivers a flip, and a leader settles the marks on a retired stream without the writer's offsets. Deleting the topic stays a Terraform change.+A leader serving a partition holds in its cache every acked write for that partition's keys that Postgres does not have yet, so a restart warms from every source that can hold one: for each draining generation, the old partitions congruent to the new one modulo `gcd(old, new)`, from the writer's committed offset to their sealed end and filtered to the keys that hash to the new partition, then the live partition itself. Marks from an old source keep their own stream (`topic`, source partition), so recovery fetches them from the old topic and the prune settles them against the writer's offsets there. A mark also names the cache partition it belongs to, and a release or prune of one partition only removes the marks and lifecycle fences homed there: after a flip, an old partition and a new one on the same pod share keys, and either may be released or pruned first. Nothing in the saga waits for the writer: it drains the oldest undrained generation first, flushes a sealed partition's tail as soon as it reaches the end, and re-subscribes in place once the group has drained the topic; its offsets carry their topic, since a resubscribe reuses the partition numbers. The control plane's retire op removes a generation from `draining` once the writer group and every group in `retire_groups`, such as the relay's, have read each partition to its end; the config watch delivers a retire as it delivers a flip, and a leader settles the marks on a retired stream without the writer's offsets. Deleting the topic stays a Terraform change.Routers park requests per key, so a key's requests keep one queue across the flip: a request goes straight to a leader only when its key has nothing parked or in flight and its partition is not stashing, and the live count decides only which partition's drain takes a key. A drain first reads the config from etcd, best effort, then takes the keys that map to its partition under the count the router has adopted; the leader bounces any key it took under an older one, and the reconcile pass drains every pending partition, with no owner when nothing names it; a write for a partition born in the flip parks until that partition's fresh handoff drains it. A router keeps a grow's floor past the retire until its own routing table has a route for every born partition, because its assignment and config watches are independent and the retire can arrive first. A leader answers a stale `x-partition` with `FAILED_PRECONDITION`, which the router treats as a bounce, so the write lands once the router's config watch delivers the flip; a router that has not adopted within its retries answers `UNAVAILABLE`, never an ack; a lifecycle batch with a member off the routed partition is refused with `INVALID_ARGUMENT`, which the identity service answers by retrying each member alone.diff --git a/rust/personhog-coordination/src/changelog.rs b/rust/personhog-coordination/src/changelog.rsindex 8a288a0118a3..194071661927 100644--- a/rust/personhog-coordination/src/changelog.rs+++ b/rust/personhog-coordination/src/changelog.rs@@ -369,6 +369,7 @@ mod tests {partitions,epoch: 9,writer_group: String::new(),+ retire_groups: vec![],draining: draining.into_iter().map(|(partitions, epoch)| Generation {diff --git a/rust/personhog-coordination/src/follow.rs b/rust/personhog-coordination/src/follow.rsnew file mode 100644index 000000000000..8823a56cf049--- /dev/null+++ b/rust/personhog-coordination/src/follow.rs@@ -0,0 +1,222 @@+//! A changelog consumer's single subscription follows `config/changelog`: the+//! oldest generation its group has not drained first, then the live topic.++use std::collections::HashSet;+use std::future::Future;+use std::ops::ControlFlow;+use std::sync::Arc;+use std::time::Duration;++use tokio_util::sync::CancellationToken;+use tracing::info;++use crate::changelog::{watch_changelog_config_loop, CHANGELOG_RECONCILE_INTERVAL};+use crate::store::PersonhogStore;+use crate::types::ChangelogConfig;++/// How often a consumer on a sealed generation re-checks its group's drain.+/// Each check reads every partition of the generation under a probe group.+pub const DRAIN_POLL_INTERVAL: Duration = Duration::from_secs(2);++/// The consumer side of a followed subscription.+pub trait ChangelogSubscription {+ fn topic(&self) -> String;+ /// Only called once the group has drained the current topic.+ fn resubscribe(&self, topic: &str) -> Result<(), String>;+ fn set_draining(&self, draining: bool);+ fn is_draining(&self) -> bool;+}++/// Chooses the subscription. A generation once proven drained is never chosen+/// again, so a slow or failed probe cannot send the consumer back to it.+#[derive(Default)]+pub struct Follower {+ drained: HashSet<String>,+}++impl Follower {+ /// The oldest draining generation not proven drained, else the live topic.+ pub async fn select_topic<F, Fut>(+ &mut self,+ changelog: &ChangelogConfig,+ drained: F,+ ) -> Result<String, String>+ where+ F: Fn(String) -> Fut,+ Fut: Future<Output = Result<bool, String>>,+ {+ let mut generations = changelog.draining.clone();+ generations.sort_by_key(|g| g.epoch);+ for generation in generations {+ if self.drained.contains(&generation.topic) {+ continue;+ }+ if !drained(generation.topic.clone()).await? {+ return Ok(generation.topic);+ }+ self.drained.insert(generation.topic);+ }+ Ok(changelog.topic.clone())+ }++ /// One pass of the follow loop; returns the topic it moved off, if it moved.+ /// The draining flag follows the config before the subscription is touched,+ /// so it holds whether or not the subscription moved.+ pub async fn follow_once<S, F, Fut>(+ &mut self,+ subscription: &S,+ changelog: &ChangelogConfig,+ drained: F,+ ) -> Result<Option<String>, String>+ where+ S: ChangelogSubscription + ?Sized,+ F: Fn(String) -> Fut,+ Fut: Future<Output = Result<bool, String>>,+ {+ let wanted = self.select_topic(changelog, &drained).await?;+ let subscribed = subscription.topic();+ // A generation can leave the config before this group drains it, when+ // retire stops waiting on the group; finish reading it before moving on.+ if subscribed != wanted+ && subscribed != changelog.topic+ && !self.drained.contains(&subscribed)+ {+ if !drained(subscribed.clone()).await? {+ subscription.set_draining(true);+ return Ok(None);+ }+ self.drained.insert(subscribed.clone());+ }+ subscription.set_draining(wanted != changelog.topic);+ if wanted == subscribed {+ return Ok(None);+ }+ subscription+ .resubscribe(&wanted)+ .map_err(|e| format!("re-subscribe to {wanted}: {e}"))?;+ info!(from = %subscribed, to = %wanted, "moved the subscription");+ Ok(Some(subscribed))+ }+}++/// Follows the config until `cancel`: re-checks on every config change, and on a+/// timer that is short while the subscription drains a sealed generation.+/// `on_move` sees each move as (from, to).+pub async fn follow_changelog<S, F, Fut, M>(+ store: Arc<PersonhogStore>,+ loaded: ChangelogConfig,+ mut follower: Follower,+ subscription: Arc<S>,+ drained: F,+ on_move: M,+ cancel: CancellationToken,+) where+ S: ChangelogSubscription + ?Sized,+ F: Fn(String) -> Fut,+ Fut: Future<Output = Result<bool, String>>,+ M: Fn(&str, &str),+{+ let (config_tx, mut config_rx) = tokio::sync::watch::channel(loaded.clone());+ let watch = watch_changelog_config_loop(+ store,+ loaded,+ CHANGELOG_RECONCILE_INTERVAL,+ cancel.clone(),+ move |next| {+ drop(config_tx.send(next.clone()));+ ControlFlow::Continue(())+ },+ );+ let follow = async {+ loop {+ let current = config_rx.borrow().clone();+ match follower+ .follow_once(subscription.as_ref(), ¤t, &drained)+ .await+ {+ Ok(Some(from)) => on_move(&from, &subscription.topic()),+ Ok(None) => {}+ Err(e) => tracing::warn!(+ error = %e,+ "could not follow the changelog config; keeping the subscription"+ ),+ }+ let poll = if subscription.is_draining() {+ DRAIN_POLL_INTERVAL+ } else {+ CHANGELOG_RECONCILE_INTERVAL+ };+ tokio::select! {+ changed = config_rx.changed() => {+ if changed.is_err() {+ return;+ }+ }+ _ = tokio::time::sleep(poll) => {}+ }+ }+ };+ tokio::select! {+ _ = cancel.cancelled() => {}+ _ = watch => {}+ _ = follow => {}+ }+}++#[cfg(test)]+mod tests {+ use std::sync::atomic::{AtomicBool, Ordering};+ use std::sync::Mutex;++ use super::*;++ struct Subscription {+ topic: Mutex<String>,+ draining: AtomicBool,+ }++ impl ChangelogSubscription for Subscription {+ fn topic(&self) -> String {+ self.topic.lock().unwrap().clone()+ }++ fn resubscribe(&self, topic: &str) -> Result<(), String> {+ *self.topic.lock().unwrap() = topic.to_string();+ Ok(())+ }++ fn set_draining(&self, draining: bool) {+ self.draining.store(draining, Ordering::SeqCst);+ }++ fn is_draining(&self) -> bool {+ self.draining.load(Ordering::SeqCst)+ }+ }++ #[tokio::test]+ async fn a_generation_retired_before_the_group_drained_it_is_read_to_the_end() {+ let subscription = Subscription {+ topic: Mutex::new("old".to_string()),+ draining: AtomicBool::new(true),+ };+ let live = ChangelogConfig::new("new", 8);+ let mut follower = Follower::default();++ let moved = follower+ .follow_once(&subscription, &live, |_| async { Ok(false) })+ .await+ .unwrap();+ assert_eq!(moved, None);+ assert_eq!(subscription.topic(), "old");+ assert!(subscription.is_draining());++ let moved = follower+ .follow_once(&subscription, &live, |_| async { Ok(true) })+ .await+ .unwrap();+ assert_eq!(moved, Some("old".to_string()));+ assert_eq!(subscription.topic(), "new");+ assert!(!subscription.is_draining());+ }+}diff --git a/rust/personhog-coordination/src/lib.rs b/rust/personhog-coordination/src/lib.rsindex 9bf795e257e8..7e19591f1bc2 100644--- a/rust/personhog-coordination/src/lib.rs+++ b/rust/personhog-coordination/src/lib.rs@@ -2,6 +2,7 @@ pub mod authority;pub mod changelog;pub mod coordinator;pub mod error;+pub mod follow;pub mod hash;pub mod migration;pub mod pod;diff --git a/rust/personhog-coordination/src/store/migration.rs b/rust/personhog-coordination/src/store/migration.rsindex 3d68d210ec5f..3e076c1a4340 100644--- a/rust/personhog-coordination/src/store/migration.rs+++ b/rust/personhog-coordination/src/store/migration.rs@@ -151,6 +151,7 @@ impl PersonhogStore {partitions: migration.to.partitions,epoch: config.epoch + 1,writer_group: config.writer_group.clone(),+ retire_groups: config.retire_groups.clone(),draining,};let mut record = migration.clone();@@ -234,14 +235,67 @@ impl PersonhogStore {Ok(self.inner.list(&key).await?)}- /// Drop a drained generation from the config, guarded on its revision.- pub async fn retire_changelog_topic(&self, topic: &str) -> Result<bool> {- count_call("retire_changelog_topic");+ /// Add a consumer group that retire waits on: false when the config is not+ /// seeded or moved. A group the config already gates on is a no-op.+ pub async fn add_changelog_retire_group(&self, group: &str) -> Result<bool> {+ count_call("add_changelog_retire_group");let Some((mut config, mod_revision)) =self.get_changelog_config_with_mod_revision().await?else {return Ok(false);};+ if config.gated_groups().any(|g| g == group) {+ return Ok(true);+ }+ config.retire_groups.push(group.to_string());+ self.put_changelog_config_if_unchanged(&config, mod_revision)+ .await+ }++ /// Stop retire waiting on a group: false when the config is not seeded or+ /// moved. A group the list does not hold is a no-op.+ pub async fn remove_changelog_retire_group(&self, group: &str) -> Result<bool> {+ count_call("remove_changelog_retire_group");+ let Some((mut config, mod_revision)) =+ self.get_changelog_config_with_mod_revision().await?+ else {+ return Ok(false);+ };+ let before = config.retire_groups.len();+ config.retire_groups.retain(|g| g != group);+ if config.retire_groups.len() == before {+ return Ok(true);+ }+ self.put_changelog_config_if_unchanged(&config, mod_revision)+ .await+ }++ async fn put_changelog_config_if_unchanged(+ &self,+ config: &ChangelogConfig,+ mod_revision: i64,+ ) -> Result<bool> {+ let key = self.key(StoreKey::ChangelogConfig);+ let txn = Txn::new()+ .when(vec![Compare::mod_revision(+ key.clone(),+ CompareOp::Equal,+ mod_revision,+ )])+ .and_then(vec![TxnOp::put(key, serde_json::to_vec(config)?, None)]);+ Ok(self.inner.txn(txn).await?.succeeded())+ }++ /// Drop a drained generation from the config the caller's drain checks+ /// read, only while that config is still current.+ pub async fn retire_changelog_topic(+ &self,+ config: &ChangelogConfig,+ mod_revision: i64,+ topic: &str,+ ) -> Result<bool> {+ count_call("retire_changelog_topic");+ let mut config = config.clone();let before = config.draining.len();config.draining.retain(|g| g.topic != topic);if config.draining.len() == before {diff --git a/rust/personhog-coordination/src/types.rs b/rust/personhog-coordination/src/types.rsindex 25e2e54be77f..f4c7d1aad5df 100644--- a/rust/personhog-coordination/src/types.rs+++ b/rust/personhog-coordination/src/types.rs@@ -327,6 +327,10 @@ pub struct ChangelogConfig {pub epoch: u64,/// The writer's consumer group; a generation retires only once it has read it to the end.pub writer_group: String,+ /// Other consumer groups of the changelog, such as the relay's; a+ /// generation retires only once each of them has read it to the end too.+ #[serde(default, skip_serializing_if = "Vec::is_empty")]+ pub retire_groups: Vec<String>,/// Generations some consumer may still be reading.pub draining: Vec<Generation>,}@@ -346,10 +350,17 @@ impl ChangelogConfig {partitions,epoch: 1,writer_group: String::new(),+ retire_groups: Vec::new(),draining: Vec::new(),}}+ /// Every consumer group a draining generation waits on before it retires.+ pub fn gated_groups(&self) -> impl Iterator<Item = &str> {+ std::iter::once(self.writer_group.as_str())+ .chain(self.retire_groups.iter().map(String::as_str))+ }+pub fn target(&self) -> ChangelogTarget {ChangelogTarget {topic: self.topic.clone(),@@ -357,9 +368,11 @@ impl ChangelogConfig {}}- /// A flip raises the epoch. Within an epoch only a retire changes the- /// object, and a retire only shrinks `draining`, so a same-epoch object- /// with more generations is an older one delivered late.+ /// A flip raises the epoch. Within an epoch a retire shrinks `draining`, so+ /// a same-epoch object with more generations is an older one delivered+ /// late. A `retire_groups` edit also changes the object within an epoch+ /// and is never adopted from a watch; nothing reads that list from an+ /// adopted copy, since the coordinator and the control plane read etcd.pub fn supersedes(&self, adopted: &ChangelogConfig) -> bool {self.epoch > adopted.epoch|| (self.epoch == adopted.epoch && self.draining.len() < adopted.draining.len())diff --git a/rust/personhog-coordination/tests/changelog.rs b/rust/personhog-coordination/tests/changelog.rsindex bc304f617a93..282c77bfa8a0 100644--- a/rust/personhog-coordination/tests/changelog.rs+++ b/rust/personhog-coordination/tests/changelog.rs@@ -103,6 +103,7 @@ async fn the_watch_delivers_newer_epochs_and_retires_but_nothing_stale() {partitions: 128,epoch: 2,writer_group: String::new(),+ retire_groups: vec![],draining: Vec::new(),};store.put_changelog_config(&flipped).await.unwrap();@@ -191,6 +192,7 @@ async fn a_severed_watch_still_adopts_the_flip_through_reconcile() {partitions: 128,epoch: 2,writer_group: String::new(),+ retire_groups: vec![],draining: Vec::new(),}).awaitdiff --git a/rust/personhog-coordination/tests/migration.rs b/rust/personhog-coordination/tests/migration.rsindex 1754ba2e35b2..4a1068df0e6a 100644--- a/rust/personhog-coordination/tests/migration.rs+++ b/rust/personhog-coordination/tests/migration.rs@@ -178,6 +178,7 @@ async fn an_acquisition_parks_until_the_config_reaches_the_handoffs_epoch() {partitions: PARTITIONS,epoch: 2,writer_group: String::new(),+ retire_groups: vec![],draining: Vec::new(),}).await@@ -353,6 +354,12 @@ async fn a_grow_migration_restarts_every_partition_over_the_new_changelog() {first.clone(),);wait_for_steady_state(&store, PARTITIONS).await;+ for group in ["relay", "relay", "writer", "gone"] {+ assert!(store.add_changelog_retire_group(group).await.unwrap());+ }+ for group in ["gone", "never-added"] {+ assert!(store.remove_changelog_retire_group(group).await.unwrap());+ }assert!(store.create_migration(&migration("m-1", 8)).await.unwrap());wait_for_phase(&store, MigrationPhase::Sealing).await;@@ -383,6 +390,7 @@ async fn a_grow_migration_restarts_every_partition_over_the_new_changelog() {assert_eq!(config.partitions, 8);assert_eq!(config.epoch, 2);assert_eq!(config.writer_group, "writer");+ assert_eq!(config.retire_groups, vec!["relay".to_string()]);assert_eq!(config.draining,vec![Generation {@@ -412,7 +420,26 @@ async fn a_grow_migration_restarts_every_partition_over_the_new_changelog() {"partition {partition} drained again for its restart: {events:?}");}- assert!(store.retire_changelog_topic(TOPIC).await.unwrap());+ // A retire lands only on the config its drain checks read.+ let (config, rev) = store+ .get_changelog_config_with_mod_revision()+ .await+ .unwrap()+ .unwrap();+ store.put_changelog_config(&config).await.unwrap();+ assert!(!store+ .retire_changelog_topic(&config, rev, TOPIC)+ .await+ .unwrap());+ let (config, rev) = store+ .get_changelog_config_with_mod_revision()+ .await+ .unwrap()+ .unwrap();+ assert!(store+ .retire_changelog_topic(&config, rev, TOPIC)+ .await+ .unwrap());assert!(store.get_changelog_config().await@@ -647,6 +674,7 @@ async fn a_request_that_fails_admission_is_rejected_with_its_reason() {partitions: PARTITIONS,epoch: 1,writer_group: "writer".to_string(),+ retire_groups: vec![],draining: vec![Generation {topic: "personhog_updates_test_v0".to_string(),partitions: PARTITIONS,@@ -771,6 +799,7 @@ async fn a_plan_lands_only_on_the_basis_it_was_read_on() {partitions: 8,epoch: 2,writer_group: String::new(),+ retire_groups: vec![],draining: vec![],};store.put_changelog_config(&flipped).await.unwrap();diff --git a/rust/personhog-leader/src/coordination/mod.rs b/rust/personhog-leader/src/coordination/mod.rsindex 3a38f9db4eeb..6c6b38db9273 100644--- a/rust/personhog-leader/src/coordination/mod.rs+++ b/rust/personhog-leader/src/coordination/mod.rs@@ -640,6 +640,7 @@ mod tests {partitions,epoch,writer_group: String::new(),+ retire_groups: vec![],draining: Vec::new(),};for (partition, cfg) in [(5, config("old", 8, 1)), (1, config("new", 4, 2))] {diff --git a/rust/personhog-leader/src/settle.rs b/rust/personhog-leader/src/settle.rsindex e2887235c611..6bcbbe4484c8 100644--- a/rust/personhog-leader/src/settle.rs+++ b/rust/personhog-leader/src/settle.rs@@ -182,6 +182,7 @@ mod tests {partitions: 8,epoch: 2,writer_group: writer_group.to_string(),+ retire_groups: vec![],draining: draining.iter().map(|t| Generation {diff --git a/rust/personhog-router/tests/stash_integration.rs b/rust/personhog-router/tests/stash_integration.rsindex a520a580ac17..b566f1de432a 100644--- a/rust/personhog-router/tests/stash_integration.rs+++ b/rust/personhog-router/tests/stash_integration.rs@@ -248,6 +248,7 @@ async fn born_partition_write_parks(retired_first: bool) {partitions: NUM_PARTITIONS,epoch: 2,writer_group: String::new(),+ retire_groups: vec![],draining: vec![Generation {topic: "personhog_updates_v1".to_string(),partitions: old_count,diff --git a/rust/personhog-stateright/src/model.rs b/rust/personhog-stateright/src/model.rsindex 57ff4d3699a2..879bac6308fd 100644--- a/rust/personhog-stateright/src/model.rs+++ b/rust/personhog-stateright/src/model.rs@@ -92,6 +92,7 @@ fn production_config(config: &Config) -> ChangelogConfig {partitions: config.live.partitions as u32,epoch: config.live.epoch as u64,writer_group: String::new(),+ retire_groups: vec![],draining: config.draining.map(|g| ProductionGeneration {diff --git a/rust/personhog-test-harness/src/stack/mod.rs b/rust/personhog-test-harness/src/stack/mod.rsindex 4c27a0030a86..b0b27d40b6cb 100644--- a/rust/personhog-test-harness/src/stack/mod.rs+++ b/rust/personhog-test-harness/src/stack/mod.rs@@ -1345,8 +1345,15 @@ impl Stack {saga_end.elapsed().as_secs_f64(),drained_at.elapsed().as_secs_f64());+ let (live, mod_revision) = self+ .store+ .get_changelog_config_with_mod_revision()+ .await?+ .context("config/changelog is not seeded")?;ensure!(- self.store.retire_changelog_topic(&old_topic).await?,+ self.store+ .retire_changelog_topic(&live, mod_revision, &old_topic)+ .await?,"{old_topic} was not listed as draining");}diff --git a/rust/personhog-writer/src/follow.rs b/rust/personhog-writer/src/follow.rsdeleted file mode 100644index 8e87a2557d5e..000000000000--- a/rust/personhog-writer/src/follow.rs+++ /dev/null@@ -1,69 +0,0 @@-//! The subscription follows `config/changelog`: the oldest generation the-//! group has not drained first, then the live topic.--use std::collections::HashSet;-use std::future::Future;--use metrics::counter;-use personhog_coordination::types::ChangelogConfig;-use tracing::info;--use crate::kafka::PersonConsumer;--/// Chooses the subscription. A generation once proven drained is never chosen-/// again, so a slow or failed probe cannot send the writer back to it.-#[derive(Default)]-pub struct Follower {- drained: HashSet<String>,-}--impl Follower {- /// The oldest draining generation not proven drained, else the live topic.- pub async fn select_topic<F, Fut>(- &mut self,- changelog: &ChangelogConfig,- drained: F,- ) -> Result<String, String>- where- F: Fn(String) -> Fut,- Fut: Future<Output = Result<bool, String>>,- {- let mut generations = changelog.draining.clone();- generations.sort_by_key(|g| g.epoch);- for generation in generations {- if self.drained.contains(&generation.topic) {- continue;- }- if !drained(generation.topic.clone()).await? {- return Ok(generation.topic);- }- self.drained.insert(generation.topic);- }- Ok(changelog.topic.clone())- }-- /// One pass of the follow loop. The draining flag follows the config before the- /// subscription is touched, so it holds whether or not the subscription moved.- pub async fn follow_once<F, Fut>(- &mut self,- consumer: &PersonConsumer,- changelog: &ChangelogConfig,- drained: F,- ) -> Result<(), String>- where- F: Fn(String) -> Fut,- Fut: Future<Output = Result<bool, String>>,- {- let wanted = self.select_topic(changelog, drained).await?;- consumer.set_draining(wanted != changelog.topic);- let subscribed = consumer.topic();- if wanted != subscribed {- consumer- .resubscribe(wanted.clone())- .map_err(|e| format!("re-subscribe to {wanted}: {e}"))?;- info!(from = %subscribed, to = %wanted, "moved the subscription");- counter!("personhog_writer_resubscriptions_total").increment(1);- }- Ok(())- }-}diff --git a/rust/personhog-writer/src/kafka.rs b/rust/personhog-writer/src/kafka.rsindex ae07952d9180..cccfe759abb0 100644--- a/rust/personhog-writer/src/kafka.rs+++ b/rust/personhog-writer/src/kafka.rs@@ -4,6 +4,7 @@ use std::sync::{Arc, Mutex, RwLock};use common_kafka::config::KafkaConfig;use metrics::counter;+use personhog_coordination::follow::ChangelogSubscription;use rdkafka::consumer::{BaseConsumer, CommitMode, Consumer, ConsumerContext, Rebalance, StreamConsumer,};@@ -262,3 +263,21 @@ impl PersonConsumer {self.topic.read().expect("topic lock poisoned").clone()}}++impl ChangelogSubscription for PersonConsumer {+ fn topic(&self) -> String {+ PersonConsumer::topic(self)+ }++ fn resubscribe(&self, topic: &str) -> Result<(), String> {+ PersonConsumer::resubscribe(self, topic.to_string()).map_err(|e| e.to_string())+ }++ fn set_draining(&self, draining: bool) {+ PersonConsumer::set_draining(self, draining)+ }++ fn is_draining(&self) -> bool {+ PersonConsumer::is_draining(self)+ }+}diff --git a/rust/personhog-writer/src/lib.rs b/rust/personhog-writer/src/lib.rsindex 26a50e12fb60..3997678bd150 100644--- a/rust/personhog-writer/src/lib.rs+++ b/rust/personhog-writer/src/lib.rs@@ -2,7 +2,6 @@ pub mod buffer;pub mod config;pub mod consumer;pub mod error;-pub mod follow;pub mod kafka;pub mod pg;pub mod store;diff --git a/rust/personhog-writer/src/main.rs b/rust/personhog-writer/src/main.rsindex 1db2b08c858d..2bbbb3d34a0d 100644--- a/rust/personhog-writer/src/main.rs+++ b/rust/personhog-writer/src/main.rs@@ -1,4 +1,3 @@-use std::ops::ControlFlow;use std::sync::Arc;use std::time::Duration;@@ -8,10 +7,9 @@ use common_database::{get_pool_with_config, PoolConfig};use common_metrics::{setup_metrics_routes_with_overrides, Matcher};use envconfig::Envconfig;use lifecycle::{ComponentOptions, Manager};-use personhog_coordination::changelog::{- load_changelog_config, watch_changelog_config_loop, CHANGELOG_BOOT_POLL_INTERVAL,- CHANGELOG_RECONCILE_INTERVAL,-};+use metrics::counter;+use personhog_coordination::changelog::{load_changelog_config, CHANGELOG_BOOT_POLL_INTERVAL};+use personhog_coordination::follow::{follow_changelog, Follower};use personhog_coordination::store::PersonhogStore;use tokio::sync::mpsc;use tokio_util::sync::CancellationToken;@@ -23,7 +21,6 @@ use tracing_subscriber::EnvFilter;use personhog_writer::config::Config;use personhog_writer::consumer::ConsumerTask;-use personhog_writer::follow::Follower;use personhog_writer::kafka::{base_client_config, PersonConsumer};use personhog_writer::pg::PgStore;use personhog_writer::store::PersonWriteStore;@@ -246,42 +243,17 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {tokio::spawn(async move {let _guard = handle.process_scope();let cancel = CancellationToken::new();- let (config_tx, mut config_rx) = tokio::sync::watch::channel(loaded.clone());- let watch = watch_changelog_config_loop(+ let follow = follow_changelog(store,- loaded.clone(),- CHANGELOG_RECONCILE_INTERVAL,+ loaded,+ follower,+ consumer,+ |topic: String| drained(&config, topic),+ |_, _| counter!("personhog_writer_resubscriptions_total").increment(1),cancel.clone(),- move |next| {- drop(config_tx.send(next.clone()));- ControlFlow::Continue(())- },);- let probe = |topic: String| drained(&config, topic);- let follow = async {- loop {- let current = config_rx.borrow().clone();- if let Err(e) = follower.follow_once(&consumer, ¤t, &probe).await {- tracing::warn!(error = %e, "could not follow the changelog config; keeping the subscription");- }- let poll = if consumer.is_draining() {- DRAIN_POLL_INTERVAL- } else {- CHANGELOG_RECONCILE_INTERVAL- };- tokio::select! {- changed = config_rx.changed() => {- if changed.is_err() {- return;- }- }- _ = tokio::time::sleep(poll) => {}- }- }- };tokio::select! {_ = handle.shutdown_signal() => cancel.cancel(),- _ = watch => {}_ = follow => {}}});@@ -291,10 +263,6 @@ async fn main() -> Result<(), Box<dyn std::error::Error>> {Ok(())}-/// How often a writer on a sealed generation re-checks the group's drain.-/// Each check reads every partition of the generation under a probe group.-const DRAIN_POLL_INTERVAL: Duration = Duration::from_secs(2);-/// Whether the writer group has read every partition of `topic` to its end.async fn drained(config: &Config, topic: String) -> Result<bool, String> {let base = base_client_config(&config.kafka);diff --git a/rust/personhog-writer/tests/integration.rs b/rust/personhog-writer/tests/integration.rsindex c4030f812f62..79766bbed2f6 100644--- a/rust/personhog-writer/tests/integration.rs+++ b/rust/personhog-writer/tests/integration.rs@@ -9,11 +9,11 @@ use common::{cleanup_team, create_local_kafka_producer, create_mock_kafka, create_test_pool, make_person,unique_team_id, KAFKA_BOOTSTRAP, TARGET_TABLE, TOPIC,};+use personhog_coordination::follow::Follower;use personhog_coordination::types::{ChangelogConfig, Generation};use personhog_proto::personhog::types::v1::Person;use personhog_writer::buffer::BufferedPerson;use personhog_writer::consumer::{ConsumerTask, FlushBatch};-use personhog_writer::follow::Follower;use personhog_writer::kafka::{PersonConsumer, Source};use personhog_writer::pg::PgStore;use personhog_writer::store::{@@ -1161,6 +1161,7 @@ async fn the_draining_flag_follows_the_subscribed_topic_not_the_move() {partitions: 1,epoch: 2,writer_group: "test-follow".to_string(),+ retire_groups: vec![],draining: vec![Generation {topic: TOPIC.to_string(),partitions: 1,