Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

112 KB
Raw Blame
//! Broker-enforced epoch fencing for the changelog, via per-partition
//! transactional producers.
//!
//! Each partition's owner holds one transactional producer per lane. Taking
//! ownership initializes every lane's id, which bumps the broker-side
//! epochs and fences every predecessor: a zombie owner's next produce or
//! commit fails at the broker instead of landing.
//!
//! Lanes exist because the coordinator holds a producer's next transaction
//! until its previous commit finishes, so consecutive windows rotate across
//! lanes; a partition still runs one window at a time. Each write keeps its
//! person's lock through the ack, so ordering is unchanged.
//!
//! Writes are grouped into transaction windows rather than one
//! transaction each. A transactional producer admits one open
//! transaction at a time, so per-write transactions would serialize a
//! partition's writes on the commit round trip. A window opens on the
//! first write, admits concurrent writes (their records ride
//! librdkafka's normal batching), then commits once and resolves every
//! waiter. The window closes on whichever comes first: its timer, which
//! bounds ack latency at light load, or its fill threshold, so a
//! backlog commits at once instead of idling out the timer. Under light
//! load a window holds one write and costs one commit; under load the
//! commit amortizes across the window and a drain cycle is bounded by
//! the commit round trip rather than the window.
//!
//! The price of sharing a transaction is shared failure: an aborted
//! window fails all of its writes together. Readers never observe
//! aborted records (consumers run `read_committed`), so the coupling is
//! visible only as grouped retryable errors.

#[cfg(any(test, feature = "test-support"))]
use std::sync::atomic::AtomicUsize;
use std::sync::atomic::{AtomicBool, AtomicU64, Ordering};
use std::sync::{Arc, Mutex, PoisonError};
use std::time::{Duration, Instant};
use std::{fmt, mem};

use common_kafka::config::KafkaConfig;
use common_kafka::transaction::{ConnectedTransactionalProducer, TransactionalProducer};
use personhog_coordination::types::ChangelogTarget;

use crate::producer_stats::FencedProducerContext;

/// The fenced producers report their statistics through one context.
type ConnectedFencedProducer = ConnectedTransactionalProducer<FencedProducerContext>;
type FencedProducer = TransactionalProducer<FencedProducerContext>;
use dashmap::mapref::entry::Entry;
use dashmap::DashMap;
use metrics::{counter, histogram};
use prost::Message as ProtoMessage;
use rdkafka::client::ClientContext;
use rdkafka::error::{KafkaError, RDKafkaErrorCode};
use rdkafka::producer::{FutureProducer, FutureRecord, Producer};
use tokio::sync::{oneshot, Notify};
use tokio::task::spawn_blocking;
use tokio::time::{sleep, timeout};
use tracing::{debug, error, warn};

use personhog_proto::personhog::types::v1::Person;

use personhog_coordination::authority::AuthorityClock;

use crate::config::{FENCING_ABORT_ATTEMPTS, FENCING_COMMIT_ATTEMPTS};
use crate::inflight::InflightTracker;
use crate::kafka::changelog_message_key;
use crate::warming::changelog_kafka;

/// The fencing scope is the partition: every owner of partition `p`
/// shares its lane ids, so a new owner's init fences the old one.
fn transactional_id(topic: &str, partition: u32, lane: usize) -> String {
    format!("personhog-changelog-{topic}-p{partition}-l{lane}")
}

#[derive(Debug)]
pub enum FencedProduceError {
    /// This pod holds no fenced producer for the partition — ownership
    /// was never taken or was released.
    NotAcquired,
    /// The broker fenced this producer: a newer owner has initialized
    /// the partition's transactional id. This pod's claim is stale, and
    /// the records demonstrably never became visible.
    Fenced,
    /// Fenced, but the window's own outcome was never settled — so the
    /// partition has moved *and* whether these records landed is unknown.
    ///
    /// The two are independent facts and the caller needs both: the
    /// router still wants the ownership answer, while the version must
    /// stay spent because a commit that was re-issued may have succeeded
    /// on an earlier attempt. Collapsing this into `Fenced` is what let a
    /// committed record's version be handed back for reuse.
    FencedUncertain(String),
    /// Send or commit failed for an ordinary, retryable reason; the
    /// window was aborted and no record became visible.
    Failed(String),
    /// The commit neither succeeded nor demonstrably aborted, so whether
    /// the records became visible is unknown.
    ///
    /// This is not the same as failure and must not be reported as one.
    /// A caller told "aborted" retries against a cache still holding the
    /// pre-write version, and produces a second record carrying the same
    /// version as the one that may already have committed — which the
    /// writer's strict version guard resolves in favour of whichever
    /// arrived first, discarding the acked one.
    Indeterminate(String),
}

impl fmt::Display for FencedProduceError {
    fn fmt(&self, f: &mut fmt::Formatter<'_>) -> fmt::Result {
        match self {
            FencedProduceError::NotAcquired => write!(f, "partition fence not acquired"),
            FencedProduceError::Fenced => {
                write!(f, "producer fenced by a newer owner of the partition")
            }
            FencedProduceError::FencedUncertain(e) => write!(
                f,
                "producer fenced by a newer owner; this window's outcome is unknown: {e}"
            ),
            FencedProduceError::Failed(e) => write!(f, "fenced produce failed: {e}"),
            FencedProduceError::Indeterminate(e) => {
                write!(f, "changelog commit outcome unknown: {e}")
            }
        }
    }
}

/// State of a partition's current transaction window.
struct Gate {
    /// A window is open and accepting joiners.
    open: bool,
    /// Sends still in flight for the open window; the committer waits
    /// for zero before committing so no record can miss its own commit.
    in_flight: usize,
    /// A send in this window failed; the committer aborts instead of
    /// committing, and every waiter fails together.
    poisoned: bool,
    /// The committer is between closing the window and finishing the
    /// commit. Once it drains `in_flight` and takes `waiters`, the other
    /// fields read as idle even though `commit_transaction` is still
    /// running — this flag keeps the next window from beginning until
    /// the producer is actually free.
    committing: bool,
    /// Commit-outcome subscribers for the open window.
    waiters: Vec<oneshot::Sender<Result<(), FencedProduceError>>>,
    /// Writes admitted to the open window. Counted against the fill
    /// threshold; unlike `in_flight` it never decrements, so a window
    /// whose early writes settle quickly still fills.
    joined: usize,
    /// Fires the open window's fill close. Armed at open, consumed by
    /// the seat that reaches the threshold, cleared when the window
    /// closes.
    fill_tx: Option<oneshot::Sender<()>>,
}

impl Gate {
    /// Count a seat toward the fill threshold, closing the window early
    /// when it is reached. A close trigger, not a hard cap: seats taken
    /// while the close propagates still ride the window.
    fn admit(&mut self, fill_threshold: usize) {
        self.joined += 1;
        if self.joined >= fill_threshold {
            if let Some(tx) = self.fill_tx.take() {
                let _ = tx.send(());
            }
        }
    }
}

/// Producers a queue budget is divided among: one per lane per partition.
pub fn fenced_producer_count(partitions: u32, lanes: usize) -> u32 {
    let lanes = u32::try_from(lanes.max(1)).unwrap_or(u32::MAX);
    partitions.max(1).saturating_mul(lanes)
}

/// One producer's share of an aggregate queue limit, never below 1.
pub fn queue_share(aggregate: u32, producers: u32) -> u32 {
    (aggregate / producers.max(1)).max(1)
}

struct Dial {
    kafka: KafkaConfig,
    topic: String,
    partition: u32,
    timeout: Duration,
    broker_txn_timeout: Duration,
    #[cfg(any(test, feature = "test-support"))]
    attempts: Arc<AtomicUsize>,
    #[cfg(any(test, feature = "test-support"))]
    hold: Arc<DialHold>,
}

#[cfg(any(test, feature = "test-support"))]
#[derive(Default)]
struct DialHold {
    held: Mutex<bool>,
    released: std::sync::Condvar,
}

/// Connect one lane's producer, with no transactional state yet.
fn connect_lane(dial: &Dial, lane: usize) -> Result<ConnectedFencedProducer, String> {
    #[cfg(any(test, feature = "test-support"))]
    dial.attempts.fetch_add(1, Ordering::SeqCst);
    #[cfg(any(test, feature = "test-support"))]
    {
        let held = dial
            .hold
            .held
            .lock()
            .unwrap_or_else(PoisonError::into_inner);
        drop(
            dial.hold
                .released
                .wait_while(held, |held| *held)
                .unwrap_or_else(PoisonError::into_inner),
        );
    }
    let tid = transactional_id(&dial.topic, dial.partition, lane);
    ConnectedFencedProducer::connect_bounded_with_context(
        &dial.kafka,
        &tid,
        dial.timeout,
        dial.broker_txn_timeout,
        FencedProducerContext::default(),
    )
    .map_err(|e| format!("lane {lane} connect: {e}"))
}

/// Claim one lane's id, which fences its previous owners. A parked
/// connection whose init fails gets one fresh connect-and-init rather
/// than failing the claim, and sets `fell_back`.
fn claim_lane(
    dial: &Dial,
    lane: usize,
    parked: Option<ConnectedFencedProducer>,
    fell_back: &AtomicBool,
) -> Result<FencedProducer, String> {
    if let Some(parked) = parked {
        match parked.init(dial.timeout) {
            Ok(ready) => return Ok(ready),
            Err((e, _)) => {
                fell_back.store(true, Ordering::Relaxed);
                warn!(
                    partition = dial.partition,
                    lane,
                    error = %e,
                    "prepared connection failed to init; connecting fresh"
                );
            }
        }
    }
    connect_lane(dial, lane)?
        .init(dial.timeout)
        .map_err(|(e, _)| format!("lane {lane} init: {e}"))
}

/// One blocking job per lane on its own thread; results in lane order.
fn per_lane<I: Send, T: Send>(inputs: Vec<I>, job: impl Fn(usize, I) -> T + Sync) -> Vec<T> {
    let job = &job;
    std::thread::scope(|scope| {
        let handles: Vec<_> = inputs
            .into_iter()
            .enumerate()
            .map(|(lane, input)| scope.spawn(move || job(lane, input)))
            .collect();
        handles
            .into_iter()
            .map(|handle| handle.join().expect("a lane job must not panic"))
            .collect()
    })
}

/// Send a fence to the blocking pool to die. Dropping the last
/// reference runs librdkafka's destroy, which blocks while the client
/// tears down — up to hundreds of milliseconds — and both removal sites
/// run on async workers. Best-effort: a caller (an in-flight write, the
/// settle wait) still holding the Arc pays the destroy wherever it drops
/// last, which is rarer than the common case this moves.
fn drop_off_worker<T: Send + 'static>(value: T) {
    tokio::task::spawn_blocking(move || drop(value));
}

/// Carries a value a cancelled future would otherwise drop on a runtime
/// worker, and sends it to the blocking pool instead.
struct DropOffWorker<T: Send + 'static>(Option<T>);

impl<T: Send + 'static> DropOffWorker<T> {
    fn take(mut self) -> T {
        self.0.take().expect("taken once")
    }
}

impl<T: Send + 'static> Drop for DropOffWorker<T> {
    fn drop(&mut self) {
        if let Some(value) = self.0.take() {
            drop_off_worker(value);
        }
    }
}

/// Stamps a lane when its commit finishes, so selection can order lanes.
static COMMIT_CLOCK: AtomicU64 = AtomicU64::new(0);

struct PartitionFence {
    producer: FencedProducer,
    lane: usize,
    /// The commit clock at this lane's last finished commit; zero until then.
    last_commit_end: AtomicU64,
    /// The partition's turn, shared by its lanes; see [`PartitionLanes::turn`].
    turn: Arc<Mutex<()>>,
    /// Writers waiting on this lane's window to close.
    #[cfg(any(test, feature = "test-support"))]
    parked: AtomicUsize,
    /// Set by a heal before it re-acquires, so a fence this lane then
    /// reports counts as the pod's own doing rather than a new owner's.
    superseded_by_heal: AtomicBool,
    /// Makes the next commit task panic, so tests can reach the arm that
    /// handles a committer which never reports. Scoped to the fence
    /// rather than a global: the tests in this binary run concurrently.
    #[cfg(any(test, feature = "test-support"))]
    panic_next_commit: AtomicBool,
    /// A std mutex, deliberately: every critical section is a handful of
    /// field updates with no await inside, and `WindowSlot`'s `Drop` must
    /// be able to release its seat synchronously.
    gate: Mutex<Gate>,
    /// Signalled when `in_flight` reaches zero.
    sends_settled: Notify,
    /// Signalled when a committing window finishes, letting blocked
    /// writers open the next one.
    window_closed: Notify,
    /// Set when the producer is left in a transaction state no later
    /// `begin_transaction` can recover from — an abort that did not land,
    /// or a commit whose outcome stayed unknown.
    ///
    /// Such a producer is still installed, so presence alone cannot tell
    /// a working fence from a dead one. This flag is what makes the
    /// distinction visible: a condemned partition answers as unowned, so
    /// the router bounces instead of retrying a dead producer, and
    /// `holds()` reports the fence missing — which is what lets
    /// `heal_fence`, on the next convergence to Serving, re-take the
    /// epoch once the pod's claim is confirmed.
    unusable: AtomicBool,
    commit_timeout: Duration,
    /// Clone of the map's repair nudge; a condemnation must be able to
    /// announce itself from wherever the commit task discovers it.
    repair_nudge: Option<Arc<Notify>>,
}

impl PartitionFence {
    /// Retire the producer: writes stop attempting it and the partition
    /// stops counting as fenced by this pod.
    fn condemn(&self, partition: u32, reason: &'static str) {
        if !self.unusable.swap(true, Ordering::Relaxed) {
            counter!(
                "personhog_leader_fence_condemned_total",
                "reason" => reason
            )
            .increment(1);
            error!(
                partition,
                lane = self.lane,
                reason,
                "changelog producer left unusable; awaiting re-acquisition"
            );
            // The only way back is a heal on a convergence to Serving, and
            // the reconcile tick that would otherwise carry it is seconds
            // away — writes bounce for that whole gap. The nudge lets the
            // coordination loop run its repair pass now; Notify stores
            // the permit, so a nudge landing before the loop listens is
            // not lost.
            if let Some(nudge) = &self.repair_nudge {
                nudge.notify_one();
            }
        }
    }

    fn is_usable(&self) -> bool {
        !self.unusable.load(Ordering::Relaxed)
    }

    fn fence_cause(&self) -> &'static str {
        if self.superseded_by_heal.load(Ordering::Relaxed) {
            "heal"
        } else {
            "takeover"
        }
    }

    fn stamp_commit_end(&self) {
        self.last_commit_end.store(
            COMMIT_CLOCK.fetch_add(1, Ordering::Relaxed) + 1,
            Ordering::Relaxed,
        );
    }
}

/// A partition's transactional producers. The coordinator refuses a
/// producer's next transaction for a while after it commits, so a write
/// takes the lane least likely to be held.
struct PartitionLanes {
    topic: String,
    lanes: Vec<Arc<PartitionFence>>,
    /// Serializes lane decisions: a writer holds it from reading the gates
    /// to moving one, and a committer while it clears `committing`, so a
    /// lane seen idle stays idle until the writer has chosen.
    turn: Arc<Mutex<()>>,
}

impl PartitionLanes {
    /// One condemned lane condemns the partition: repair re-acquires
    /// whole partitions, which aborts the other lanes' open windows, and
    /// their writers answer as fenced and retry.
    fn is_usable(&self) -> bool {
        self.lanes.iter().all(|lane| lane.is_usable())
    }

    fn contains(&self, fence: &Arc<PartitionFence>) -> bool {
        self.lanes.iter().any(|lane| Arc::ptr_eq(lane, fence))
    }

    /// The lane a write takes, or none when a gate is poisoned: that lane
    /// is condemned, so the partition heals like any dead producer.
    fn pick(&self, partition: u32) -> Option<Arc<PartitionFence>> {
        let mut states = Vec::with_capacity(self.lanes.len());
        for lane in &self.lanes {
            let Ok(gate) = lane.gate.lock() else {
                lane.condemn(partition, "gate_poisoned");
                return None;
            };
            states.push(LaneState {
                open: gate.open,
                committing: gate.committing,
                last_commit_end: lane.last_commit_end.load(Ordering::Relaxed),
            });
        }
        Some(Arc::clone(&self.lanes[pick_lane(&states)]))
    }
}

#[derive(Clone, Copy, Debug)]
struct LaneState {
    open: bool,
    committing: bool,
    last_commit_end: u64,
}

/// One window at a time per partition: join an open one, park behind a
/// committing one, else open on the lane whose last commit is oldest.
/// Concurrent commits on one partition stretch the coordinator's hold.
fn pick_lane(states: &[LaneState]) -> usize {
    let oldest = |wanted: fn(&LaneState) -> bool| {
        states
            .iter()
            .enumerate()
            .filter(|(_, state)| wanted(state))
            .map(|(index, state)| (state.last_commit_end, index))
            .min()
            .map(|(_, index)| index)
    };
    oldest(|state| state.open)
        .or_else(|| oldest(|state| state.committing))
        .or_else(|| oldest(|_| true))
        .unwrap_or(0)
}

/// One seat in the open window, released on drop.
///
/// A produce future can be dropped at any await — a task unwinding, the
/// runtime shutting down, a caller giving up — and the seat has to come
/// back even then, or the committer waits on an in-flight count that
/// never reaches zero and every later write on the partition parks
/// forever behind it.
struct WindowSlot {
    fence: Arc<PartitionFence>,
    released: bool,
}

impl WindowSlot {
    /// Take a seat in a window already counted as in-flight.
    fn held(fence: Arc<PartitionFence>) -> Self {
        Self {
            fence,
            released: false,
        }
    }

    /// Release the seat, optionally poisoning the window, and report
    /// whether the window is now settled.
    fn release_inner(&mut self, poison: bool) -> bool {
        let mut gate = self.fence.gate.lock().unwrap();
        gate.in_flight -= 1;
        if poison {
            gate.poisoned = true;
        }
        let settled = gate.in_flight == 0;
        drop(gate);
        if settled {
            self.fence.sends_settled.notify_waiters();
        }
        self.released = true;
        settled
    }

    /// Subscribe to the window's commit outcome and release the seat in
    /// one critical section. Both must be atomic: if the committer could
    /// take the waiter list between them, this write's record would be
    /// committed with no one left to tell.
    fn subscribe(mut self) -> oneshot::Receiver<Result<(), FencedProduceError>> {
        let (tx, rx) = oneshot::channel();
        let mut gate = self.fence.gate.lock().unwrap();
        gate.waiters.push(tx);
        gate.in_flight -= 1;
        let settled = gate.in_flight == 0;
        drop(gate);
        if settled {
            self.fence.sends_settled.notify_waiters();
        }
        self.released = true;
        rx
    }

    /// Release the seat after a failed send: the window aborts, failing
    /// this write and its window-mates together.
    fn poison(mut self) {
        self.release_inner(true);
    }
}

impl Drop for WindowSlot {
    fn drop(&mut self) {
        if !self.released {
            // A dropped produce: release the seat but do not poison.
            // Its record may already be enqueued and will ride the
            // commit; nobody is waiting for the ack, and failing the
            // window would punish the writes that are still waiting.
            //
            // The record therefore becomes durable without ever being
            // acked, and it can still sit in an open window when the
            // partition moves. That is safe because the new owner
            // acquires this partition's transactional id before reading
            // the changelog: the abandoned window can no longer commit,
            // so the record is either already below the new owner's
            // cutoff or never visible at all.
            counter!("personhog_leader_fence_slots_abandoned_total").increment(1);
            self.release_inner(false);
        }
    }
}

/// Marks a window as committing, and clears the mark on drop.
///
/// Between closing a window and finishing its commit the gate reads as
/// idle — `in_flight` is zero and the waiters have been taken — so only
/// this mark keeps the next writer from beginning a transaction the
/// producer is not free for. If the committer unwinds while it is set,
/// nothing else ever clears it: every later write parks on
/// `window_closed` until its own deadline expires, forever, and no
/// repair path looks at a gate.
struct CommittingMark {
    fence: Arc<PartitionFence>,
    partition: u32,
}

impl CommittingMark {
    fn take(fence: Arc<PartitionFence>, partition: u32) -> Self {
        {
            let mut gate = fence.gate.lock().unwrap();
            gate.open = false;
            gate.committing = true;
            // The window is closed; an unfired fill sender is stale and
            // must not linger into the next window's gate state.
            gate.fill_tx = None;
        }
        Self { fence, partition }
    }
}

impl Drop for CommittingMark {
    fn drop(&mut self) {
        // Stamped and cleared in one turn, so a writer that finds this
        // lane idle also finds it the newest commit.
        let turn = self
            .fence
            .turn
            .lock()
            .unwrap_or_else(PoisonError::into_inner);
        self.fence.stamp_commit_end();
        // Deliberately tolerant of a poisoned gate: the mark existing is
        // what wedges the partition, so it has to come off even when the
        // lock's last holder panicked.
        let orphans = match self.fence.gate.lock() {
            Ok(mut gate) => {
                gate.committing = false;
                mem::take(&mut gate.waiters)
            }
            Err(poisoned) => {
                let mut gate = poisoned.into_inner();
                gate.committing = false;
                mem::take(&mut gate.waiters)
            }
        };
        drop(turn);
        // On the ordinary path the committer took the waiters before this
        // drops, so the list is empty. Waiters still here mean the
        // committer unwound between taking the mark and answering anyone
        // — their outcomes are unobservable, and leaving them queued
        // would park every later write forever: nothing can open a window
        // while waiters remain, and nothing would notify again. Dropping
        // their senders answers each with the doubt it earned (a closed
        // channel reads as an unobserved commit), and condemning routes
        // the partition into the same bounce-and-recover story as every
        // other producer no window can be begun from.
        if !orphans.is_empty() {
            self.fence.condemn(self.partition, "committer_unwound");
        }
        drop(orphans);
        self.fence.window_closed.notify_waiters();
    }
}

/// Holds a freshly acquired fence until the work that justified taking
/// it succeeds, and gives it back otherwise.
///
/// A warm can end without returning — its future is dropped when the
/// pod's coordination attempt is torn down, which is exactly what a lost
/// lease does. The pod records a partition as held only once the warm
/// returns, so a fence taken by a warm that never finishes belongs to no
/// partition the local self-fence knows to release: the process would
/// keep the partition's broker epoch while owning nothing, and the real
/// owner's writes would fail as fenced until it re-acquired.
pub struct FenceGuard {
    fenced: Arc<FencedChangelogProducers>,
    partition: u32,
    /// The fence this guard is answerable for. Releasing by partition
    /// alone would drop whatever happens to be installed at drop time,
    /// which after a release and a re-acquire is somebody else's
    /// producer — the same hazard `forget_fence` checks for.
    taken: Option<Arc<PartitionLanes>>,
    /// Names the work the fence was taken for, so an abandon names the
    /// path that dropped mid-flight.
    context: &'static str,
    armed: bool,
}

impl FenceGuard {
    pub fn new(
        fenced: Arc<FencedChangelogProducers>,
        partition: u32,
        context: &'static str,
    ) -> Self {
        let taken = fenced.installed(partition);
        Self {
            fenced,
            partition,
            taken,
            context,
            armed: true,
        }
    }

    /// The work succeeded; the fence is now the caller's to hold.
    pub fn keep(mut self) {
        self.armed = false;
    }
}

impl Drop for FenceGuard {
    fn drop(&mut self) {
        if self.armed {
            counter!("personhog_leader_fence_abandoned_total").increment(1);
            warn!(
                partition = self.partition,
                context = self.context,
                "releasing a fence taken for work that did not finish"
            );
            match &self.taken {
                Some(fence) => self.fenced.forget_fence(self.partition, fence),
                None => self.fenced.release(self.partition),
            }
        }
    }
}

/// A partition's slot in the pre-connect pipeline: claimed while a
/// dial is in flight, then holding the parked connection.
enum Prepared {
    /// A preconnect owns the slot and is dialing. Concurrent
    /// preconnects seeing this return immediately, so at most one dial
    /// runs per partition. The claim is what bounds connection churn:
    /// the trigger fires on every convergence pass through a drain
    /// window, and without it those passes stack concurrent dials
    /// until the pod runs out of memory.
    ///
    /// The token identifies the owning dial. A dial resolves only its
    /// own claim: a stale dial whose claim was removed mid-flight (by
    /// an acquire, a release, or the sweep) must neither usurp a
    /// replacement dial's claim on success nor release it on failure —
    /// either would let convergence start extra dials, which is the
    /// churn the claim exists to prevent.
    Connecting { token: u64, since: Instant },
    /// A finished dial, one connection per lane, parked for the next
    /// acquire to consume.
    Ready {
        topic: String,
        connected: Vec<ConnectedFencedProducer>,
    },
}

/// Per-partition fenced producers for the changelog. Constructed once
/// and shared; partitions are acquired at warm completion and released
/// with ownership.
pub struct FencedChangelogProducers {
    /// Queue limits here are the aggregate every producer shares (see `queue_share`).
    kafka: KafkaConfig,
    init_timeout: Duration,
    commit_timeout: Duration,
    /// How long the broker keeps one of our windows open before
    /// abandoning it; distinct from `commit_timeout`, which bounds only
    /// this process's wait on the commit call.
    broker_txn_timeout: Duration,
    /// How long an open window admits joiners before committing, when
    /// it does not fill first.
    window: Duration,
    /// Fill threshold: joiners that close the window early. Bounds a
    /// backlog's drain cycle by the commit round trip instead of the
    /// window.
    window_max_writes: usize,
    /// Ceiling on the drain's wait for an open window to commit. Set
    /// well under the pre-revoke self-fence's allowance for a whole
    /// drain, so a shutdown with an open window does not truncate here
    /// and report a failed drain.
    settle_budget: Duration,
    /// Transactional producers per partition.
    lanes: usize,
    /// Ceiling on one produce, past every stage's own timeout, so a
    /// wedged lane cannot hold a write's lock and in-flight slot forever.
    produce_bound: Duration,
    partitions: DashMap<u32, Arc<PartitionLanes>>,
    /// Nudged on condemnation, so the coordination loop can run a
    /// repair pass now instead of on its next reconcile tick. Carries no
    /// payload: the pass re-derives what needs converging, the same way
    /// the tick does. `None` leaves repair to the tick alone.
    repair_nudge: Option<Arc<Notify>>,
    /// Connected-but-uninitialized producers, one set per partition at
    /// most, parked by `preconnect` for a later `acquire` to claim. Connection
    /// setup is the slow half of acquisition and touches no broker
    /// transactional state, so it can run ahead of the authority
    /// transition; `init_transactions` — the fencing action — still
    /// happens only inside `acquire`.
    prepared: DashMap<u32, Prepared>,
    /// Distinguishes dials, so each resolves only its own claim.
    dial_token: AtomicU64,

    /// Dials attempted, so tests can pin single-flight: the storm this
    /// guards against is invisible through public behavior until the
    /// pod is already dying.
    #[cfg(any(test, feature = "test-support"))]
    connect_attempts: Arc<AtomicUsize>,
    #[cfg(any(test, feature = "test-support"))]
    dial_hold: Arc<DialHold>,
    #[cfg(any(test, feature = "test-support"))]
    dial_waits: AtomicUsize,

    /// Outcomes a test stages for the next produce on a partition.
    ///
    /// The uncertain outcomes need a broker fault landing inside a
    /// transaction, which no test can stage against a healthy cluster —
    /// but what the caller *does* with them is the reason they are
    /// distinguished at all, so the answers have to be reachable.
    #[cfg(any(test, feature = "test-support"))]
    staged_failures: DashMap<u32, FencedProduceError>,
}

/// Construction parameters for [`FencedChangelogProducers`], named
/// because five of them are `Duration`s: a transposed pair at a call
/// site compiles, and `main`'s wiring is executed by no test — a swapped
/// window and settle budget would ship as a 2s admission window on
/// every fenced write.
pub struct FencedProducerConfig {
    pub kafka: KafkaConfig,
    pub init_timeout: Duration,
    pub commit_timeout: Duration,
    pub broker_txn_timeout: Duration,
    pub window: Duration,
    pub window_max_writes: usize,
    pub settle_budget: Duration,
    pub lanes: usize,
}

impl FencedChangelogProducers {
    pub fn new(config: FencedProducerConfig) -> Self {
        let FencedProducerConfig {
            kafka,
            init_timeout,
            commit_timeout,
            broker_txn_timeout,
            window,
            window_max_writes,
            settle_budget,
            lanes,
        } = config;
        assert!(lanes >= 1, "a partition needs at least one lane");
        // Two queued writes' worth of window, send and commit attempts,
        // the same shape the config sizes the lease runway against.
        let produce_bound = (window + commit_timeout * (COMMIT_RETRIES as u32 + 2)) * 2;
        Self {
            kafka,
            init_timeout,
            commit_timeout,
            broker_txn_timeout,
            window,
            window_max_writes,
            settle_budget,
            lanes,
            produce_bound,
            partitions: DashMap::new(),
            repair_nudge: None,
            prepared: DashMap::new(),
            dial_token: AtomicU64::new(0),
            #[cfg(any(test, feature = "test-support"))]
            connect_attempts: Arc::new(AtomicUsize::new(0)),
            #[cfg(any(test, feature = "test-support"))]
            dial_hold: Arc::default(),
            #[cfg(any(test, feature = "test-support"))]
            dial_waits: AtomicUsize::new(0),
            #[cfg(any(test, feature = "test-support"))]
            staged_failures: DashMap::new(),
        }
    }

    /// Announce condemnations on `nudge`, so the coordination loop can
    /// run a repair pass immediately instead of on its next reconcile
    /// tick. The listening end is [`PodHandle::
    /// with_repair_nudge`](personhog_coordination::pod::PodHandle).
    pub fn with_repair_nudge(mut self, nudge: Arc<Notify>) -> Self {
        self.repair_nudge = Some(nudge);
        self
    }

    /// Take the partition's fence: claim every lane's id, which fences
    /// every previous owner of them. The claims run as one blocking task
    /// and their producers travel back in a guard, so a cancelled acquire
    /// never destroys a producer on a runtime worker.
    async fn acquire_installed(
        &self,
        partition: u32,
        target: &ChangelogTarget,
    ) -> Result<Arc<PartitionLanes>, String> {
        let topic = target.topic.as_str();
        let start = Instant::now();
        self.await_dial_in_flight(partition).await;
        // A claim still Connecting has outlived its dial; removing it discards the late result.
        let mut parked = self.take_prepared(partition, topic);
        let path = if parked.is_empty() {
            "cold"
        } else {
            "prepared"
        };
        parked.resize_with(self.lanes, || None);
        let dial = self.dial(partition, target);
        let claimed = spawn_blocking(move || {
            let mut producers = Vec::with_capacity(parked.len());
            let mut failure = None;
            let fell_back = AtomicBool::new(false);
            let results = per_lane(parked, |lane, parked| {
                claim_lane(&dial, lane, parked, &fell_back)
            });
            if fell_back.load(Ordering::Relaxed) {
                counter!("personhog_leader_fence_preconnect_total", "outcome" => "init_failed")
                    .increment(1);
            }
            for (lane, result) in results.into_iter().enumerate() {
                match result {
                    Ok(producer) => producers.push(producer),
                    Err(e) => {
                        warn!(partition, lane, error = %e, "lane claim failed");
                        failure.get_or_insert(e);
                    }
                }
            }
            match failure {
                Some(e) => Err(e),
                None => Ok(DropOffWorker(Some(producers))),
            }
        })
        .await;
        let producers = match claimed {
            Ok(Ok(producers)) => producers.take(),
            Ok(Err(e)) => {
                counter!("personhog_leader_fence_init_total", "outcome" => "error").increment(1);
                return Err(e);
            }
            Err(join) => {
                counter!("personhog_leader_fence_init_total", "outcome" => "error").increment(1);
                return Err(format!("fence claim join: {join}"));
            }
        };
        counter!("personhog_leader_fence_init_total", "outcome" => "ok").increment(1);
        histogram!("personhog_leader_fence_init_ms", "path" => path)
            .record(start.elapsed().as_secs_f64() * 1000.0);
        let turn = Arc::new(Mutex::new(()));
        let lanes = producers
            .into_iter()
            .enumerate()
            .map(|(lane, producer)| {
                Arc::new(PartitionFence {
                    producer,
                    lane,
                    last_commit_end: AtomicU64::new(0),
                    turn: Arc::clone(&turn),
                    #[cfg(any(test, feature = "test-support"))]
                    parked: AtomicUsize::new(0),
                    superseded_by_heal: AtomicBool::new(false),
                    gate: Mutex::new(Gate {
                        open: false,
                        in_flight: 0,
                        joined: 0,
                        fill_tx: None,
                        poisoned: false,
                        committing: false,
                        waiters: Vec::new(),
                    }),
                    sends_settled: Notify::new(),
                    window_closed: Notify::new(),
                    #[cfg(any(test, feature = "test-support"))]
                    panic_next_commit: AtomicBool::new(false),
                    unusable: AtomicBool::new(false),
                    commit_timeout: self.commit_timeout,
                    repair_nudge: self.repair_nudge.clone(),
                })
            })
            .collect();
        let installed = Arc::new(PartitionLanes {
            topic: topic.to_string(),
            lanes,
            turn,
        });
        if let Some(replaced) = self.partitions.insert(partition, Arc::clone(&installed)) {
            // The heal path installs over a still-present condemned
            // fence, and by then the commit task has usually dropped its
            // clones — making this insert the last reference and its
            // drop a blocking librdkafka destroy. Send it to the
            // blocking pool like every other eviction site.
            drop_off_worker(replaced);
        }
        Ok(installed)
    }

    /// Each producer's queue is its share of the aggregate under the generation's partition count.
    fn dial(&self, partition: u32, target: &ChangelogTarget) -> Dial {
        let producers = fenced_producer_count(target.partitions, self.lanes);
        let kafka = changelog_kafka(&self.kafka, target.cluster.as_ref());
        Dial {
            kafka: KafkaConfig {
                kafka_producer_queue_mib: queue_share(kafka.kafka_producer_queue_mib, producers),
                kafka_producer_queue_messages: queue_share(
                    kafka.kafka_producer_queue_messages,
                    producers,
                ),
                ..kafka
            },
            topic: target.topic.clone(),
            partition,
            timeout: self.init_timeout,
            broker_txn_timeout: self.broker_txn_timeout,
            #[cfg(any(test, feature = "test-support"))]
            attempts: Arc::clone(&self.connect_attempts),
            #[cfg(any(test, feature = "test-support"))]
            hold: Arc::clone(&self.dial_hold),
        }
    }

    /// Wait for the first dial seen to finish, up to its own bound, so a claim reuses its connection.
    async fn await_dial_in_flight(&self, partition: u32) {
        let mut followed = None;
        let mut waited = false;
        loop {
            let dial = self
                .prepared
                .get(&partition)
                .and_then(|slot| match slot.value() {
                    Prepared::Connecting { token, since } => Some((*token, *since)),
                    Prepared::Ready { .. } => None,
                });
            let first = *followed.get_or_insert(dial);
            match (dial, first) {
                (Some((token, since)), Some((first_token, _)))
                    if token == first_token && since.elapsed() < self.init_timeout =>
                {
                    #[cfg(any(test, feature = "test-support"))]
                    if !waited {
                        self.dial_waits.fetch_add(1, Ordering::SeqCst);
                    }
                    waited = true;
                    sleep(Duration::from_millis(10)).await;
                }
                _ => break,
            }
        }
        if waited {
            counter!("personhog_leader_fence_dial_waits_total").increment(1);
        }
    }

    /// The partition's parked connections for `topic`, removing its slot; one
    /// parked for another topic is discarded.
    fn take_prepared(&self, partition: u32, topic: &str) -> Vec<Option<ConnectedFencedProducer>> {
        match self.prepared.remove(&partition) {
            Some((
                _,
                Prepared::Ready {
                    topic: parked_topic,
                    connected,
                },
            )) if parked_topic == topic => connected.into_iter().map(Some).collect(),
            Some((_, Prepared::Ready { connected, .. })) => {
                counter!("personhog_leader_fence_preconnect_total", "outcome" => "stale_topic")
                    .increment(1);
                spawn_blocking(move || drop(connected));
                Vec::new()
            }
            _ => Vec::new(),
        }
    }

    /// Take the partition's fence, discarding the handle. The caller
    /// relies on the map rather than on holding the fence itself. Ids are
    /// topic-scoped: a new generation fences nothing on the old one.
    pub async fn acquire(&self, partition: u32, target: &ChangelogTarget) -> Result<(), String> {
        self.acquire_installed(partition, target).await.map(|_| ())
    }

    /// Claim and drop each lane's id; a dial in flight is waited for, not duplicated.
    pub async fn seal(
        &self,
        partition: u32,
        target: &ChangelogTarget,
    ) -> Result<&'static str, String> {
        let topic = target.topic.as_str();
        let start = Instant::now();
        self.await_dial_in_flight(partition).await;
        let mut parked = self.take_prepared(partition, topic);
        let path = if parked.is_empty() {
            "cold"
        } else {
            "prepared"
        };
        parked.resize_with(self.lanes, || None);
        let dial = self.dial(partition, target);
        let claimed = spawn_blocking(move || {
            let fell_back = AtomicBool::new(false);
            let results = per_lane(parked, |lane, parked| {
                claim_lane(&dial, lane, parked, &fell_back)
            });
            let mut producers = Vec::with_capacity(results.len());
            let mut failure = None;
            for result in results {
                match result {
                    Ok(producer) => producers.push(producer),
                    Err(e) => {
                        failure.get_or_insert(e);
                    }
                }
            }
            // The ids are claimed once init returns; the destroy that
            // follows takes hundreds of milliseconds and nobody waits for it.
            drop_off_worker(producers);
            match failure {
                Some(e) => Err(e),
                None => Ok(()),
            }
        })
        .await
        .map_err(|join| format!("seal join: {join}"))?;
        histogram!("personhog_leader_fence_seal_ms", "path" => path)
            .record(start.elapsed().as_secs_f64() * 1000.0);
        counter!(
            "personhog_leader_fence_seal_total",
            "outcome" => if claimed.is_ok() { "ok" } else { "error" }
        )
        .increment(1);
        claimed.map(|()| path)
    }

    /// Mark the installed lanes as about to be fenced by this pod's own
    /// re-acquisition, so the fences their writers then see count as a
    /// heal rather than a takeover.
    pub fn expect_self_fence(&self, partition: u32) {
        if let Some(lanes) = self.installed(partition) {
            for lane in &lanes.lanes {
                lane.superseded_by_heal.store(true, Ordering::Relaxed);
            }
        }
    }

    /// The lanes currently installed for a partition, if any.
    fn installed(&self, partition: u32) -> Option<Arc<PartitionLanes>> {
        self.partitions.get(&partition).map(|f| Arc::clone(&f))
    }

    #[cfg(any(test, feature = "test-support"))]
    fn lane(&self, partition: u32, lane: usize) -> Option<Arc<PartitionFence>> {
        self.installed(partition)
            .map(|lanes| Arc::clone(&lanes.lanes[lane]))
    }

    /// Drop the partition's fence with ownership. The broker-side epoch
    /// survives; only a future owner's init advances it.
    pub fn release(&self, partition: u32) {
        if let Some((_, lanes)) = self.partitions.remove(&partition) {
            drop_off_worker(lanes);
        }
        self.discard_prepared(partition);
    }

    /// Connect the partition's producers ahead of acquisition, so the
    /// acquire that follows pays only the init round trip. Runs the
    /// connections on the blocking pool and parks them; a failure is only
    /// logged and counted, because the cold acquire path covers it.
    /// Single-flight per partition: the slot is claimed before the dial
    /// begins, so however often convergence fires this through a drain
    /// window, one dial runs and the rest return as coalesced.
    pub async fn preconnect(&self, partition: u32, target: &ChangelogTarget) {
        // The claim must precede the dial, and the entry guard must not
        // be held across it: the guard holds a shard lock.
        let token = self.dial_token.fetch_add(1, Ordering::Relaxed);
        match self.prepared.entry(partition) {
            Entry::Occupied(_) => {
                // Another preconnect is dialing or already parked a
                // connection; either way this call has nothing to add.
                counter!("personhog_leader_fence_preconnect_total", "outcome" => "coalesced")
                    .increment(1);
                return;
            }
            Entry::Vacant(slot) => {
                slot.insert(Prepared::Connecting {
                    token,
                    since: Instant::now(),
                });
            }
        }
        let start = Instant::now();
        let dial = self.dial(partition, target);
        let lanes = self.lanes;
        let dialed = spawn_blocking(move || {
            // All or nothing: each connection carries its own lane's id.
            let mut connected = Vec::with_capacity(lanes);
            for result in per_lane(vec![(); lanes], |lane, ()| connect_lane(&dial, lane)) {
                connected.push(result?);
            }
            Ok(connected)
        })
        .await
        .unwrap_or_else(|join| Err(format!("fence dial join: {join}")));
        match dialed {
            Ok(connected) => {
                counter!("personhog_leader_fence_preconnect_total", "outcome" => "ok").increment(1);
                histogram!("personhog_leader_fence_preconnect_ms")
                    .record(start.elapsed().as_secs_f64() * 1000.0);
                self.park(partition, token, &target.topic, connected);
            }
            Err(e) => {
                counter!("personhog_leader_fence_preconnect_total", "outcome" => "error")
                    .increment(1);
                warn!(partition, error = %e, "fence preconnect failed; acquisition will connect cold");
                // Release the claim so a later preconnect can retry —
                // but only this dial's own claim. A replacement dial's
                // claim, or a parked connection, stays.
                self.prepared.remove_if(
                    &partition,
                    |_, slot| matches!(slot, Prepared::Connecting { token: t, .. } if *t == token),
                );
            }
        }
    }

    /// Park a finished dial, unless its claim is gone (a release, a
    /// sweep, or an acquire that went cold discarded it): a discarded
    /// partition must not resurrect, so the late connection is dropped
    /// instead. The token check makes ownership exact — a stale dial
    /// finding a replacement dial's claim here must not usurp it, or
    /// the replacement's own park would discard the newer connection
    /// while later convergence passes see whatever raced in.
    fn park(
        &self,
        partition: u32,
        token: u64,
        topic: &str,
        connected: Vec<ConnectedFencedProducer>,
    ) {
        match self.prepared.entry(partition) {
            Entry::Occupied(mut slot) if matches!(slot.get(), Prepared::Connecting { token: t, .. } if *t == token) =>
            {
                slot.insert(Prepared::Ready {
                    topic: topic.to_string(),
                    connected,
                });
            }
            _ => {
                counter!("personhog_leader_fence_preconnect_total", "outcome" => "discarded")
                    .increment(1);
                spawn_blocking(move || drop(connected));
            }
        }
    }

    /// Drop a parked connection on the blocking pool — librdkafka's
    /// client teardown blocks, same as a full fence's. A Connecting
    /// claim is removed without a drop: its dial discards on
    /// completion once the claim is gone.
    fn discard_prepared(&self, partition: u32) {
        if let Some((_, Prepared::Ready { connected, .. })) = self.prepared.remove(&partition) {
            spawn_blocking(move || drop(connected));
        }
    }

    /// Discard every parked connection. A connection is normally
    /// consumed within its drain window, seconds after parking; one
    /// still here at the periodic sweep belongs to an acquire that went
    /// cold first or to a cancelled inbound handoff — and a cancelled
    /// inbound handoff leaves no convergence behind to notice it, so
    /// the sweep is the only owner its lifetime has. A drain window
    /// that happens to straddle the sweep tick loses its head start and
    /// acquires cold, which is the behavior this whole path improves on
    /// rather than a failure.
    pub fn sweep_prepared(&self) {
        // Every live dial is bounded by the init timeout, so a claim
        // twice that old has no owner left to resolve it (its task
        // panicked or was torn down). Clearing it un-wedges preconnect
        // for the partition; correctness never depended on the claim,
        // acquisition just goes cold.
        let stale_claim_bound = self.init_timeout * 2;
        let now = Instant::now();
        let slots: Vec<u32> = self.prepared.iter().map(|entry| *entry.key()).collect();
        for partition in slots {
            // remove_if re-checks the slot under the shard lock, so a
            // dial that parks between the scan and this removal is not
            // swept as a stale claim.
            match self.prepared.remove_if(&partition, |_, slot| match slot {
                Prepared::Ready { .. } => true,
                Prepared::Connecting { since, .. } => {
                    now.duration_since(*since) > stale_claim_bound
                }
            }) {
                Some((_, Prepared::Ready { connected, .. })) => {
                    counter!("personhog_leader_fence_preconnect_total", "outcome" => "swept")
                        .increment(1);
                    warn!(
                        partition,
                        "swept a parked changelog connection nothing consumed"
                    );
                    spawn_blocking(move || drop(connected));
                }
                Some((_, Prepared::Connecting { .. })) => {
                    counter!("personhog_leader_fence_preconnect_total", "outcome" => "orphaned")
                        .increment(1);
                    warn!(partition, "cleared an orphaned preconnect claim");
                }
                None => {}
            }
        }
    }

    /// Whether a parked connection exists for the partition.
    #[cfg(any(test, feature = "test-support"))]
    pub fn has_prepared(&self, partition: u32) -> bool {
        self.prepared
            .get(&partition)
            .is_some_and(|slot| matches!(*slot, Prepared::Ready { .. }))
    }

    /// How many dials have been attempted.
    #[cfg(any(test, feature = "test-support"))]
    pub fn connect_attempts_for_test(&self) -> usize {
        self.connect_attempts.load(Ordering::SeqCst)
    }

    #[cfg(any(test, feature = "test-support"))]
    pub fn hold_dials_for_test(&self) {
        *self
            .dial_hold
            .held
            .lock()
            .unwrap_or_else(PoisonError::into_inner) = true;
    }

    #[cfg(any(test, feature = "test-support"))]
    pub fn release_dials_for_test(&self) {
        *self
            .dial_hold
            .held
            .lock()
            .unwrap_or_else(PoisonError::into_inner) = false;
        self.dial_hold.released.notify_all();
    }

    #[cfg(any(test, feature = "test-support"))]
    pub fn dial_waits_for_test(&self) -> usize {
        self.dial_waits.load(Ordering::SeqCst)
    }

    /// Stage a Connecting claim of a given age, standing in for a dial
    /// whose task died without resolving it.
    #[cfg(any(test, feature = "test-support"))]
    pub fn stage_connecting_for_test(&self, partition: u32, age: Duration) {
        let token = self.dial_token.fetch_add(1, Ordering::Relaxed);
        self.prepared.insert(
            partition,
            Prepared::Connecting {
                token,
                since: Instant::now() - age,
            },
        );
    }

    /// Whether a Connecting claim holds the partition's slot.
    #[cfg(any(test, feature = "test-support"))]
    pub fn has_connecting_claim_for_test(&self, partition: u32) -> bool {
        self.prepared
            .get(&partition)
            .is_some_and(|slot| matches!(*slot, Prepared::Connecting { .. }))
    }

    /// Whether this pod holds a *usable* fence for the partition.
    ///
    /// Deliberately not mere presence. A condemned producer is still
    /// installed, and answering "yes" for one would tell the repair pass
    /// that a partition it must re-acquire needs nothing — which is
    /// exactly how such a partition stayed unwritable until a handoff
    /// moved it.
    pub fn holds(&self, partition: u32) -> bool {
        self.partitions
            .get(&partition)
            .is_some_and(|lanes| lanes.is_usable())
    }

    /// Commit the partition's open window before its owner gives it up.
    ///
    /// A request cancelled mid-produce takes its handler — and the
    /// drain's in-flight count — with it, leaving the record it enqueued
    /// in a window nobody is waiting on. Left alone, that window's fate
    /// falls to whichever acts first: this pod's own committer, or the
    /// successor's `init_transactions` aborting it at the broker. Waiting
    /// here settles it in the successor's favour, so a cancelled client's
    /// changes land instead of being discarded by a race.
    ///
    /// Deliberately best-effort. Refusing to hand over a partition whose
    /// window did not settle would buy nothing: a producer that cannot
    /// report its outcome is a producer the broker will not accept
    /// another record from, so the records cannot grow after this point
    /// whatever we do — and the successor's init aborts what is left,
    /// exactly as it did before this wait existed. What refusing *does*
    /// cost is the partition: the drain has already fenced writes, and
    /// there is no branch that un-fences one whose handoff never
    /// completed.
    pub async fn settle(&self, partition: u32) {
        let Some(lanes) = self.installed(partition) else {
            // A write that met a condemned producer already gave the
            // fence up, which is the ordinary shape after a condemnation
            // — recorded rather than silent, or the series has no
            // denominator.
            counter!("personhog_leader_fence_settle_total", "outcome" => "absent").increment(1);
            return;
        };
        // Bounded well under the broker's own patience for the window.
        // The wait exists to catch a committer that is about to fire
        // anyway — it sleeps for the window, five milliseconds by
        // default, then commits — so seconds of budget only help when
        // the commit is retrying, which is the case where the
        // successor's abort is the right answer regardless. The cap
        // matters because the pre-revoke self-fence allows three seconds
        // for a whole drain: a budget derived from the lease alone
        // reaches about 6s at the production TTL, so every shutdown with an
        // open window would truncate here and report a failed drain.
        let budget = self.settle_budget;
        // Every lane waits under the one budget at once, so a lane stuck
        // in its commit cannot spend it before the others are looked at.
        let settled: Vec<AtomicBool> = lanes.lanes.iter().map(|_| AtomicBool::new(false)).collect();
        let waited = timeout(budget, async {
            let waits = lanes
                .lanes
                .iter()
                .zip(&settled)
                .map(|(fence, settled)| async move {
                    loop {
                        // Register before inspecting, or a close landing
                        // between the two is lost and this waits on a wakeup
                        // already spent.
                        let closed = fence.window_closed.notified();
                        tokio::pin!(closed);
                        closed.as_mut().enable();
                        {
                            let gate = fence.gate.lock().unwrap_or_else(PoisonError::into_inner);
                            if !gate.open && !gate.committing && gate.in_flight == 0 {
                                break;
                            }
                        }
                        closed.await;
                    }
                    settled.store(true, Ordering::Relaxed);
                });
            futures::future::join_all(waits).await;
        })
        .await;
        let unsettled: Vec<usize> = settled
            .iter()
            .enumerate()
            .filter(|(_, settled)| !settled.load(Ordering::Relaxed))
            .map(|(lane, _)| lane)
            .collect();

        let outcome = if waited.is_err() {
            "timeout"
        } else if !lanes.is_usable() {
            // Condemned somewhere in this partition's history, so this
            // producer has nothing left to give the changelog.
            "unusable"
        } else {
            "settled"
        };
        counter!("personhog_leader_fence_settle_total", "outcome" => outcome).increment(1);
        if outcome != "settled" {
            warn!(
                partition,
                outcome,
                unsettled_lanes = ?unsettled,
                "changelog window did not settle before the drain; leaving what remains to the \
                 incoming owner's init"
            );
        }
    }

    /// Put the partition's producer into the state a failed abort or an
    /// unknown commit leaves it in.
    ///
    /// Reaching that state for real takes a broker fault landing inside a
    /// transaction, which no test can stage against a healthy cluster —
    /// but what happens *afterwards* is the entire reason the state is
    /// tracked, so the aftermath has to be reachable.
    /// Hold the partition's gate in the state a commit in flight leaves
    /// it, so a writer arriving now parks instead of opening a window.
    ///
    /// The wake path is otherwise unreachable from a test: parking
    /// requires arriving inside the commit's own round trip, which is
    /// milliseconds wide and not something a test can aim at. Staging the
    /// gate directly is what makes the interleaving deterministic rather
    /// than statistical.
    #[cfg(any(test, feature = "test-support"))]
    pub fn begin_committing_for_test(&self, partition: u32, lane: usize) {
        if let Some(fence) = self.lane(partition, lane) {
            let mut gate = fence.gate.lock().unwrap();
            gate.open = false;
            gate.committing = true;
        }
    }

    /// Release that gate, stamp the lane's commit clock, and wake the
    /// parked writers, as a finishing commit does.
    #[cfg(any(test, feature = "test-support"))]
    pub fn finish_committing_for_test(&self, partition: u32, lane: usize) {
        if let Some(fence) = self.lane(partition, lane) {
            {
                let _turn = fence.turn.lock().unwrap_or_else(PoisonError::into_inner);
                fence.stamp_commit_end();
                fence.gate.lock().unwrap().committing = false;
            }
            fence.window_closed.notify_waiters();
        }
    }

    /// Close the lane's open window now, as its fill threshold would.
    #[cfg(any(test, feature = "test-support"))]
    pub fn close_window_for_test(&self, partition: u32, lane: usize) {
        if let Some(fence) = self.lane(partition, lane) {
            if let Some(fill) = fence.gate.lock().unwrap().fill_tx.take() {
                let _ = fill.send(());
            }
        }
    }

    /// Poison a lane's gate, as a panic under its lock would.
    #[cfg(any(test, feature = "test-support"))]
    pub fn poison_gate_for_test(&self, partition: u32, lane: usize) {
        if let Some(fence) = self.lane(partition, lane) {
            let poisoner = std::thread::spawn(move || {
                let _held = fence.gate.lock().unwrap();
                panic!("poisoning the gate");
            });
            assert!(poisoner.join().is_err(), "the poisoner must panic");
        }
    }

    /// Writers parked on a lane's window closing.
    #[cfg(any(test, feature = "test-support"))]
    pub fn parked_writers_for_test(&self, partition: u32, lane: usize) -> usize {
        self.lane(partition, lane)
            .map(|fence| fence.parked.load(Ordering::SeqCst))
            .unwrap_or(0)
    }

    /// Writers subscribed to a lane's open window for its commit outcome.
    #[cfg(any(test, feature = "test-support"))]
    pub fn waiting_writers_for_test(&self, partition: u32, lane: usize) -> usize {
        self.lane(partition, lane)
            .map(|fence| fence.gate.lock().unwrap().waiters.len())
            .unwrap_or(0)
    }

    /// Poison the partition's open window, as a send that failed inside
    /// it does.
    ///
    /// A real poisoning needs a produce to fail mid-window against a
    /// broker that is otherwise healthy enough to have opened one, which
    /// is not stageable — but what the commit path does with a poisoned
    /// window is the whole reason the flag exists.
    #[cfg(any(test, feature = "test-support"))]
    pub fn poison_window_for_test(&self, partition: u32) {
        if let Some(fence) = self.lane(partition, 0) {
            fence.gate.lock().unwrap().poisoned = true;
        }
    }

    /// Stage a committer that unwound between taking the mark and
    /// answering its subscribers — the shape a runtime teardown or a
    /// panic in the settle wait leaves behind. Real unwinds are not
    /// stageable deterministically; what matters is the mark's cleanup.
    #[cfg(any(test, feature = "test-support"))]
    pub fn orphan_committer_for_test(&self, partition: u32) {
        if let Some(fence) = self.lane(partition, 0) {
            let mark = CommittingMark::take(Arc::clone(&fence), partition);
            let (tx, _rx) = oneshot::channel();
            fence.gate.lock().unwrap().waiters.push(tx);
            drop(mark);
        }
    }

    /// Stage a committer that never reports its outcome. The real path
    /// needs the runtime to tear the task down mid-commit, which no test
    /// can arrange against a live producer.
    #[cfg(any(test, feature = "test-support"))]
    pub fn panic_next_commit_for_test(&self, partition: u32) {
        if let Some(fence) = self.lane(partition, 0) {
            fence.panic_next_commit.store(true, Ordering::SeqCst);
        }
    }

    /// Drop the open window's commit-outcome subscribers without
    /// answering them, as a committer that vanished mid-commit leaves
    /// them. `spawn_blocking` work is not cancelled when its handle is
    /// dropped, so the commit it was running may well have landed.
    #[cfg(any(test, feature = "test-support"))]
    pub fn abandon_waiters_for_test(&self, partition: u32) {
        if let Some(fence) = self.lane(partition, 0) {
            let taken = mem::take(&mut fence.gate.lock().unwrap().waiters);
            drop(taken);
        }
    }

    /// Make the next produce on this partition answer with `outcome`.
    #[cfg(any(test, feature = "test-support"))]
    pub fn stage_failure_for_test(&self, partition: u32, outcome: FencedProduceError) {
        self.staged_failures.insert(partition, outcome);
    }

    #[cfg(any(test, feature = "test-support"))]
    pub fn condemn_for_test(&self, partition: u32, lane: usize) {
        if let Some(fence) = self.lane(partition, lane) {
            fence.condemn(partition, "test");
        }
    }

    /// Produce on a specific lane, bypassing selection.
    #[cfg(any(test, feature = "test-support"))]
    pub async fn produce_on_lane_for_test(
        &self,
        partition: u32,
        lane: usize,
        person: &Person,
    ) -> Result<i64, FencedProduceError> {
        let lanes = self
            .installed(partition)
            .ok_or(FencedProduceError::NotAcquired)?;
        let fence = Arc::clone(&lanes.lanes[lane]);
        self.produce_on(partition, lanes, Some(fence), person).await
    }

    /// Each lane's commit clock stamp; zero for a lane that never
    /// committed.
    #[cfg(any(test, feature = "test-support"))]
    pub fn lane_commit_marks_for_test(&self, partition: u32) -> Vec<u64> {
        self.installed(partition)
            .map(|lanes| {
                lanes
                    .lanes
                    .iter()
                    .map(|lane| lane.last_commit_end.load(Ordering::Relaxed))
                    .collect()
            })
            .unwrap_or_default()
    }

    /// Produce one changelog record inside a transaction window on one of
    /// the partition's lanes, returning its offset once the window
    /// commits.
    pub async fn produce(
        &self,
        partition: u32,
        person: &Person,
    ) -> Result<i64, FencedProduceError> {
        #[cfg(any(test, feature = "test-support"))]
        if let Some((_, staged)) = self.staged_failures.remove(&partition) {
            return Err(staged);
        }
        let lanes = self.installed(partition).ok_or_else(|| {
            counter!("personhog_leader_kafka_produce_errors_total").increment(1);
            FencedProduceError::NotAcquired
        })?;
        match timeout(
            self.produce_bound,
            self.produce_on(partition, lanes, None, person),
        )
        .await
        {
            Ok(outcome) => outcome,
            // The record may be enqueued or committed by now, so the
            // outcome is unknown rather than failed.
            Err(_) => {
                counter!("personhog_leader_fence_produce_bound_exceeded_total").increment(1);
                error!(
                    partition,
                    bound_ms = self.produce_bound.as_millis() as u64,
                    "changelog produce exceeded its bound; outcome unknown"
                );
                Err(FencedProduceError::Indeterminate(
                    "produce exceeded its bound".to_string(),
                ))
            }
        }
    }

    /// Produce on the partition, picking a lane per attempt unless one is
    /// pinned.
    async fn produce_on(
        &self,
        partition: u32,
        lanes: Arc<PartitionLanes>,
        pinned: Option<Arc<PartitionFence>>,
        person: &Person,
    ) -> Result<i64, FencedProduceError> {
        let durable_start = Instant::now();

        // Join the open window, or open one. A window mid-commit admits
        // no joiners; wait for it to close and pick again, since the lane
        // that just committed is the one the coordinator is holding.
        let join_start = Instant::now();
        let (fence, opened) = loop {
            // Checked every iteration, not only on the way in. A
            // condemned producer cannot begin another transaction, and
            // the commit that condemns it is the same one whose end wakes
            // whoever is parked below — so a check placed after the park
            // is skipped by exactly the writer it exists for, which then
            // opens a window on a dead producer and answers with a
            // retryable failure instead of the ownership bounce that gets
            // the partition re-acquired.
            if !lanes.is_usable() {
                self.forget_fence(partition, &lanes);
                counter!("personhog_leader_kafka_produce_errors_total").increment(1);
                return Err(FencedProduceError::NotAcquired);
            }
            // Held until this writer has joined, opened, or decided to
            // park: no lane can open or finish a commit in between.
            let turn = lanes.turn.lock().unwrap_or_else(PoisonError::into_inner);
            let fence = match &pinned {
                Some(fence) => Arc::clone(fence),
                None => match lanes.pick(partition) {
                    Some(fence) => fence,
                    None => {
                        self.forget_fence(partition, &lanes);
                        counter!("personhog_leader_kafka_produce_errors_total").increment(1);
                        return Err(FencedProduceError::NotAcquired);
                    }
                },
            };
            // Register interest before inspecting the gate: a close that
            // fires between the check and the await must not be lost.
            let closed = fence.window_closed.notified();
            tokio::pin!(closed);
            closed.as_mut().enable();
            // Only the gate's own fields are touched under its lock.
            // Classifying a failure consults the producer's fatal state
            // and can evict the fence, so it happens after the guard is
            // gone: the gate is a std mutex, and a panic while it is held
            // poisons it for every writer on the partition.
            let begin_failed = {
                let mut gate = fence.gate.lock().unwrap();
                if gate.open {
                    gate.in_flight += 1;
                    gate.admit(self.window_max_writes);
                    counter!("personhog_leader_fence_lane_choices_total", "choice" => "join")
                        .increment(1);
                    break (Arc::clone(&fence), None);
                }
                if !gate.committing && gate.in_flight == 0 && gate.waiters.is_empty() {
                    // Idle: open a new window. BeginTxn is a local
                    // librdkafka state transition, safe inline.
                    match fence.producer.inner().begin_transaction() {
                        Ok(()) => {
                            let (fill_tx, fill_rx) = oneshot::channel();
                            gate.open = true;
                            gate.in_flight = 1;
                            gate.joined = 0;
                            gate.poisoned = false;
                            gate.fill_tx = Some(fill_tx);
                            gate.admit(self.window_max_writes);
                            counter!("personhog_leader_fence_lane_choices_total", "choice" => "open")
                                .increment(1);
                            break (Arc::clone(&fence), Some(fill_rx));
                        }
                        Err(e) => Some(e),
                    }
                } else {
                    None
                }
            };
            drop(turn);
            if let Some(e) = begin_failed {
                counter!("personhog_leader_kafka_produce_errors_total").increment(1);
                return Err(self.classify(&fence, partition, e));
            }
            counter!("personhog_leader_fence_lane_choices_total", "choice" => "park").increment(1);
            #[cfg(any(test, feature = "test-support"))]
            fence.parked.fetch_add(1, Ordering::SeqCst);
            closed.await;
            #[cfg(any(test, feature = "test-support"))]
            fence.parked.fetch_sub(1, Ordering::SeqCst);
        };
        // Near-zero when a window was open or the producer idle; the tail
        // is time parked behind a draining or committing window — the
        // queueing term of the fencing tax, and the first thing to grow
        // if commits slow down.
        // From here the window counts this write as in flight; the slot
        // returns the seat on every exit, including cancellation.
        let slot = WindowSlot::held(Arc::clone(&fence));
        histogram!("personhog_leader_fence_window_wait_ms", "partition" => partition.to_string())
            .record(join_start.elapsed().as_secs_f64() * 1000.0);

        if let Some(fill) = opened {
            let fence_for_commit = Arc::clone(&fence);
            let window = self.window;
            let topic = lanes.topic.clone();
            let part = partition;
            tokio::spawn(async move {
                commit_window_after(fence_for_commit, window, fill, topic, part).await;
            });
        }

        // Send within the open window; the record rides librdkafka's
        // ordinary batching alongside its window-mates.
        let key = changelog_message_key(person.team_id, person.id);
        let payload = person.encode_to_vec();
        // Same metric as the unfenced path in kafka.rs, so the size
        // distribution covers all changelog produces regardless of the
        // fencing rollout. Recorded before the send so a payload rejected
        // by the broker (message.max.bytes) still shows up.
        histogram!("personhog_leader_kafka_produce_bytes").record(payload.len() as f64);
        let record = FutureRecord::to(&lanes.topic)
            .partition(partition as i32)
            .key(&key)
            .payload(&payload);
        let produce_start = Instant::now();
        let delivery = fence.producer.inner().send_result(record);

        let offset = match delivery {
            Ok(fut) => match fut.await {
                Ok(Ok((_, offset))) => {
                    histogram!("personhog_leader_fence_send_ms")
                        .record(produce_start.elapsed().as_secs_f64() * 1000.0);
                    Ok(offset)
                }
                // These fire per write, and a broker outage fails every
                // write, so the aggregate is the useful signal and the
                // per-request detail belongs at debug. The caller logs
                // the classified outcome, and the counters below carry
                // the rate.
                Ok(Err((e, _))) => {
                    debug!(partition, error = %e, "fenced send delivery failed");
                    Err(Some(e))
                }
                Err(_cancelled) => {
                    debug!(partition, "fenced send cancelled");
                    Err(None)
                }
            },
            Err((e, _)) => {
                debug!(partition, error = %e, "fenced send enqueue failed");
                Err(Some(e))
            }
        };

        // A failed send poisons the window (the committer aborts it) and
        // returns immediately with its own classified error — a fenced
        // rejection must surface as `Fenced`, not as the window's
        // generic abort.
        if let Err(send_err) = offset {
            counter!("personhog_leader_kafka_produce_errors_total").increment(1);
            slot.poison();
            return Err(match send_err {
                Some(e) => self.classify(&fence, partition, e),
                None => FencedProduceError::Failed("send cancelled (timeout)".to_string()),
            });
        }
        let offset = offset.expect("checked above");

        let rx = slot.subscribe();

        // The record exists but is invisible until the window commits;
        // the ack must wait for the commit outcome. Together with the
        // window-wait and send spans this completes the caller-visible
        // decomposition of a fenced produce.
        let ack_wait_start = Instant::now();
        let outcome = rx.await;
        histogram!("personhog_leader_fence_ack_wait_ms", "partition" => partition.to_string())
            .record(ack_wait_start.elapsed().as_secs_f64() * 1000.0);
        match outcome {
            Ok(Ok(())) => {
                counter!("personhog_leader_kafka_produces_total").increment(1);
                // The same series the unfenced path records: time from
                // entering produce to a durable record, so the two flag
                // arms stay comparable.
                histogram!("personhog_leader_kafka_produce_duration_ms")
                    .record(durable_start.elapsed().as_secs_f64() * 1000.0);
                Ok(offset)
            }
            Ok(Err(e)) => {
                counter!("personhog_leader_kafka_produce_errors_total").increment(1);
                // The commit reported the fence, so the eviction has to
                // happen here: `commit_window_after` runs detached and
                // holds no handle to the fence map. Without this the dead
                // producer stays installed, later writes discover the
                // fence one failed `begin_transaction` at a time, and
                // `holds()` reports authority this pod no longer has.
                if matches!(
                    e,
                    FencedProduceError::Fenced | FencedProduceError::FencedUncertain(_)
                ) {
                    self.forget_lane(partition, &fence);
                }
                Err(e)
            }
            // The committer dropped its senders without answering, so
            // nobody observed whether the commit landed — and
            // `spawn_blocking` work is not cancelled when its handle is
            // dropped, so it may well have. Reporting a definite abort
            // here frees a version whose record may exist.
            Err(_) => {
                counter!("personhog_leader_kafka_produce_errors_total").increment(1);
                Err(FencedProduceError::Indeterminate(
                    "commit task dropped without reporting an outcome".to_string(),
                ))
            }
        }
    }

    /// Classify a transaction error, removing the fence on a broker
    /// fence rejection so subsequent writes fail fast as stale. The
    /// fenced signal often hides behind a local symptom — librdkafka
    /// purges queued messages once the producer turns fatal — so the
    /// producer's fatal error is consulted alongside the surface error.
    fn classify(
        &self,
        fence: &Arc<PartitionFence>,
        partition: u32,
        e: KafkaError,
    ) -> FencedProduceError {
        if is_fenced(&e) || producer_fenced(fence.producer.inner()) {
            let cause = fence.fence_cause();
            counter!(
                "personhog_leader_produce_fenced_total",
                "partition" => partition.to_string(),
                "cause" => cause
            )
            .increment(1);
            error!(
                partition,
                lane = fence.lane,
                cause,
                error = %e,
                "changelog producer fenced by a newer claim on its id"
            );
            self.forget_lane(partition, fence);
            FencedProduceError::Fenced
        } else {
            FencedProduceError::Failed(e.to_string())
        }
    }

    /// Drop a fence that the broker rejected, but only if it is still the
    /// one installed: a write can be in flight across a release and a
    /// re-acquire, and the stale producer's failure must not evict its
    /// live replacement.
    fn forget_fence(&self, partition: u32, lanes: &Arc<PartitionLanes>) {
        self.evict(partition, |installed| Arc::ptr_eq(installed, lanes));
    }

    /// [`Self::forget_fence`] for the lane that reported the fence: a
    /// newer owner claimed the partition's ids, so the whole set is stale.
    fn forget_lane(&self, partition: u32, fence: &Arc<PartitionFence>) {
        self.evict(partition, |installed| installed.contains(fence));
    }

    fn evict(&self, partition: u32, stale: impl Fn(&Arc<PartitionLanes>) -> bool) {
        let removed = self
            .partitions
            .remove_if(&partition, |_, installed| stale(installed));
        if let Some((_, evicted)) = removed {
            drop_off_worker(evicted);
            // The escalation signal for a partition giving up its fence
            // outside the orderly release path — the series exists so a
            // deploy-window burst of these is visible.
            counter!(
                "personhog_leader_fenced_partition_drops_total",
                "partition" => partition.to_string()
            )
            .increment(1);
        }
    }
}

/// How many times a retriable commit is re-attempted before its outcome
/// is declared unknown.
///
/// Derived from the attempt budget the lease runway affords, rather than
/// chosen independently: `validate_fencing_timescales` sizes the fencing
/// timeouts so that exactly this many attempts still fit inside the
/// window the keepalive reserves for self-fencing.
const COMMIT_RETRIES: usize = FENCING_COMMIT_ATTEMPTS as usize - 1;

/// What is known about a window's records after a failed commit, and
/// whether the producer survived learning it.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum CommitOutcome {
    /// The transaction is definitely aborted; nothing became visible.
    Aborted,
    /// Definitely aborted, but the abort never landed, so the producer is
    /// stuck in a state it cannot begin another transaction from.
    AbortedProducerDead,
    /// Whether the records committed is not known.
    ///
    /// There is no healthy-producer form of this: a commit fails without
    /// a definite outcome only when the error is neither retriable nor
    /// abort-requiring, which is librdkafka's fatal class, or when the
    /// follow-up abort itself failed. Both leave the producer unusable.
    Unknown,
}

/// What a failed window means for its writers.
#[derive(Clone, Copy, Debug, PartialEq, Eq)]
enum WindowVerdict {
    /// A newer owner holds the partition; the records never landed.
    Fenced,
    /// A newer owner holds the partition and the records' fate is
    /// unknown — both facts, because they answer different questions.
    FencedUncertain,
    /// The window aborted for an ordinary reason; safe to retry.
    Aborted,
    /// Whether the records landed is unknown.
    Indeterminate,
}

/// Whether the partition moved and whether the records landed are
/// independent, so the verdict carries both.
///
/// A commit can time out, be re-issued by librdkafka itself, and only
/// then report the fence — by which point an earlier attempt may already
/// have succeeded. Reporting that as a plain fence tells the caller the
/// record does not exist, which frees its version and puts a second
/// record at the same number behind one that may be committed. Reporting
/// it as merely indeterminate throws away the ownership answer the router
/// bounces on. `FencedUncertain` is both.
/// Every reason production code passes to `condemn`, in one place so the
/// preregistration below cannot drift from the call sites: a reason
/// missing here first appears as a brand-new series mid-incident instead
/// of rising from zero.
const CONDEMN_REASONS: [&str; 7] = [
    "abort_fenced",
    "abort_failed",
    "commit_fenced",
    "commit_indeterminate",
    "commit_task_lost",
    "committer_unwound",
    "gate_poisoned",
];

/// Why a window's outcome leaves the producer unusable, if it does.
///
/// Extracted so the arrow *into* the condemned state is reachable: the
/// tests could reach the aftermath through a staging hook, but the branch
/// that decides it could be deleted outright with the suite still green,
/// and it is the only path in production that ever sets the flag.
///
/// A dead producer carries two very different stories, split by
/// `fenced`: a producer fenced by a newer owner's init lost the epoch (a
/// handoff, a heal, a zombie being cut off — coordination working), while
/// an abort that failed on a producer nobody fenced means the broker
/// could not be reached in time (an infrastructure excursion). The panel
/// reading this label has to tell those apart without the logs.
fn condemn_reason(outcome: CommitOutcome, fenced: bool) -> Option<&'static str> {
    match outcome {
        CommitOutcome::Aborted => None,
        CommitOutcome::AbortedProducerDead if fenced => Some("abort_fenced"),
        CommitOutcome::AbortedProducerDead => Some("abort_failed"),
        // A commit killed by a newer owner's init reports librdkafka's
        // fatal class and lands here, not in AbortedProducerDead — the
        // fenced split must cover this arm or every deploy-shaped
        // condemnation reads as a broker excursion.
        CommitOutcome::Unknown if fenced => Some("commit_fenced"),
        CommitOutcome::Unknown => Some("commit_indeterminate"),
    }
}

fn window_verdict(outcome: CommitOutcome, fenced: bool) -> WindowVerdict {
    match (outcome.is_aborted(), fenced) {
        (true, true) => WindowVerdict::Fenced,
        (false, true) => WindowVerdict::FencedUncertain,
        (true, false) => WindowVerdict::Aborted,
        (false, false) => WindowVerdict::Indeterminate,
    }
}

impl CommitOutcome {
    /// Whether the records are known not to have become visible.
    fn is_aborted(self) -> bool {
        matches!(self, Self::Aborted | Self::AbortedProducerDead)
    }
}

fn is_fenced(e: &KafkaError) -> bool {
    matches!(
        e.rdkafka_error_code(),
        Some(RDKafkaErrorCode::Fenced) | Some(RDKafkaErrorCode::ProducerFenced)
    )
}

/// Whether the producer has entered the fatally-fenced state. Once a
/// newer owner initializes the transactional id, librdkafka marks this
/// client fatal and individual operations report local symptoms (purged
/// queues, aborted transactions) rather than the fence itself.
fn producer_fenced<C: ClientContext>(producer: &FutureProducer<C>) -> bool {
    producer
        .client()
        .fatal_error()
        .map(|(code, _)| {
            matches!(
                code,
                RDKafkaErrorCode::Fenced | RDKafkaErrorCode::ProducerFenced
            )
        })
        .unwrap_or(false)
}

/// Abort the window, up to the number of attempts the lease runway was
/// sized for.
///
/// Derived from the constant rather than written out, so that raising the
/// budget raises the attempts and vice versa. The two drifted apart once
/// already: the loop spent four transaction timeouts while the bound that
/// had to contain them counted one.
fn abort_window<C: ClientContext>(
    producer: &FutureProducer<C>,
    timeout: Duration,
) -> Result<(), KafkaError> {
    let mut last = producer.abort_transaction(timeout);
    let mut spent = 1;
    while last.is_err() && spent < FENCING_ABORT_ATTEMPTS {
        last = producer.abort_transaction(timeout);
        spent += 1;
    }
    last
}

/// Close the window after its admission interval: stop admitting, wait
/// for in-flight sends to settle, then commit (or abort a poisoned
/// window) and resolve every waiter with the shared outcome.
async fn commit_window_after(
    fence: Arc<PartitionFence>,
    window: Duration,
    fill: oneshot::Receiver<()>,
    topic: String,
    partition: u32,
) {
    // The fill branch also covers the sender dropping unfired, which
    // cannot happen while this window is open: the sender lives in the
    // gate of a fence this task holds, and nothing replaces it until
    // the CommittingMark below has been dropped.
    tokio::select! {
        _ = sleep(window) => {
            counter!("personhog_leader_fence_window_closed_total", "reason" => "timer")
                .increment(1);
        }
        _ = fill => {
            counter!("personhog_leader_fence_window_closed_total", "reason" => "filled")
                .increment(1);
        }
    }

    // Stop admitting joiners, then wait for outstanding sends. The mark
    // holds until this function returns — by any path — so no writer can
    // begin the next transaction while this one is still resolving, and
    // none is stranded if it unwinds.
    let _committing = CommittingMark::take(Arc::clone(&fence), partition);
    loop {
        // Register interest before inspecting the gate: a settle that
        // fires between the check and the await must not be lost.
        let settled = fence.sends_settled.notified();
        tokio::pin!(settled);
        settled.as_mut().enable();
        {
            let gate = fence.gate.lock().unwrap();
            if gate.in_flight == 0 {
                break;
            }
        }
        settled.await;
    }

    let (waiters, poisoned) = {
        let mut gate = fence.gate.lock().unwrap();
        (mem::take(&mut gate.waiters), gate.poisoned)
    };
    histogram!("personhog_leader_fence_window_writes", "partition" => partition.to_string())
        .record(waiters.len() as f64);

    // Commit and abort block up to the transaction timeout; keep them
    // off the runtime threads.
    let producer = fence.producer.inner().clone();
    let timeout = fence.commit_timeout;
    let commit_start = Instant::now();
    #[cfg(any(test, feature = "test-support"))]
    let panic_next = fence.panic_next_commit.swap(false, Ordering::SeqCst);
    let result = spawn_blocking(move || {
        #[cfg(any(test, feature = "test-support"))]
        if panic_next {
            panic!("staged commit task failure");
        }
        if poisoned {
            // One attempt. A producer left in its abortable state fails
            // every later `begin_transaction`, so a failed abort costs
            // the producer itself rather than just this window — and the
            // recovery for that is re-acquisition, whose init aborts the
            // pending transaction at the broker anyway.
            match abort_window(&producer, timeout) {
                Ok(()) => Err((KafkaError::Canceled, CommitOutcome::Aborted)),
                // The records are still definitely not visible — an abort
                // that did not land leaves the transaction open, and open
                // is not committed. What is lost is the producer: it stays
                // in its abortable state, where every later
                // `begin_transaction` fails.
                Err(e) => {
                    counter!("personhog_leader_fence_abort_failed_total").increment(1);
                    Err((e, CommitOutcome::AbortedProducerDead))
                }
            }
        } else {
            // A commit that fails is not the same as a commit that did
            // not happen. librdkafka distinguishes three cases and the
            // caller's correct behaviour differs for each: a retriable
            // failure leaves the outcome unknown and must be re-attempted
            // (commit is idempotent within the transaction); an
            // abort-requiring failure is a definite abort once the abort
            // is issued; anything else leaves the fate of the records in
            // doubt, which is a distinct answer from "failed".
            let mut attempt = producer.commit_transaction(timeout);
            for _ in 0..COMMIT_RETRIES {
                match &attempt {
                    Err(KafkaError::Transaction(e)) if e.is_retriable() => {
                        counter!("personhog_leader_fence_commit_retries_total").increment(1);
                        warn!(error = %e, "changelog commit failed; retrying");
                        attempt = producer.commit_transaction(timeout);
                    }
                    _ => break,
                }
            }
            match attempt {
                Ok(()) => Ok(()),
                Err(KafkaError::Transaction(e)) if e.txn_requires_abort() => {
                    match abort_window(&producer, timeout) {
                        Ok(()) => Err((KafkaError::Transaction(e), CommitOutcome::Aborted)),
                        // The abort itself is now in doubt, so the
                        // records are too.
                        Err(abort_err) => Err((abort_err, CommitOutcome::Unknown)),
                    }
                }
                // Neither retriable nor abort-requiring is librdkafka's
                // fatal class, which includes the fence. The transaction
                // is left open with its fate unknown and this producer
                // has no way back to a state where it can begin another;
                // whether it was *also* fenced is answered separately,
                // by the verdict, because the two facts serve different
                // callers.
                //
                // Deliberately not inferring "the first attempt cannot
                // have landed" from this loop never re-issuing:
                // `commit_transaction` re-sends EndTxn internally, so one
                // call here is not one round trip at the broker.
                Err(e) => Err((e, CommitOutcome::Unknown)),
            }
        }
    })
    .await;

    let outcome: Result<(), FencedProduceError> = match result {
        Ok(Ok(())) => {
            histogram!("personhog_leader_fence_commit_ms", "outcome" => "committed")
                .record(commit_start.elapsed().as_secs_f64() * 1000.0);
            Ok(())
        }
        Ok(Err((e, outcome))) => {
            histogram!("personhog_leader_fence_commit_ms", "outcome" => "failed")
                .record(commit_start.elapsed().as_secs_f64() * 1000.0);
            let fenced_now = is_fenced(&e) || producer_fenced(fence.producer.inner());
            if let Some(reason) = condemn_reason(outcome, fenced_now) {
                fence.condemn(partition, reason);
            }
            let verdict = window_verdict(outcome, fenced_now);
            let cause = fence.fence_cause();
            if verdict == WindowVerdict::FencedUncertain {
                counter!(
                    "personhog_leader_produce_fenced_total",
                    "partition" => partition.to_string(),
                    "cause" => cause
                )
                .increment(1);
                error!(
                    partition,
                    lane = fence.lane,
                    cause,
                    topic,
                    error = %e,
                    "changelog window fenced by a newer claim on its id, with its own outcome unknown"
                );
                Err(FencedProduceError::FencedUncertain(e.to_string()))
            } else if verdict == WindowVerdict::Fenced {
                // A poisoned window's fence was already classified and
                // counted where the send failed; counting again here
                // would report two fences for one event.
                if !poisoned {
                    counter!(
                        "personhog_leader_produce_fenced_total",
                        "partition" => partition.to_string(),
                        "cause" => cause
                    )
                    .increment(1);
                }
                error!(
                    partition,
                    lane = fence.lane,
                    cause,
                    topic,
                    error = %e,
                    "changelog window fenced by a newer claim on its id"
                );
                Err(FencedProduceError::Fenced)
            } else if verdict == WindowVerdict::Aborted {
                counter!(
                    "personhog_leader_fence_aborts_total",
                    "partition" => partition.to_string()
                )
                .increment(1);
                error!(
                    partition,
                    lane = fence.lane,
                    topic,
                    error = %e,
                    "changelog window aborted"
                );
                Err(FencedProduceError::Failed(format!("aborted: {e}")))
            } else {
                // Retries are exhausted and the transaction's fate is
                // unknown. Saying "aborted" here would invite a retry
                // that collides with a record that may already be
                // committed, so the doubt is reported as doubt.
                counter!(
                    "personhog_leader_fence_commit_indeterminate_total",
                    "partition" => partition.to_string()
                )
                .increment(1);
                error!(
                    partition,
                    lane = fence.lane,
                    topic,
                    error = %e,
                    "changelog commit outcome unknown; the window may or may not have committed"
                );
                Err(FencedProduceError::Indeterminate(e.to_string()))
            }
        }
        // The commit task itself vanished, so nothing observed the
        // transaction's fate — which leaves the transaction open and this
        // producer unable to begin another. Condemning it is what stops
        // the partition counting as fenced; without it every later write
        // retries a producer that cannot serve one, for the life of the
        // process, with nothing to escalate.
        Err(join) => {
            fence.condemn(partition, "commit_task_lost");
            Err(FencedProduceError::Indeterminate(format!(
                "commit join: {join}"
            )))
        }
    };

    for waiter in waiters {
        // A dropped receiver means the writer's request already ended;
        // nothing to deliver.
        waiter.send(clone_outcome(&outcome)).ok();
    }
}

/// Materialize every fencing series at startup so deploy-window bursts
/// land in an already-scraped series instead of being swallowed by
/// first-increment lazy registration — fence acquisition in particular
/// fires exactly during deploys. A touched histogram renders with zero
/// count until its first sample.
pub fn preregister_fencing_metrics() {
    for outcome in ["ok", "error"] {
        counter!("personhog_leader_fence_init_total", "outcome" => outcome).increment(0);
    }
    // Settling fires only during handoffs, so every one of its samples
    // lands in a deploy window — the case this preregistration exists
    // for.
    for outcome in ["settled", "timeout", "unusable", "absent"] {
        counter!("personhog_leader_fence_settle_total", "outcome" => outcome).increment(0);
    }
    counter!("personhog_leader_fence_slots_abandoned_total").increment(0);
    for reason in ["timer", "filled"] {
        counter!("personhog_leader_fence_window_closed_total", "reason" => reason).increment(0);
    }
    counter!("personhog_leader_fence_abandoned_total").increment(0);
    counter!("personhog_leader_fence_abort_failed_total").increment(0);
    for choice in ["join", "open", "park"] {
        counter!("personhog_leader_fence_lane_choices_total", "choice" => choice).increment(0);
    }
    for outcome in [
        "ok",
        "error",
        "coalesced",
        "discarded",
        "swept",
        "orphaned",
        "init_failed",
    ] {
        counter!("personhog_leader_fence_preconnect_total", "outcome" => outcome).increment(0);
    }
    // The healing counters fire exactly during the incidents an operator
    // would reach for them in, and rarely enough that lazy registration
    // can swallow the first burst between scrapes.
    counter!("personhog_leader_fence_healed_total").increment(0);
    counter!("personhog_leader_fence_heal_abandoned_total").increment(0);
    for reason in CONDEMN_REASONS {
        counter!("personhog_leader_fence_condemned_total", "reason" => reason).increment(0);
    }
    counter!("personhog_leader_fence_commit_retries_total").increment(0);
    // The coordination loop's repair-pass counter fires exclusively
    // mid-incident, exactly when a first-increment series would be
    // swallowed between scrapes.
    for outcome in ["run", "suppressed"] {
        counter!("personhog_coordination_repair_passes_total", "outcome" => outcome).increment(0);
    }
    counter!("personhog_leader_kafka_produce_errors_total").increment(0);
    counter!("personhog_leader_fence_produce_bound_exceeded_total").increment(0);
    // Histograms are deliberately absent: the Prometheus exporter
    // renders one only once it has a sample, so there is nothing a
    // startup call can materialize.
}

/// Registers one partition's fencing series when it is captured.
pub fn preregister_partition_fencing_metrics(partition: u32) {
    let p = partition.to_string();
    counter!("personhog_leader_fence_aborts_total", "partition" => p.clone()).increment(0);
    counter!(
        "personhog_leader_fenced_partition_drops_total",
        "partition" => p.clone()
    )
    .increment(0);
    counter!(
        "personhog_leader_fence_heal_failures_total",
        "partition" => p.clone()
    )
    .increment(0);
    counter!(
        "personhog_leader_fence_commit_indeterminate_total",
        "partition" => p.clone()
    )
    .increment(0);
    for cause in ["takeover", "heal"] {
        counter!(
            "personhog_leader_produce_fenced_total",
            "partition" => p.clone(),
            "cause" => cause
        )
        .increment(0);
    }
}

/// `FencedProduceError` holds a String and cannot derive Clone cheaply
/// through the oneshot fan-out; rebuild the outcome per waiter.
fn clone_outcome(outcome: &Result<(), FencedProduceError>) -> Result<(), FencedProduceError> {
    match outcome {
        Ok(()) => Ok(()),
        Err(FencedProduceError::NotAcquired) => Err(FencedProduceError::NotAcquired),
        Err(FencedProduceError::Fenced) => Err(FencedProduceError::Fenced),
        Err(FencedProduceError::FencedUncertain(e)) => {
            Err(FencedProduceError::FencedUncertain(e.clone()))
        }
        Err(FencedProduceError::Failed(e)) => Err(FencedProduceError::Failed(e.clone())),
        Err(FencedProduceError::Indeterminate(e)) => {
            Err(FencedProduceError::Indeterminate(e.clone()))
        }
    }
}

/// Re-take the partition's fence if this pod is serving it without one.
///
/// A fence can go missing under a pod that legitimately owns its
/// partition: a broker rejection evicted it, an abort exhausted its
/// retries and left the producer unusable, or a stale pod took the epoch
/// and stepped back. Nothing in the handoff protocol repairs that —
/// convergence sees the partition warmed and unfenced and does nothing —
/// so without this the partition stays unwritable until a handoff moves
/// it.
///
/// Re-acquisition is safe only because of where this is called from and
/// what it checks. The caller is the convergence to `Serving`, so the
/// durable assignment says this pod owns the partition; the claim must
/// still be valid, because taking a fence moves the broker's epoch away
/// from whoever holds it; and a partition being locally fenced is
/// disqualifying, since that means a handoff is moving it and the
/// incoming owner's fence is the one that should stand.
/// What the healing pass did, for the caller to act on: a healed
/// partition is freshly fenced — the epoch just moved, and the same
/// convergence must not move it again — and a failure is the caller's
/// to surface, because a partition that cannot regain its fence has no
/// other repair path.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub enum HealOutcome {
    /// Nothing to do: the fence is present and usable, the partition is
    /// under handoff, or this pod cannot vouch for its claim.
    Intact,
    /// A fence was re-taken and is serving.
    Healed,
    /// A fence was taken but given back — standing lapsed or a handoff
    /// began during the broker round trip.
    Abandoned,
}

pub async fn heal_fence(
    fenced: &Arc<FencedChangelogProducers>,
    inflight: &InflightTracker,
    authority: &AuthorityClock,
    partition: u32,
    target: &ChangelogTarget,
) -> Result<HealOutcome, String> {
    let lost_standing = !authority.is_valid();
    if lost_standing || fenced.holds(partition) || inflight.is_fenced(partition) {
        return Ok(HealOutcome::Intact);
    }
    fenced.expect_self_fence(partition);
    let taken = match fenced.acquire_installed(partition, target).await {
        Ok(taken) => taken,
        Err(e) => {
            // Partition-labeled: a wedged partition must be attributable
            // from the series alone, because this is deliberately not a
            // run failure (see `verify_serving`) and the counter is the
            // alerting surface for it.
            counter!(
                "personhog_leader_fence_heal_failures_total",
                "partition" => partition.to_string()
            )
            .increment(1);
            error!(partition, error = %e, "failed to re-take the changelog fence");
            return Err(e);
        }
    };
    // Answerable for the installed fence from here until a deliberate
    // branch takes over. A drop *during* the acquire needs no guard —
    // the producer is discarded before it is installed, `holds()` stays
    // false, and the next convergence heals again. What the guard covers
    // is the stretch after installation: the checks below await nothing
    // today, so its teeth are a panic between them and any await a
    // future change adds. Same shape as the warm path.
    let guard = FenceGuard::new(Arc::clone(fenced), partition, "heal");
    // The round trip is long enough for the ground to move: the claim can
    // lapse, or a handoff can start draining the partition. Holding a
    // fence taken without standing is not passive — the write path trusts
    // the broker epoch rather than re-checking the claim, so a request
    // landing here would ack a mutation with an epoch taken from the
    // partition's real owner.
    let lost_standing = !authority.is_valid();
    if lost_standing || inflight.is_fenced(partition) {
        guard.keep();
        // The fence this call installed, not whatever is installed now:
        // re-reading the map would match by construction and evict a
        // replacement just as readily as its own.
        fenced.forget_fence(partition, &taken);
        counter!("personhog_leader_fence_heal_abandoned_total").increment(1);
        warn!(
            partition,
            "released a fence taken while standing lapsed mid-acquire"
        );
        return Ok(HealOutcome::Abandoned);
    }
    guard.keep();
    counter!("personhog_leader_fence_healed_total").increment(1);
    warn!(
        partition,
        "re-took the changelog fence for a served partition"
    );
    Ok(HealOutcome::Healed)
}

#[cfg(test)]
mod tests {
    use super::*;
    use envconfig::Envconfig;

    #[test]
    fn a_producer_dials_its_generations_cluster_with_its_queue_share() {
        let mut kafka = KafkaConfig::init_from_hashmap(&std::collections::HashMap::new()).unwrap();
        kafka.kafka_producer_queue_mib = 1024;
        kafka.kafka_producer_queue_messages = 64_000;
        let producers = FencedChangelogProducers::new(FencedProducerConfig {
            kafka,
            init_timeout: Duration::from_secs(1),
            commit_timeout: Duration::from_secs(1),
            broker_txn_timeout: Duration::from_secs(10),
            window: Duration::from_millis(5),
            window_max_writes: 32,
            settle_budget: Duration::from_secs(1),
            lanes: 2,
        });
        for (partitions, mib, messages) in [(8, 64, 4_000), (16, 32, 2_000)] {
            let target = ChangelogTarget {
                topic: format!("g{partitions}"),
                partitions,
                cluster: None,
            };
            let dial = producers.dial(0, &target);
            assert_eq!(
                (
                    dial.kafka.kafka_producer_queue_mib,
                    dial.kafka.kafka_producer_queue_messages
                ),
                (mib, messages),
                "{partitions} partitions"
            );
        }
        let moved = ChangelogTarget {
            topic: "g8".to_string(),
            partitions: 8,
            cluster: Some(personhog_coordination::types::ClusterEndpoint {
                name: "dedicated".to_string(),
                hosts: "d:9092".to_string(),
                tls: true,
            }),
        };
        let dial = producers.dial(0, &moved);
        assert_eq!(
            (
                dial.kafka.kafka_hosts.as_str(),
                dial.kafka.kafka_tls,
                dial.kafka.kafka_producer_queue_mib
            ),
            ("d:9092", true, 64)
        );
    }

    /// The verdict the caller acts on, as a truth table over the two
    /// independent facts it combines.
    ///
    /// The cell that matters is (unknown, fenced). A commit that timed
    /// out, was re-issued — by librdkafka itself, not just by this loop —
    /// and only then discovered the fence may already have landed. Its
    /// own verdict keeps both facts: the router still gets the ownership
    /// answer it bounces on, and the version stays spent. Collapsing it
    /// into `Fenced` frees a version whose record may be committed, and
    /// collapsing it into `Indeterminate` throws away the bounce.
    /// The arrow *into* the condemned state. A producer left in a
    /// transaction it cannot leave must be given up, or the partition
    /// keeps counting as fenced and every later write retries a producer
    /// that cannot serve one, for the life of the process.
    #[test]
    fn a_producer_that_cannot_begin_another_window_is_condemned() {
        for fenced in [false, true] {
            assert_eq!(condemn_reason(CommitOutcome::Aborted, fenced), None);
        }
        assert_eq!(
            condemn_reason(CommitOutcome::Unknown, true),
            Some("commit_fenced")
        );
        assert_eq!(
            condemn_reason(CommitOutcome::Unknown, false),
            Some("commit_indeterminate")
        );
        // The same dead producer, split by what killed it: an epoch lost
        // to a newer owner is coordination working, an abort nobody
        // fenced failing is the broker unreachable — the operator's first
        // question, answered from the series alone.
        assert_eq!(
            condemn_reason(CommitOutcome::AbortedProducerDead, true),
            Some("abort_fenced")
        );
        assert_eq!(
            condemn_reason(CommitOutcome::AbortedProducerDead, false),
            Some("abort_failed")
        );
    }

    #[test]
    fn a_partition_runs_one_window_at_a_time_across_its_lanes() {
        let idle = |last_commit_end| LaneState {
            open: false,
            committing: false,
            last_commit_end,
        };
        let open = |last_commit_end| LaneState {
            open: true,
            committing: false,
            last_commit_end,
        };
        let committing = |last_commit_end| LaneState {
            open: false,
            committing: true,
            last_commit_end,
        };
        // Idle partition: the lane that committed longest ago.
        assert_eq!(pick_lane(&[idle(5), idle(2)]), 1);
        assert_eq!(pick_lane(&[idle(0)]), 0);
        // An open window is joined even on the lane that committed last.
        assert_eq!(pick_lane(&[idle(1), open(9)]), 1);
        // A committing lane parks the write instead of opening another lane.
        assert_eq!(pick_lane(&[committing(1), idle(9)]), 0);
        assert_eq!(pick_lane(&[idle(9), committing(4), committing(2)]), 2);
    }

    /// A condemn reason absent from the preregistration list first
    /// appears as a new series mid-incident instead of rising from zero.
    #[test]
    fn every_condemn_reason_is_preregistered() {
        for outcome in [
            CommitOutcome::Aborted,
            CommitOutcome::AbortedProducerDead,
            CommitOutcome::Unknown,
        ] {
            for fenced in [false, true] {
                if let Some(reason) = condemn_reason(outcome, fenced) {
                    assert!(
                        CONDEMN_REASONS.contains(&reason),
                        "condemn_reason produced {reason:?}, missing from CONDEMN_REASONS"
                    );
                }
            }
        }
        for direct in ["commit_task_lost", "committer_unwound"] {
            assert!(
                CONDEMN_REASONS.contains(&direct),
                "direct condemn call site uses {direct:?}, missing from CONDEMN_REASONS"
            );
        }
    }

    #[test]
    fn a_fence_only_settles_the_records_when_the_window_also_aborted() {
        for (outcome, fenced, expected) in [
            (CommitOutcome::Aborted, true, WindowVerdict::Fenced),
            (
                CommitOutcome::AbortedProducerDead,
                true,
                WindowVerdict::Fenced,
            ),
            (CommitOutcome::Aborted, false, WindowVerdict::Aborted),
            (
                CommitOutcome::AbortedProducerDead,
                false,
                WindowVerdict::Aborted,
            ),
            (CommitOutcome::Unknown, true, WindowVerdict::FencedUncertain),
            (CommitOutcome::Unknown, false, WindowVerdict::Indeterminate),
        ] {
            assert_eq!(
                window_verdict(outcome, fenced),
                expected,
                "outcome={outcome:?} fenced={fenced}"
            );
        }
    }
}