Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

29.2 KB
Raw Blame
//! Lifecycle fence state: which persons are frozen by a live lifecycle
//! operation, and how a leader rebuilds that knowledge across ownership
//! changes.
//!
//! The fence's source of truth is not this map — it is the saga's mark row
//! (`lifecycle_op_person` in `marked`/`sealed`), committed on the persons
//! primary before `FencePerson` is ever called. The map is the leader's
//! in-process copy, and on the write path it is authoritative: a fenced
//! write is rejected from memory alone, never a database read.
//!
//! Postgres is read off the write path only: the takeover scan before a
//! partition serves, the heal below, and a release's mark check. The scan
//! and the `FencePerson` RPC are the only ways an entry gets in; a
//! `ReleaseFence`, a partition drop, or the heal takes it out.
//!
//! The leader never reclaims a fence on its own, and deliberately so.
//! personhog-identity owns lifecycle correctness: every op is driven to a
//! terminal state (lease steal, sweeper resumption), so a `ReleaseFence`
//! always eventually arrives, with one exception: an op whose committed
//! release this leader definitively refused (a semantic refusal) is
//! parked by the saga engine and stops retrying, so its fences hold
//! until an operator re-drives it. The person stays frozen rather than
//! half-destroyed. An entry that outlives a crashed saga is not stale —
//! the mark is still live, the op really is unfinished, and the person
//! really should stay frozen. For that guarantee to hold, a
//! release must never *vacuously* succeed: both fence RPCs verify this
//! pod serves the partition, so a misrouted call fails and identity's
//! retry reaches the pod whose map actually gates the writes.
//!
//! The map is NOT a cache: it has no capacity limit and no eviction path
//! — losing an entry while continuing to serve would violate consistency,
//! not degrade it.
//!
//! One known window: a takeover that runs between a release being acked
//! and the saga settling its mark row installs a fence for an op that is
//! already done — a ghost no `ReleaseFence` will ever clear. The
//! [`FenceHealer`] closes it lazily: a write rejected by a fence triggers
//! a non-blocking mark-row read, and a fence whose op has settled is
//! dropped, so the next write goes through instead of waiting for the
//! partition to change hands.

use std::collections::HashMap;
use std::sync::atomic::{AtomicBool, Ordering};
use std::sync::{Arc, Mutex, PoisonError};
use std::time::{Duration, Instant};

use async_trait::async_trait;
use dashmap::DashMap;
use metrics::{counter, gauge, histogram};
use sqlx::{PgPool, Row};
use tokio::sync::OnceCell;
use tonic::Status;
use uuid::Uuid;

use personhog_common::partitioning::partition_for_person;
use personhog_proto::personhog::types::v1::LifecycleOpType;

use crate::cache::PersonCacheKey;
use crate::pg::PgFallback;

/// A person's live fence: the operation that froze it, and when this pod
/// sealed it. A takeover fence, rebuilt from the rows, has no seal time.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
pub struct FenceState {
    pub op_id: Uuid,
    pub op_type: LifecycleOpType,
    pub sealed_at: Option<Instant>,
}

pub type FenceMap = Arc<DashMap<PersonCacheKey, FenceState>>;

/// Metadata key carried on fenced-write rejections: `delete` or `merge`.
/// Callers distinguish "this person is being destroyed, back off" from
/// ordinary precondition failures by this key's presence.
pub const FENCED_METADATA_KEY: &str = "x-person-fenced";
/// Metadata key carrying the fencing operation's id on rejections.
pub const FENCED_OP_ID_METADATA_KEY: &str = "x-person-fenced-op-id";

/// A definitive FAILED_PRECONDITION the router passes through to the
/// caller instead of bouncing. Bare FAILED_PRECONDITION classifies as a
/// routing-race bounce and exhausts into retriable UNAVAILABLE, which
/// would turn a fail-closed verification refusal into an infinite saga
/// retry loop; the marker (see
/// `personhog_common::grpc::SEMANTIC_REFUSAL_METADATA_KEY`) makes the
/// refusal survive the trip.
pub use personhog_common::grpc::semantic_refusal;

/// The typed rejection for a write to a fenced person: PERSON_DELETING /
/// PERSON_MERGING per the RFC, encoded as FAILED_PRECONDITION plus
/// metadata (gRPC has no custom codes).
pub fn fenced_status(state: &FenceState) -> Status {
    let what = match state.op_type {
        LifecycleOpType::Delete => "PERSON_DELETING",
        LifecycleOpType::Merge => "PERSON_MERGING",
        LifecycleOpType::Unspecified => "PERSON_FENCED",
    };
    let label = state.op_type.as_op_type_str();
    let mut status = Status::failed_precondition(format!(
        "{what}: person is fenced by lifecycle op {}",
        state.op_id
    ));
    if let Ok(value) = label.parse() {
        status.metadata_mut().insert(FENCED_METADATA_KEY, value);
    }
    if let Ok(value) = state.op_id.to_string().parse() {
        status
            .metadata_mut()
            .insert(FENCED_OP_ID_METADATA_KEY, value);
    }
    status
}

/// How long the takeover scan may run before the handoff gives up. The
/// scan gates a partition's return to service, and it reads a set whose
/// size depends on another service's liveness (marks stay live while
/// their op does), so it is bounded here rather than trusted to be
/// small. Expiry fails the handoff — a partition whose fences cannot be
/// known must not serve writes.
const SCAN_TIMEOUT: Duration = Duration::from_secs(10);

/// The takeover scan: load every live mark belonging to `partition` and
/// install it in the fence map, before the partition accepts writes.
///
/// The scan cannot target a partition (the partition is a murmur2 hash
/// Postgres cannot compute), so it reads the whole live-mark set — the
/// partial mark index contains nothing but live marks — and filters
/// in-process with the same partition function request validation uses.
/// The join to `lifecycle_op` for the op type costs a lookup per row, so
/// this is not an index-only read; it stays cheap because the live-mark
/// set is small while ops complete. Merge targets are claimed but never
/// fenced, so they are excluded.
///
/// The partition's existing entries are dropped first, so a re-warm
/// (a handoff cancelled after warming, then re-acquired) converges
/// instead of accumulating. Returns how many fences were installed.
pub async fn rebuild_partition_fences(
    fallback: &PgFallback,
    fences: &FenceMap,
    partition: u32,
    num_partitions: u32,
) -> Result<usize, sqlx::Error> {
    let start = std::time::Instant::now();
    let tables = &fallback.lifecycle;
    let sql = format!(
        r#"
        SELECT lop.team_id, lop.person_id, lop.op_id, o.op_type
        FROM {op_person} lop
        JOIN {op} o ON o.op_id = lop.op_id
        WHERE lop.mark_active
          AND lop.role <> 'target'
        "#,
        op_person = tables.op_person,
        op = tables.op,
    );
    let query = sqlx::query(&sql).fetch_all(&fallback.pool);

    let rows = match tokio::time::timeout(SCAN_TIMEOUT, query).await {
        Ok(result) => result?,
        Err(_) => {
            counter!("personhog_leader_fence_scan_timeouts_total").increment(1);
            return Err(sqlx::Error::PoolTimedOut);
        }
    };
    histogram!("personhog_leader_fence_scan_duration_seconds")
        .record(start.elapsed().as_secs_f64());
    // Read next to installed is the scan's amplification: it reads the
    // whole live-mark set (the partition is a hash Postgres cannot
    // compute) and keeps ~1/num_partitions of it. This ratio growing is
    // the signal that the scan-the-world design needs revisiting.
    counter!("personhog_leader_fence_scan_rows_read_total").increment(rows.len() as u64);

    // Converge rather than accumulate: this partition's fences are
    // exactly what the marks say, not that plus whatever a previous warm
    // left behind.
    drop_partition_fences(fences, partition, num_partitions, |_| false);

    let mut installed = 0usize;
    for row in rows {
        let team_id: i32 = row.get("team_id");
        let person_id: i64 = row.get("person_id");
        if partition_for_person(team_id as i64, person_id, num_partitions) != partition {
            continue;
        }
        let op_type: String = row.get("op_type");
        fences.insert(
            PersonCacheKey {
                team_id: team_id as i64,
                person_id,
            },
            FenceState {
                op_id: row.get("op_id"),
                op_type: LifecycleOpType::from_op_type_str(&op_type),
                sealed_at: None,
            },
        );
        installed += 1;
    }
    counter!("personhog_leader_fence_scan_fences_installed_total").increment(installed as u64);
    gauge!("personhog_leader_fences_active").set(fences.len() as f64);
    Ok(installed)
}

/// Drop the fences homed on `partition`, except keys `keep` claims for another
/// partition this pod serves. The new owner rebuilds its own.
pub fn drop_partition_fences(
    fences: &FenceMap,
    partition: u32,
    num_partitions: u32,
    keep: impl Fn(&PersonCacheKey) -> bool,
) -> usize {
    let before = fences.len();
    fences.retain(|key, _| {
        partition_for_person(key.team_id, key.person_id, num_partitions) != partition || keep(key)
    });
    gauge!("personhog_leader_fences_active").set(fences.len() as f64);
    before - fences.len()
}

/// How long a person's heal verdict stands before a rejected write may
/// trigger another mark-row read. Bounds the PG read rate for a person
/// that is legitimately fenced and still receiving writes: at most one
/// point read per person per cooldown, however hot the write storm.
const HEAL_COOLDOWN: Duration = Duration::from_secs(5);

/// Above this many tracked persons, expired cooldown stamps are pruned on
/// the next trigger. Keeps the tracker bounded without a sweeper task.
const HEAL_TRACKER_PRUNE_THRESHOLD: usize = 10_000;

/// Lazily removes ghost fences — entries whose op has already settled its
/// mark row, so no `ReleaseFence` will ever arrive for them (see the
/// module docs for how the takeover scan creates these).
///
/// Triggered from the write path when a fence rejects a write, but never
/// on it: the check runs on a spawned task, the write fails as fenced
/// either way, and the caller's retry finds the fence gone. Fail-closed
/// by construction — the fence is only dropped when Postgres, the source
/// of truth, says the mark is no longer live, and only if the entry still
/// belongs to the op that was checked (a newer op's fence is never
/// touched).
pub struct FenceHealer {
    fallback: PgFallback,
    fences: FenceMap,
    /// Per-person stamp of the last triggered check, for the cooldown.
    last_checked: DashMap<PersonCacheKey, std::time::Instant>,
}

impl FenceHealer {
    pub fn new(fallback: PgFallback, fences: FenceMap) -> Self {
        Self {
            fallback,
            fences,
            last_checked: DashMap::new(),
        }
    }

    /// Kick off a background ghost check for `key`, unless one ran within
    /// the cooldown. Called after a write was rejected by `state`.
    pub fn maybe_heal(self: &Arc<Self>, key: PersonCacheKey, state: FenceState) {
        let now = std::time::Instant::now();
        if self.last_checked.len() > HEAL_TRACKER_PRUNE_THRESHOLD {
            self.last_checked
                .retain(|_, stamp| now.duration_since(*stamp) < HEAL_COOLDOWN);
        }
        // The entry API keeps check-and-stamp atomic: one write storm
        // triggers one check per cooldown, not one per rejection.
        match self.last_checked.entry(key.clone()) {
            dashmap::mapref::entry::Entry::Occupied(mut entry)
                if now.duration_since(*entry.get()) >= HEAL_COOLDOWN =>
            {
                entry.insert(now);
            }
            dashmap::mapref::entry::Entry::Vacant(entry) => {
                entry.insert(now);
            }
            dashmap::mapref::entry::Entry::Occupied(_) => return,
        }
        let healer = Arc::clone(self);
        tokio::spawn(async move {
            healer.check_and_heal(key, state).await;
        });
    }

    async fn check_and_heal(&self, key: PersonCacheKey, state: FenceState) {
        let status = match mark_status(&self.fallback, state.op_id, key.team_id, key.person_id)
            .await
        {
            Ok(status) => status,
            Err(e) => {
                counter!("personhog_leader_fence_heals_total", "outcome" => "error").increment(1);
                tracing::warn!(
                    team_id = key.team_id,
                    person_id = key.person_id,
                    op_id = %state.op_id,
                    error = %e,
                    "ghost-fence check failed; fence kept"
                );
                return;
            }
        };
        // Live is what the takeover scan installs from; anything else —
        // a terminal status or no row at all — means the op has settled
        // and its release already happened somewhere.
        if matches!(status.as_deref(), Some("marked") | Some("sealed")) {
            counter!("personhog_leader_fence_heals_total", "outcome" => "still_live").increment(1);
            return;
        }
        // Conditional on the op id: a fence re-installed by a newer op in
        // the meantime is someone else's and stays.
        let removed = self
            .fences
            .remove_if(&key, |_, current| current.op_id == state.op_id)
            .is_some();
        if removed {
            self.last_checked.remove(&key);
            gauge!("personhog_leader_fences_active").set(self.fences.len() as f64);
            counter!("personhog_leader_fence_heals_total", "outcome" => "removed").increment(1);
            tracing::info!(
                team_id = key.team_id,
                person_id = key.person_id,
                op_id = %state.op_id,
                "removed a ghost fence: its op has already settled"
            );
        }
    }
}

/// The committed-release check: the status of the op's mark row for this
/// person, straight from the source of truth. `None` when the op never
/// claimed the person (or claimed it only as a merge target — targets are
/// never destroyed). A committed release must find a live mark here before
/// it may produce a death document: the request alone must never be enough
/// to destroy a person.
pub async fn mark_status(
    fallback: &PgFallback,
    op_id: Uuid,
    team_id: i64,
    person_id: i64,
) -> Result<Option<String>, sqlx::Error> {
    let mut conn = crate::pg::acquire_timed(&fallback.pool, "mark_status").await?;
    let sql = format!(
        "SELECT status FROM {} \
         WHERE op_id = $1 AND team_id = $2 AND person_id = $3 AND role <> 'target'",
        fallback.lifecycle.op_person
    );
    sqlx::query_scalar(&sql)
        .bind(op_id)
        .bind(team_id as i32)
        .bind(person_id)
        .fetch_optional(&mut *conn)
        .await
}

/// The fold's check: the status of the op's mark row claiming this person
/// as its merge target. `None` when the op never claimed the person as
/// target — including when it holds the person under another role, which
/// would make a fold into it a saga bug. A fold must find a live mark here
/// before it may write. The check is read-time only: a fold already past
/// it when the op settles can still land (the window spans the fold
/// computation and the produce), so it closes late re-drives, not the
/// full race — the same at-least-once residual the op_id proto comment
/// states.
pub async fn target_mark_status(
    fallback: &PgFallback,
    op_id: Uuid,
    team_id: i64,
    person_id: i64,
) -> Result<Option<String>, sqlx::Error> {
    let mut conn = crate::pg::acquire_timed(&fallback.pool, "target_mark_status").await?;
    let sql = format!(
        "SELECT status FROM {} \
         WHERE op_id = $1 AND team_id = $2 AND person_id = $3 AND role = 'target'",
        fallback.lifecycle.op_person
    );
    sqlx::query_scalar(&sql)
        .bind(op_id)
        .bind(team_id as i32)
        .bind(person_id)
        .fetch_optional(&mut *conn)
        .await
}

/// A memory bound only; what a snapshot may vouch for does not depend on its age.
const MARK_SNAPSHOT_TTL: Duration = Duration::from_secs(30);

/// Keeps the snapshot map bounded without a sweeper task.
const MARK_SNAPSHOT_PRUNE_THRESHOLD: usize = 1_000;

/// Paces the prune scan under a burst of fresh snapshots.
const MARK_SNAPSHOT_PRUNE_INTERVAL: Duration = Duration::from_secs(5);

const MARK_SNAPSHOTS_TOTAL: &str = "personhog_leader_mark_snapshots_total";

#[derive(Debug, Clone, PartialEq, Eq)]
pub struct MarkRow {
    pub team_id: i64,
    pub person_id: i64,
    pub status: String,
}

#[async_trait]
pub trait MarkSource: Send + Sync {
    async fn load_op(&self, op_id: Uuid) -> Result<Vec<MarkRow>, sqlx::Error>;
}

/// Merge targets are excluded because targets are never destroyed.
pub struct PgMarkSource {
    pool: PgPool,
    op_person_table: String,
}

#[async_trait]
impl MarkSource for PgMarkSource {
    async fn load_op(&self, op_id: Uuid) -> Result<Vec<MarkRow>, sqlx::Error> {
        let mut conn = crate::pg::acquire_timed(&self.pool, "mark_snapshot").await?;
        let sql = format!(
            "SELECT team_id, person_id, status FROM {} \
             WHERE op_id = $1 AND role <> 'target'",
            self.op_person_table
        );
        let rows = sqlx::query(&sql).bind(op_id).fetch_all(&mut *conn).await?;
        Ok(rows
            .into_iter()
            .map(|row| MarkRow {
                team_id: i64::from(row.get::<i32, _>("team_id")),
                person_id: row.get("person_id"),
                status: row.get("status"),
            })
            .collect())
    }
}

/// The op's mark rows as of one read.
#[derive(Debug)]
pub struct MarkSnapshot {
    fetched_at: Instant,
    rows: HashMap<(i64, i64), String>,
}

impl MarkSnapshot {
    /// A fence this pod sealed before the read and that still stands; a
    /// takeover fence never qualifies.
    pub fn vouches_for(&self, fence: &FenceState) -> bool {
        fence
            .sealed_at
            .is_some_and(|sealed_at| sealed_at < self.fetched_at)
    }

    pub fn status(&self, team_id: i64, person_id: i64) -> Option<&str> {
        self.rows.get(&(team_id, person_id)).map(String::as_str)
    }
}

/// The op's mark rows, read once per op per pod. The row still has to
/// vouch for the (op, person) pair.
pub struct MarkVerifier {
    source: Arc<dyn MarkSource>,
    ttl: Duration,
    snapshots: DashMap<Uuid, Arc<OnceCell<Arc<MarkSnapshot>>>>,
    last_prune: Mutex<Instant>,
}

impl MarkVerifier {
    pub fn new(fallback: &PgFallback) -> Self {
        let source = PgMarkSource {
            pool: fallback.pool.clone(),
            op_person_table: fallback.lifecycle.op_person.clone(),
        };
        Self::with_source(Arc::new(source), MARK_SNAPSHOT_TTL)
    }

    pub fn with_source(source: Arc<dyn MarkSource>, ttl: Duration) -> Self {
        Self {
            source,
            ttl,
            snapshots: DashMap::new(),
            last_prune: Mutex::new(Instant::now()),
        }
    }

    /// The op's snapshot, shared by every release that arrives while it is
    /// fresh. A failed load leaves the cell empty for the next call to retry.
    pub async fn snapshot(&self, op_id: Uuid) -> Result<Arc<MarkSnapshot>, sqlx::Error> {
        loop {
            let cell = self.cell(op_id);
            let loaded_now = AtomicBool::new(false);
            let snapshot = cell
                .get_or_try_init(|| async {
                    loaded_now.store(true, Ordering::Relaxed);
                    counter!(MARK_SNAPSHOTS_TOTAL, "outcome" => "load").increment(1);
                    // Taken before the read, so a fence sealed earlier predates every row.
                    let fetched_at = Instant::now();
                    let rows = self.source.load_op(op_id).await?;
                    Ok::<_, sqlx::Error>(Arc::new(MarkSnapshot {
                        fetched_at,
                        rows: rows
                            .into_iter()
                            .map(|row| ((row.team_id, row.person_id), row.status))
                            .collect(),
                    }))
                })
                .await?;
            if loaded_now.load(Ordering::Relaxed) {
                return Ok(Arc::clone(snapshot));
            }
            // Only a snapshot someone else loaded can expire here, so a TTL
            // shorter than one load cannot make this loop spin.
            if snapshot.fetched_at.elapsed() > self.ttl {
                counter!(MARK_SNAPSHOTS_TOTAL, "outcome" => "expired").increment(1);
                self.snapshots
                    .remove_if(&op_id, |_, current| Arc::ptr_eq(current, &cell));
                continue;
            }
            counter!(MARK_SNAPSHOTS_TOTAL, "outcome" => "hit").increment(1);
            return Ok(Arc::clone(snapshot));
        }
    }

    fn cell(&self, op_id: Uuid) -> Arc<OnceCell<Arc<MarkSnapshot>>> {
        if let Some(cell) = self.snapshots.get(&op_id) {
            return Arc::clone(&cell);
        }
        if self.snapshots.len() >= MARK_SNAPSHOT_PRUNE_THRESHOLD {
            let mut last_prune = self
                .last_prune
                .lock()
                .unwrap_or_else(PoisonError::into_inner);
            if last_prune.elapsed() >= MARK_SNAPSHOT_PRUNE_INTERVAL {
                let ttl = self.ttl;
                // A cell still loading goes too; its waiters hold their own handle.
                self.snapshots.retain(|_, cell| {
                    cell.get()
                        .is_some_and(|snapshot| snapshot.fetched_at.elapsed() <= ttl)
                });
                *last_prune = Instant::now();
            }
        }
        Arc::clone(&self.snapshots.entry(op_id).or_default())
    }
}

#[cfg(test)]
mod mark_verifier_tests {
    use std::sync::atomic::AtomicUsize;

    use super::*;

    struct CountingSource {
        rows: Vec<MarkRow>,
        loads: AtomicUsize,
        fail_next_load: AtomicBool,
    }

    impl CountingSource {
        fn sealed(persons: i64) -> Arc<Self> {
            Arc::new(Self {
                rows: (1..=persons)
                    .map(|person_id| MarkRow {
                        team_id: 1,
                        person_id,
                        status: "sealed".to_string(),
                    })
                    .collect(),
                loads: AtomicUsize::new(0),
                fail_next_load: AtomicBool::new(false),
            })
        }

        fn loads(&self) -> usize {
            self.loads.load(Ordering::SeqCst)
        }
    }

    #[async_trait]
    impl MarkSource for CountingSource {
        async fn load_op(&self, _op_id: Uuid) -> Result<Vec<MarkRow>, sqlx::Error> {
            self.loads.fetch_add(1, Ordering::SeqCst);
            if self.fail_next_load.swap(false, Ordering::SeqCst) {
                return Err(sqlx::Error::PoolTimedOut);
            }
            Ok(self.rows.clone())
        }
    }

    #[tokio::test]
    async fn releases_of_one_op_share_one_load_and_unknown_persons_stay_unverified() {
        let source = CountingSource::sealed(50);
        let verifier = Arc::new(MarkVerifier::with_source(source.clone(), MARK_SNAPSHOT_TTL));
        let op = Uuid::now_v7();

        let mut releases = tokio::task::JoinSet::new();
        for person_id in 1..=50 {
            let verifier = Arc::clone(&verifier);
            releases.spawn(async move {
                verifier
                    .snapshot(op)
                    .await
                    .map(|snapshot| snapshot.status(1, person_id).map(str::to_owned))
            });
        }
        while let Some(joined) = releases.join_next().await {
            let status = joined.expect("lookup task").expect("lookup succeeds");
            assert_eq!(status.as_deref(), Some("sealed"));
        }
        assert_eq!(source.loads(), 1, "one load serves every victim of the op");

        let snapshot = verifier.snapshot(op).await.expect("lookup succeeds");
        assert_eq!(
            snapshot.status(1, 51),
            None,
            "a person the op never claimed is unverified"
        );
        assert_eq!(
            snapshot.status(2, 1),
            None,
            "the row must match the team too"
        );
        assert_eq!(source.loads(), 1);
    }

    #[test]
    fn a_snapshot_vouches_only_for_a_fence_sealed_before_its_read() {
        let read = Instant::now();
        let snapshot = MarkSnapshot {
            fetched_at: read,
            rows: HashMap::new(),
        };
        let fence = |sealed_at| FenceState {
            op_id: Uuid::nil(),
            op_type: LifecycleOpType::Delete,
            sealed_at,
        };
        assert!(snapshot.vouches_for(&fence(Some(read - Duration::from_millis(1)))));
        assert!(!snapshot.vouches_for(&fence(Some(read + Duration::from_millis(1)))));
        assert!(!snapshot.vouches_for(&fence(None)));
    }

    #[tokio::test]
    async fn an_expired_snapshot_is_read_again() {
        let source = CountingSource::sealed(1);
        let verifier = MarkVerifier::with_source(source.clone(), Duration::ZERO);
        let op = Uuid::now_v7();

        verifier.snapshot(op).await.expect("first lookup");
        assert_eq!(source.loads(), 1, "the loader keeps its own snapshot");
        let snapshot = verifier.snapshot(op).await.expect("second lookup");
        assert_eq!(snapshot.status(1, 1), Some("sealed"));
        assert_eq!(source.loads(), 2, "an expired snapshot is replaced");
    }

    #[tokio::test]
    async fn a_failed_load_is_retried_by_the_next_release() {
        let source = CountingSource::sealed(1);
        source.fail_next_load.store(true, Ordering::SeqCst);
        let verifier = MarkVerifier::with_source(source.clone(), MARK_SNAPSHOT_TTL);
        let op = Uuid::now_v7();

        verifier
            .snapshot(op)
            .await
            .expect_err("the failed load surfaces to the release");
        let snapshot = verifier.snapshot(op).await.expect("second lookup");
        assert_eq!(snapshot.status(1, 1), Some("sealed"));
        assert_eq!(source.loads(), 2, "nothing from the failed load is trusted");
    }

    #[tokio::test]
    async fn a_fence_sealed_during_the_read_is_not_vouched_for() {
        struct RecordingSource(Mutex<Option<Instant>>);

        #[async_trait]
        impl MarkSource for RecordingSource {
            async fn load_op(&self, _op_id: Uuid) -> Result<Vec<MarkRow>, sqlx::Error> {
                *self.0.lock().unwrap() = Some(Instant::now());
                tokio::time::sleep(Duration::from_millis(5)).await;
                Ok(Vec::new())
            }
        }

        let source = Arc::new(RecordingSource(Mutex::new(None)));
        let verifier = MarkVerifier::with_source(source.clone(), MARK_SNAPSHOT_TTL);
        let snapshot = verifier.snapshot(Uuid::now_v7()).await.expect("load");
        let sealed_during_read = source.0.lock().unwrap().expect("the source was read");
        assert!(!snapshot.vouches_for(&FenceState {
            op_id: Uuid::nil(),
            op_type: LifecycleOpType::Delete,
            sealed_at: Some(sealed_during_read),
        }));
    }

    #[tokio::test]
    async fn the_prune_runs_on_an_interval_and_drops_cells_that_never_loaded() {
        let source = CountingSource::sealed(1);
        let verifier = MarkVerifier::with_source(source, Duration::ZERO);
        for _ in 0..MARK_SNAPSHOT_PRUNE_THRESHOLD {
            verifier.snapshot(Uuid::now_v7()).await.expect("load");
        }
        verifier.cell(Uuid::now_v7());
        assert_eq!(verifier.snapshots.len(), MARK_SNAPSHOT_PRUNE_THRESHOLD + 1);

        verifier.snapshot(Uuid::now_v7()).await.expect("load");
        assert_eq!(
            verifier.snapshots.len(),
            MARK_SNAPSHOT_PRUNE_THRESHOLD + 2,
            "no scan inside the interval"
        );

        *verifier.last_prune.lock().unwrap() = Instant::now() - MARK_SNAPSHOT_PRUNE_INTERVAL;
        verifier.snapshot(Uuid::now_v7()).await.expect("load");
        assert_eq!(
            verifier.snapshots.len(),
            1,
            "expired snapshots and the cell that never loaded go; the new op's stays"
        );
    }
}

#[cfg(test)]
mod partition_fence_tests {
    use super::*;

    #[test]
    fn a_drop_spares_the_keys_another_home_still_claims() {
        let fences: FenceMap = Arc::new(DashMap::new());
        let homed_on = |partition: u32| {
            (1..)
                .map(|person_id| PersonCacheKey {
                    team_id: 1,
                    person_id,
                })
                .filter(move |k| partition_for_person(k.team_id, k.person_id, 8) == partition)
        };
        let [kept, dropped, elsewhere] = {
            let mut on_five = homed_on(5);
            [
                on_five.next().unwrap(),
                on_five.next().unwrap(),
                homed_on(6).next().unwrap(),
            ]
        };
        for key in [&kept, &dropped, &elsewhere] {
            fences.insert(
                key.clone(),
                FenceState {
                    op_id: Uuid::now_v7(),
                    op_type: LifecycleOpType::Delete,
                    sealed_at: None,
                },
            );
        }

        let removed = drop_partition_fences(&fences, 5, 8, |key| *key == kept);

        assert_eq!(removed, 1);
        assert!(fences.contains_key(&kept));
        assert!(!fences.contains_key(&dropped));
        assert!(fences.contains_key(&elsewhere));
    }
}