Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

70.1 KB
Raw Blame
//! Shadow-compare support for the `flags-cache-builder` consumer.
//!
//! A `shadow: true` invalidation asks the builder to prove parity, not to serve:
//! build the payload exactly as a real invalidation would, diff it against the
//! live cache entry Python (Celery) owns, and record the result — never write.
//! This module holds the semantic diff and the repeat-offender suppression. The
//! diff is pure; the suppression keeps its pending state in the flags Redis
//! tier, because it has to outlive the process. Reading the live entry and
//! emitting metrics stay in the binary.
//!
//! The diff covers all three top-level payload keys, because all three feed the
//! serve path (`flag_definitions_cache.rs` evaluates from `flags`,
//! `evaluation_metadata`, and `cohorts` as cached, with no read-time repair):
//!
//! - `flags`: per-flag comparison mirroring the Python verifier's definition of
//!   parity (`verify_team_flags` in `products/feature_flags/backend/flags_cache.py`)
//!   — flag-set equality by id, field-level comparison, missing-in-cache and
//!   stale-in-cache detection, an unevaluable stale row tolerated, and `filters`
//!   exempted when both sides agree the flag is unevaluable (the writers blank
//!   those; entries predating blanking still hold the full blob).
//! - `evaluation_metadata`: full value comparison, not just the presence check
//!   the Python verifier does. That verifier compares Python-written state
//!   against a Python DB read, so its metadata can't disagree with itself; here
//!   the Rust reimplementation of dependency staging is exactly what's under test.
//! - `cohorts`: per-cohort comparison by id, for the same reason — the Rust
//!   cohort BFS is a reimplementation that can truncate or diverge, and the
//!   serve path evaluates cohort-filtered flags from this cached array. A live
//!   entry with no `cohorts` key predates the field and is skipped, since the
//!   service falls back to its own cohort load for such entries.
//!
//! Parity is per team, not per byte: both sides are deserialized through the
//! same typed models before comparing, so JSON key order, absent-vs-null
//! optional fields, and serialization formatting can never register as
//! mismatches. Unlike the Python verifier's "compare only DB-side keys" rule
//! (which tolerates stale extra keys in raw cache JSON — the typed round-trip
//! already drops those), the field loop here walks the union of both sides'
//! keys: the typed model omits `None` fields on serialize, so a built-side
//! `None` against a cached value would otherwise be invisible.

use std::collections::HashMap;
use std::sync::Arc;
use std::time::{Duration, SystemTime, UNIX_EPOCH};

use common_redis::{Client, CustomRedisError};
use common_types::TeamId;
use serde::{Deserialize, Serialize};
use sha2::{Digest, Sha256};
use strum::EnumIter;

use crate::cohorts::cohort_models::Cohort;
use crate::flags::cache_builder::is_evaluable;
use crate::flags::flag_models::{
    EvaluationMetadata, FeatureFlag, FeatureFlagId, HypercacheFlagsWrapper,
};

/// The slice of a live cache entry the shadow diff reads. Deserialized leniently
/// (unknown fields are ignored; `evaluation_metadata` and `cohorts` may be
/// absent in entries predating them) so an old-shape entry surfaces as a
/// semantic mismatch or a skip below rather than a parse failure.
#[derive(Debug, Deserialize)]
pub struct ShadowLiveEntry {
    #[serde(default)]
    pub flags: Vec<FeatureFlag>,
    #[serde(default)]
    pub evaluation_metadata: Option<EvaluationMetadata>,
    #[serde(default)]
    pub cohorts: Option<Vec<Cohort>>,
}

/// One issue class per diff entry; used as the `issue_type` metric label, so the
/// set must stay small and static. `EnumIter` is derived so the builder's
/// startup pre-creation walks the variants instead of a hand-written list of
/// labels, which would keep compiling after a variant is added here.
#[derive(Debug, Clone, Copy, PartialEq, Eq, EnumIter)]
pub enum ShadowIssueType {
    /// Live entry has no `evaluation_metadata` key (pre-metadata entry shape).
    MissingEvaluationMetadata,
    /// Live entry's `evaluation_metadata` value differs from the fresh build's.
    EvaluationMetadataMismatch,
    /// Flag present in the fresh build but absent from the live entry.
    MissingInCache,
    /// Evaluable flag present in the live entry but absent from the fresh build.
    StaleInCache,
    /// Flag present on both sides with differing field values.
    FieldMismatch,
    /// Cohort present in the fresh build but absent from the live entry.
    CohortMissingInCache,
    /// Cohort present in the live entry but absent from the fresh build.
    CohortStaleInCache,
    /// Cohort present on both sides with differing field values.
    CohortFieldMismatch,
}

impl ShadowIssueType {
    pub fn as_label(&self) -> &'static str {
        match self {
            Self::MissingEvaluationMetadata => "missing_evaluation_metadata",
            Self::EvaluationMetadataMismatch => "evaluation_metadata_mismatch",
            Self::MissingInCache => "missing_in_cache",
            Self::StaleInCache => "stale_in_cache",
            Self::FieldMismatch => "field_mismatch",
            Self::CohortMissingInCache => "cohort_missing_in_cache",
            Self::CohortStaleInCache => "cohort_stale_in_cache",
            Self::CohortFieldMismatch => "cohort_field_mismatch",
        }
    }
}

/// One semantic difference between the fresh build and the live entry.
#[derive(Debug, Clone)]
pub struct ShadowDiff {
    pub issue_type: ShadowIssueType,
    /// What differs: `"flag 42 (checkout)"` or `"cohort 7"`. `None` for
    /// team-level issues (the evaluation-metadata ones).
    pub subject: Option<String>,
    /// Differing field names; populated for the field-mismatch issue types.
    pub fields: Vec<String>,
    /// Content hash of the disagreement, used by `MismatchTracker` to decide
    /// whether the *same* mismatch persisted across two consecutive shadow
    /// builds. Hashing the values (not just the subject + issue type) keeps an
    /// edit session — where each save races Python's rebuild with *different*
    /// content — from ever counting as a repeat.
    fingerprint: u64,
}

/// Truncated SHA-256 rather than `DefaultHasher`, because the value is written
/// to Redis and compared by a later process, which can be a different build of
/// this binary. The standard library does not promise `DefaultHasher` gives the
/// same output across Rust versions, so a toolchain bump would change every
/// fingerprint and demote each team's pending mismatch to a first sighting for
/// one build after the deploy. SHA-256 of the same input is the same value in
/// every build, which is what a persisted comparison needs.
///
/// 64 bits of the digest is enough, because a collision would have to be two
/// different disagreements for the same team inside the TTL.
fn fingerprint(
    issue_type: ShadowIssueType,
    subject: Option<&str>,
    built: Option<&serde_json::Value>,
    cached: Option<&serde_json::Value>,
) -> u64 {
    let mut hasher = Sha256::new();
    hasher.update(issue_type.as_label().as_bytes());
    for part in [
        subject.map(|s| s.to_string()),
        built.map(|v| v.to_string()),
        cached.map(|v| v.to_string()),
    ] {
        // The marker byte separates the parts and tells an absent part from an
        // empty one. It is not an escape: `subject` is a formatted string that
        // carries whatever bytes a flag key carries, so a part can contain a
        // marker byte and the encoding is not self-delimiting. The part count is
        // fixed at three, which is what keeps distinct disagreements apart in
        // practice. A collision would need a contrived flag key, and would cost
        // one wrongly paired confirmation for one team.
        match part {
            Some(part) => {
                hasher.update([0x01]);
                hasher.update(part.as_bytes());
            }
            None => hasher.update([0x00]),
        }
    }
    let digest = hasher.finalize();
    u64::from_be_bytes(digest[..8].try_into().expect("SHA-256 digest is 32 bytes"))
}

fn diff(
    issue_type: ShadowIssueType,
    subject: String,
    built: Option<&serde_json::Value>,
    cached: Option<&serde_json::Value>,
    fields: Vec<String>,
) -> ShadowDiff {
    ShadowDiff {
        issue_type,
        fingerprint: fingerprint(issue_type, Some(&subject), built, cached),
        subject: Some(subject),
        fields,
    }
}

/// Field names whose values differ between two serialized objects, over the
/// union of both sides' keys (see the module doc for why the union), skipping
/// `skip_key` when given. Sorted for deterministic logs and fingerprints.
fn differing_fields(
    built: &serde_json::Value,
    cached: &serde_json::Value,
    skip_key: Option<&str>,
) -> Vec<String> {
    let (Some(built_map), Some(cached_map)) = (built.as_object(), cached.as_object()) else {
        return Vec::new();
    };
    let mut fields: Vec<String> = built_map
        .keys()
        .chain(cached_map.keys().filter(|k| !built_map.contains_key(*k)))
        .filter(|key| Some(key.as_str()) != skip_key)
        .filter(|key| built_map.get(*key) != cached_map.get(*key))
        .cloned()
        .collect();
    fields.sort_unstable();
    fields
}

/// Semantically diff a fresh build against the live cache entry. Empty result
/// means parity across all three payload keys.
pub fn diff_live_entry(built: &HypercacheFlagsWrapper, live: &ShadowLiveEntry) -> Vec<ShadowDiff> {
    // Mirrors the Python verifier's MISSING_EVALUATION_METADATA early return:
    // an entry that old predates the current payload shape, so a per-flag diff
    // would only bury the actual finding.
    let Some(live_metadata) = &live.evaluation_metadata else {
        return vec![ShadowDiff {
            issue_type: ShadowIssueType::MissingEvaluationMetadata,
            subject: None,
            fields: Vec::new(),
            fingerprint: fingerprint(ShadowIssueType::MissingEvaluationMetadata, None, None, None),
        }];
    };

    let mut diffs: Vec<ShadowDiff> = Vec::new();

    if *live_metadata != built.evaluation_metadata {
        let built_value = serde_json::to_value(&built.evaluation_metadata).unwrap_or_default();
        let cached_value = serde_json::to_value(live_metadata).unwrap_or_default();
        diffs.push(ShadowDiff {
            issue_type: ShadowIssueType::EvaluationMetadataMismatch,
            subject: None,
            fields: differing_fields(&built_value, &cached_value, None),
            fingerprint: fingerprint(
                ShadowIssueType::EvaluationMetadataMismatch,
                None,
                Some(&built_value),
                Some(&cached_value),
            ),
        });
    }

    diffs.extend(diff_flags(&built.flags, &live.flags));

    // A live entry without the `cohorts` key predates the field; the service
    // falls back to its own cohort load for such entries, so there is nothing
    // wrong being served from it and nothing to compare.
    if let Some(live_cohorts) = &live.cohorts {
        let built_cohorts = built.cohorts.as_deref().unwrap_or(&[]);
        diffs.extend(diff_cohorts(built_cohorts, live_cohorts));
    }

    diffs
}

fn diff_flags(built: &[FeatureFlag], live: &[FeatureFlag]) -> Vec<ShadowDiff> {
    let built_by_id: HashMap<FeatureFlagId, &FeatureFlag> =
        built.iter().map(|f| (f.id, f)).collect();
    let cached_by_id: HashMap<FeatureFlagId, &FeatureFlag> =
        live.iter().map(|f| (f.id, f)).collect();

    let mut diffs: Vec<(FeatureFlagId, ShadowDiff)> = Vec::new();

    for flag in built {
        let subject = format!("flag {} ({})", flag.id, flag.key);
        match cached_by_id.get(&flag.id) {
            None => {
                let built_value = serde_json::to_value(flag).unwrap_or_default();
                diffs.push((
                    flag.id,
                    diff(
                        ShadowIssueType::MissingInCache,
                        subject,
                        Some(&built_value),
                        None,
                        Vec::new(),
                    ),
                ));
            }
            Some(cached) => {
                if let Some(d) = compare_flag_fields(flag, cached) {
                    diffs.push((flag.id, d));
                }
            }
        }
    }

    for flag in live {
        if built_by_id.contains_key(&flag.id) {
            continue;
        }
        // An unevaluable cached row is invisible to the matcher, so its
        // presence is not drift — same tolerance as the Python verifier.
        if !is_evaluable(flag) {
            continue;
        }
        let cached_value = serde_json::to_value(flag).unwrap_or_default();
        diffs.push((
            flag.id,
            diff(
                ShadowIssueType::StaleInCache,
                format!("flag {} ({})", flag.id, flag.key),
                None,
                Some(&cached_value),
                Vec::new(),
            ),
        ));
    }

    diffs.sort_by_key(|(id, _)| *id);
    diffs.into_iter().map(|(_, d)| d).collect()
}

/// Field-level comparison of one flag present on both sides. `filters` is
/// exempt when both sides agree the flag is unevaluable — only the cache
/// writers blank filters, and entries predating blanking still hold the full
/// blob the matcher never reads.
fn compare_flag_fields(built: &FeatureFlag, cached: &FeatureFlag) -> Option<ShadowDiff> {
    let built_value = serde_json::to_value(built).unwrap_or_default();
    let cached_value = serde_json::to_value(cached).unwrap_or_default();
    let both_unevaluable = !is_evaluable(built) && !is_evaluable(cached);
    let skip = both_unevaluable.then_some("filters");

    let fields = differing_fields(&built_value, &cached_value, skip);
    if fields.is_empty() {
        return None;
    }
    Some(diff(
        ShadowIssueType::FieldMismatch,
        format!("flag {} ({})", built.id, built.key),
        Some(&built_value),
        Some(&cached_value),
        fields,
    ))
}

/// Per-cohort comparison by id, order-insensitive (the Rust builder sorts by
/// id; Python appends in load order). No unevaluable tolerance: both writers
/// emit exactly the referenced set, so any set or field difference is drift.
fn diff_cohorts(built: &[Cohort], live: &[Cohort]) -> Vec<ShadowDiff> {
    let built_by_id: HashMap<i32, &Cohort> = built.iter().map(|c| (c.id, c)).collect();
    let cached_by_id: HashMap<i32, &Cohort> = live.iter().map(|c| (c.id, c)).collect();

    let mut diffs: Vec<(i32, ShadowDiff)> = Vec::new();

    for cohort in built {
        let subject = format!("cohort {}", cohort.id);
        match cached_by_id.get(&cohort.id) {
            None => {
                let built_value = serde_json::to_value(cohort).unwrap_or_default();
                diffs.push((
                    cohort.id,
                    diff(
                        ShadowIssueType::CohortMissingInCache,
                        subject,
                        Some(&built_value),
                        None,
                        Vec::new(),
                    ),
                ));
            }
            Some(cached) => {
                let built_value = serde_json::to_value(cohort).unwrap_or_default();
                let cached_value = serde_json::to_value(cached).unwrap_or_default();
                let fields = differing_fields(&built_value, &cached_value, None);
                if !fields.is_empty() {
                    diffs.push((
                        cohort.id,
                        diff(
                            ShadowIssueType::CohortFieldMismatch,
                            subject,
                            Some(&built_value),
                            Some(&cached_value),
                            fields,
                        ),
                    ));
                }
            }
        }
    }

    for cohort in live {
        if built_by_id.contains_key(&cohort.id) {
            continue;
        }
        let cached_value = serde_json::to_value(cohort).unwrap_or_default();
        diffs.push((
            cohort.id,
            diff(
                ShadowIssueType::CohortStaleInCache,
                format!("cohort {}", cohort.id),
                None,
                Some(&cached_value),
                Vec::new(),
            ),
        ));
    }

    diffs.sort_by_key(|(id, _)| *id);
    diffs.into_iter().map(|(_, d)| d).collect()
}

/// The tracker's judgement on one shadow build's diffs: which mismatches were
/// also present (same fingerprint) on the team's previous shadow build — those
/// count — and which are first sightings, suppressed as probable races against
/// Python's own rebuild of the team.
#[derive(Debug, Default)]
pub struct ShadowObservation {
    pub confirmed: Vec<ShadowDiff>,
    pub first_sight: Vec<ShadowDiff>,
    /// Tracker store accesses that failed while this observation was made. The
    /// caller counts them. They never move a diff into `confirmed`.
    pub store_errors: Vec<TrackerStoreOp>,
}

impl ShadowObservation {
    pub fn is_match(&self) -> bool {
        self.confirmed.is_empty() && self.first_sight.is_empty()
    }
}

/// Which tracker store access failed. Used as a metric label, so the set must
/// stay small and static. `EnumIter` is derived for the same reason as
/// `ShadowIssueType`: startup pre-creation iterates the variants.
#[derive(Debug, Clone, Copy, PartialEq, Eq, EnumIter)]
pub enum TrackerStoreOp {
    /// Read of the team's previous observation. The build falls back to first
    /// sight, so confirmations are lost while reads fail.
    Read,
    /// Write of this observation. The previous observation is cleared in its place,
    /// so the team's next shadow build finds nothing to compare against and also
    /// reports first sight.
    Write,
    /// Clear of the team's pending state, either after a clean build or in place of
    /// a write that failed.
    Clear,
}

impl TrackerStoreOp {
    pub fn as_label(&self) -> &'static str {
        match self {
            Self::Read => "read",
            Self::Write => "write",
            Self::Clear => "clear",
        }
    }
}

/// Redis key for a team's pending mismatch. The `posthog:1:` prefix is how
/// django-redis addresses keys in this tier (see `warm_run_status`). The
/// `feature_flags/shadow_mismatch/` path keeps these keys clear of the cache
/// entries the serve path reads, which are
/// `posthog:1:cache/teams/{team}/feature_flags/flags.json` (see
/// `HYPERCACHE_NAMESPACE` and `HYPERCACHE_OBJECT_NAME` in `cache_writer`): those
/// hold `cache/` in the segment where these hold `feature_flags/`, and Python's
/// hypercache scans match on the `cache/` form.
fn pending_key(team_id: TeamId) -> String {
    format!("posthog:1:feature_flags/shadow_mismatch/{team_id}")
}

/// Cap on one tracker Redis call. `common_hypercache` puts the same bound around
/// its own Redis access, and the tracker needs it for the same reason: a shadow
/// team that waits on Redis delays the real invalidations in the batches behind
/// it. The client's own response timeout cannot carry this, because it is
/// configurable and `FLAGS_REDIS_RESPONSE_TIMEOUT_MS=0` disables it entirely.
const STORE_TIMEOUT: Duration = Duration::from_millis(500);

/// Cap on the fingerprints stored for one team. The state shadow mode exists to
/// find is a systematic builder bug, which makes every flag on every team
/// disagree at once, and the whole set would then be written, held for the TTL,
/// and read back on the team's next build, against the Redis tier the serve path
/// reads from. Past the cap a team confirms only the disagreements that fit, which
/// is the right trade: a team with hundreds of confirmed diffs has already told
/// you what you needed to know. The confirmed log line is capped for the same
/// reason.
///
/// The cap takes the first entries of `diff_live_entry`'s stable order, so a given
/// disagreement either fits on every build or on none, which makes its
/// confirmation deterministic. An unstable subset would leave the aggregate
/// confirmed count about the same, because the next build compares every diff it
/// finds against whatever was stored, but a specific flag would then confirm on
/// some builds and not others.
const MAX_STORED_FINGERPRINTS: usize = 256;

/// The shortest gap between two observations that can confirm a mismatch, which is
/// the lower bound on the window `FLAGS_CACHE_SHADOW_MISMATCH_TTL` bounds above.
/// Two invalidations for one team that fall just outside
/// `FLAGS_CACHE_COALESCE_WINDOW_MS` become two builds a fraction of a second apart,
/// and a row written between their two Postgres reads then confirms as a parity
/// defect. Sized above one cohort recalculation episode, which holds
/// `is_calculating` for seconds to minutes, and far below the TTL so a quiet team
/// still confirms persistent drift.
///
/// A constant where the TTL beside it is a setting, because this bound tunes shadow
/// telemetry on a path that never writes the cache and never serves a request. A
/// change to it can wait for a deploy of this binary.
pub const MIN_CONFIRM_INTERVAL: Duration = Duration::from_secs(60);

/// The tracker's stored state for one team: each fingerprint from the team's last
/// shadow build, against the time the first build that saw it observed it.
///
/// The time is per fingerprint and not per build, because a team's diff set
/// changes while one disagreement inside it stays. One timestamp for the whole
/// set restarts on every such change, and the disagreement that never went away
/// then never reaches the interval. Teams that rebuild most often would suffer
/// that most, and those are the teams this instrument exists to watch.
///
/// The times are unix seconds and not `Instant`s, because the value is written to
/// Redis and read by a later process on another pod. An `Instant` is
/// process-local and cannot be serialized.
///
/// Pairs and not a map: `StoredPending` is untagged, so serde buffers the value
/// before it picks a variant, and that buffer holds every map key as a string. A
/// `HashMap<u64, _>` reads back from it as a type error, which would report every
/// stored value as unparseable.
#[derive(Serialize, Deserialize, Default)]
struct PendingObservation {
    first_seen: Vec<(u64, u64)>,
}

/// The shapes a stored value can have. `Legacy` is the bare fingerprint array
/// written before the confirmation window had a lower bound, and reading it stops
/// a deploy from counting every pre-deploy key as a parse failure on
/// `flags_cache_shadow_tracker_store_errors_total{op="read"}`, which would report
/// a TTL's worth of read errors that are really a shape change. The variant can
/// go once a full TTL has elapsed past that deploy.
///
/// A legacy value carries no times, so it reads as an empty observation. It must
/// not confirm: a fingerprint this build cannot date would otherwise count as
/// arbitrarily old and confirm at once, which is the race the interval filters.
///
/// `Current` is first because an untagged enum takes the first variant that
/// matches, and a bare array cannot deserialize into `PendingObservation`.
#[derive(Deserialize)]
#[serde(untagged)]
enum StoredPending {
    Current(PendingObservation),
    /// The fingerprints are never read, but the variant still names their type
    /// rather than accepting anything. A value that is neither shape has to stay
    /// a parse error, so that a genuinely corrupt entry keeps reaching the failure
    /// counter instead of passing as an empty observation.
    Legacy(#[allow(dead_code)] Vec<u64>),
}

/// Wall-clock seconds since the unix epoch. A clock set before 1970 resolves to
/// zero, which stores a time the next build reads as arbitrarily old. That needs
/// a system clock decades wrong, and the alternative is to fail a diagnostics
/// build on a clock read.
fn unix_seconds(now: SystemTime) -> u64 {
    now.duration_since(UNIX_EPOCH).map_or(0, |d| d.as_secs())
}

/// Bound one tracker call and report an elapsed timer as the client's own timeout
/// error, so callers have a single error type to handle.
async fn with_timeout<T>(
    call: impl std::future::Future<Output = Result<T, CustomRedisError>>,
) -> Result<T, CustomRedisError> {
    match tokio::time::timeout(STORE_TIMEOUT, call).await {
        Ok(result) => result,
        Err(_elapsed) => Err(CustomRedisError::Timeout),
    }
}

/// Repeat-offender suppression: a mismatch counts only when the tracker has held
/// the same fingerprint for at least `MIN_CONFIRM_INTERVAL`, across builds no more
/// than `ttl` apart. A shadow build races Python's own rebuild of the same team,
/// so a single-shot mismatch is expected noise; Python repairs the entry and the
/// next shadow build of the team comes back clean, clearing the pending state.
///
/// The window needs both bounds. Without the lower bound, a repeat means only
/// "the build before this one", and the builder coalesces a team's invalidations
/// over a window of a few hundred milliseconds, so two invalidations just outside
/// it become two builds a fraction of a second apart. A row that is being written
/// between those two Postgres reads then confirms as a parity defect: cohort
/// recalculation state and a referencing flag's `version` both move on their own
/// schedule. Without the upper bound, a disagreement Python has since repaired
/// stays eligible to confirm forever.
///
/// The pending state lives in the flags Redis tier rather than in the process,
/// because builder pods roll and Kafka partitions move between them far more
/// often than the TTL elapses. Process-local state made the real confirmation
/// window "as long as this pod lives", so a team that rebuilds less often than
/// that could never confirm, however wrong its cache entry was.
///
/// Expiry is the TTL on the Redis key. Nothing in this process sweeps or bounds
/// the pending set.
pub struct MismatchTracker {
    /// The same client the hypercache writer and reader use.
    redis: Arc<dyn Client + Send + Sync>,
    ttl: Duration,
}

impl MismatchTracker {
    pub fn new(redis: Arc<dyn Client + Send + Sync>, ttl: Duration) -> Self {
        Self { redis, ttl }
    }

    /// Fold one shadow build's diffs into the store and split them into confirmed
    /// and first-sight. A diff confirms when its fingerprint was also in the
    /// team's previous, unexpired observation *and* that observation is at least
    /// `MIN_CONFIRM_INTERVAL` old. An empty `diffs` clears the team's pending
    /// state.
    ///
    /// `now` comes from the caller rather than from a clock read here, so the
    /// elapsed comparison is stated at the call site and tests can pass literal
    /// instants.
    ///
    /// Store failures resolve towards first sight and none of them fail the shadow
    /// build: a mismatch we cannot prove is a repeat is reported as a first
    /// sighting, so an unreachable store loses confirmations rather than inventing
    /// them. The failures come back in `store_errors` to be counted.
    ///
    /// That fallback needs the team's stale pending state gone, and removing it is
    /// itself a store access that can fail. When a clean build's clear fails, or
    /// when a write and the clear that compensates for it both fail, the previous
    /// fingerprints stand until the TTL. A disagreement that then returns byte for
    /// byte can confirm one build early. The counter shows it: `op="clear"` alone,
    /// or `op="write"` together with `op="clear"`.
    pub async fn observe(
        &self,
        team_id: TeamId,
        diffs: Vec<ShadowDiff>,
        now: SystemTime,
    ) -> ShadowObservation {
        let key = pending_key(team_id);

        if diffs.is_empty() {
            return ShadowObservation {
                store_errors: self.clear_pending(team_id, key).await,
                ..ShadowObservation::default()
            };
        }

        let now_secs = unix_seconds(now);
        let mut store_errors = Vec::new();
        let prior = match self.read_pending(key.clone()).await {
            Ok(prior) => prior,
            Err(e) => {
                tracing::warn!(team_id, error = %e, "Shadow mismatch tracker read failed; counting this build as a first sighting");
                store_errors.push(TrackerStoreOp::Read);
                HashMap::new()
            }
        };

        // A fingerprint the tracker already holds keeps the time it was first
        // stored with, so a disagreement accumulates elapsed time even while the
        // rest of the team's diff set changes around it. A fingerprint this build
        // has not seen before starts its window now.
        //
        // A stored time can sit ahead of this pod's clock, and it is kept as it
        // is. Do not clamp it to `now_secs`. The builder pods share no clock, so
        // "the pod that wrote this ran ahead" and "my own clock is behind" both
        // read as `at > now_secs`, and a clamp answers both the same way. A pod
        // running behind would then pull the stored time back to its own clock,
        // the fingerprint would look older than it is, and a later pod would
        // confirm before the interval had really elapsed. Keeping the stored time
        // costs a confirmation delayed by the skew, which is the direction every
        // other tracker failure takes. See the test named for this.
        let first_seen: HashMap<u64, u64> = diffs
            .iter()
            .take(MAX_STORED_FINGERPRINTS)
            .map(|d| {
                let at = prior.get(&d.fingerprint).copied().unwrap_or(now_secs);
                (d.fingerprint, at)
            })
            .collect();

        if let Err(e) = self.write_pending(key.clone(), &first_seen).await {
            tracing::warn!(team_id, error = %e, "Shadow mismatch tracker write failed; clearing the team's pending state instead");
            store_errors.push(TrackerStoreOp::Write);
            // A failed SETEX leaves the previous build's fingerprints in place with
            // their own TTL, so the team's next build would compare against an
            // observation from two builds ago and could confirm a disagreement that
            // the build in between did not see. Clearing keeps the fallback at first
            // sight. Redis serves DEL when it refuses a write for `maxmemory`, which
            // is the case that produces this.
            store_errors.extend(self.clear_pending(team_id, key).await);
        }

        // `saturating_sub` covers a stored time ahead of this pod's clock, which
        // resolves to no elapsed time and therefore to first sight.
        let min_interval = MIN_CONFIRM_INTERVAL.as_secs();
        let (confirmed, first_sight) = diffs.into_iter().partition(|d| {
            prior
                .get(&d.fingerprint)
                .is_some_and(|at| now_secs.saturating_sub(*at) >= min_interval)
        });
        ShadowObservation {
            confirmed,
            first_sight,
            store_errors,
        }
    }

    /// The team's previous shadow build, as each fingerprint against the time it
    /// was first seen. An absent key is how the TTL expires the window, so it
    /// reads as "no previous build" and not as an error. A value that does not
    /// parse is an error, because a stored shape this build cannot read is worth
    /// seeing on the failure counter.
    async fn read_pending(&self, key: String) -> Result<HashMap<u64, u64>, CustomRedisError> {
        match with_timeout(self.redis.get(key)).await {
            Ok(raw) => {
                let stored: StoredPending = serde_json::from_str(&raw)
                    .map_err(|e| CustomRedisError::ParseError(e.to_string()))?;
                Ok(match stored {
                    StoredPending::Current(pending) => pending.first_seen.into_iter().collect(),
                    StoredPending::Legacy(_) => HashMap::new(),
                })
            }
            Err(CustomRedisError::NotFound) => Ok(HashMap::new()),
            Err(e) => Err(e),
        }
    }

    async fn write_pending(
        &self,
        key: String,
        first_seen: &HashMap<u64, u64>,
    ) -> Result<(), CustomRedisError> {
        let payload = serde_json::to_string(&PendingObservation {
            first_seen: first_seen.iter().map(|(fp, at)| (*fp, *at)).collect(),
        })
        .map_err(|e| CustomRedisError::ParseError(e.to_string()))?;
        with_timeout(self.redis.setex(key, payload, self.ttl.as_secs())).await
    }

    /// Drop the team's pending state, so the next mismatch starts over as a first
    /// sighting. Called after a clean build, and in place of a write that failed.
    ///
    /// A failed clear is the residual described on `observe`. It needs the same
    /// disagreement to come back byte for byte inside the TTL, and the fingerprint
    /// hashes the content, so that is a real recurrence rather than the rebuild
    /// race the suppression exists to filter.
    async fn clear_pending(&self, team_id: TeamId, key: String) -> Vec<TrackerStoreOp> {
        match with_timeout(self.redis.del(key)).await {
            Ok(()) => Vec::new(),
            Err(e) => {
                tracing::warn!(team_id, error = %e, "Shadow mismatch tracker clear failed; the team's pending state stands until it expires");
                vec![TrackerStoreOp::Clear]
            }
        }
    }
}

/// Render diffs for a log line, capped in entries and bytes: at full shadow
/// volume an unbounded dump of a large team's field diffs would swamp the
/// log pipeline.
pub fn summarize_diffs(diffs: &[ShadowDiff], max_entries: usize, max_bytes: usize) -> String {
    let mut out = String::new();
    let mut rendered = 0;
    for diff in diffs.iter().take(max_entries) {
        let label = diff.issue_type.as_label();
        let head = match &diff.subject {
            Some(subject) => format!("{subject}: {label}"),
            None => label.to_string(),
        };
        let entry = if diff.fields.is_empty() {
            head
        } else {
            format!("{head}[{}]", diff.fields.join(","))
        };
        let separator = if out.is_empty() { 0 } else { 2 };
        if out.len() + separator + entry.len() > max_bytes {
            break;
        }
        if !out.is_empty() {
            out.push_str("; ");
        }
        out.push_str(&entry);
        rendered += 1;
    }
    if rendered < diffs.len() {
        out.push_str(&format!("; +{} more", diffs.len() - rendered));
    }
    out
}

#[cfg(test)]
mod tests {
    use std::time::Duration;

    use common_redis::{MockRedisCall, MockRedisClient, MockRedisValue};
    use serde_json::json;

    use super::*;

    fn flag_from_json(value: serde_json::Value) -> FeatureFlag {
        serde_json::from_value(value).expect("flag json must parse")
    }

    fn base_flag_json(id: i32) -> serde_json::Value {
        json!({
            "id": id,
            "team_id": 1,
            "key": format!("flag-{id}"),
            "filters": {"groups": [{"properties": [], "rollout_percentage": 100.0}]},
            "active": true,
            "deleted": false,
            "has_experiment": false,
        })
    }

    fn base_cohort_json(id: i32) -> serde_json::Value {
        json!({
            "id": id,
            "team_id": 1,
            "name": format!("cohort-{id}"),
            "deleted": false,
            "filters": {"properties": {"type": "OR", "values": []}},
            "is_calculating": false,
            "is_static": false,
            "errors_calculating": 0,
            "groups": [],
        })
    }

    fn cohort_from_json(value: serde_json::Value) -> Cohort {
        serde_json::from_value(value).expect("cohort json must parse")
    }

    fn wrapper(flags: Vec<FeatureFlag>) -> HypercacheFlagsWrapper {
        HypercacheFlagsWrapper {
            flags,
            evaluation_metadata: EvaluationMetadata::default(),
            cohorts: Some(Vec::new()),
        }
    }

    fn live(flags: Vec<FeatureFlag>) -> ShadowLiveEntry {
        ShadowLiveEntry {
            flags,
            evaluation_metadata: Some(EvaluationMetadata::default()),
            cohorts: Some(Vec::new()),
        }
    }

    fn issue_types(diffs: &[ShadowDiff]) -> Vec<ShadowIssueType> {
        diffs.iter().map(|d| d.issue_type).collect()
    }

    #[test]
    fn identical_sides_match() {
        let built = wrapper(vec![flag_from_json(base_flag_json(1))]);
        let cached = live(vec![flag_from_json(base_flag_json(1))]);
        assert!(diff_live_entry(&built, &cached).is_empty());
    }

    #[test]
    fn key_order_only_difference_is_a_match() {
        // Same content, keys in a different order and rollout as int vs float —
        // parity is semantic, not byte-for-byte.
        let built = wrapper(vec![flag_from_json(json!({
            "id": 1,
            "team_id": 1,
            "key": "flag-1",
            "active": true,
            "deleted": false,
            "has_experiment": false,
            "filters": {"groups": [{"rollout_percentage": 100.0, "properties": []}]},
        }))]);
        let cached: ShadowLiveEntry = serde_json::from_value(json!({
            "evaluation_metadata": {
                "dependency_stages": [],
                "flags_with_missing_deps": [],
                "transitive_deps": {},
            },
            "cohorts": [],
            "flags": [{
                "filters": {"groups": [{"properties": [], "rollout_percentage": 100}]},
                "has_experiment": false,
                "deleted": false,
                "active": true,
                "key": "flag-1",
                "team_id": 1,
                "id": 1,
            }],
        }))
        .expect("live entry must parse");
        assert!(diff_live_entry(&built, &cached).is_empty());
    }

    #[test]
    fn null_optional_field_matches_absent_field() {
        // Python's serializer can emit explicit nulls where the typed model
        // omits the key; both deserialize to None, so this must not be drift.
        let built = wrapper(vec![flag_from_json(base_flag_json(1))]);
        let mut cached_json = base_flag_json(1);
        cached_json["filters"]["payloads"] = serde_json::Value::Null;
        cached_json["evaluation_runtime"] = serde_json::Value::Null;
        let cached = live(vec![flag_from_json(cached_json)]);
        assert!(diff_live_entry(&built, &cached).is_empty());
    }

    #[test]
    fn field_change_reports_field_mismatch_with_field_names() {
        let built = wrapper(vec![flag_from_json(base_flag_json(1))]);
        let mut cached_json = base_flag_json(1);
        cached_json["filters"]["groups"][0]["rollout_percentage"] = json!(50.0);
        cached_json["has_experiment"] = json!(true);
        let cached = live(vec![flag_from_json(cached_json)]);

        let diffs = diff_live_entry(&built, &cached);
        assert_eq!(issue_types(&diffs), vec![ShadowIssueType::FieldMismatch]);
        assert_eq!(diffs[0].fields, vec!["filters", "has_experiment"]);
        assert_eq!(diffs[0].subject.as_deref(), Some("flag 1 (flag-1)"));
    }

    #[test]
    fn built_none_field_against_cached_value_is_a_mismatch() {
        // The typed model omits None fields on serialize, so a union-of-keys
        // walk is required to see a cached-only value like this one.
        let built = wrapper(vec![flag_from_json(base_flag_json(1))]);
        let mut cached_json = base_flag_json(1);
        cached_json["bucketing_identifier"] = json!("device_id");
        let cached = live(vec![flag_from_json(cached_json)]);

        let diffs = diff_live_entry(&built, &cached);
        assert_eq!(issue_types(&diffs), vec![ShadowIssueType::FieldMismatch]);
        assert_eq!(diffs[0].fields, vec!["bucketing_identifier"]);
    }

    #[test]
    fn missing_flag_reports_missing_in_cache() {
        let built = wrapper(vec![
            flag_from_json(base_flag_json(1)),
            flag_from_json(base_flag_json(2)),
        ]);
        let cached = live(vec![flag_from_json(base_flag_json(1))]);

        let diffs = diff_live_entry(&built, &cached);
        assert_eq!(issue_types(&diffs), vec![ShadowIssueType::MissingInCache]);
        assert_eq!(diffs[0].subject.as_deref(), Some("flag 2 (flag-2)"));
    }

    #[test]
    fn stale_active_flag_reports_stale_in_cache() {
        let built = wrapper(vec![flag_from_json(base_flag_json(1))]);
        let cached = live(vec![
            flag_from_json(base_flag_json(1)),
            flag_from_json(base_flag_json(2)),
        ]);

        let diffs = diff_live_entry(&built, &cached);
        assert_eq!(issue_types(&diffs), vec![ShadowIssueType::StaleInCache]);
        assert_eq!(diffs[0].subject.as_deref(), Some("flag 2 (flag-2)"));
    }

    #[test]
    fn stale_unevaluable_flag_is_tolerated() {
        // Matcher-invisible rows aren't drift — mirrors the Python verifier.
        let built = wrapper(vec![flag_from_json(base_flag_json(1))]);
        let mut stale = base_flag_json(2);
        stale["active"] = json!(false);
        let cached = live(vec![
            flag_from_json(base_flag_json(1)),
            flag_from_json(stale),
        ]);

        assert!(diff_live_entry(&built, &cached).is_empty());
    }

    #[test]
    fn blanked_filters_on_unevaluable_flag_are_tolerated() {
        // Fresh build blanks unevaluable filters; a live entry predating
        // blanking still holds the full blob. Both agree the flag is
        // unevaluable, so `filters` is exempt.
        let mut built_json = base_flag_json(1);
        built_json["active"] = json!(false);
        built_json["filters"] = json!({"groups": []});
        let mut cached_json = base_flag_json(1);
        cached_json["active"] = json!(false);
        let built = wrapper(vec![flag_from_json(built_json)]);
        let cached = live(vec![flag_from_json(cached_json)]);

        assert!(diff_live_entry(&built, &cached).is_empty());
    }

    #[test]
    fn active_disagreement_still_reports() {
        // The blanked-filters exemption requires both sides unevaluable; a
        // disagreement about `active` itself is always drift.
        let mut cached_json = base_flag_json(1);
        cached_json["active"] = json!(false);
        let built = wrapper(vec![flag_from_json(base_flag_json(1))]);
        let cached = live(vec![flag_from_json(cached_json)]);

        let diffs = diff_live_entry(&built, &cached);
        assert_eq!(issue_types(&diffs), vec![ShadowIssueType::FieldMismatch]);
        assert!(diffs[0].fields.contains(&"active".to_string()));
    }

    #[test]
    fn missing_evaluation_metadata_short_circuits() {
        let built = wrapper(vec![flag_from_json(base_flag_json(1))]);
        let cached = ShadowLiveEntry {
            flags: vec![],
            evaluation_metadata: None,
            cohorts: Some(Vec::new()),
        };

        let diffs = diff_live_entry(&built, &cached);
        assert_eq!(
            issue_types(&diffs),
            vec![ShadowIssueType::MissingEvaluationMetadata]
        );
        assert_eq!(diffs[0].subject, None);
    }

    #[test]
    fn evaluation_metadata_value_difference_reports() {
        // The Python verifier only checks presence; here the Rust dependency
        // staging is under test, so the value must match too.
        let mut built = wrapper(vec![flag_from_json(base_flag_json(1))]);
        built.evaluation_metadata = EvaluationMetadata {
            dependency_stages: vec![vec![1]],
            flags_with_missing_deps: vec![],
            transitive_deps: [(1, Default::default())].into(),
            ..Default::default()
        };
        let mut cached = live(vec![flag_from_json(base_flag_json(1))]);
        cached.evaluation_metadata = Some(EvaluationMetadata {
            dependency_stages: vec![vec![1]],
            flags_with_missing_deps: vec![1],
            transitive_deps: [(1, Default::default())].into(),
            ..Default::default()
        });

        let diffs = diff_live_entry(&built, &cached);
        assert_eq!(
            issue_types(&diffs),
            vec![ShadowIssueType::EvaluationMetadataMismatch]
        );
        assert_eq!(diffs[0].fields, vec!["flags_with_missing_deps"]);
        assert_eq!(diffs[0].subject, None);
    }

    #[test]
    fn equal_evaluation_metadata_matches_regardless_of_map_order() {
        // transitive_deps is a map and stages are sorted by construction, so
        // equal metadata built independently must compare equal.
        let metadata = || EvaluationMetadata {
            dependency_stages: vec![vec![1, 2]],
            flags_with_missing_deps: vec![],
            transitive_deps: [(1, Default::default()), (2, Default::default())].into(),
            ..Default::default()
        };
        let mut built = wrapper(vec![]);
        built.evaluation_metadata = metadata();
        let mut cached = live(vec![]);
        cached.evaluation_metadata = Some(metadata());

        assert!(diff_live_entry(&built, &cached).is_empty());
    }

    #[test]
    fn cohort_field_difference_reports_with_field_names() {
        let mut built = wrapper(vec![]);
        built.cohorts = Some(vec![cohort_from_json(base_cohort_json(7))]);
        let mut cached_json = base_cohort_json(7);
        cached_json["filters"] = json!({"properties": {"type": "AND", "values": []}});
        let mut cached = live(vec![]);
        cached.cohorts = Some(vec![cohort_from_json(cached_json)]);

        let diffs = diff_live_entry(&built, &cached);
        assert_eq!(
            issue_types(&diffs),
            vec![ShadowIssueType::CohortFieldMismatch]
        );
        assert_eq!(diffs[0].fields, vec!["filters"]);
        assert_eq!(diffs[0].subject.as_deref(), Some("cohort 7"));
    }

    #[test]
    fn cohort_set_differences_report_missing_and_stale() {
        // A truncated Rust cohort BFS shows up as cohort_missing_in_cache on
        // the live side or stale on the built side — both must be visible.
        let mut built = wrapper(vec![]);
        built.cohorts = Some(vec![
            cohort_from_json(base_cohort_json(1)),
            cohort_from_json(base_cohort_json(2)),
        ]);
        let mut cached = live(vec![]);
        cached.cohorts = Some(vec![
            cohort_from_json(base_cohort_json(1)),
            cohort_from_json(base_cohort_json(3)),
        ]);

        let diffs = diff_live_entry(&built, &cached);
        assert_eq!(
            issue_types(&diffs),
            vec![
                ShadowIssueType::CohortMissingInCache,
                ShadowIssueType::CohortStaleInCache,
            ]
        );
        assert_eq!(diffs[0].subject.as_deref(), Some("cohort 2"));
        assert_eq!(diffs[1].subject.as_deref(), Some("cohort 3"));
    }

    #[test]
    fn cohort_null_optional_field_matches_absent_field() {
        // Python's _serialize_cohort emits explicit nulls for empty optionals;
        // the typed model omits them. Not drift.
        let mut cached_json = base_cohort_json(7);
        cached_json["description"] = serde_json::Value::Null;
        cached_json["query"] = serde_json::Value::Null;
        let mut built = wrapper(vec![]);
        built.cohorts = Some(vec![cohort_from_json(base_cohort_json(7))]);
        let mut cached = live(vec![]);
        cached.cohorts = Some(vec![cohort_from_json(cached_json)]);

        assert!(diff_live_entry(&built, &cached).is_empty());
    }

    #[test]
    fn live_entry_without_cohorts_key_skips_cohort_compare() {
        // Entries predating the cohorts field carry none; the service falls
        // back to its own cohort load for them, so absence is not drift.
        let mut built = wrapper(vec![]);
        built.cohorts = Some(vec![cohort_from_json(base_cohort_json(1))]);
        let mut cached = live(vec![]);
        cached.cohorts = None;

        assert!(diff_live_entry(&built, &cached).is_empty());
    }

    /// Pinned so a change to the hashed parts, the marker bytes, or the digest has
    /// to be made on purpose. The value is written to Redis and compared by a later
    /// process, so its encoding is a cross-process contract like the key string. A
    /// silent change stops every team's pending state from matching for one TTL
    /// window after the deploy, and confirmations drop to zero with nothing saying
    /// why. Called with literal parts rather than through `diff_live_entry`, so a
    /// new field on `FeatureFlag` does not fail it.
    #[test]
    fn fingerprint_encoding_is_pinned() {
        assert_eq!(
            fingerprint(
                ShadowIssueType::FieldMismatch,
                Some("flag 1 (flag-1)"),
                Some(&json!({"has_experiment": false})),
                Some(&json!({"has_experiment": true})),
            ),
            0x3e37_1bcf_8d31_4c14
        );

        // The all-absent shape, which exercises the other marker byte.
        assert_eq!(
            fingerprint(ShadowIssueType::MissingEvaluationMetadata, None, None, None),
            0xdcb8_1973_c8ce_a9d4
        );
    }

    fn mismatch_diffs() -> Vec<ShadowDiff> {
        let built = wrapper(vec![flag_from_json(base_flag_json(1))]);
        let mut cached_json = base_flag_json(1);
        cached_json["has_experiment"] = json!(true);
        let cached = live(vec![flag_from_json(cached_json)]);
        diff_live_entry(&built, &cached)
    }

    const TRACKER_TTL: Duration = Duration::from_secs(3600);
    const PENDING_KEY_TEAM_7: &str = "posthog:1:feature_flags/shadow_mismatch/7";

    /// A wall clock reading `secs` after the epoch. The tracker compares the
    /// stored timestamp against the one the caller passes, so a test states both
    /// instead of reading a real clock.
    fn at(secs: u64) -> SystemTime {
        SystemTime::UNIX_EPOCH + Duration::from_secs(secs)
    }

    /// The first build in a test. Not the epoch itself, so a test that subtracts
    /// from it stays a valid instant.
    fn first_build() -> SystemTime {
        at(1_000)
    }

    /// The instant a fingerprint first seen at `first_build` becomes eligible to
    /// confirm. Tests that are not about timing use this for their second build so
    /// the interval never masks what they do assert.
    fn window_opens_at() -> SystemTime {
        first_build() + MIN_CONFIRM_INTERVAL
    }

    /// One disagreement per flag, for a flag count the caller picks. Used to cross
    /// the store cap.
    fn many_mismatch_diffs(count: i32) -> Vec<ShadowDiff> {
        let built = (1..=count).map(|id| flag_from_json(base_flag_json(id)));
        let cached = (1..=count).map(|id| {
            let mut cached_json = base_flag_json(id);
            cached_json["has_experiment"] = json!(true);
            flag_from_json(cached_json)
        });
        diff_live_entry(&wrapper(built.collect()), &live(cached.collect()))
    }

    /// A disagreement on the same flag with different content, so the fingerprint
    /// differs from `mismatch_diffs`.
    fn other_mismatch_diffs() -> Vec<ShadowDiff> {
        let built = wrapper(vec![flag_from_json(base_flag_json(1))]);
        let mut cached_json = base_flag_json(1);
        cached_json["filters"]["groups"][0]["rollout_percentage"] = json!(25.0);
        diff_live_entry(&built, &live(vec![flag_from_json(cached_json)]))
    }

    fn tracker(redis: &MockRedisClient) -> MismatchTracker {
        MismatchTracker::new(Arc::new(redis.clone()), TRACKER_TTL)
    }

    /// The store as the team's next shadow build finds it, in a new process.
    /// `MockRedisClient` records writes but serves no reads, so replaying its
    /// recorded writes into a fresh client is what models state that outlived the
    /// process which wrote it. The tracker holds nothing in memory, so a later
    /// build in the same pod and one in a new pod run the same code path.
    ///
    /// It replays *attempted* writes. The mock records a call whether or not it
    /// returned an error, so a test that injects a write failure must build the
    /// next store itself rather than call this.
    ///
    /// `shadow_observation` in `bin/flags_cache_builder.rs` models the same thing
    /// inline for the outcome-label test. A change to what the mock records has to
    /// reach both.
    fn next_build(prior: &MockRedisClient) -> MockRedisClient {
        let mut next = MockRedisClient::new();
        for call in prior.get_calls() {
            match (call.op.as_str(), call.value) {
                ("setex", MockRedisValue::StringWithTTL(value, _)) => {
                    next.get_ret(&call.key, Ok(value));
                }
                ("del", _) => {
                    next.get_ret(&call.key, Err(CustomRedisError::NotFound));
                }
                _ => {}
            }
        }
        next
    }

    fn calls_with_op(redis: &MockRedisClient, op: &str) -> Vec<MockRedisCall> {
        redis
            .get_calls()
            .into_iter()
            .filter(|call| call.op == op)
            .collect()
    }

    #[tokio::test]
    async fn tracker_suppresses_first_sight_and_confirms_repeat_across_a_restart() {
        let first_pod = MockRedisClient::new();
        let first = tracker(&first_pod)
            .observe(7, mismatch_diffs(), first_build())
            .await;
        assert!(first.confirmed.is_empty());
        assert_eq!(first.first_sight.len(), 1);
        assert!(first.store_errors.is_empty());

        let second_pod = next_build(&first_pod);
        let second = tracker(&second_pod)
            .observe(7, mismatch_diffs(), window_opens_at())
            .await;
        assert_eq!(second.confirmed.len(), 1);
        assert!(second.first_sight.is_empty());
    }

    #[tokio::test]
    async fn tracker_does_not_confirm_inside_the_minimum_interval() {
        // Two of a team's invalidations that fall either side of the coalesce
        // window become two builds a fraction of a second apart. A row that is
        // written between their two Postgres reads disagrees on both, and with no
        // lower bound on the window that counted as a confirmed parity defect.
        let first_pod = MockRedisClient::new();
        tracker(&first_pod)
            .observe(7, mismatch_diffs(), first_build())
            .await;

        let second_pod = next_build(&first_pod);
        let second = tracker(&second_pod)
            .observe(
                7,
                mismatch_diffs(),
                first_build() + Duration::from_millis(600),
            )
            .await;
        assert!(second.confirmed.is_empty());
        assert_eq!(second.first_sight.len(), 1);
    }

    #[tokio::test]
    async fn tracker_keeps_the_window_open_across_builds_inside_the_interval() {
        // The middle build must not restart the window. A team that invalidates
        // faster than the interval would otherwise never accumulate elapsed time
        // and could never confirm anything, however wrong its cache entry was.
        let first_pod = MockRedisClient::new();
        tracker(&first_pod)
            .observe(7, mismatch_diffs(), at(1_000))
            .await;

        let second_pod = next_build(&first_pod);
        let second = tracker(&second_pod)
            .observe(7, mismatch_diffs(), at(1_030))
            .await;
        assert!(second.confirmed.is_empty());

        let third_pod = next_build(&second_pod);
        let third = tracker(&third_pod)
            .observe(7, mismatch_diffs(), at(1_061))
            .await;
        assert_eq!(third.confirmed.len(), 1);
    }

    #[tokio::test]
    async fn tracker_confirms_a_lasting_diff_while_another_comes_and_goes() {
        // A team's diff set changes while one disagreement inside it stays. Timing
        // the whole set instead of each fingerprint restarts the window on every
        // such change, so the disagreement that never went away never confirms —
        // and a team whose set churns is a team that rebuilds often, which is the
        // one this instrument most needs to report on.
        let lasting = mismatch_diffs();
        let both = [mismatch_diffs(), other_mismatch_diffs()].concat();

        let first_pod = MockRedisClient::new();
        tracker(&first_pod)
            .observe(7, lasting.clone(), at(1_000))
            .await;

        // The set grows, then shrinks back, both inside the interval.
        let second_pod = next_build(&first_pod);
        tracker(&second_pod).observe(7, both, at(1_030)).await;

        let third_pod = next_build(&second_pod);
        let third = tracker(&third_pod).observe(7, lasting, at(1_061)).await;
        assert_eq!(third.confirmed.len(), 1);
    }

    #[tokio::test]
    async fn tracker_does_not_confirm_against_a_clock_that_went_backwards() {
        // Builder pods do not share a clock. A build on a pod running behind the
        // one that wrote the observation must resolve to first sight rather than
        // underflow the elapsed check.
        let first_pod = MockRedisClient::new();
        tracker(&first_pod)
            .observe(7, mismatch_diffs(), first_build())
            .await;

        let second_pod = next_build(&first_pod);
        let second = tracker(&second_pod)
            .observe(7, mismatch_diffs(), first_build() - Duration::from_secs(10))
            .await;
        assert!(second.confirmed.is_empty());
        assert_eq!(second.first_sight.len(), 1);
    }

    #[tokio::test]
    async fn tracker_does_not_confirm_before_the_interval_really_elapsed() {
        // A pod running behind must not age a fingerprint on the pods that come
        // after it. Clamping the stored time to the writing pod's own clock looks
        // like the fix for a time that sits in the future, but the pods share no
        // clock: a pod that is behind cannot tell itself apart from a writer that
        // ran ahead. It would store its own earlier clock, and the build after it
        // would then measure elapsed time from a start that never happened.
        let ahead_pod = MockRedisClient::new();
        tracker(&ahead_pod)
            .observe(7, mismatch_diffs(), at(1_000))
            .await;

        // 70 seconds behind. Correct on its own build, which reports first sight.
        let behind_pod = next_build(&ahead_pod);
        let behind = tracker(&behind_pod)
            .observe(7, mismatch_diffs(), at(930))
            .await;
        assert!(behind.confirmed.is_empty());

        // 40 seconds of real time after the first sighting, against a 60 second
        // interval. Confirming here would report a race as a parity defect, which
        // is the whole failure this interval exists to remove.
        let later_pod = next_build(&behind_pod);
        let later = tracker(&later_pod)
            .observe(7, mismatch_diffs(), at(1_040))
            .await;
        assert!(later.confirmed.is_empty());
        assert_eq!(later.first_sight.len(), 1);
    }

    #[tokio::test]
    async fn tracker_restarts_the_window_when_the_disagreement_changes() {
        // Different content is a different disagreement, so its window starts at
        // the build that first saw that content and not at the build before it.
        let first_pod = MockRedisClient::new();
        tracker(&first_pod)
            .observe(7, mismatch_diffs(), at(1_000))
            .await;

        let second_pod = next_build(&first_pod);
        tracker(&second_pod)
            .observe(7, other_mismatch_diffs(), at(1_030))
            .await;

        // 70 seconds past the first build, but only 40 past the build that first
        // saw this content.
        let third_pod = next_build(&second_pod);
        let third = tracker(&third_pod)
            .observe(7, other_mismatch_diffs(), at(1_070))
            .await;
        assert!(third.confirmed.is_empty());

        let fourth_pod = next_build(&third_pod);
        let fourth = tracker(&fourth_pod)
            .observe(7, other_mismatch_diffs(), at(1_100))
            .await;
        assert_eq!(fourth.confirmed.len(), 1);
    }

    #[tokio::test]
    async fn tracker_reads_a_stored_value_written_before_the_window_existed() {
        // The bare fingerprint array is the shape written before `observed_at`
        // existed. It has to parse, or every key that predates the deploy reports
        // as a read failure for a whole TTL. It also has to not confirm, because
        // it carries no timestamp, and treating an absent one as time zero would
        // make it infinitely old and confirm on this very build.
        let legacy =
            serde_json::to_string(&vec![mismatch_diffs()[0].fingerprint]).expect("must serialize");
        let mut legacy_pod = MockRedisClient::new();
        legacy_pod.get_ret(PENDING_KEY_TEAM_7, Ok(legacy));

        let first = tracker(&legacy_pod)
            .observe(7, mismatch_diffs(), at(1_000))
            .await;
        assert!(first.store_errors.is_empty());
        assert!(first.confirmed.is_empty());
        assert_eq!(first.first_sight.len(), 1);

        // The rewrite carries a timestamp, so the window opens from this build.
        let next_pod = next_build(&legacy_pod);
        let later = tracker(&next_pod)
            .observe(7, mismatch_diffs(), at(1_000) + MIN_CONFIRM_INTERVAL)
            .await;
        assert_eq!(later.confirmed.len(), 1);
    }

    #[tokio::test]
    async fn tracker_clears_pending_state_on_match() {
        let first_pod = MockRedisClient::new();
        tracker(&first_pod)
            .observe(7, mismatch_diffs(), first_build())
            .await;

        let mut second_pod = next_build(&first_pod);
        second_pod.del_ret(PENDING_KEY_TEAM_7, Ok(()));

        let clean = tracker(&second_pod)
            .observe(7, Vec::new(), window_opens_at())
            .await;
        assert!(clean.is_match());
        assert!(clean.store_errors.is_empty());

        // Asserted against the store, not against the next build: an unseeded mock
        // reads as an absent key, so a later first sighting proves nothing about
        // whether the clean build cleared anything.
        assert_eq!(
            calls_with_op(&second_pod, "del")
                .iter()
                .map(|call| call.key.as_str())
                .collect::<Vec<_>>(),
            vec![PENDING_KEY_TEAM_7]
        );

        // The earlier sighting no longer counts — mismatch starts over. Observed
        // well past the interval, so the window being closed cannot be what makes
        // this a first sighting.
        let third_pod = next_build(&second_pod);
        let after = tracker(&third_pod)
            .observe(
                7,
                mismatch_diffs(),
                window_opens_at() + MIN_CONFIRM_INTERVAL,
            )
            .await;
        assert!(after.confirmed.is_empty());
        assert_eq!(after.first_sight.len(), 1);
    }

    #[tokio::test]
    async fn tracker_expires_pending_state_after_ttl() {
        let first_pod = MockRedisClient::new();
        tracker(&first_pod)
            .observe(7, mismatch_diffs(), first_build())
            .await;

        // The TTL on this write is what bounds the confirmation window above, and
        // the key is pinned because a tracker write must never land on a cache
        // entry the serve path reads.
        let writes = calls_with_op(&first_pod, "setex");
        assert_eq!(writes.len(), 1);
        assert_eq!(writes[0].key, PENDING_KEY_TEAM_7);
        match writes[0].value.clone() {
            MockRedisValue::StringWithTTL(_, ttl) => assert_eq!(ttl, TRACKER_TTL.as_secs()),
            other => panic!("expected a value with a TTL, got {other:?}"),
        }

        // The mock has no clock, so expiry is modelled as the key being gone by the
        // time the next build reads it. Stated rather than left to the mock's
        // default, so the test still means this if that default changes.
        let mut expired = next_build(&first_pod);
        expired.get_ret(PENDING_KEY_TEAM_7, Err(CustomRedisError::NotFound));

        let late = tracker(&expired)
            .observe(7, mismatch_diffs(), window_opens_at())
            .await;
        assert!(late.confirmed.is_empty());
        assert_eq!(late.first_sight.len(), 1);
        assert!(late.store_errors.is_empty());
    }

    #[tokio::test]
    async fn tracker_treats_different_content_as_first_sight() {
        // An edit session produces a different disagreement each save — the
        // fingerprint hashes the content, so those never confirm.
        let first_pod = MockRedisClient::new();
        tracker(&first_pod)
            .observe(7, mismatch_diffs(), first_build())
            .await;

        let second_pod = next_build(&first_pod);
        let second = tracker(&second_pod)
            .observe(7, other_mismatch_diffs(), window_opens_at())
            .await;
        assert!(second.confirmed.is_empty());
        assert_eq!(second.first_sight.len(), 1);
    }

    #[tokio::test]
    async fn tracker_is_per_team() {
        let first_pod = MockRedisClient::new();
        tracker(&first_pod)
            .observe(7, mismatch_diffs(), first_build())
            .await;

        let second_pod = next_build(&first_pod);
        let other_team = tracker(&second_pod)
            .observe(8, mismatch_diffs(), window_opens_at())
            .await;
        assert!(other_team.confirmed.is_empty());
        assert_eq!(other_team.first_sight.len(), 1);
    }

    #[tokio::test]
    async fn tracker_read_failure_falls_back_to_first_sight() {
        let first_pod = MockRedisClient::new();
        tracker(&first_pod)
            .observe(7, mismatch_diffs(), first_build())
            .await;

        let mut second_pod = next_build(&first_pod);
        second_pod.get_ret(PENDING_KEY_TEAM_7, Err(CustomRedisError::Timeout));

        let second = tracker(&second_pod)
            .observe(7, mismatch_diffs(), window_opens_at())
            .await;
        assert!(second.confirmed.is_empty());
        assert_eq!(second.first_sight.len(), 1);
        assert_eq!(second.store_errors, vec![TrackerStoreOp::Read]);
    }

    #[tokio::test]
    async fn tracker_write_failure_clears_the_stale_observation() {
        // A failed write leaves the previous build's fingerprints in place with
        // their own TTL. Without the compensating clear, the team's next build
        // compares against an observation from two builds ago and can confirm a
        // disagreement the build in between did not see.
        let mut second_pod = next_build(&{
            let first_pod = MockRedisClient::new();
            tracker(&first_pod)
                .observe(7, mismatch_diffs(), first_build())
                .await;
            first_pod
        });
        second_pod.set_ret(PENDING_KEY_TEAM_7, Err(CustomRedisError::Timeout));
        second_pod.del_ret(PENDING_KEY_TEAM_7, Ok(()));

        let second = tracker(&second_pod)
            .observe(7, other_mismatch_diffs(), window_opens_at())
            .await;
        assert_eq!(second.store_errors, vec![TrackerStoreOp::Write]);
        assert_eq!(
            calls_with_op(&second_pod, "del")
                .iter()
                .map(|call| call.key.as_str())
                .collect::<Vec<_>>(),
            vec![PENDING_KEY_TEAM_7]
        );
    }

    #[tokio::test]
    async fn tracker_write_and_clear_failure_leaves_the_stale_observation() {
        // Both accesses failing is the residual `observe` documents: the previous
        // fingerprints stand until the TTL, so a disagreement that returns byte for
        // byte can confirm one build early. The counter pair is the only signal.
        let mut second_pod = next_build(&{
            let first_pod = MockRedisClient::new();
            tracker(&first_pod)
                .observe(7, mismatch_diffs(), first_build())
                .await;
            first_pod
        });
        second_pod.set_ret(PENDING_KEY_TEAM_7, Err(CustomRedisError::Timeout));
        second_pod.del_ret(PENDING_KEY_TEAM_7, Err(CustomRedisError::Timeout));

        let second = tracker(&second_pod)
            .observe(7, other_mismatch_diffs(), window_opens_at())
            .await;
        assert_eq!(
            second.store_errors,
            vec![TrackerStoreOp::Write, TrackerStoreOp::Clear]
        );
        assert!(second.confirmed.is_empty());
    }

    #[tokio::test]
    async fn tracker_unreadable_stored_value_is_reported_not_ignored() {
        // A stored value this build cannot parse counts as a read failure rather
        // than an absent key, so a value shape that changed across deploys shows on
        // the counter instead of looking like an expired window.
        let mut redis = MockRedisClient::new();
        redis.get_ret(PENDING_KEY_TEAM_7, Ok("not json".to_string()));

        let observation = tracker(&redis)
            .observe(7, mismatch_diffs(), first_build())
            .await;
        assert!(observation.confirmed.is_empty());
        assert_eq!(observation.first_sight.len(), 1);
        assert_eq!(observation.store_errors, vec![TrackerStoreOp::Read]);
    }

    #[tokio::test]
    async fn tracker_confirms_the_diffs_that_fit_the_stored_cap() {
        let over_cap = MAX_STORED_FINGERPRINTS as i32 + 20;
        let first_pod = MockRedisClient::new();
        let first = tracker(&first_pod)
            .observe(7, many_mismatch_diffs(over_cap), first_build())
            .await;
        assert_eq!(first.first_sight.len(), over_cap as usize);

        let second_pod = next_build(&first_pod);
        let second = tracker(&second_pod)
            .observe(7, many_mismatch_diffs(over_cap), window_opens_at())
            .await;

        // Pins the cap, and that a diff inside it confirms. It does not pin which
        // diffs are stored: the next build compares all of its diffs against the
        // stored set, so any 256 of them produce 256 confirmations. Taking the
        // first entries of a stable order is what makes one flag's disagreement
        // confirm on every build rather than on some.
        assert_eq!(second.confirmed.len(), MAX_STORED_FINGERPRINTS);
        assert_eq!(second.first_sight.len(), 20);
    }

    #[tokio::test]
    async fn tracker_clear_failure_is_reported() {
        let mut redis = MockRedisClient::new();
        redis.del_ret(PENDING_KEY_TEAM_7, Err(CustomRedisError::Timeout));

        let clean = tracker(&redis).observe(7, Vec::new(), first_build()).await;
        assert!(clean.is_match());
        assert_eq!(clean.store_errors, vec![TrackerStoreOp::Clear]);
    }

    #[test]
    fn summarize_caps_entries_and_reports_remainder() {
        let diffs = mismatch_diffs();
        let mut many: Vec<ShadowDiff> = Vec::new();
        for _ in 0..5 {
            many.extend(diffs.iter().cloned());
        }

        let summary = summarize_diffs(&many, 2, 4096);
        assert_eq!(summary.matches("flag 1").count(), 2);
        assert!(summary.ends_with("; +3 more"));
    }

    #[test]
    fn summarize_caps_bytes() {
        let diffs = mismatch_diffs();
        let summary = summarize_diffs(&diffs, 10, 8);
        assert_eq!(summary, "; +1 more");
        assert!(summary.len() <= 8 + "; +1 more".len());
    }

    #[test]
    fn summarize_renders_fields() {
        let summary = summarize_diffs(&mismatch_diffs(), 10, 4096);
        assert_eq!(summary, "flag 1 (flag-1): field_mismatch[has_experiment]");
    }
}