Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

39.5 KB
Raw Blame
//! Internal batch flag evaluation endpoint for flag-driven static cohort generation.
//!
//! Django's `get_cohort_actors_for_feature_flag` Celery task drives this endpoint with
//! strictly sequential, cursor-paged requests: each page scans a slice of a team's persons
//! and evaluates a single (pinned-version) flag for every person, reusing the exact same
//! `FeatureFlagMatcher` machinery as live `/flags` evaluation. The response carries the
//! matched person UUIDs, which Django inserts into the static cohort.
//!
//! Deliberate differences from the live `/flags` pipeline:
//! - Auth is the `INTERNAL_REQUEST_TOKEN` bearer token only — no SDK token, no team lookup
//!   by token. The team is still loaded by id once per page to read its timezone and
//!   property-matching version, matching live `/flags`.
//! - The handler bypasses the `/flags` request pipeline entirely, so no billing or
//!   flag-analytics counters are emitted, and dedicated `flags_batch_eval_*` ops metrics
//!   track batch traffic. The per-person matcher still emits its own evaluation metrics
//!   (`flags_evaluation_time`, dependency-graph build counters), which share the same
//!   series as live `/flags`, so a large cohort run dominates those on its pod.
//! - The matcher always runs with `skip_writes(true)`: experience-continuity hash key
//!   overrides are read but never written.
//! - The matcher compares dependency answers by match only for the target and for each
//!   dependency that no evaluated flag reads by variant (`enabled_only_flag_keys`), because the
//!   caller reads only the target's `enabled`.
//! - Flags are always read fresh from Postgres (never the hypercache) so the
//!   `expected_version` optimistic-lock check is meaningful.
//!
//! The paged scan walks `posthog_person.id` ascending across a live table, so the run sees
//! a moving snapshot rather than a point-in-time one: persons inserted above the current
//! cursor mid-run are included, persons inserted below it (or deleted) are not. Flag
//! definitions and property-matching semantics are pinned, but person membership is eventually
//! consistent with the table at the time each page is scanned. This matches the existing
//! cohort-generation contract (a later refresh reconciles drift).

use std::collections::HashSet;
use std::sync::Arc;

use axum::{
    body::Body,
    extract::State,
    http::{HeaderMap, StatusCode},
    response::{IntoResponse, Response},
    Json,
};
use common_database::PostgresReader;
use common_metrics::inc;
use common_types::PersonId;
use serde::{Deserialize, Serialize};
use sqlx::{Acquire, FromRow};
use tracing::warn;
use uuid::Uuid;

use crate::{
    api::{
        errors::{FlagError, CODE_DEPENDENCY_FAILED, CODE_FLAG_DATA_PARSING},
        types::FlagsResponse,
    },
    database::{get_connection_with_metrics, PostgresRouter},
    flags::{
        cache_builder::compute_flag_dependencies_or_single_stage,
        feature_flag_list::PreparedFlags,
        flag_matching::FeatureFlagMatcher,
        flag_models::{EvaluationMetadata, FeatureFlag, FeatureFlagId, FeatureFlagList},
    },
    handler::authentication::is_internal_request_inner,
    metrics::consts::{
        FLAG_BATCH_EVAL_PERSONS_COUNTER, FLAG_BATCH_EVAL_PERSON_RETRIES_COUNTER,
        FLAG_BATCH_EVAL_REQUESTS_COUNTER, FLAG_BATCH_EVAL_TIME,
    },
    router,
    team::team_models::{PropertyMatchingVersion, Team},
};

#[derive(Debug, Deserialize)]
pub struct BatchFlagEvaluationRequest {
    pub team_id: i32,
    /// The project the flag belongs to. Accepted for contract parity with the Django
    /// caller; flag lookup is team-scoped (matching live Rust evaluation), so this is
    /// currently informational only.
    #[serde(default)]
    pub project_id: Option<i64>,
    pub flag_key: String,
    /// Optimistic-lock pin: the flag `version` Django read before starting the run.
    /// Nullable versions coerce to 0 on both sides, so JSON `null` is accepted and
    /// treated as 0.
    #[serde(default)]
    pub expected_version: Option<i32>,
    /// Optimistic-lock pin for the team's property matching semantics.
    #[serde(default)]
    pub expected_property_matching_version: Option<i16>,
    /// Exclusive lower bound on `posthog_person.id`; 0 (default) starts from the beginning.
    #[serde(default)]
    pub cursor: i64,
    /// Page size; defaults to `DEFAULT_LIMIT` (1000), capped by `BATCH_FLAG_EVAL_MAX_LIMIT`.
    pub limit: Option<i64>,
}

#[derive(Debug, Serialize, Deserialize)]
pub struct BatchFlagEvaluationResponse {
    pub matched_person_uuids: Vec<Uuid>,
    /// Cursor for the next page, or `None` when this page was the last one.
    pub next_cursor: Option<i64>,
    /// Number of persons in this page whose evaluation failed (they are skipped, not matched).
    pub errors_count: u64,
    /// Matching semantics used for this page.
    pub property_matching_version: i16,
}

const DEFAULT_LIMIT: i64 = 1_000;

/// A page evaluates its persons one at a time under `BATCH_FLAG_EVAL_TIMEOUT_MS`. A retry of a
/// person that timed out waits on the persons DB a second time. The cap keeps a sustained stall
/// from spending the page timeout on retries.
const MAX_PERSON_RETRIES_PER_PAGE: u32 = 20;

#[derive(Debug)]
pub enum BatchFlagEvaluationError {
    Unauthorized,
    InvalidRequest(String),
    PayloadTooLarge,
    FlagNotFound,
    FlagInactive,
    GroupAggregatedFlag,
    UnsupportedConfigFormat,
    VersionConflict { expected: i32, actual: i32 },
    PropertyMatchingVersionConflict { expected: i16, actual: i16 },
    Upstream(FlagError),
}

impl BatchFlagEvaluationError {
    fn outcome_label(&self) -> &'static str {
        match self {
            Self::Unauthorized => "unauthorized",
            Self::InvalidRequest(_) => "invalid_request",
            Self::PayloadTooLarge => "payload_too_large",
            Self::FlagNotFound => "flag_not_found",
            Self::FlagInactive => "flag_inactive",
            Self::GroupAggregatedFlag => "group_aggregated_flag",
            Self::UnsupportedConfigFormat => "unsupported_config_format",
            Self::VersionConflict { .. } => "version_conflict",
            Self::PropertyMatchingVersionConflict { .. } => "property_matching_version_conflict",
            Self::Upstream(_) => "upstream_error",
        }
    }
}

impl IntoResponse for BatchFlagEvaluationError {
    fn into_response(self) -> Response {
        let (status, error, detail, actual_version, actual_property_matching_version) = match self {
            Self::Unauthorized => (
                StatusCode::UNAUTHORIZED,
                "unauthorized",
                "Missing or invalid internal request token".to_string(),
                None,
                None,
            ),
            Self::InvalidRequest(detail) => {
                (StatusCode::BAD_REQUEST, "invalid_request", detail, None, None)
            }
            Self::PayloadTooLarge => (
                StatusCode::PAYLOAD_TOO_LARGE,
                "payload_too_large",
                format!("Request body exceeds the {MAX_BATCH_BODY_BYTES}-byte limit"),
                None,
                None,
            ),
            Self::FlagNotFound => (
                StatusCode::NOT_FOUND,
                "flag_not_found",
                "No flag with this key exists for this team".to_string(),
                None,
                None,
            ),
            Self::FlagInactive => (
                StatusCode::BAD_REQUEST,
                "flag_inactive",
                "Flag is not active".to_string(),
                None,
                None,
            ),
            Self::GroupAggregatedFlag => (
                StatusCode::BAD_REQUEST,
                "group_aggregated_flag",
                "Group-aggregated flags are not supported for batch evaluation".to_string(),
                None,
                None,
            ),
            Self::UnsupportedConfigFormat => (
                StatusCode::BAD_REQUEST,
                "unsupported_config_format",
                "Flag is stored in a configuration format this service does not evaluate".to_string(),
                None,
                None,
            ),
            Self::VersionConflict { expected, actual } => (
                StatusCode::CONFLICT,
                "version_conflict",
                format!("Flag version is {actual}, expected {expected}; the flag changed during cohort generation"),
                Some(actual),
                None,
            ),
            Self::PropertyMatchingVersionConflict { expected, actual } => (
                StatusCode::CONFLICT,
                "property_matching_version_conflict",
                format!(
                    "Property matching version is {actual}, expected {expected}; matching semantics changed during cohort generation"
                ),
                None,
                Some(actual),
            ),
            Self::Upstream(e) => {
                // Delegate status selection to FlagError's own mapping, but keep the
                // JSON envelope consistent with the other variants.
                let status = StatusCode::from_u16(e.status_code())
                    .unwrap_or(StatusCode::INTERNAL_SERVER_ERROR);
                (status, "upstream_error", e.to_string(), None, None)
            }
        };

        let mut body = serde_json::json!({ "error": error, "detail": detail });
        if let Some(actual) = actual_version {
            body["actual_version"] = serde_json::json!(actual);
        }
        if let Some(actual) = actual_property_matching_version {
            body["actual_property_matching_version"] = serde_json::json!(actual);
        }
        (status, Json(body)).into_response()
    }
}

/// One row from the person page scan: the person plus its deterministic
/// (minimum `posthog_persondistinctid.id`) distinct_id, when one exists.
#[derive(Debug, FromRow)]
struct PersonScanRow {
    id: PersonId,
    uuid: Uuid,
    distinct_id: Option<String>,
}

/// Scans one page of a team's persons in `id` order, attaching the distinct_id with the
/// lowest `posthog_persondistinctid.id` (a deterministic choice). Persons with no
/// distinct_ids come back with `distinct_id = NULL` and still advance the cursor, so a
/// run of distinct_id-less persons cannot stall paging.
async fn scan_persons_page(
    reader: PostgresReader,
    team_id: i32,
    cursor: i64,
    limit: i64,
    statement_timeout_ms: u64,
) -> Result<Vec<PersonScanRow>, FlagError> {
    let mut conn =
        get_connection_with_metrics(&reader, "persons_reader", "batch_eval_person_scan").await?;
    let scan_failed = |e: sqlx::Error| {
        warn!(team_id, cursor, "Batch eval person scan failed: {e}");
        let message = format!("person scan query failed: {e}");
        FlagError::internal(anyhow::Error::new(e).context(message))
    };

    // SET LOCAL lasts only until the transaction ends, so the connection goes back to the
    // pool with the pool's statement timeout.
    let mut tx = conn.begin().await.map_err(scan_failed)?;
    // SET does not accept a bind parameter. The value is a u64, so it cannot inject SQL.
    sqlx::query(&format!(
        "SET LOCAL statement_timeout = {statement_timeout_ms}"
    ))
    .execute(&mut *tx)
    .await
    .map_err(scan_failed)?;

    // Sort-free PK range scan on the partitioned persons table; the lateral subquery is
    // covered by the existing person_id index on posthog_persondistinctid.
    let query = r#"
        SELECT p.id, p.uuid, d.distinct_id
        FROM posthog_person p
        LEFT JOIN LATERAL (
            SELECT pd.distinct_id
            FROM posthog_persondistinctid pd
            WHERE pd.person_id = p.id
              AND pd.team_id = p.team_id
              AND pd.is_deleted = false
            ORDER BY pd.id
            LIMIT 1
        ) d ON true
        WHERE p.team_id = $1
          AND p.id > $2
          AND p.is_deleted = false
        ORDER BY p.id
        LIMIT $3
    "#;

    let rows = sqlx::query_as::<_, PersonScanRow>(query)
        .bind(team_id)
        .bind(cursor)
        .bind(limit)
        .fetch_all(&mut *tx)
        .await
        .map_err(scan_failed)?;
    tx.commit().await.map_err(scan_failed)?;
    Ok(rows)
}

/// Fetches the team's flags fresh from Postgres and locates the target flag by key.
/// Returns the full flag set (needed for flag-dependency evaluation) and the target's index.
async fn fetch_flags_with_target(
    client: PostgresReader,
    team_id: i32,
    flag_key: &str,
) -> Result<(Vec<FeatureFlag>, usize), BatchFlagEvaluationError> {
    let flags = FeatureFlagList::from_pg(client, team_id)
        .await
        .map_err(BatchFlagEvaluationError::Upstream)?;
    let target_index = flags
        .iter()
        .position(|f| f.key == flag_key)
        .ok_or(BatchFlagEvaluationError::FlagNotFound)?;
    Ok((flags, target_index))
}

/// The request is a small JSON object (~200 bytes typical), so 64 KiB is generous
/// while protecting against oversized payloads.
pub const MAX_BATCH_BODY_BYTES: usize = 64 * 1024;

pub async fn batch_flag_evaluation(
    State(state): State<router::State>,
    headers: HeaderMap,
    body: Body,
) -> Result<Json<BatchFlagEvaluationResponse>, BatchFlagEvaluationError> {
    let timer = common_metrics::timing_guard(FLAG_BATCH_EVAL_TIME, &[]);

    // Authenticate before reading the body so unauthenticated clients cannot
    // tie up resources by streaming large or slow request bodies.
    if !is_internal_request_inner(state.config.internal_request_token.as_deref(), &headers) {
        let outcome = "unauthorized";
        inc(
            FLAG_BATCH_EVAL_REQUESTS_COUNTER,
            &[("outcome".to_string(), outcome.to_string())],
            1,
        );
        timer.label("outcome", outcome).fin();
        return Err(BatchFlagEvaluationError::Unauthorized);
    }

    let result = match axum::body::to_bytes(body, MAX_BATCH_BODY_BYTES).await {
        Ok(body_bytes) => handle_batch_flag_evaluation(&state, &body_bytes).await,
        // `to_bytes` wraps the underlying http_body_util error; walk the full source
        // chain (not just the first level) to classify an oversized body as 413, the
        // same way `body_read_metrics::record_body_read` does for `/flags`.
        Err(e) => {
            let too_large = std::iter::successors(std::error::Error::source(&e), |s| s.source())
                .any(|s| s.is::<http_body_util::LengthLimitError>());
            Err(if too_large {
                BatchFlagEvaluationError::PayloadTooLarge
            } else {
                BatchFlagEvaluationError::InvalidRequest(format!("body read error: {e}"))
            })
        }
    };

    let outcome = match &result {
        Ok(_) => "success",
        Err(e) => e.outcome_label(),
    };
    inc(
        FLAG_BATCH_EVAL_REQUESTS_COUNTER,
        &[("outcome".to_string(), outcome.to_string())],
        1,
    );
    timer.label("outcome", outcome).fin();

    result.map(Json)
}

/// Keys of the flags the matcher compares by match only: the target, and each of its dependencies
/// that no evaluated flag reads by variant. A person joins the cohort on the target's `enabled`
/// alone. A dependency that only `true` or `false` filters read cannot change that through its
/// variant.
fn enabled_only_flag_keys(
    flags: &[FeatureFlag],
    target_id: FeatureFlagId,
    evaluation_metadata: &EvaluationMetadata,
) -> HashSet<String> {
    let dependency_ids = evaluation_metadata.transitive_deps.get(&target_id);
    let evaluated: Vec<&FeatureFlag> = flags
        .iter()
        .filter(|flag| {
            flag.id == target_id || dependency_ids.is_some_and(|ids| ids.contains(&flag.id))
        })
        .collect();
    let read_by_variant: HashSet<FeatureFlagId> = evaluated
        .iter()
        .flat_map(|flag| flag.filters.flag_ids_read_by_variant())
        .collect();
    evaluated
        .into_iter()
        .filter(|flag| !read_by_variant.contains(&flag.id))
        .map(|flag| flag.key.clone())
        .collect()
}

fn failure_code(result: &Result<FlagsResponse, FlagError>, target_key: &str) -> Option<String> {
    let response = match result {
        Ok(response) => response,
        Err(e) => return Some(e.evaluation_error_code()),
    };
    let target = response
        .flags
        .get(target_key)
        .filter(|details| details.failed)?;
    // `dependency_failed` does not say whether a retry can help, so report the code of a failed
    // dependency. The request names only the target, so the other flags in the response are its
    // dependencies. A retry helps only when no dependency failed permanently. Dependents of a flag
    // with an unsupported format read it as false, so that failure cannot fail the target.
    if target.reason.code == CODE_DEPENDENCY_FAILED {
        let root_codes: Vec<&str> = response
            .flags
            .values()
            .filter(|details| details.failed)
            .map(|details| details.reason.code.as_str())
            .filter(|code| *code != CODE_DEPENDENCY_FAILED && *code != CODE_FLAG_DATA_PARSING)
            .collect();
        if let Some(code) = root_codes
            .iter()
            .find(|code| !is_transient_failure(code))
            .or(root_codes.first())
        {
            return Some(code.to_string());
        }
    }
    Some(target.reason.code.clone())
}

/// Whether a failure code comes from a transient database fault or a timeout, which a second
/// evaluation can fix. Other failures, such as a missing dependency, an unsupported filter, or a
/// `database_error` from a query that cannot succeed, fail the same way on every attempt. An
/// unknown code gets no retry, so a new kind of failure cannot double the load on the persons
/// pools.
fn is_transient_failure(code: &str) -> bool {
    code.starts_with("timeout")
        || matches!(
            code,
            "database_unavailable"
                | "no_more_connections"
                | "query_wait_timeout"
                | "hash_key_override_error"
        )
}

/// Evaluates a person a second time when the first evaluation fails with a transient code. The
/// persons pools size their timeouts for /flags. Django leaves a failed person out of the cohort.
/// Each retry uses one unit of `retries_left`. When the budget is empty, a failed person gets no
/// second attempt, so a degraded replica cannot double the queries of a whole page.
async fn evaluate_with_one_retry<F, Fut>(
    target_key: &str,
    retries_left: &mut u32,
    mut evaluate: F,
) -> Result<FlagsResponse, FlagError>
where
    F: FnMut() -> Fut,
    Fut: std::future::Future<Output = Result<FlagsResponse, FlagError>>,
{
    let first = evaluate().await;
    if *retries_left == 0
        || !failure_code(&first, target_key).is_some_and(|code| is_transient_failure(&code))
    {
        return first;
    }
    *retries_left -= 1;

    let second = evaluate().await;
    let outcome = if failure_code(&second, target_key).is_some() {
        "failed"
    } else {
        "recovered"
    };
    inc(
        FLAG_BATCH_EVAL_PERSON_RETRIES_COUNTER,
        &[("outcome".to_string(), outcome.to_string())],
        1,
    );
    second
}

async fn handle_batch_flag_evaluation(
    state: &router::State,
    body: &[u8],
) -> Result<BatchFlagEvaluationResponse, BatchFlagEvaluationError> {
    let request: BatchFlagEvaluationRequest = serde_json::from_slice(body)
        .map_err(|e| BatchFlagEvaluationError::InvalidRequest(format!("invalid JSON body: {e}")))?;

    let max_limit = state.config.batch_flag_eval_max_limit;
    let limit = request.limit.unwrap_or(DEFAULT_LIMIT.min(max_limit));
    if limit < 1 || limit > max_limit {
        return Err(BatchFlagEvaluationError::InvalidRequest(format!(
            "limit must be between 1 and {max_limit}"
        )));
    }
    if request.flag_key.is_empty() {
        return Err(BatchFlagEvaluationError::InvalidRequest(
            "flag_key must not be empty".to_string(),
        ));
    }
    if request.cursor < 0 {
        return Err(BatchFlagEvaluationError::InvalidRequest(
            "cursor must not be negative".to_string(),
        ));
    }

    // Read the flag set from the replica first; on a version mismatch, re-fetch from the
    // primary to rule out replica lag before declaring a conflict. No evaluation happens
    // under the wrong version.
    let (mut flags_vec, mut target_index) = fetch_flags_with_target(
        state.database_pools.non_persons_reader.clone(),
        request.team_id,
        &request.flag_key,
    )
    .await?;

    let expected_version = request.expected_version.unwrap_or(0);
    if flags_vec[target_index].version.unwrap_or(0) != expected_version {
        (flags_vec, target_index) = fetch_flags_with_target(
            state.database_pools.non_persons_writer.clone(),
            request.team_id,
            &request.flag_key,
        )
        .await?;
        let actual = flags_vec[target_index].version.unwrap_or(0);
        if actual != expected_version {
            return Err(BatchFlagEvaluationError::VersionConflict {
                expected: expected_version,
                actual,
            });
        }
    }

    let target = &flags_vec[target_index];
    // The Django caller already returns [] for group-aggregated and inactive flags
    // without calling us; those two guards are defensive. The format check is not: nothing
    // upstream filters a non-v1 stored config.
    target
        .filters
        .require_v1()
        .map_err(|_| BatchFlagEvaluationError::UnsupportedConfigFormat)?;
    if target.get_group_type_index().is_some() {
        return Err(BatchFlagEvaluationError::GroupAggregatedFlag);
    }
    if !target.active {
        return Err(BatchFlagEvaluationError::FlagInactive);
    }
    let target_key = target.key.clone();

    let expected_property_matching_version = request
        .expected_property_matching_version
        .unwrap_or(PropertyMatchingVersion::LEGACY.0);
    let mut team = Team::from_pg_by_id(
        state.database_pools.non_persons_reader.clone(),
        request.team_id,
    )
    .await
    .map_err(BatchFlagEvaluationError::Upstream)?;
    if team.property_matching_version.0 != expected_property_matching_version {
        team = Team::from_pg_by_id(
            state.database_pools.non_persons_writer.clone(),
            request.team_id,
        )
        .await
        .map_err(BatchFlagEvaluationError::Upstream)?;
        if team.property_matching_version.0 != expected_property_matching_version {
            return Err(BatchFlagEvaluationError::PropertyMatchingVersionConflict {
                expected: expected_property_matching_version,
                actual: team.property_matching_version.0,
            });
        }
    }

    // Mirror the live pipeline's exclusion of inactive flags so dependency conditions on
    // them pre-seed as false instead of evaluating.
    let filtered_out_flag_ids: HashSet<i32> = flags_vec
        .iter()
        .filter(|f| !f.active)
        .map(|f| f.id)
        .collect();

    let evaluation_metadata =
        compute_flag_dependencies_or_single_stage(request.team_id, &flags_vec);
    let enabled_only_flag_keys =
        enabled_only_flag_keys(&flags_vec, flags_vec[target_index].id, &evaluation_metadata);

    let flag_list = FeatureFlagList {
        flags: PreparedFlags::seal(flags_vec),
        filtered_out_flag_ids,
        evaluation_metadata: Arc::new(evaluation_metadata),
        cohorts: None,
    };

    // The service exposes no dedicated non-critical persons pool, so the scan shares the
    // live persons reader.
    let rows = scan_persons_page(
        state.database_pools.persons_reader.clone(),
        request.team_id,
        request.cursor,
        limit,
        state.config.batch_flag_eval_scan_statement_timeout_ms,
    )
    .await
    .map_err(BatchFlagEvaluationError::Upstream)?;

    let next_cursor = if rows.len() as i64 == limit {
        rows.last().map(|r| r.id)
    } else {
        None
    };

    let pg_router = PostgresRouter::new(
        state.database_pools.persons_reader.clone(),
        state.database_pools.persons_writer.clone(),
        state.database_pools.non_persons_reader.clone(),
        state.database_pools.non_persons_writer.clone(),
    );
    let enable_realtime_cohort_evaluation =
        team.evaluates_realtime_cohorts(&state.config.realtime_cohort_evaluation_team_ids);
    let use_explicit_exact_matching = team.property_matching_version.uses_explicit_matching();
    let team_timezone = team.parsed_timezone();
    let optimize_experience_continuity_lookups =
        state.config.optimize_experience_continuity_lookups.0;

    let mut matched_person_uuids: Vec<Uuid> = Vec::new();
    let mut errors_count: u64 = 0;
    let mut retries_left = MAX_PERSON_RETRIES_PER_PAGE;

    for row in rows {
        // Persons with zero distinct_ids (almost-deleted) are skipped.
        let Some(distinct_id) = row.distinct_id else {
            record_person_outcome("skipped_no_distinct_id");
            continue;
        };

        let evaluate = || {
            // Per-evaluation so each one is independently traceable in canonical logs.
            let request_id = Uuid::new_v4();

            let mut matcher = FeatureFlagMatcher::new(
                distinct_id.clone(),
                None,
                request.team_id,
                pg_router.clone(),
                state.cohort_cache_manager.clone(),
                state.group_type_cache_manager.clone(),
                None,
            )
            .with_cohort_membership_provider(state.cohort_membership_provider.clone())
            .with_realtime_cohort_evaluation(enable_realtime_cohort_evaluation)
            .with_explicit_exact_matching(use_explicit_exact_matching)
            .with_membership_stamp_policy(state.config.realtime_cohort_membership_stamp_policy)
            .with_rayon_dispatcher(state.rayon_dispatcher.clone())
            .with_parallel_eval_threshold(state.config.parallel_eval_threshold)
            // Read-only: experience-continuity overrides are consulted but never written.
            .with_skip_writes(true)
            .with_enabled_only_flag_keys(enabled_only_flag_keys.clone())
            .with_timezone(team_timezone)
            // The deadline starts when the matcher is built. Building the matcher in this
            // closure gives the retry in `evaluate_with_one_retry` its own deadline.
            .with_persons_db_deadline(
                state
                    .config
                    .persons_db_deadline(tokio::time::Instant::now(), None)
                    .map(|deadline| deadline.at),
            );

            let flag_list = flag_list.clone();
            let target_key = target_key.clone();
            async move {
                matcher
                    .evaluate_all_feature_flags(
                        flag_list,
                        None,
                        None,
                        None,
                        request_id,
                        Some(vec![target_key]),
                        optimize_experience_continuity_lookups,
                    )
                    .await
            }
        };
        let evaluation = evaluate_with_one_retry(&target_key, &mut retries_left, evaluate).await;

        match evaluation {
            Ok(response) => match response.flags.get(&target_key) {
                Some(details) if details.failed => {
                    errors_count += 1;
                    record_person_outcome("error");
                }
                Some(details) if details.enabled => {
                    matched_person_uuids.push(row.uuid);
                    record_person_outcome("matched");
                }
                Some(_) => record_person_outcome("not_matched"),
                None => {
                    warn!(
                        team_id = request.team_id,
                        flag_key = %target_key,
                        person_uuid = %row.uuid,
                        "Batch eval response missing the target flag"
                    );
                    errors_count += 1;
                    record_person_outcome("error");
                }
            },
            Err(e) => {
                warn!(
                    team_id = request.team_id,
                    flag_key = %target_key,
                    person_uuid = %row.uuid,
                    "Batch eval failed for person: {e}"
                );
                errors_count += 1;
                record_person_outcome("error");
            }
        }
    }

    Ok(BatchFlagEvaluationResponse {
        matched_person_uuids,
        next_cursor,
        errors_count,
        property_matching_version: team.property_matching_version.0,
    })
}

fn record_person_outcome(result: &str) {
    inc(
        FLAG_BATCH_EVAL_PERSONS_COUNTER,
        &[("result".to_string(), result.to_string())],
        1,
    );
}

#[cfg(test)]
mod tests {
    use super::*;
    use crate::api::types::{FlagDetails, FromFeatureAndMatch};
    use crate::config::DEFAULT_TEST_CONFIG;
    use crate::flags::config_format::decode_filters;
    use crate::flags::flag_match_reason::FeatureFlagMatchReason;
    use crate::flags::flag_matching::FeatureFlagMatch;
    use crate::mock;
    use crate::utils::graph_utils::DependencyType;
    use crate::utils::test_utils::{counter_total, TestContext};
    use common_database::{get_pool_with_config, PoolConfig};
    use metrics_util::debugging::DebuggingRecorder;
    use serde_json::{json, Value};
    use std::collections::HashMap;
    use std::sync::atomic::{AtomicUsize, Ordering};
    use std::time::Duration;

    #[test]
    fn enabled_only_flag_keys_skip_dependencies_read_by_variant() {
        let flag = |id: i32, key: &str, properties: Value| {
            mock!(FeatureFlag,
                id: id,
                key: key.to_string(),
                filters: decode_filters(json!({"groups": [{"properties": properties}]})).unwrap()
            )
        };
        let reads = |flag_id: i32, value: Value| json!({"key": flag_id.to_string(), "type": "flag", "value": value, "operator": "flag_evaluates_to"});
        let flags = vec![
            flag(1, "failed", json!([])),
            flag(2, "read_by_match", json!([reads(1, json!(true))])),
            flag(3, "read_by_variant", json!([reads(1, json!(true))])),
            flag(
                4,
                "target",
                json!([reads(2, json!(true)), reads(3, json!("control"))]),
            ),
            flag(5, "unrelated", json!([])),
        ];
        let metadata = compute_flag_dependencies_or_single_stage(1, &flags);

        assert_eq!(
            enabled_only_flag_keys(&flags, 4, &metadata),
            HashSet::from(["failed", "read_by_match", "target"].map(String::from))
        );
    }

    fn target_flag_response(details: FlagDetails) -> Result<FlagsResponse, FlagError> {
        Ok(FlagsResponse::new(
            false,
            HashMap::from([("target".to_string(), details)]),
            None,
            Uuid::nil(),
        ))
    }

    fn failed_target_flag(error: FlagError) -> Result<FlagsResponse, FlagError> {
        let flag = mock!(FeatureFlag, key: "target".to_string());
        target_flag_response(FlagDetails::create_error(&flag, &error, None))
    }

    fn target_with_failed_dependencies(errors: &[FlagError]) -> Result<FlagsResponse, FlagError> {
        let target = mock!(FeatureFlag, key: "target".to_string());
        let mut flags = HashMap::from([(
            "target".to_string(),
            FlagDetails::create_error(&target, &FlagError::DependencyFailed(2), None),
        )]);
        for (dependency_id, error) in (2..).zip(errors) {
            let key = format!("dependency_{dependency_id}");
            let dependency = mock!(FeatureFlag, id: dependency_id, key: key.clone());
            flags.insert(key, FlagDetails::create_error(&dependency, error, None));
        }
        let healthy = mock!(FeatureFlag, id: 99, key: "healthy_dependency".to_string());
        flags.insert(
            "healthy_dependency".to_string(),
            FlagDetails::create(
                &healthy,
                &FeatureFlagMatch {
                    matches: false,
                    variant: None,
                    reason: FeatureFlagMatchReason::NoConditionMatch,
                    condition_index: None,
                    payload: None,
                    evaluation_v2: None,
                },
            ),
        );
        Ok(FlagsResponse::new(true, flags, None, Uuid::nil()))
    }

    fn evaluated() -> Result<FlagsResponse, FlagError> {
        let flag = mock!(FeatureFlag, key: "target".to_string());
        target_flag_response(FlagDetails::create(
            &flag,
            &FeatureFlagMatch {
                matches: false,
                variant: None,
                reason: FeatureFlagMatchReason::NoConditionMatch,
                condition_index: None,
                payload: None,
                evaluation_v2: None,
            },
        ))
    }

    fn database_unavailable() -> Result<FlagsResponse, FlagError> {
        Err(FlagError::DatabaseUnavailable)
    }

    #[rstest::rstest]
    #[case::error(database_unavailable, evaluated, 1, false, Some("recovered"))]
    #[case::transient_failed_flag(
        || failed_target_flag(FlagError::DatabaseUnavailable),
        evaluated,
        1,
        false,
        Some("recovered")
    )]
    #[case::timeout_error(
        || Err(FlagError::TimeoutError(Some("query_canceled".to_string()))),
        evaluated,
        1,
        false,
        Some("recovered")
    )]
    #[case::timeout_failed_flag(
        || failed_target_flag(FlagError::TimeoutError(None)),
        evaluated,
        1,
        false,
        Some("recovered")
    )]
    #[case::missing_dependency_flag(
        || failed_target_flag(FlagError::DependencyNotFound(DependencyType::Cohort, 1)),
        evaluated,
        1,
        true,
        None
    )]
    #[case::transient_failure_in_dependency(
        || target_with_failed_dependencies(&[FlagError::TimeoutError(Some("pool_timeout".to_string()))]),
        evaluated,
        1,
        false,
        Some("recovered")
    )]
    #[case::permanent_failure_in_dependency(
        || target_with_failed_dependencies(&[FlagError::DependencyNotFound(DependencyType::Cohort, 1)]),
        evaluated,
        1,
        true,
        None
    )]
    #[case::transient_and_permanent_failures_in_dependencies(
        || target_with_failed_dependencies(&[
            FlagError::TimeoutError(Some("pool_timeout".to_string())),
            FlagError::DependencyNotFound(DependencyType::Cohort, 1),
        ]),
        evaluated,
        1,
        true,
        None
    )]
    #[case::transient_failure_beside_unsupported_dependency(
        || target_with_failed_dependencies(&[
            FlagError::TimeoutError(Some("pool_timeout".to_string())),
            FlagError::flag_data_parsing("unsupported feature flag configuration format"),
        ]),
        evaluated,
        1,
        false,
        Some("recovered")
    )]
    #[case::database_error(
        || Err(FlagError::DatabaseError(sqlx::Error::ColumnNotFound("id".to_string()), None)),
        evaluated,
        1,
        true,
        None
    )]
    #[case::evaluated(evaluated, evaluated, 1, false, None)]
    #[case::retry_fails(database_unavailable, database_unavailable, 1, true, Some("failed"))]
    #[case::budget_spent(database_unavailable, evaluated, 0, true, None)]
    #[tokio::test]
    async fn test_evaluate_with_one_retry(
        #[case] first_attempt: fn() -> Result<FlagsResponse, FlagError>,
        #[case] second_attempt: fn() -> Result<FlagsResponse, FlagError>,
        #[case] retry_budget: u32,
        #[case] expected_failed: bool,
        #[case] expected_outcome: Option<&str>,
    ) {
        let recorder = DebuggingRecorder::new();
        let snapshotter = recorder.snapshotter();
        let _guard = metrics::set_default_local_recorder(&recorder);
        let attempts = AtomicUsize::new(0);
        let mut retries_left = retry_budget;

        let result = evaluate_with_one_retry("target", &mut retries_left, || {
            let attempt = attempts.fetch_add(1, Ordering::SeqCst);
            async move {
                if attempt == 0 {
                    first_attempt()
                } else {
                    second_attempt()
                }
            }
        })
        .await;

        let attempts = attempts.load(Ordering::SeqCst);
        assert_eq!(attempts, 1 + usize::from(expected_outcome.is_some()));
        assert_eq!(retries_left, retry_budget + 1 - attempts as u32);
        let failed = match result {
            Ok(response) => response.flags["target"].failed,
            Err(_) => true,
        };
        assert_eq!(failed, expected_failed);
        for outcome in ["recovered", "failed"] {
            assert_eq!(
                counter_total(
                    &snapshotter,
                    FLAG_BATCH_EVAL_PERSON_RETRIES_COUNTER,
                    &[("outcome", outcome)]
                ),
                u64::from(expected_outcome == Some(outcome)),
                "{outcome}"
            );
        }
    }

    #[tokio::test]
    async fn test_person_scan_outlasts_the_pool_statement_timeout() {
        let context = TestContext::new(None).await;
        let team = context.insert_new_team(None).await.unwrap();
        context
            .insert_person(team.id, "user".to_string(), None)
            .await
            .unwrap();

        let reader: PostgresReader = Arc::new(
            get_pool_with_config(
                &DEFAULT_TEST_CONFIG.get_persons_read_database_url(),
                PoolConfig {
                    statement_timeout_ms: Some(100),
                    max_connections: 1,
                    ..PoolConfig::default()
                },
            )
            .unwrap(),
        );

        let mut lock_conn = context.persons_writer.get_connection().await.unwrap();
        let mut lock_tx = lock_conn.begin().await.unwrap();
        sqlx::query("LOCK TABLE posthog_person IN ACCESS EXCLUSIVE MODE")
            .execute(&mut *lock_tx)
            .await
            .unwrap();

        let scan = tokio::spawn(scan_persons_page(reader.clone(), team.id, 0, 10, 10_000));

        // Release the lock only after the scan query waits on it. Otherwise the scan can start
        // after the rollback and pass without the SET LOCAL timeout.
        let deadline = tokio::time::Instant::now() + Duration::from_secs(10);
        loop {
            // pg_stat_activity keeps one snapshot for the whole transaction, so clear it
            // before each read.
            sqlx::query("SELECT pg_stat_clear_snapshot()")
                .execute(&mut *lock_tx)
                .await
                .unwrap();
            let scan_blocked: bool = sqlx::query_scalar(
                "SELECT EXISTS (
                    SELECT 1 FROM pg_stat_activity
                    WHERE pg_backend_pid() = ANY(pg_blocking_pids(pid))
                      AND query LIKE '%JOIN LATERAL%'
                )",
            )
            .fetch_one(&mut *lock_tx)
            .await
            .unwrap();
            if scan_blocked {
                break;
            }
            assert!(
                tokio::time::Instant::now() < deadline,
                "the person scan never waited on the table lock"
            );
            tokio::time::sleep(Duration::from_millis(10)).await;
        }

        // Hold the lock for longer than the pool's 100ms statement timeout.
        tokio::time::sleep(Duration::from_millis(300)).await;
        lock_tx.rollback().await.unwrap();

        let rows = scan.await.unwrap().unwrap();
        assert_eq!(rows.len(), 1);

        // The pool has one connection, so this acquire returns the connection the scan used.
        let mut conn = reader.get_connection().await.unwrap();
        let statement_timeout: String = sqlx::query_scalar("SHOW statement_timeout")
            .fetch_one(&mut *conn)
            .await
            .unwrap();
        assert_eq!(statement_timeout, "100ms");
    }
}