:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(flags): cap the persons db deadline at the request timeout (#113502)
Phil Haack364e5d65c10b
diff --git a/docs/internal/feature-flags/database-interaction-patterns.md b/docs/internal/feature-flags/database-interaction-patterns.mdindex ce9a4e5b7c04..54c595894d06 100644--- a/docs/internal/feature-flags/database-interaction-patterns.md+++ b/docs/internal/feature-flags/database-interaction-patterns.md@@ -79,7 +79,24 @@ The same hook also increments the `db_connection_created_total` counter when `po`PERSONS_DB_DEADLINE_MS` (default 2500ms) bounds all persons DB work in one flag evaluation.The hash key override check, write, and read, the group type mapping lookup, and the person, cohort, and group properties fetch share one deadline, so sequential calls cannot add up past it.-The deadline starts when the matcher is built, just before evaluation.+The budget starts when the matcher is built, just before evaluation.+The request timeout (`REQUEST_TIMEOUT_MS`) starts earlier, when the request arrives and before it waits for a concurrency permit.+So the deadline is the earlier of two instants: the budget's end, and `PERSONS_DB_DEADLINE_RESERVE_MS` (default 500ms) before the request timeout ends.+After a long permit wait, the second instant comes first.+A request that starts evaluation before that instant has the full reserve to finish after its persons DB calls stop.+A request that waited past that instant has less than the reserve left, so the request timeout can still return a 503.+Its persons DB calls fail without taking a connection.+Those flags report `timeout:persons_db_deadline` even when the persons DB is healthy.+`persons_db_deadline_bound` on the canonical log line says which limit set the deadline: `request_timeout` for the cap, `budget` otherwise.+`request_timeout` means time spent before evaluation shortened the deadline.+It does not show that the persons DB was healthy, because a stall lengthens permit waits and makes the cap set the deadline more often.+When the cap sets the deadline, raising `PERSONS_DB_DEADLINE_MS` has no effect.+Setting it to `0` removes the cap, and it also removes the bound on a stalled persons DB.+The realtime cohort lookup runs after the persons DB calls, and only `REALTIME_COHORT_LOOKUP_TIMEOUT_MS` (default 1000ms) bounds it.+The default lookup timeout exceeds the default reserve, so a request near the cap that evaluates realtime cohorts can still reach the request timeout.+Startup clamps the reserve to between 100ms and 2000ms.+Startup logs a warning when `REQUEST_TIMEOUT_MS` is not more than the reserve, because every `/flags` persons DB call then fails.+It also logs a warning when `PERSONS_DB_DEADLINE_MS` plus the reserve is at least `REQUEST_TIMEOUT_MS`, because the budget then has no effect on `/flags`.Postgres `statement_timeout` cannot cancel a query on a database that has stopped answering, so once a query is in flight this client-side timer is the only bound.When the pool has no free connection, the pool acquire timeout (`ACQUIRE_TIMEOUT_SECS`) can fire first, and the call fails with `timeout:pool_timeout` instead.@@ -106,6 +123,7 @@ The canonical log line records the first stopped call in `persons_db_deadline_exThe internal batch evaluation endpoint applies the deadline to each person's evaluation.A person that fails at the deadline is evaluated once more with a fresh deadline, up to `MAX_PERSON_RETRIES_PER_PAGE` retries per page.The deadline bounds each person, not the page.+`REQUEST_TIMEOUT_MS` does not cover this endpoint, so its deadline is the budget alone, with no reserve.A persons database that stays unresponsive still ends the page at `BATCH_FLAG_EVAL_TIMEOUT_MS`.When a person's evaluation still returns an error, Django leaves that person out of the static cohort, keeps the persons that matched, and records the run as failed.Set `PERSONS_DB_DEADLINE_MS=0` to disable the deadline.@@ -418,6 +436,7 @@ Queries exceeding 500ms are logged at WARN level with timing information.| `BEHAVIORAL_COHORTS_READ_DATABASE_URL` | empty | Behavioral cohorts database (enables realtime cohort evaluation) || `BATCH_FLAG_EVAL_SCAN_STATEMENT_TIMEOUT_MS` | 10000 | Statement timeout for the batch evaluation person scan || `PERSONS_DB_DEADLINE_MS` | 2500 | Deadline shared by all persons DB calls in one evaluation |+| `PERSONS_DB_DEADLINE_RESERVE_MS` | 500 | Time left for the rest of the request after that deadline |### Tuning guidancediff --git a/rust/feature-flags/src/api/batch_flag_evaluation.rs b/rust/feature-flags/src/api/batch_flag_evaluation.rsindex 036cc663aed5..673bff1cf6f8 100644--- a/rust/feature-flags/src/api/batch_flag_evaluation.rs+++ b/rust/feature-flags/src/api/batch_flag_evaluation.rs@@ -659,7 +659,12 @@ async fn handle_batch_flag_evaluation(.with_timezone(team_timezone)// The deadline starts when the matcher is built. Building the matcher in this// closure gives the retry in `evaluate_with_one_retry` its own deadline.- .with_persons_db_deadline(state.config.persons_db_deadline());+ .with_persons_db_deadline(+ state+ .config+ .persons_db_deadline(tokio::time::Instant::now(), None)+ .map(|deadline| deadline.at),+ );let flag_list = flag_list.clone();let target_key = target_key.clone();diff --git a/rust/feature-flags/src/api/concurrency_metrics.rs b/rust/feature-flags/src/api/concurrency_metrics.rsindex d656a85a5d3a..b39015caa47d 100644--- a/rust/feature-flags/src/api/concurrency_metrics.rs+++ b/rust/feature-flags/src/api/concurrency_metrics.rs@@ -33,13 +33,16 @@//! `flags_concurrency_limit_wait_ms` histogram collapsing to ~0 in//! production dashboards.-use std::time::{Duration, Instant};+use std::time::Duration;use axum::{extract::Request, middleware::Next, response::Response};+use tokio::time::Instant;/// Wall-clock instant captured immediately before the request enters/// `ConcurrencyLimitLayer`. Read by [`record_concurrency_wait`] to compute-/// permit-acquisition latency. `Copy` so the shim doesn't need to clone.+/// permit-acquisition latency. The `flags` handler also reads it to cap the+/// persons DB deadline at the request timeout. `Copy` so the shim doesn't+/// need to clone.#[derive(Clone, Copy, Debug)]pub struct ConcurrencyEnterTime(pub Instant);diff --git a/rust/feature-flags/src/api/endpoint.rs b/rust/feature-flags/src/api/endpoint.rsindex e33b87669633..94d21c9b57af 100644--- a/rust/feature-flags/src/api/endpoint.rs+++ b/rust/feature-flags/src/api/endpoint.rs@@ -1,7 +1,7 @@use crate::{api::{body_read_metrics::BodyReadDuration,- concurrency_metrics::ConcurrencyLimitWait,+ concurrency_metrics::{ConcurrencyEnterTime, ConcurrencyLimitWait},errors::{ClientFacingError, FlagError},flags_rate_limiter::RateLimitResult,types::{@@ -276,6 +276,9 @@ pub async fn flags<C>(Extension(rate_limiters): Extension<router::FlagsEndpointRateLimiters<C>>,InsecureClientIp(direct_ip): InsecureClientIp,Query(query_params): Query<FlagsQueryParams>,+ // Populated by the `record_concurrency_enter` middleware before the+ // permit wait. Optional for the same reason as `concurrency_wait`.+ concurrency_enter: Option<Extension<ConcurrencyEnterTime>>,// Populated by the `record_concurrency_wait` middleware after// `ConcurrencyLimitLayer` hands off a permit. Optional so the handler// tolerates the layer pair being removed or temporarily disabled.@@ -294,6 +297,9 @@ whereC: clock::Clock + Clone + Send + Sync + 'static,{let request_id = extract_request_id(&headers);+ let request_entered_at = concurrency_enter+ .map(|Extension(enter)| enter.0)+ .unwrap_or_else(tokio::time::Instant::now);// Extract client IP, checking X-Forwarded-For header firstlet ip = extract_client_ip(&headers, direct_ip);@@ -485,6 +491,7 @@ wherelet context = RequestContext {request_id,+ request_entered_at,state: state.clone(),ip,headers: headers.clone(),diff --git a/rust/feature-flags/src/config.rs b/rust/feature-flags/src/config.rsindex 4e8ded785fd3..33aeffb11c24 100644--- a/rust/feature-flags/src/config.rs+++ b/rust/feature-flags/src/config.rs@@ -439,11 +439,24 @@ pub struct Config {// check, write, and read, the group type mapping lookup, and the properties fetch.// statement_timeout cannot cancel a query on a database that has stopped answering.// Only this timer bounds those calls then. On expiry, the flags that need persons data- // return an error and the other flags evaluate normally. The default leaves 2s of the- // 4.5s REQUEST_TIMEOUT_MS for the rest of a /flags request. 0 disables the deadline.+ // return an error and the other flags evaluate normally. The budget starts just before+ // evaluation. With no permit wait, the default leaves 2s of the 4.5s REQUEST_TIMEOUT_MS for+ // the rest of a /flags request. `Config::persons_db_deadline` moves the deadline earlier+ // when the wait for a concurrency permit leaves less than PERSONS_DB_DEADLINE_RESERVE_MS of+ // that 2s. 0 disables the deadline.#[envconfig(from = "PERSONS_DB_DEADLINE_MS", default = "2500")]pub persons_db_deadline_ms: u64,+ // The rest of a /flags request has at least this long to finish after the persons DB+ // deadline. `Config::persons_db_deadline` never sets the deadline later than this long before+ // REQUEST_TIMEOUT_MS ends. Startup clamps it between `Config::MIN_PERSONS_DB_DEADLINE_RESERVE_MS`+ // and `Config::MAX_PERSONS_DB_DEADLINE_RESERVE_MS`. The realtime cohort lookup runs after the+ // persons DB calls, and only REALTIME_COHORT_LOOKUP_TIMEOUT_MS bounds it. The default lookup+ // timeout exceeds the default reserve, so a request near the cap can still reach the request+ // timeout.+ #[envconfig(from = "PERSONS_DB_DEADLINE_RESERVE_MS", default = "500")]+ pub persons_db_deadline_reserve_ms: u64,+#[envconfig(default = "1000")]pub max_concurrency: usize,@@ -1080,10 +1093,43 @@ impl ThreadCounts {}}+/// The instant after which persons DB calls in one evaluation fail, and the limit that set it.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub struct PersonsDbDeadline {+ pub at: tokio::time::Instant,+ pub bound: PersonsDbDeadlineBound,+}++/// `RequestTimeout` means the time before evaluation shortened the deadline. Persons DB calls can+/// then fail on a healthy persons DB, so the canonical log line records the bound.+#[derive(Clone, Copy, Debug, PartialEq, Eq)]+pub enum PersonsDbDeadlineBound {+ /// `PERSONS_DB_DEADLINE_MS` after evaluation starts.+ Budget,+ /// `PERSONS_DB_DEADLINE_RESERVE_MS` before `REQUEST_TIMEOUT_MS` ends.+ RequestTimeout,+}++impl PersonsDbDeadlineBound {+ pub fn as_str(&self) -> &'static str {+ match self {+ Self::Budget => "budget",+ Self::RequestTimeout => "request_timeout",+ }+ }+}+impl Config {const MAX_RESPONSE_TIMEOUT_MS: u64 = 30_000; // 30 secondsconst MAX_CONNECTION_TIMEOUT_MS: u64 = 60_000; // 60 seconds+ /// With less time, the flags evaluated after the persons DB calls may not finish before the+ /// request timeout returns a 503.+ const MIN_PERSONS_DB_DEADLINE_RESERVE_MS: u64 = 100;+ /// The work after the persons DB calls needs far less time. A larger reserve only takes time+ /// from the persons DB budget.+ const MAX_PERSONS_DB_DEADLINE_RESERVE_MS: u64 = 2_000;+/// Validate and fix timeout configuration, logging warnings and applying defaults for invalid values////// This method checks timeout values and relationships, applying safe defaults when invalid@@ -1122,6 +1168,47 @@ impl Config {self.redis_connection_timeout_ms);}++ let reserve_ms = self.persons_db_deadline_reserve_ms.clamp(+ Self::MIN_PERSONS_DB_DEADLINE_RESERVE_MS,+ Self::MAX_PERSONS_DB_DEADLINE_RESERVE_MS,+ );+ if reserve_ms != self.persons_db_deadline_reserve_ms {+ tracing::warn!(+ configured = self.persons_db_deadline_reserve_ms,+ using = reserve_ms,+ "PERSONS_DB_DEADLINE_RESERVE_MS is outside {}..={}ms, clamping",+ Self::MIN_PERSONS_DB_DEADLINE_RESERVE_MS,+ Self::MAX_PERSONS_DB_DEADLINE_RESERVE_MS+ );+ self.persons_db_deadline_reserve_ms = reserve_ms;+ }++ if let Some(problem) = self.persons_db_deadline_misconfiguration() {+ tracing::warn!(+ persons_db_deadline_ms = self.persons_db_deadline_ms,+ persons_db_deadline_reserve_ms = self.persons_db_deadline_reserve_ms,+ request_timeout_ms = self.request_timeout_ms,+ "{problem}"+ );+ }+ }++ /// Why the persons DB deadline settings defeat the /flags deadline, or `None` when they do not.+ fn persons_db_deadline_misconfiguration(&self) -> Option<&'static str> {+ if self.persons_db_deadline_ms == 0 {+ None+ } else if self.request_timeout_ms <= self.persons_db_deadline_reserve_ms {+ Some("REQUEST_TIMEOUT_MS is not more than PERSONS_DB_DEADLINE_RESERVE_MS, so every /flags persons DB call fails without running")+ } else if self+ .persons_db_deadline_ms+ .saturating_add(self.persons_db_deadline_reserve_ms)+ >= self.request_timeout_ms+ {+ Some("PERSONS_DB_DEADLINE_MS plus PERSONS_DB_DEADLINE_RESERVE_MS is at least REQUEST_TIMEOUT_MS, so the request timeout always sets the /flags persons DB deadline and PERSONS_DB_DEADLINE_MS has no effect on /flags")+ } else {+ None+ }}/// Baseline config for the integration test harness.@@ -1178,6 +1265,7 @@ impl Config {cohort_membership_cache_max_entries: 50_000,realtime_cohort_lookup_timeout_ms: 1000,persons_db_deadline_ms: 30_000,+ persons_db_deadline_reserve_ms: 500,max_concurrency: 1000,max_pg_connections: 10,min_non_persons_reader_connections: 0,@@ -1352,10 +1440,39 @@ impl Config {}}- /// The budget for all persons DB work in one flag evaluation, or `None` when disabled.- pub fn persons_db_deadline(&self) -> Option<std::time::Duration> {- (self.persons_db_deadline_ms > 0)- .then(|| std::time::Duration::from_millis(self.persons_db_deadline_ms))+ /// The instant when all persons DB work in one flag evaluation stops and the limit that set it,+ /// or `None` when disabled.+ /// The `persons_db_deadline_ms` budget counts from `now`. The request timeout counts from+ /// `request_entered_at`, before the wait for a concurrency permit. The deadline is never later+ /// than `persons_db_deadline_reserve_ms` before the request timeout ends, so the rest of the+ /// request has that long to finish. When `request_timeout_ms` is not more than the reserve,+ /// the deadline is the arrival time and every persons DB call fails. `request_entered_at` is+ /// `None` for a caller that `request_timeout_ms` does not cover, such as batch evaluation.+ /// The deadline is then the budget's end.+ pub fn persons_db_deadline(+ &self,+ now: tokio::time::Instant,+ request_entered_at: Option<tokio::time::Instant>,+ ) -> Option<PersonsDbDeadline> {+ if self.persons_db_deadline_ms == 0 {+ return None;+ }+ let budget_end = PersonsDbDeadline {+ at: now + std::time::Duration::from_millis(self.persons_db_deadline_ms),+ bound: PersonsDbDeadlineBound::Budget,+ };+ let request_cap = request_entered_at.map(|entered| PersonsDbDeadline {+ at: entered+ + std::time::Duration::from_millis(+ self.request_timeout_ms+ .saturating_sub(self.persons_db_deadline_reserve_ms),+ ),+ bound: PersonsDbDeadlineBound::RequestTimeout,+ });+ Some(match request_cap {+ Some(cap) if cap.at < budget_end.at => cap,+ _ => budget_end,+ })}/// Check if persons database routing is enabled@@ -1720,6 +1837,76 @@ mod tests {);}+ #[rstest::rstest]+ #[case::below_minimum(0, 100)]+ #[case::in_range(500, 500)]+ #[case::above_maximum(10_000, 2_000)]+ fn test_validate_and_fix_timeouts_clamps_persons_db_deadline_reserve(+ #[case] configured_ms: u64,+ #[case] expected_ms: u64,+ ) {+ let mut config = Config::default_test_config();+ config.persons_db_deadline_reserve_ms = configured_ms;+ config.validate_and_fix_timeouts();+ assert_eq!(config.persons_db_deadline_reserve_ms, expected_ms);+ }++ #[rstest::rstest]+ #[case::fits(2500, 500, 4500, false)]+ #[case::budget_reaches_the_cap(2500, 2000, 4500, true)]+ #[case::request_timeout_within_the_reserve(2500, 500, 400, true)]+ #[case::disabled(0, 500, 100, false)]+ fn test_persons_db_deadline_misconfiguration(+ #[case] persons_db_deadline_ms: u64,+ #[case] reserve_ms: u64,+ #[case] request_timeout_ms: u64,+ #[case] misconfigured: bool,+ ) {+ let mut config = Config::default_test_config();+ config.persons_db_deadline_ms = persons_db_deadline_ms;+ config.persons_db_deadline_reserve_ms = reserve_ms;+ config.request_timeout_ms = request_timeout_ms;+ assert_eq!(+ config.persons_db_deadline_misconfiguration().is_some(),+ misconfigured+ );+ }++ // The cap is the request timeout minus the reserve, counted from arrival.+ #[rstest::rstest]+ #[case::short_permit_wait(2500, 500, 4500, true, 1000, Some((3500, PersonsDbDeadlineBound::Budget)))]+ #[case::long_permit_wait(2500, 500, 4500, true, 3000, Some((4000, PersonsDbDeadlineBound::RequestTimeout)))]+ #[case::larger_reserve(2500, 1000, 4500, true, 3000, Some((3500, PersonsDbDeadlineBound::RequestTimeout)))]+ #[case::waited_past_the_cap(2500, 500, 4500, true, 4200, Some((4000, PersonsDbDeadlineBound::RequestTimeout)))]+ #[case::request_timeout_within_the_reserve(2500, 500, 400, true, 0, Some((0, PersonsDbDeadlineBound::RequestTimeout)))]+ #[case::no_request_timeout(2500, 500, 400, false, 0, Some((2500, PersonsDbDeadlineBound::Budget)))]+ #[case::disabled(0, 500, 4500, true, 0, None)]+ fn test_persons_db_deadline_is_capped_by_the_request_timeout(+ #[case] persons_db_deadline_ms: u64,+ #[case] reserve_ms: u64,+ #[case] request_timeout_ms: u64,+ #[case] request_timeout_applies: bool,+ #[case] permit_wait_ms: u64,+ #[case] expected: Option<(u64, PersonsDbDeadlineBound)>,+ ) {+ use std::time::Duration;++ let mut config = Config::default_test_config();+ config.persons_db_deadline_ms = persons_db_deadline_ms;+ config.persons_db_deadline_reserve_ms = reserve_ms;+ config.request_timeout_ms = request_timeout_ms;+ let arrival = tokio::time::Instant::now();+ let now = arrival + Duration::from_millis(permit_wait_ms);++ assert_eq!(+ config.persons_db_deadline(now, request_timeout_applies.then_some(arrival)),+ expected.map(|(ms, bound)| PersonsDbDeadline {+ at: arrival + Duration::from_millis(ms),+ bound,+ })+ );+ }+#[test]fn test_validate_and_fix_timeouts_any_relationship_allowed() {let mut config = Config::default_test_config();diff --git a/rust/feature-flags/src/flags/flag_matching.rs b/rust/feature-flags/src/flags/flag_matching.rsindex f7dcf650405b..3f081e5a9b5b 100644--- a/rust/feature-flags/src/flags/flag_matching.rs+++ b/rust/feature-flags/src/flags/flag_matching.rs@@ -55,7 +55,6 @@ use serde_json::Value;use std::collections::{HashMap, HashSet};use std::future::Future;use std::sync::Arc;-use std::time::Duration;use tokio::time::Instant;use tracing::{debug, error, instrument, warn};use uuid::Uuid;@@ -675,9 +674,9 @@ impl FeatureFlagMatcher {self}- /// Gives all persons DB work in this evaluation one shared budget, which starts now.- pub fn with_persons_db_deadline(mut self, budget: Option<Duration>) -> Self {- self.persons_db_deadline = budget.map(|budget| Instant::now() + budget);+ /// Gives all persons DB work in this evaluation one shared deadline.+ pub fn with_persons_db_deadline(mut self, deadline: Option<Instant>) -> Self {+ self.persons_db_deadline = deadline;self}diff --git a/rust/feature-flags/src/flags/test_flag_matching.rs b/rust/feature-flags/src/flags/test_flag_matching.rsindex c51ad15ce472..efa17fe31448 100644--- a/rust/feature-flags/src/flags/test_flag_matching.rs+++ b/rust/feature-flags/src/flags/test_flag_matching.rs@@ -8238,7 +8238,7 @@ mod tests {Arc::new(GroupTypeCacheManager::new(stalled_db.clone(), None, None)),Some(HashMap::from([("project".to_string(), json!("p1"))])),)- .with_persons_db_deadline(Some(deadline));+ .with_persons_db_deadline(Some(tokio::time::Instant::now() + deadline));let rollout_flag = mock!(FeatureFlag, id: 1, key: "rollout_flag".mock_into());let person_flag = mock!(FeatureFlag,diff --git a/rust/feature-flags/src/handler/canonical_log.rs b/rust/feature-flags/src/handler/canonical_log.rsindex b2e59d95272f..3499a91c8dd2 100644--- a/rust/feature-flags/src/handler/canonical_log.rs+++ b/rust/feature-flags/src/handler/canonical_log.rs@@ -1,4 +1,5 @@use crate::api::errors::FlagError;+use crate::config::PersonsDbDeadlineBound;use crate::flags::flag_matching::EvaluationType;use crate::handler::phases::PhaseDurations;use crate::metrics::consts::{@@ -255,6 +256,9 @@ pub struct FlagsCanonicalLogLine {/// The first persons DB call that the persons DB deadline stopped, such as/// "fetch_properties". The flags that needed persons data returned an error.pub persons_db_deadline_exceeded: Option<&'static str>,+ /// Which limit set the persons DB deadline. With `RequestTimeout`, a+ /// `persons_db_deadline_exceeded` on the same line can occur on a healthy persons DB.+ pub persons_db_deadline_bound: Option<PersonsDbDeadlineBound>,/// Which evaluation strategy was used for this request./// Set to `Parallel` if any dependency level triggered parallel evaluation.@@ -357,6 +361,7 @@ impl Default for FlagsCanonicalLogLine {dependency_graph_errors: 0,hash_key_override_status: None,persons_db_deadline_exceeded: None,+ persons_db_deadline_bound: None,evaluation_type: None,rate_limited: false,rate_limit_warned: false,@@ -433,6 +438,7 @@ impl FlagsCanonicalLogLine {dependency_graph_errors = self.dependency_graph_errors,hash_key_override_status = self.hash_key_override_status,persons_db_deadline_exceeded = self.persons_db_deadline_exceeded,+ persons_db_deadline_bound = self.persons_db_deadline_bound.map(|b| b.as_str()),evaluation_type = self.evaluation_type.map(|t| t.as_str()),rate_limited = self.rate_limited,rate_limit_warned = self.rate_limit_warned,diff --git a/rust/feature-flags/src/handler/flags.rs b/rust/feature-flags/src/handler/flags.rsindex 993cb3e3907b..3251cab00b55 100644--- a/rust/feature-flags/src/handler/flags.rs+++ b/rust/feature-flags/src/handler/flags.rs@@ -342,6 +342,7 @@ pub async fn evaluate_for_request(groups: Option<HashMap<String, Value>>,hash_key_override: Option<String>,request_id: Uuid,+ request_entered_at: tokio::time::Instant,disable_flags: bool,flag_keys: Option<Vec<String>>,detailed_analysis: Option<bool>,@@ -362,6 +363,13 @@ pub async fn evaluate_for_request(return Ok(FlagsResponse::new(false, HashMap::new(), None, request_id));}+ let persons_db_deadline = state+ .config+ .persons_db_deadline(tokio::time::Instant::now(), Some(request_entered_at));+ with_canonical_log(|log| {+ log.persons_db_deadline_bound = persons_db_deadline.map(|deadline| deadline.bound)+ });+let ctx = FeatureFlagEvaluationContext {team_id,team_timezone,@@ -397,7 +405,7 @@ pub async fn evaluate_for_request(membership_stamp_policy: state.config.realtime_cohort_membership_stamp_policy,detailed_analysis: detailed_analysis.unwrap_or(false),only_use_override_person_properties: only_use_override_person_properties.unwrap_or(false),- persons_db_deadline: state.config.persons_db_deadline(),+ persons_db_deadline: persons_db_deadline.map(|deadline| deadline.at),};evaluation::evaluate_feature_flags(ctx, request_id).awaitdiff --git a/rust/feature-flags/src/handler/mod.rs b/rust/feature-flags/src/handler/mod.rsindex 6ff4be534ae5..9eb9057d9cd2 100644--- a/rust/feature-flags/src/handler/mod.rs+++ b/rust/feature-flags/src/handler/mod.rs@@ -273,6 +273,7 @@ async fn process_request_inner(property_overrides.groups,property_overrides.hash_key,context.request_id,+ context.request_entered_at,request.is_flags_disabled(),request.flag_keys.clone(),Some(is_internal && context.meta.detailed_analysis.unwrap_or(false)),diff --git a/rust/feature-flags/src/handler/types.rs b/rust/feature-flags/src/handler/types.rsindex 1af10d30feba..09eaffe49f51 100644--- a/rust/feature-flags/src/handler/types.rs+++ b/rust/feature-flags/src/handler/types.rs@@ -42,6 +42,9 @@ pub struct RequestContext {/// Request IDpub request_id: Uuid,+ /// When the request arrived, before it waited for a concurrency permit.+ pub request_entered_at: tokio::time::Instant,+/// Side channel for body logging: when at least one team is opted into/// `BodyLogger`, the endpoint installs an `Arc<OnceLock<Bytes>>` here and/// keeps a clone. The decode step in `parse_and_authenticate` fills it@@ -103,8 +106,8 @@ pub struct FeatureFlagEvaluationContext {pub detailed_analysis: bool,/// Whether to only use person properties from request payload, ignoring database properties.pub only_use_override_person_properties: bool,- /// Budget shared by all persons DB work in this evaluation. `None` disables the deadline.- pub persons_db_deadline: Option<std::time::Duration>,+ /// Persons DB calls in this evaluation fail after this instant. `None` disables the deadline.+ pub persons_db_deadline: Option<tokio::time::Instant>,}/// SDK type classification based on user-agent parsing.diff --git a/rust/feature-flags/src/main.rs b/rust/feature-flags/src/main.rsindex 745c407df337..a0beb94175d5 100644--- a/rust/feature-flags/src/main.rs+++ b/rust/feature-flags/src/main.rs@@ -51,8 +51,6 @@ fn init_tracer(}async fn async_main(mut config: Config, rayon_dispatcher: RayonDispatcher) {- config.validate_and_fix_timeouts();-// Instantiate tracing outputs following Django's DEBUG-based approach:// - stdout with a level configured by the RUST_LOG envvar// - OpenTelemetry if enabled, for levels INFO and higher@@ -133,6 +131,10 @@ async fn async_main(mut config: Config, rayon_dispatcher: RayonDispatcher) {.with(otel_layer).init();+ // This call must follow tracing init. Before init, tracing has no subscriber and drops the+ // warnings.+ config.validate_and_fix_timeouts();+// Start continuous profiling if enabled (keep _agent alive for the duration of the program)// NOTE: Must be after tracing is initialized so logs are visiblelet _profiling_agent = match config.continuous_profiling.start_agent() {diff --git a/rust/feature-flags/src/router.rs b/rust/feature-flags/src/router.rsindex 08ad99934287..a2d647bc31b2 100644--- a/rust/feature-flags/src/router.rs+++ b/rust/feature-flags/src/router.rs@@ -570,7 +570,9 @@ where.layer(axum::middleware::from_fn(record_concurrency_wait)).layer(ConcurrencyLimitLayer::new(config.max_concurrency))// Stamps `Instant::now()` after timeout-deadline propagation but- // before permit acquisition.+ // before permit acquisition. The flags handler caps the persons DB+ // deadline relative to this instant. That cap holds only while this+ // layer sits directly inside `TimeoutLayer`..layer(axum::middleware::from_fn(record_concurrency_enter)).layer(ServiceBuilder::new()diff --git a/rust/feature-flags/tests/test_flags.rs b/rust/feature-flags/tests/test_flags.rsindex d6967f30b539..b59560afa9aa 100644--- a/rust/feature-flags/tests/test_flags.rs+++ b/rust/feature-flags/tests/test_flags.rs@@ -6346,7 +6346,13 @@ async fn it_degrades_person_flags_when_the_persons_db_stops_answering() -> Resul);// The deadline is shorter than the pool acquire timeout. Only the deadline can then produce// the reason code that the test asserts below.- config.persons_db_deadline_ms = 500;+ config.persons_db_deadline_ms = 1000;+ // With one permit, one request waits about 1s for the other to finish. The waiting request+ // must stop its persons DB work by 1s after it arrived, which is the 1.5s request timeout+ // minus the 500ms reserve. If its deadline counted 1s from when it got the permit, the+ // request timeout would end it with a 503.+ config.max_concurrency = 1;+ config.request_timeout_ms = 1500;let client = setup_redis_client(Some(config.redis_url.clone())).await;let team = insert_new_team_in_redis(client.clone()).await.unwrap();@@ -6377,22 +6383,25 @@ async fn it_degrades_person_flags_when_the_persons_db_stops_answering() -> Resulinsert_flags_for_team_in_redis(client, team.id, Some(flag_json.to_string())).await?;let server = ServerHandle::for_config(config).await;- let payload = json!({"token": team.api_token, "distinct_id": "stalled_user"});- let res = server- .send_flags_request(payload.to_string(), Some("2"), None)- .await;-- assert_eq!(res.status(), StatusCode::OK);- assert_json_include!(- actual: res.json::<Value>().await?,- expected: json!({- "errorsWhileComputingFlags": true,- "flags": {- "rollout-flag": {"enabled": true},- "person-flag": {"enabled": false, "reason": {"code": "timeout:persons_db_deadline"}}- }- })+ let payload = json!({"token": team.api_token, "distinct_id": "stalled_user"}).to_string();+ let (res_a, res_b) = tokio::join!(+ server.send_flags_request(payload.clone(), Some("2"), None),+ server.send_flags_request(payload, Some("2"), None),);+ for res in [res_a, res_b] {+ assert_eq!(res.status(), StatusCode::OK);+ assert_json_include!(+ actual: res.json::<Value>().await?,+ expected: json!({+ "errorsWhileComputingFlags": true,+ "flags": {+ "rollout-flag": {"enabled": true},+ "person-flag": {"enabled": false, "reason": {"code": "timeout:persons_db_deadline"}}+ }+ })+ );+ }+Ok(())}