:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
//! `BillingAggregator` — in-process per-pod aggregation with periodic flush.
//!
//! `record()` is synchronous, lock-scoped, and returns in microseconds. A
//! background task wakes every `flush_interval`, atomically swaps the inner
//! map, and issues pipelined `HINCRBY`s to Redis. On graceful shutdown a
//! final flush runs before the service exits.
//!
//! The aggregator is the authoritative writer for the production billing
//! keyspace — there is no per-request synchronous HINCRBY. That makes
//! `flags_billing_pending_records` and
//! `flags_billing_seconds_since_successful_flush` billing-critical signals.
//!
//! # Durability trade-off
//!
//! Aggregation defers writes until the next flush, so failure modes differ
//! from a per-request synchronous write:
//!
//! - **Redis error on a flush chunk** (normal tick): failing chunk plus
//! any unattempted remainder are re-queued into `pending` and retried on
//! the next tick. Incident magnitude surfaces on
//! `flags_billing_flush_blocked_requests_total{error_type=…}` — *not*
//! `flags_billing_unflushed_requests_total`, since the records are not
//! terminally lost. Re-queued entries are **rebucketed to the current
//! bucket** so a sustained outage does not accumulate one new
//! `AggregationKey` per (team, request_type, library) on every
//! `CACHE_BUCKET_SIZE` rollover — cardinality stays bounded by the
//! active (team × request_type × library) surface regardless of outage
//! duration. Counts are conserved; per-bucket time attribution during
//! the outage is reattributed to the bucket current at retry time. The
//! `max_pending_entries` cap remains as a safety tripwire (not a
//! working-set estimate) — `flags_billing_unflushed_requests_total{cause="cap_drop"}`
//! firing now signals exotic failure (e.g. a single sustained-outage
//! pod with cardinality already at cap before the outage), not the
//! normal outage trajectory.
//! - **Graceful shutdown** (SIGTERM): a final best-effort flush runs
//! within `shutdown_flush_timeout`. Anything not flushed is reported
//! under `flags_billing_unflushed_requests_total{cause="shutdown_drop"}`.
//! - **Ungraceful termination** (SIGKILL past the grace window, OOM, node
//! loss, panic): **up to one `flush_interval` of records is lost.** The
//! `pending` map lives only in process memory; there is no WAL. The
//! crashed pod cannot emit at the moment of loss, so detection relies
//! on the **last scrape before the crash**: `flags_billing_pending_records`
//! gauges the per-pod count of at-risk records, and a non-zero value
//! correlated with a pod restart in
//! `kube_pod_container_status_restarts_total` attributes the loss. The
//! fleet-wide `flags_billing_records_total` vs.
//! `flags_billing_entries_flushed_total` rate ratio is a secondary
//! signal for the same gap.
//!
//! ## Detecting a wedged flusher before it drops counts
//!
//! - `flags_billing_pending_entries` — live gauge; sustained growth
//! is the leading indicator.
//! - `flags_billing_seconds_since_successful_flush` — stale-flush
//! alarm; rises even when `flags_billing_flush_errors_total` stays
//! flat (the signature of a hung `execute_pipeline`).
//! - `flags_billing_flush_errors_total` — labeled by `error_type`;
//! alert via `rate(flags_billing_flush_errors_total[1m]) > 0 for: 30s`
//! to catch a wedged Redis link.
//!
//! ## Knob-sizing guidance
//!
//! `flush_interval` directly bounds the crash-loss window. Shorter = less
//! worst-case data loss, more Redis RTTs; longer = better aggregation,
//! bigger worst-case loss. The default (10s) is sized so a typical SIGKILL
//! loses well under a minute of billable activity per pod while still
//! giving the aggregation meaningful compression.
//!
//! See `validate` for knob bounds. Mis-setting any knob to zero silently
//! breaks billing in a different way (panic, cap every record, instant
//! shutdown loss) — `validate` rejects them at boot.
use std::collections::HashMap;
use std::sync::atomic::{AtomicU64, Ordering};
use std::sync::{Arc, Mutex};
use std::time::Duration;
use common_metrics::{gauge, histogram, inc};
use common_redis::{Client as RedisClient, CustomRedisError, PipelineCommand};
use rand::Rng;
use strum::IntoEnumIterator;
use tokio::sync::Notify;
use tokio::task::JoinHandle;
use crate::billing::usage_reporter::UsageReporter;
use crate::flags::flag_analytics::{
current_bucket, get_team_request_key, get_team_request_library_key,
};
use crate::flags::flag_request::FlagRequestType;
use crate::handler::types::Library;
use crate::metrics::consts::{
FLAGS_BILLING_ENTRIES_FLUSHED, FLAGS_BILLING_FLUSH_BLOCKED_REQUESTS,
FLAGS_BILLING_FLUSH_DURATION_MS, FLAGS_BILLING_FLUSH_ERRORS, FLAGS_BILLING_PENDING_ENTRIES,
FLAGS_BILLING_PENDING_RECORDS, FLAGS_BILLING_RECORDS, FLAGS_BILLING_RECORD_DURATION_US,
FLAGS_BILLING_SECONDS_SINCE_SUCCESSFUL_FLUSH, FLAGS_BILLING_UNFLUSHED_REQUESTS,
};
fn record_labels_for(request_type: FlagRequestType) -> Vec<(String, String)> {
vec![(
"request_type".to_string(),
request_type.as_str().to_string(),
)]
}
/// Cause label for `flags_billing_unflushed_requests_total`. Every variant is
/// a terminal loss; transient retries surface on
/// `FLAGS_BILLING_FLUSH_BLOCKED_REQUESTS` instead. Kept as a closed enum so a
/// typo can't quietly create a new label value.
#[derive(Debug, Clone, Copy)]
enum UnflushedCause {
CapDrop,
FlushDroppedOnError,
ShutdownDrop,
}
impl UnflushedCause {
fn as_str(self) -> &'static str {
match self {
UnflushedCause::CapDrop => "cap_drop",
UnflushedCause::FlushDroppedOnError => "flush_dropped_on_error",
UnflushedCause::ShutdownDrop => "shutdown_drop",
}
}
}
fn inc_unflushed(cause: UnflushedCause, count: u64) {
if count == 0 {
return;
}
inc(
FLAGS_BILLING_UNFLUSHED_REQUESTS,
&[("cause".to_string(), cause.as_str().to_string())],
count,
);
}
/// Record counts blocked by a `BailOnError` flush failure. These records are
/// requeued for the next tick — this is incident-magnitude, not terminal
/// loss. The `error_type` value matches `classify_redis_error`, so per-type
/// rates line up with `FLAGS_BILLING_FLUSH_ERRORS`.
fn inc_flush_blocked(error_type: &'static str, count: u64) {
if count == 0 {
return;
}
inc(
FLAGS_BILLING_FLUSH_BLOCKED_REQUESTS,
&[("error_type".to_string(), error_type.to_string())],
count,
);
}
/// Whether a failing chunk's HINCRBYs likely reached Redis. Drives the
/// flusher's decision of "credit it" vs "retry it" vs "drop it" — see the
/// branch in `flush_chunk`.
///
/// `Applied`: Redis received and processed the commands; the failure was on
/// the response path (e.g. malformed reply we couldn't parse). Crediting
/// the chunk to `entries_flushed_total` matches Redis's actual state. A
/// retry here would double-count.
///
/// `Ambiguous`: timeouts and generic transport failures. The bytes may or
/// may not have reached Redis. Today's policy is to retry (matches the
/// pre-classification behavior, which optimizes for the much larger
/// under-count risk from dropping). The label is mostly for visibility:
/// once we have a reconciliation against the per-request shadow keys, this
/// is the bucket whose retry-vs-drop default we'd revisit.
///
/// `NotApplied`: the call never reached Redis (e.g. configuration errors,
/// connection refused). Retry is unambiguously safe.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum FlushOutcome {
Applied,
Ambiguous,
NotApplied,
}
impl FlushOutcome {
fn as_str(self) -> &'static str {
match self {
FlushOutcome::Applied => "applied",
FlushOutcome::Ambiguous => "ambiguous",
FlushOutcome::NotApplied => "not_applied",
}
}
}
/// Map a Redis error to whether the call likely reached Redis. Conservative
/// by default: anything we can't classify confidently lands in `Ambiguous`.
fn classify_flush_outcome(err: &CustomRedisError) -> FlushOutcome {
// Exhaustive on purpose: same compile-error discipline as
// `classify_redis_error`. A new `CustomRedisError` variant must pick an
// outcome explicitly rather than defaulting silently.
match err {
// Redis sent us bytes back — we couldn't parse them. For
// pipelined HINCRBYs, the only way we get to "tried to parse a
// response" is if Redis applied all commands before sending it.
CustomRedisError::ParseError(_) => FlushOutcome::Applied,
// HINCRBY on a pre-existing hash can't actually return NotFound, but
// if it did, Redis processed the command and returned an answer.
CustomRedisError::NotFound => FlushOutcome::Applied,
// Pod can't open a connection — bytes never left.
CustomRedisError::InvalidConfiguration(_) => FlushOutcome::NotApplied,
// Timeout could fire on the write deadline (NotApplied) or the read
// deadline (Applied-with-lost-ack). Without protocol-level
// distinction we default to the safer label.
CustomRedisError::Timeout => FlushOutcome::Ambiguous,
// Transport-level failures: cluster redirects, connection resets,
// partial-pipeline aborts. Some are NotApplied (connection refused
// before any byte sent), some are Applied-with-lost-ack. Inspecting
// the inner kind to split them is a future refinement; today we
// pick the safe label.
CustomRedisError::Redis(_) => FlushOutcome::Ambiguous,
}
}
/// Key used to aggregate repeated `record()` calls in-process.
///
/// `bucket` is computed from the record time, not the flush time — late-flushed
/// records must still land in the bucket they arrived in.
#[derive(Debug, Clone, Copy, PartialEq, Eq, Hash)]
pub struct AggregationKey {
pub team_id: i32,
pub request_type: FlagRequestType,
pub library: Option<Library>,
pub bucket: u64,
}
/// Policy for handling a chunk error during a flush. The flusher's normal
/// tick uses `BailOnError` (if Redis is rejecting, save RTTs and retry next
/// interval). The shutdown path uses `BestEffort` — it's our last chance to
/// land writes before the pod exits, so don't abandon trailing chunks over a
/// single transient failure.
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum FlushPolicy {
BailOnError,
BestEffort,
}
#[derive(Debug, Clone)]
pub struct BillingAggregatorConfig {
/// Interval between flushes. Must be non-zero — see `validate`.
pub flush_interval: Duration,
/// Safety tripwire — expected steady-state size is orders of magnitude
/// smaller. A non-zero rate of
/// `flags_billing_unflushed_requests_total{cause="cap_drop"}` is an alert.
/// Must be non-zero — see `validate`.
pub max_pending_entries: usize,
/// Must be non-zero — see `validate`.
pub per_flush_batch_size: usize,
/// Must be non-zero — see `validate`.
pub shutdown_flush_timeout: Duration,
/// Override the random jitter with a fixed duration. Used by tests to make
/// timing deterministic under paused tokio time. Not intended for production.
#[doc(hidden)]
pub jitter_override: Option<Duration>,
}
impl Default for BillingAggregatorConfig {
fn default() -> Self {
Self {
flush_interval: Duration::from_secs(10),
max_pending_entries: 500_000,
per_flush_batch_size: 200,
shutdown_flush_timeout: Duration::from_secs(15),
jitter_override: None,
}
}
}
impl BillingAggregatorConfig {
/// Reject zero/degenerate knob values that would break the flusher or
/// silently disable billing. Called by `BillingAggregator::start` before
/// spawning the background task so a misconfigured deploy fails loudly at
/// boot instead of crashing the flusher task minutes later.
pub fn validate(&self) -> Result<(), String> {
if self.flush_interval.is_zero() {
return Err("flush_interval must be > 0 (tokio::time::interval panics on zero)".into());
}
if self.max_pending_entries == 0 {
return Err(
"max_pending_entries must be > 0 (zero caps every new key immediately)".into(),
);
}
if self.per_flush_batch_size == 0 {
return Err("per_flush_batch_size must be > 0".into());
}
if self.shutdown_flush_timeout.is_zero() {
return Err(
"shutdown_flush_timeout must be > 0 (zero loses every pending count on shutdown)"
.into(),
);
}
Ok(())
}
}
/// In-process counter aggregation for billable flag requests.
pub struct BillingAggregator {
inner: Arc<Inner>,
flusher: Mutex<Option<JoinHandle<()>>>,
metrics_sampler: Mutex<Option<JoinHandle<()>>>,
}
struct Inner {
config: BillingAggregatorConfig,
redis: Arc<dyn RedisClient + Send + Sync>,
pending: Mutex<HashMap<AggregationKey, u64>>,
/// Running sum of `pending.values()`. Updated under the `pending` lock so
/// increments and the lock-protected HashMap mutation are guaranteed
/// consistent. The flush drain swaps it to zero, the requeue path adds
/// back what's reinjected, and shutdown reads it as the residual count of
/// records still in `pending`.
pending_total: AtomicU64,
/// Counts drained from `pending` but not yet credited to any terminal
/// counter (`entries_flushed_total` or
/// `unflushed_requests_total{cause="flush_dropped_on_error"}`).
/// Non-zero only while `flush_once` is mid-flight. If the flusher task is
/// aborted mid-flush (e.g. shutdown timeout), this preserves the count of
/// lost records so `record_shutdown_drops` can report it.
in_flight_uncredited: AtomicU64,
/// Unix epoch (ms) of the last successful flush. Zero = no successful
/// flush has occurred yet. Read by the metrics sampler to compute
/// `seconds_since_successful_flush`. A hung flusher shows up here as a
/// monotonically increasing age while `flush_errors_total` stays at 0
/// (a hung flush never completes its success/failure accounting).
last_successful_flush_epoch_ms: AtomicU64,
/// Per-request-type record counters. Bumped on every `record()` call and
/// drained at flush time, so the hot path does one atomic increment
/// instead of an `inc()` call that allocates label strings every time.
record_count_decide: AtomicU64,
record_count_flag_definitions: AtomicU64,
record_count_flag_definitions_not_modified: AtomicU64,
shutdown_signal: Notify,
usage_reporter: Option<Arc<UsageReporter>>,
}
impl Inner {
fn new(
redis: Arc<dyn RedisClient + Send + Sync>,
config: BillingAggregatorConfig,
usage_reporter: Option<Arc<UsageReporter>>,
) -> Arc<Self> {
Arc::new(Self {
usage_reporter,
config,
redis,
pending: Mutex::new(HashMap::new()),
pending_total: AtomicU64::new(0),
in_flight_uncredited: AtomicU64::new(0),
last_successful_flush_epoch_ms: AtomicU64::new(0),
record_count_decide: AtomicU64::new(0),
record_count_flag_definitions: AtomicU64::new(0),
record_count_flag_definitions_not_modified: AtomicU64::new(0),
shutdown_signal: Notify::new(),
})
}
fn record_counter(&self, request_type: FlagRequestType) -> &AtomicU64 {
match request_type {
FlagRequestType::Decide => &self.record_count_decide,
FlagRequestType::FlagDefinitions => &self.record_count_flag_definitions,
FlagRequestType::FlagDefinitionsNotModified => {
&self.record_count_flag_definitions_not_modified
}
}
}
/// Sum any remaining entries in `pending` plus any drained-but-uncredited
/// counts from an interrupted flush, and emit them as shutdown drops.
fn record_shutdown_drops(&self) {
let remaining_pending = self.pending_total.load(Ordering::Relaxed);
// `Acquire` pairs with the `Release` stores on the per-chunk and
// trailing-chunk refresh sites in `flush_once`, plus the `AcqRel`
// swap at the end. Without this, on weak-memory architectures
// (ARM64) a shutdown abort firing between a per-chunk refresh and
// the swap could observe a stale `in_flight` and either miscount
// shutdown drops or double-credit them on top of the swap path.
let in_flight = self.in_flight_uncredited.load(Ordering::Acquire);
inc_unflushed(UnflushedCause::ShutdownDrop, remaining_pending + in_flight);
}
}
impl BillingAggregator {
/// Construct an aggregator and spawn its flusher task.
///
/// The flusher ticks on `flush_interval + random jitter` so a fleet-wide
/// deploy doesn't synchronize every pod's flush into the same Redis burst.
pub fn start(
redis: Arc<dyn RedisClient + Send + Sync>,
config: BillingAggregatorConfig,
) -> Arc<Self> {
Self::start_with_usage_reporter(redis, config, None)
}
/// As `start`, but also mirrors every credited flush into usage-ingestion.
pub fn start_with_usage_reporter(
redis: Arc<dyn RedisClient + Send + Sync>,
config: BillingAggregatorConfig,
usage_reporter: Option<Arc<UsageReporter>>,
) -> Arc<Self> {
// Fail fast on misconfiguration. A misconfigured FLAGS_BILLING_*
// env var would otherwise panic the flusher task silently, or silently
// disable billing — both worse than refusing to boot.
if let Err(e) = config.validate() {
panic!("invalid BillingAggregatorConfig: {e}");
}
let flush_interval_ms = config.flush_interval.as_millis() as u64;
let max_pending_entries = config.max_pending_entries;
let per_flush_batch_size = config.per_flush_batch_size;
let inner = Inner::new(redis, config, usage_reporter);
let flusher = tokio::spawn(run_flusher(inner.clone()));
let metrics_sampler = tokio::spawn(run_metrics_sampler(inner.clone()));
tracing::info!(
flush_interval_ms,
max_pending_entries,
per_flush_batch_size,
"BillingAggregator started"
);
Arc::new(Self {
inner,
flusher: Mutex::new(Some(flusher)),
metrics_sampler: Mutex::new(Some(metrics_sampler)),
})
}
/// Record a single billable request. Non-blocking; no Redis I/O.
///
/// **Lifecycle constraint:** must not be called after `shutdown()` has
/// returned. The flusher task is consumed by shutdown, so a post-shutdown
/// record lands in `pending` and is never flushed or accounted as a drop —
/// it is silently lost with no metric. In the normal server lifecycle
/// `serve()` calls `shutdown()` only after `axum::serve()` has fully
/// drained, which means every request handler that holds the aggregator
/// `Arc` has already returned. Background tasks or deferred work that
/// outlive `axum::serve()` and still hold the aggregator `Arc` would
/// violate this constraint — don't do that.
pub fn record(&self, team_id: i32, request_type: FlagRequestType, library: Option<Library>) {
// Time the body so the duration histogram can surface `pending` mutex
// contention. Empty label slice so the per-call emission doesn't
// allocate inside `apply_label_filter`.
let start = std::time::Instant::now();
// Bump the per-request-type atomic counter; the flusher emits the
// `FLAGS_BILLING_RECORDS` metric in batches at flush time so the
// hot path doesn't pay the per-call label clone in `inc()`.
self.inner
.record_counter(request_type)
.fetch_add(1, Ordering::Relaxed);
let key = AggregationKey {
team_id,
request_type,
library,
bucket: current_bucket(),
};
{
let mut pending = self.inner.pending.lock().unwrap();
if pending.len() >= self.inner.config.max_pending_entries && !pending.contains_key(&key)
{
// Cap hit on a new key: drop the incoming record rather than
// evicting an existing entry. Eviction would be O(n) under
// the hot-path mutex (a scan to find the oldest bucket). The
// cap is a tripwire, not a steady-state path —
// `flags_billing_unflushed_requests_total{cause="cap_drop"}`
// alerts on any non-zero rate — and the flusher drains the
// map every `flush_interval`, so the cap state is transient.
// Existing-key increments are always allowed (they don't
// grow the map).
inc_unflushed(UnflushedCause::CapDrop, 1);
} else {
*pending.entry(key).or_insert(0) += 1;
// Update the running total under the same lock so a
// concurrent reader can never observe `pending` and
// `pending_total` out of step.
self.inner.pending_total.fetch_add(1, Ordering::Relaxed);
}
}
// `as_nanos() / 1000.0` preserves sub-microsecond resolution that
// `as_micros()` would truncate, since uncontended record() is
// typically a few hundred nanoseconds.
let elapsed_us = start.elapsed().as_nanos() as f64 / 1000.0;
histogram(FLAGS_BILLING_RECORD_DURATION_US, &[], elapsed_us);
}
/// Perform a final flush and stop the flusher task.
///
/// Must be called after `axum::serve(...).await` resolves so all
/// in-flight requests have already recorded. Times out after
/// `shutdown_flush_timeout`; any remaining entries are counted in
/// `flags_billing_unflushed_requests_total{cause="shutdown_drop"}`.
///
/// Concurrent callers: only the first caller performs the flush; any
/// other caller racing into `shutdown()` returns immediately without
/// waiting for the first caller's flush to complete. If multiple tasks
/// need to await flush completion, wrap the call in a `tokio::sync::OnceCell`
/// (or similar) outer guard. Sequential calls are safe — the second is
/// a no-op. Call exactly once from the task that owns the server lifecycle.
pub async fn shutdown(&self) {
// Sampler is best-effort emission — `abort` is fine; a lost last
// tick doesn't matter operationally.
if let Some(handle) = self.metrics_sampler.lock().unwrap().take() {
handle.abort();
}
// `notify_one()` (not `notify_waiters()`): if the flusher happens to be
// mid-tick when we fire, the notify must be stored so the next
// `notified().await` returns immediately. `notify_waiters()` would be
// a no-op in that window.
self.inner.shutdown_signal.notify_one();
let handle = self.flusher.lock().unwrap().take();
if let Some(handle) = handle {
// Hold onto an abort handle: dropping a `JoinHandle` does NOT
// cancel the task, and we need to be sure a timed-out flusher
// can't keep running after the process-level shutdown has moved
// on (and can't later double-credit counts we just recorded as
// shutdown drops).
let abort_handle = handle.abort_handle();
let timeout = self.inner.config.shutdown_flush_timeout;
match tokio::time::timeout(timeout, handle).await {
Ok(Ok(())) => {
tracing::info!("BillingAggregator: shutdown flush completed");
}
Ok(Err(join_err)) => {
tracing::error!(error = %join_err, "BillingAggregator flusher panicked during shutdown");
self.inner.record_shutdown_drops();
}
Err(_) => {
tracing::warn!(
timeout_ms = timeout.as_millis() as u64,
"BillingAggregator: shutdown flush timed out"
);
abort_handle.abort();
self.inner.record_shutdown_drops();
}
}
}
// After the final flush, so the records it just queued are drained rather than
// dropped with the runtime.
if let Some(reporter) = &self.inner.usage_reporter {
reporter
.shutdown(self.inner.config.shutdown_flush_timeout)
.await;
}
}
#[cfg(test)]
pub fn pending_len(&self) -> usize {
self.inner.pending.lock().unwrap().len()
}
#[cfg(test)]
pub fn in_flight_uncredited(&self) -> u64 {
self.inner.in_flight_uncredited.load(Ordering::Relaxed)
}
#[cfg(test)]
pub fn pending_total(&self) -> u64 {
self.inner.pending_total.load(Ordering::Relaxed)
}
/// Tests sometimes need to set up `pending` directly (e.g. to inject
/// non-`current_bucket()` buckets, or to construct an empty-drain
/// scenario by clearing previously-recorded data). Going through this
/// helper keeps `pending_total` in step with `pending` — `record()`
/// maintains that invariant on the production path, and the metrics
/// sampler reads `pending_total` lock-free assuming it holds.
#[cfg(test)]
fn seed_pending(&self, entries: impl IntoIterator<Item = (AggregationKey, u64)>) {
let mut pending = self.inner.pending.lock().unwrap();
let mut total = self.inner.pending_total.load(Ordering::Relaxed);
for (key, count) in entries {
*pending.entry(key).or_insert(0) += count;
total += count;
}
self.inner.pending_total.store(total, Ordering::Relaxed);
}
/// Construct an aggregator without spawning the background flusher. For
/// test harnesses that don't exercise periodic flushing — `record()` still
/// fills the pending map, but counts only land in Redis if the test calls
/// `flush_once` directly. `shutdown()` is a no-op (no task to join), so
/// nothing leaks at runtime teardown. Not for production use.
#[doc(hidden)]
pub fn for_tests(
redis: Arc<dyn RedisClient + Send + Sync>,
config: BillingAggregatorConfig,
) -> Arc<Self> {
Arc::new(Self {
inner: Inner::new(redis, config, None),
flusher: Mutex::new(None),
metrics_sampler: Mutex::new(None),
})
}
}
/// Spawned task: tick → drain → flush, until the shutdown signal fires.
async fn run_flusher(inner: Arc<Inner>) {
// Initial jitter desynchronizes fleet-wide flushes after a coordinated
// deploy. Race it against shutdown so a quick SIGTERM doesn't wait for the
// jitter to elapse before flushing.
let jitter = inner
.config
.jitter_override
.unwrap_or_else(|| pick_jitter(inner.config.flush_interval));
tokio::select! {
_ = tokio::time::sleep(jitter) => {}
_ = inner.shutdown_signal.notified() => {
flush_once(&inner, FlushPolicy::BestEffort).await;
return;
}
}
let mut interval = tokio::time::interval(inner.config.flush_interval);
// `Delay` (not `Skip`): if a flush runs longer than the interval, the next
// tick still fires — buffered records keep accumulating and the cap
// tripwire handles unbounded growth. `Skip` would silently drop ticks
// during sustained slowness, leaving counts stranded longer than
// `flush_interval`.
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
// First tick fires immediately — swallow it so the first real flush
// happens one interval from now, after traffic has had a chance to
// accumulate.
interval.tick().await;
loop {
tokio::select! {
_ = interval.tick() => {
flush_once(&inner, FlushPolicy::BailOnError).await;
}
_ = inner.shutdown_signal.notified() => {
tracing::info!("BillingAggregator: shutdown signal received, performing final flush");
flush_once(&inner, FlushPolicy::BestEffort).await;
return;
}
}
}
}
/// Outcome of a single pipeline-chunk execution. `ErrBail` is how
/// `flush_chunk` signals to its caller that the current flush should stop
/// attempting further chunks (the caller is responsible for requeuing any
/// entries it hadn't yet handed off).
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
enum ChunkOutcome {
Ok,
Err,
/// `error_type` is `classify_redis_error()` for the failing chunk —
/// threaded out so `flush_once` can label the remainder-emit on
/// `FLAGS_BILLING_FLUSH_BLOCKED_REQUESTS` with the same error type as
/// the chunk that bailed.
ErrBail {
error_type: &'static str,
},
}
/// Execute one pipeline chunk and classify the result. Moves `chunk_entries`
/// into `requeue` on a `BailOnError` error so those entries retry next tick;
/// clears `chunk_entries` otherwise. `flushed_counts` is updated in place so
/// the caller can reconcile `in_flight_uncredited`. Dropped counts in
/// `BestEffort` mode are not tracked here — the residual in
/// `in_flight_uncredited` after the loop equals the total dropped, and the
/// caller credits it under `flush_dropped_on_error`.
///
/// Treats any `Ok(_)` from `execute_pipeline` as a fully-successful chunk —
/// per-command inner errors inside the returned vector are not inspected.
/// Safe today because every command in the pipeline is `HINCRBY` on a
/// pre-existing hash key, which cannot fail at the per-command level.
/// Adding non-`HINCRBY` operations to this pipeline requires revisiting this
/// assumption to avoid silent over-crediting of `entries_flushed_total`.
async fn flush_chunk(
inner: &Arc<Inner>,
commands: Vec<PipelineCommand>,
chunk_counts: u64,
chunk_entries: &mut Vec<(AggregationKey, u64)>,
policy: FlushPolicy,
flushed_counts: &mut u64,
requeue: &mut Vec<(AggregationKey, u64)>,
) -> ChunkOutcome {
match inner.redis.execute_pipeline(commands).await {
Ok(_) => {
*flushed_counts += chunk_counts;
// Credit per-chunk so a shutdown abort between chunks doesn't
// strand counts already written to Redis under an unincremented
// `entries_flushed_total`.
inc(FLAGS_BILLING_ENTRIES_FLUSHED, &[], chunk_counts);
report_usage(inner, chunk_entries);
chunk_entries.clear();
ChunkOutcome::Ok
}
Err(e) => {
let error_type = classify_redis_error(&e);
let outcome = classify_flush_outcome(&e);
record_chunk_error(&e, error_type, outcome, chunk_counts, policy);
// `Applied`: Redis processed the chunk's commands; the failure
// was on the response path. Credit the chunk as flushed and
// continue — retrying would double-credit the bucket. The
// chunk's records are gone from `in_flight_uncredited` via the
// bump to `flushed_counts`, the same path a successful chunk
// takes.
if outcome == FlushOutcome::Applied {
*flushed_counts += chunk_counts;
inc(FLAGS_BILLING_ENTRIES_FLUSHED, &[], chunk_counts);
report_usage(inner, chunk_entries);
chunk_entries.clear();
// Under `BailOnError` we still need to signal an error so
// the caller stops attempting subsequent chunks on a
// possibly-degraded connection. Under `BestEffort` the
// return is informational; the caller treats it the same
// as `Ok`. The staleness epoch is not affected by this
// return value — it's gated on `flushed_counts ==
// total_counts`, which holds for an Applied-only tick and
// stamps by design.
return match policy {
FlushPolicy::BailOnError => ChunkOutcome::ErrBail { error_type },
FlushPolicy::BestEffort => ChunkOutcome::Err,
};
}
// `Ambiguous` and `NotApplied`: today's behavior — under
// `BailOnError` the chunk's records are requeued for the next
// tick; under `BestEffort` they're terminally dropped. Keeping
// Ambiguous on the retry path matches the pre-classification
// policy: the under-count risk from dropping ambiguous chunks is
// much larger in practice than the over-count risk from
// retrying them (see the `outcome` label docs in
// `metrics::consts`). The `outcome` label is what lets us
// revisit that policy once reconciliation data lands.
match policy {
FlushPolicy::BailOnError => {
requeue.append(chunk_entries);
ChunkOutcome::ErrBail { error_type }
}
FlushPolicy::BestEffort => {
chunk_entries.clear();
ChunkOutcome::Err
}
}
}
}
}
fn report_usage(inner: &Arc<Inner>, chunk_entries: &[(AggregationKey, u64)]) {
if let Some(reporter) = &inner.usage_reporter {
reporter.report(chunk_entries, now_epoch_ms() as i64);
}
}
/// Swap the pending map, flush the drained batch to Redis, record metrics.
///
/// Chunks at `AggregationKey` boundaries (never splits a key's team-level
/// and SDK-level writes across chunks). Per-chunk success credits
/// `entries_flushed_total` so successful chunks stay credited even when a
/// later chunk fails. `in_flight_uncredited` tracks what's still unaccounted
/// for so a mid-flush abort (e.g. shutdown timeout) is still reportable via
/// shutdown drops.
///
/// On a chunk error: `BailOnError` (normal tick) stops attempting further
/// chunks and merges the failing chunk's entries plus the unattempted
/// remainder back into `pending` for the next tick, emitting
/// `flush_blocked_requests_total{error_type=…}` so the incident magnitude
/// is visible without conflating with terminal loss. `BestEffort`
/// (shutdown) keeps attempting and records unrecoverable losses under
/// `unflushed_requests_total{cause="flush_dropped_on_error"}` — the process
/// is exiting, there is no next tick.
async fn flush_once(inner: &Arc<Inner>, policy: FlushPolicy) {
// Drain the per-request-type record counters and emit
// FLAGS_BILLING_RECORDS. Doing this here instead of in `record()`
// keeps the hot path free of the per-call `apply_label_filter`
// allocation that `inc()` triggers.
for request_type in FlagRequestType::iter() {
let records = inner
.record_counter(request_type)
.swap(0, Ordering::Relaxed);
if records > 0 {
inc(
FLAGS_BILLING_RECORDS,
&record_labels_for(request_type),
records,
);
}
}
// Swap pending and zero `pending_total` under the same lock so a
// concurrent reader can never see pending empty with the atomic non-zero
// (or vice-versa). The atomic's value at swap time equals the sum of the
// drained map's values, since they're maintained together.
let (drained, total_counts): (HashMap<AggregationKey, u64>, u64) = {
let mut pending = inner.pending.lock().unwrap();
let drained = std::mem::take(&mut *pending);
let total = inner.pending_total.swap(0, Ordering::Relaxed);
(drained, total)
};
if drained.is_empty() {
// An empty drain is still evidence the flusher loop is alive. Stamp
// the epoch so an idle pod doesn't trip the stale-flush alarm — a
// low-traffic pod that flushed once at boot and then went quiet
// would otherwise show monotonically rising
// `seconds_since_successful_flush` while remaining perfectly healthy.
inner
.last_successful_flush_epoch_ms
.store(now_epoch_ms(), Ordering::Relaxed);
return;
}
let start = std::time::Instant::now();
// Publish in-flight *before* the first await so an abort anywhere in
// the flush still reports the correct residual. `total_counts` came
// from `pending_total.swap(0)` above, so it equals the sum of drained
// values without re-walking them.
inner
.in_flight_uncredited
.store(total_counts, Ordering::Relaxed);
let batch_size = inner.config.per_flush_batch_size.max(1);
// Each key produces 1-2 commands, so over-provision by 1 to avoid a
// realloc when the last key pushes the buffer past batch_size.
let mut buffer: Vec<PipelineCommand> = Vec::with_capacity(batch_size + 1);
let mut chunk_entries: Vec<(AggregationKey, u64)> = Vec::with_capacity(batch_size);
let mut buffer_counts: u64 = 0;
let mut flushed_counts: u64 = 0;
let mut requeue: Vec<(AggregationKey, u64)> = Vec::new();
let mut iter = drained.into_iter();
while let Some((key, count)) = iter.next() {
// Clamp to i64::MAX for the HINCRBY signed parameter. The pending
// map's u64 counts are bounded by `max_pending_entries * (records
// per key per flush interval)`, well below i64::MAX in practice.
let count_i64 = i64::try_from(count).unwrap_or(i64::MAX);
let field = key.bucket.to_string();
// Push the library command first so the `field` String can be moved
// into the team command without an extra clone.
if let Some(library) = key.library {
buffer.push(PipelineCommand::HIncrBy {
key: get_team_request_library_key(key.team_id, key.request_type, library),
field: field.clone(),
count: count_i64,
});
}
buffer.push(PipelineCommand::HIncrBy {
key: get_team_request_key(key.team_id, key.request_type),
field,
count: count_i64,
});
chunk_entries.push((key, count));
buffer_counts += count;
if buffer.len() >= batch_size {
let outcome = flush_chunk(
inner,
std::mem::take(&mut buffer),
buffer_counts,
&mut chunk_entries,
policy,
&mut flushed_counts,
&mut requeue,
)
.await;
buffer_counts = 0;
buffer.reserve(batch_size + 1);
// Always refresh `in_flight_uncredited` from `flushed_counts` so
// an `Applied`-classified chunk (which returns `Err`/`ErrBail`
// but still bumps `flushed_counts`) deducts its credit from the
// residual. For `NotApplied`/`Ambiguous` errors `flushed_counts`
// is unchanged, so this is a no-op and preserves the
// pre-classification behavior. Without this refresh an Applied
// chunk would be credited twice: once on
// `entries_flushed_total` and again on `flush_dropped_on_error`
// via the residual.
//
// `saturating_sub` + `debug_assert!` defends against a
// refactor that double-bumps `flushed_counts` for a single
// chunk: in release the value clamps to 0 (under-credit, the
// billing-safe direction) instead of wrapping to ~u64::MAX
// and being credited as `shutdown_drop`. `Release` pairs with
// `record_shutdown_drops`'s `Acquire` load so a concurrent
// shutdown abort observes the latest credit.
debug_assert!(
flushed_counts <= total_counts,
"flushed_counts={flushed_counts} exceeded total_counts={total_counts}"
);
inner.in_flight_uncredited.store(
total_counts.saturating_sub(flushed_counts),
Ordering::Release,
);
match outcome {
ChunkOutcome::Ok => {}
ChunkOutcome::Err => {}
ChunkOutcome::ErrBail { error_type } => {
// Drain the remainder of the iterator into requeue so
// unattempted entries retry on the next tick. Also emit
// `flush_blocked_requests_total` for the remainder so the
// rate reflects all requests blocked by the error, not
// just the chunk that hit Redis. These records aren't
// terminally lost (they'll retry next tick) — that's why
// the metric is `flush_blocked_*`, not `unflushed_*` —
// but the incident-magnitude signal would otherwise hide
// everything past the first failing chunk in bail mode.
let remainder: Vec<(AggregationKey, u64)> = iter.by_ref().collect();
let remainder_counts: u64 = remainder.iter().map(|(_, c)| *c).sum();
inc_flush_blocked(error_type, remainder_counts);
requeue.extend(remainder);
break;
}
}
}
}
// Trailing partial chunk. In bail mode we broke above with an empty
// buffer; in best-effort mode this may reveal a final error.
if !buffer.is_empty() {
let outcome = flush_chunk(
inner,
std::mem::take(&mut buffer),
buffer_counts,
&mut chunk_entries,
policy,
&mut flushed_counts,
&mut requeue,
)
.await;
// Same refresh as the per-chunk path: keep `in_flight_uncredited`
// in step with `flushed_counts` so an `Applied` trailing chunk
// deducts itself from the residual instead of being double-credited.
// See the per-chunk store above for ordering / underflow rationale.
debug_assert!(
flushed_counts <= total_counts,
"flushed_counts={flushed_counts} exceeded total_counts={total_counts}"
);
inner.in_flight_uncredited.store(
total_counts.saturating_sub(flushed_counts),
Ordering::Release,
);
match outcome {
ChunkOutcome::Ok => {}
ChunkOutcome::Err | ChunkOutcome::ErrBail { .. } => {}
}
}
let elapsed_ms = start.elapsed().as_millis() as f64;
histogram(FLAGS_BILLING_FLUSH_DURATION_MS, &[], elapsed_ms);
// Re-merge failed entries into `pending` so they retry next tick.
// Ordering: drop `in_flight_uncredited` *before* inserting into
// `pending`. A mid-requeue abort then under-counts (some entries in
// neither place) rather than double-counting (counted in both). Under-
// count is the preferable failure mode for billing.
//
// Lock discipline: the O(N) merge runs *outside* the pending lock.
// We swap out the fresh pending (small — only records added during
// the flush), merge locally, and swap back. This keeps the hot-path
// `record()` critical section unblocked under a Redis outage, when
// `requeue` can be up to `max_pending_entries` large.
//
// Rebucket-on-requeue: entries are reattributed to `current_bucket()`
// before being merged. Without this, a sustained Redis outage would
// accumulate one new `AggregationKey` per (team, request_type, library)
// every `CACHE_BUCKET_SIZE` (120s) as the wall clock advances —
// cardinality scales with outage minutes regardless of traffic, and the
// `max_pending_entries` cap eventually trips `cap_drop` on new keys.
// Collapsing forward bounds cardinality to (team × request_type ×
// library) and conserves the count, at the cost of per-bucket time
// attribution during the outage. Billing aggregates over months, so the
// attribution slip is invisible at invoice time; per-bucket quota
// enforcement during the outage itself is impossible anyway because
// the data isn't in Redis.
let requeued_counts: u64 = requeue.iter().map(|(_, c)| *c).sum();
if !requeue.is_empty() {
// Under the loop's accounting invariant `requeued_counts <=
// in_flight_uncredited`, so this never underflows. Use a saturating
// update anyway so the clamp-on-drift defense at the per-chunk
// store sites holds end-to-end: a refactor that pushed
// `requeued_counts` past the residual would otherwise wrap to
// ~u64::MAX here and be re-credited as `shutdown_drop`.
let _ =
inner
.in_flight_uncredited
.fetch_update(Ordering::Relaxed, Ordering::Relaxed, |v| {
Some(v.saturating_sub(requeued_counts))
});
let mut merged: HashMap<AggregationKey, u64> = {
let mut pending = inner.pending.lock().unwrap();
std::mem::take(&mut *pending)
};
let now_bucket = current_bucket();
for (mut key, count) in requeue {
key.bucket = now_bucket;
*merged.entry(key).or_insert(0) += count;
}
{
let mut pending = inner.pending.lock().unwrap();
// Reconcile any records that arrived during the merge.
for (key, count) in pending.drain() {
*merged.entry(key).or_insert(0) += count;
}
*pending = merged;
// The requeued counts weren't in `pending_total` (they came from
// `drained`, which already swapped to 0). Records that arrived
// during the flush are already in `pending_total` from their
// `record()` calls. Add the requeued sum back inside this lock so
// a sampler tick can never see `pending_total` lagging the map.
inner
.pending_total
.fetch_add(requeued_counts, Ordering::Relaxed);
}
}
// Zero `in_flight_uncredited` *before* crediting `flush_dropped_on_error`
// so a concurrent `record_shutdown_drops` cannot read both: a non-zero
// residual *and* the about-to-happen FlushDroppedOnError credit. Without
// this swap-before-credit ordering, a shutdown deadline firing between
// the credit and the zero would re-credit `dropped_counts` as
// `shutdown_drop`, double-counting in
// `flags_billing_unflushed_requests_total`.
//
// After the swap, exactly one of the two paths sees the residual:
// - this thread credits FlushDroppedOnError(dropped),
// - or `record_shutdown_drops` reads in_flight=0 and credits only
// remaining_pending under ShutdownDrop.
//
// `AcqRel` because this is a read-modify-write whose Release half pairs
// with any later observer of `in_flight_uncredited` — Relaxed would
// permit the credit to be reordered before the swap on weak-memory
// architectures, reintroducing the race the swap is meant to close.
let in_flight_residual = inner.in_flight_uncredited.swap(0, Ordering::AcqRel);
if in_flight_residual > 0 {
inc_unflushed(UnflushedCause::FlushDroppedOnError, in_flight_residual);
}
// Stamp the successful-flush timestamp when every drained record
// landed in Redis. That happens via `Ok` chunks (clean success) AND
// `Applied` chunks (Redis processed the commands but returned an
// unusable reply — the records are in Redis, we just credited them
// through the error path). Using `flushed_counts == total_counts`
// rather than `!any_error` so a tick where every chunk was Applied
// — billing is whole — doesn't keep `seconds_since_successful_flush`
// climbing as if the flusher were wedged. A BailOnError tick that
// requeued anything (`flushed_counts < total_counts`) or a
// BestEffort tick that dropped anything (`in_flight_residual > 0`)
// does NOT stamp — both are real degradation. The empty-drain branch
// earlier in `flush_once` stamps for a different reason: a tick with
// nothing to flush is still proof the loop is alive.
if flushed_counts == total_counts {
inner
.last_successful_flush_epoch_ms
.store(now_epoch_ms(), Ordering::Relaxed);
}
}
/// Emit `FLUSH_ERRORS` (1 per failed chunk, labeled by `error_type` and
/// `outcome`) and — when the chunk is going to be requeued (`BailOnError` +
/// non-`Applied`) — `flush_blocked_requests_total{error_type=…}` for the
/// chunk's aggregated request count. Also logs the raw error at warn level
/// (never on a metric label — unbounded cardinality risk).
///
/// `Applied` chunks are NOT emitted on `flush_blocked_requests_total` because
/// they were credited to `entries_flushed_total` rather than retried — there
/// is nothing "blocked" about them. Under `BestEffort` (shutdown) the
/// blocked metric is always suppressed because errors there either
/// terminally land in `flags_billing_unflushed_requests_total{cause="flush_dropped_on_error"}`
/// (`Ambiguous` and `NotApplied`) or were credited to
/// `entries_flushed_total` (`Applied`) — neither is "blocked, retry next
/// tick" semantics.
fn record_chunk_error(
e: &CustomRedisError,
error_type: &'static str,
outcome: FlushOutcome,
requests_in_chunk: u64,
policy: FlushPolicy,
) {
inc(
FLAGS_BILLING_FLUSH_ERRORS,
&[
("error_type".to_string(), error_type.to_string()),
("outcome".to_string(), outcome.as_str().to_string()),
],
1,
);
// Applied chunks are credited as flushed in the caller — they aren't
// blocked or retried, so emitting `flush_blocked_requests_total` for
// them would overstate incident magnitude.
let will_requeue = policy == FlushPolicy::BailOnError && outcome != FlushOutcome::Applied;
if will_requeue {
inc_flush_blocked(error_type, requests_in_chunk);
}
tracing::warn!(
error = %e,
error_type,
outcome = outcome.as_str(),
requests_in_chunk,
"BillingAggregator: flush pipeline failed"
);
}
pub(crate) fn classify_redis_error(err: &CustomRedisError) -> &'static str {
// Exhaustive on purpose: a new `CustomRedisError` variant should produce a
// compile error here so a new error type gets a deliberate
// `error_type` label rather than disappearing into "other".
match err {
CustomRedisError::Timeout => "timeout",
CustomRedisError::Redis(_) => "transport",
CustomRedisError::NotFound => "not_found",
CustomRedisError::ParseError(_) => "parse",
CustomRedisError::InvalidConfiguration(_) => "config",
}
}
/// Interval between live-gauge samples. Sized to the Prometheus scrape
/// cadence (15s default): faster sampling produces values nothing reads,
/// while still surfacing a wedged flusher within a couple of scrape
/// windows rather than waiting for the next flush interval. Each tick
/// briefly takes the `pending` mutex, which is the same lock `record()`
/// holds on the hot path, so cadence is a contention budget — not just a
/// CPU one.
const METRICS_SAMPLE_INTERVAL: Duration = Duration::from_secs(5);
fn now_epoch_ms() -> u64 {
std::time::SystemTime::now()
.duration_since(std::time::UNIX_EPOCH)
.map(|d| d.as_millis() as u64)
.unwrap_or(0)
}
/// Spawned task: periodically samples `pending.len()`, `in_flight_uncredited`,
/// and time-since-last-successful-flush, and emits them as gauges.
///
/// Runs independently of `run_flusher` so a hung `execute_pipeline` can't
/// freeze these gauges — the whole point of sampling outside the flush
/// path is to surface a wedged flusher. Exits via `abort()` at shutdown.
async fn run_metrics_sampler(inner: Arc<Inner>) {
let mut interval = tokio::time::interval(METRICS_SAMPLE_INTERVAL);
interval.set_missed_tick_behavior(tokio::time::MissedTickBehavior::Delay);
// Swallow the immediate first tick so startup doesn't emit zero-valued
// samples before the aggregator has seen any traffic.
interval.tick().await;
loop {
interval.tick().await;
sample_metrics(&inner);
}
}
fn sample_metrics(inner: &Arc<Inner>) {
let pending_len = inner.pending.lock().unwrap().len();
gauge(FLAGS_BILLING_PENDING_ENTRIES, &[], pending_len as f64);
// Lock-free read: `pending_total` is maintained under the same lock as
// `pending` mutations, so its value is a consistent snapshot from the
// sampler's perspective. Slight skew vs. `pending_len` is fine — both
// gauges are scrape-cadence approximations.
let pending_records = inner.pending_total.load(Ordering::Relaxed);
gauge(FLAGS_BILLING_PENDING_RECORDS, &[], pending_records as f64);
let last_ms = inner.last_successful_flush_epoch_ms.load(Ordering::Relaxed);
gauge(
FLAGS_BILLING_SECONDS_SINCE_SUCCESSFUL_FLUSH,
&[],
compute_staleness_seconds(last_ms, now_epoch_ms()),
);
}
/// Stale-flush gauge math, factored out so tests can pin the invariants
/// without observing global metric state. `last_ms == 0` means no successful
/// flush has happened yet — report 0 rather than "now - 0 = decades since
/// 1970", so a freshly booted pod with no traffic doesn't trip alerts.
/// `saturating_sub` clamps the backward-NTP-step case (`now < last`) to 0.
fn compute_staleness_seconds(last_ms: u64, now_ms: u64) -> f64 {
if last_ms == 0 {
0.0
} else {
now_ms.saturating_sub(last_ms) as f64 / 1000.0
}
}
fn pick_jitter(flush_interval: Duration) -> Duration {
// Up to 10% of the flush interval, capped at 1s — enough to desynchronize
// fleet-wide flushes without leaving records stranded for multiple
// seconds at startup.
const MAX_JITTER_MS: u64 = 1_000;
let max_jitter_ms = ((flush_interval.as_millis() / 10) as u64).clamp(1, MAX_JITTER_MS);
let jitter_ms = rand::thread_rng().gen_range(0..max_jitter_ms);
Duration::from_millis(jitter_ms)
}
#[cfg(test)]
mod tests {
use super::*;
use crate::billing::usage_test_support::{serve, RecordingIngestion};
use crate::config::TeamIdCollection;
use common_redis::{MockRedisClient, MockRedisValue};
use rstest::rstest;
fn test_config() -> BillingAggregatorConfig {
BillingAggregatorConfig {
flush_interval: Duration::from_millis(50),
max_pending_entries: 10_000,
per_flush_batch_size: 200,
shutdown_flush_timeout: Duration::from_secs(1),
jitter_override: None,
}
}
/// Build an aggregator without spawning the flusher, so tests can drive
/// flushes deterministically via `flush_once`.
fn new_test_aggregator(
config: BillingAggregatorConfig,
) -> (Arc<MockRedisClient>, Arc<BillingAggregator>) {
new_test_aggregator_with_redis(MockRedisClient::new(), config)
}
fn new_test_aggregator_with_redis(
redis: MockRedisClient,
config: BillingAggregatorConfig,
) -> (Arc<MockRedisClient>, Arc<BillingAggregator>) {
new_test_aggregator_with_reporter(redis, config, None)
}
fn new_test_aggregator_with_reporter(
redis: MockRedisClient,
config: BillingAggregatorConfig,
usage_reporter: Option<Arc<UsageReporter>>,
) -> (Arc<MockRedisClient>, Arc<BillingAggregator>) {
let redis = Arc::new(redis);
let agg = Arc::new(BillingAggregator {
inner: Inner::new(redis.clone(), config, usage_reporter),
flusher: Mutex::new(None),
metrics_sampler: Mutex::new(None),
});
(redis, agg)
}
fn hincrby_calls(redis: &MockRedisClient) -> Vec<(String, i64)> {
redis
.get_calls()
.into_iter()
.filter(|c| c.op == "pipeline_hincrby")
.filter_map(|c| match c.value {
MockRedisValue::I64(v) => Some((c.key, v)),
_ => None,
})
.collect()
}
/// Poll a condition every 1ms up to 1s. For tests that need to observe
/// a specific in-flight state of the flusher (e.g. "drain has happened
/// and Redis is awaiting"). Sleeping a fixed wall-clock duration races
/// scheduler latency on slow CI; polling on the deterministic signal
/// is what `test_shutdown_timeout_credits_residual_to_in_flight`
/// already does, factored out here so other race-window tests share
/// the same pattern.
async fn wait_until(mut cond: impl FnMut() -> bool) {
for _ in 0..1000 {
if cond() {
return;
}
tokio::time::sleep(Duration::from_millis(1)).await;
}
panic!("wait_until: condition not satisfied within 1s");
}
/// Paused-time counterpart of `wait_until`. `tokio::time::advance` wakes
/// timer-blocked tasks but does not poll them; each `yield_now` lets the
/// runtime schedule a ready task. `wait_until`'s 1ms sleep would itself
/// block forever under paused time. Bounded so a stuck condition fails
/// loudly rather than hanging the test.
async fn yield_until(mut cond: impl FnMut() -> bool) {
for _ in 0..1000 {
if cond() {
return;
}
tokio::task::yield_now().await;
}
panic!("yield_until: condition not satisfied after 1000 yields");
}
/// Record `n` Decide requests for teams `1..=n` with the PosthogJs library.
/// Each call generates one team-level + one library-level entry under the
/// same bucket. Used by the flush-policy tests to set up a known shape.
fn record_n_decide_with_library(agg: &Arc<BillingAggregator>, n: i32) {
for team_id in 1..=n {
agg.record(team_id, FlagRequestType::Decide, Some(Library::PosthogJs));
}
}
#[rstest]
// Repeated record on the same key collapses into one entry with the summed count.
#[case::same_key_aggregates(
&[(1, Some(Library::PosthogJs)), (1, Some(Library::PosthogJs)), (1, Some(Library::PosthogJs))],
&[((1, Some(Library::PosthogJs)), 3)],
)]
// Same team but distinct libraries (including no library) split into separate keys.
#[case::distinct_libraries_split(
&[(1, Some(Library::PosthogJs)), (1, Some(Library::PosthogNode)), (1, None)],
&[
((1, Some(Library::PosthogJs)), 1),
((1, Some(Library::PosthogNode)), 1),
((1, None), 1),
],
)]
fn test_record_aggregation(
#[case] records: &[(i32, Option<Library>)],
#[case] expected_counts: &[((i32, Option<Library>), u64)],
) {
let (_, agg) = new_test_aggregator(test_config());
for &(team_id, library) in records {
agg.record(team_id, FlagRequestType::Decide, library);
}
assert_eq!(
agg.pending_len(),
expected_counts.len(),
"pending_len must match number of expected distinct keys",
);
let bucket = current_bucket();
let pending = agg.inner.pending.lock().unwrap();
for &((team_id, library), expected) in expected_counts {
let key = AggregationKey {
team_id,
request_type: FlagRequestType::Decide,
library,
bucket,
};
assert_eq!(
pending.get(&key).copied(),
Some(expected),
"team {team_id} library {library:?} count mismatch",
);
}
}
/// `pending_total` is maintained alongside `pending` so the metrics
/// sampler can read it lock-free. If `record()`, the flush drain, and
/// the requeue path don't keep them in step, the `pending_counts` gauge
/// drifts away from reality. This test pins the invariant
/// `pending_total == sum(pending.values())` after each kind of
/// transition.
#[tokio::test]
async fn test_pending_total_matches_pending_values_sum() {
let (_, agg) = new_test_aggregator(test_config());
let assert_consistent = |agg: &Arc<BillingAggregator>, label: &str| {
let pending = agg.inner.pending.lock().unwrap();
let walked: u64 = pending.values().sum();
let atomic = agg.pending_total();
assert_eq!(
walked, atomic,
"{label}: pending_total ({atomic}) drifted from pending.values().sum() ({walked})"
);
};
// After records on distinct keys.
agg.record(1, FlagRequestType::Decide, None);
agg.record(2, FlagRequestType::Decide, None);
assert_consistent(&agg, "after distinct-key records");
// After repeated records on the same key.
agg.record(1, FlagRequestType::Decide, None);
agg.record(1, FlagRequestType::Decide, None);
assert_consistent(&agg, "after repeated-key records");
// After a successful flush.
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
assert_consistent(&agg, "after successful flush");
assert_eq!(agg.pending_total(), 0, "pending_total must zero on drain");
// After records following a flush.
agg.record(3, FlagRequestType::FlagDefinitions, None);
agg.record(3, FlagRequestType::FlagDefinitions, None);
assert_consistent(&agg, "after post-flush records");
}
/// `pending_total` must absorb requeued counts when a flush bails out.
/// Without this, a Redis outage would drain pending to "empty" while
/// records actually pile back into `pending` from the requeue path.
#[tokio::test]
async fn test_pending_total_absorbs_requeued_counts_on_bail() {
let mut mock = MockRedisClient::new();
mock.pipeline_error(CustomRedisError::Timeout); // every flush attempt fails
let redis: Arc<dyn RedisClient + Send + Sync> = Arc::new(mock);
let agg = BillingAggregator::for_tests(redis, test_config());
// Three records on distinct keys (so they don't collapse).
agg.record(1, FlagRequestType::Decide, None);
agg.record(2, FlagRequestType::Decide, None);
agg.record(3, FlagRequestType::Decide, None);
assert_eq!(agg.pending_total(), 3);
// BailOnError flush against an erroring mock: the failing chunk
// and unattempted remainder both requeue. After flush, pending
// should hold the same 3 records and pending_total should match.
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
let pending = agg.inner.pending.lock().unwrap();
let walked: u64 = pending.values().sum();
drop(pending);
assert_eq!(
walked,
agg.pending_total(),
"pending_total drifted from pending after bail+requeue"
);
assert_eq!(agg.pending_total(), 3, "all three records must requeue");
}
/// The reporter only ever sees a chunk Redis already credited, so this asserts the
/// mirror of a successful flush rather than the reporter's own behavior.
#[tokio::test]
async fn test_flush_once_mirrors_credited_counts_to_usage_ingestion() {
let service = RecordingIngestion::default();
let addr = serve(service.clone()).await;
let reporter = UsageReporter::new(&addr.to_string(), false, TeamIdCollection::All, 1_000)
.unwrap()
.unwrap();
let (_redis, agg) = new_test_aggregator_with_reporter(
MockRedisClient::new(),
test_config(),
Some(reporter.clone()),
);
agg.record(42, FlagRequestType::Decide, Some(Library::PosthogJs));
agg.record(42, FlagRequestType::Decide, Some(Library::PosthogJs));
agg.record(7, FlagRequestType::Decide, None);
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
reporter.shutdown(Duration::from_secs(5)).await;
let records: Vec<_> = service.requests().into_iter().flatten().collect();
// One record per aggregation entry, not per Redis write: an entry that carries a
// library writes both a team key and an SDK key, and billing counts the request once.
assert_eq!(records.len(), 2);
assert!(records
.iter()
.any(|record| record.team_id == 42 && record.quantity == 2));
assert!(records
.iter()
.any(|record| record.team_id == 7 && record.quantity == 1));
}
#[tokio::test]
async fn test_flush_once_writes_pipelined_hincrby() {
let (redis, agg) = new_test_aggregator(test_config());
agg.record(42, FlagRequestType::Decide, Some(Library::PosthogJs));
agg.record(42, FlagRequestType::Decide, Some(Library::PosthogJs));
agg.record(7, FlagRequestType::Decide, None);
let bucket = current_bucket();
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
let calls = hincrby_calls(&redis);
// 3 HIncrBys: team-key for team 42, sdk-key for team 42, team-key for
// team 7. Team 7 has no library, so no sdk-key.
assert_eq!(calls.len(), 3);
let expected_team_42_team = format!(
"{}:{bucket}",
get_team_request_key(42, FlagRequestType::Decide)
);
let expected_team_42_sdk = format!(
"{}:{bucket}",
get_team_request_library_key(42, FlagRequestType::Decide, Library::PosthogJs)
);
let expected_team_7_team = format!(
"{}:{bucket}",
get_team_request_key(7, FlagRequestType::Decide)
);
assert!(
calls.contains(&(expected_team_42_team.clone(), 2)),
"expected team 42 team key with count 2, got {:?}",
calls
);
assert!(
calls.contains(&(expected_team_42_sdk.clone(), 2)),
"expected team 42 sdk key with count 2, got {:?}",
calls
);
assert!(
calls.contains(&(expected_team_7_team.clone(), 1)),
"expected team 7 team key with count 1, got {:?}",
calls
);
}
#[tokio::test]
async fn test_flush_once_writes_flag_definitions_keys() {
let (redis, agg) = new_test_aggregator(test_config());
agg.record(
42,
FlagRequestType::FlagDefinitions,
Some(Library::PosthogJs),
);
agg.record(7, FlagRequestType::Decide, None);
agg.record(9, FlagRequestType::FlagDefinitionsNotModified, None);
let bucket = current_bucket();
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
let calls = hincrby_calls(&redis);
let expected_team_42_def = format!("posthog:local_evaluation_requests:42:{bucket}");
let expected_sdk_42_def =
format!("posthog:local_evaluation_requests:sdk:42:posthog-js:{bucket}");
let expected_team_7_decide = format!("posthog:decide_requests:7:{bucket}");
let expected_team_9_not_modified =
format!("posthog:local_evaluation_not_modified_requests:9:{bucket}");
assert_eq!(calls.len(), 4);
assert!(calls.contains(&(expected_team_42_def, 1)));
assert!(calls.contains(&(expected_sdk_42_def, 1)));
assert!(calls.contains(&(expected_team_7_decide, 1)));
assert!(calls.contains(&(expected_team_9_not_modified, 1)));
// Belt-and-braces: the aggregator must never write `:shadow`-suffixed
// keys. A regression here would mean we're writing into the
// reconciliation keyspace that has no reader.
for (key, _) in &calls {
assert!(
!key.contains(":shadow"),
"aggregator wrote a shadow-suffixed key: {key}"
);
}
}
#[tokio::test]
async fn test_flush_once_drains_pending() {
let (_, agg) = new_test_aggregator(test_config());
agg.record(1, FlagRequestType::Decide, None);
assert_eq!(agg.pending_len(), 1);
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
assert_eq!(agg.pending_len(), 0);
}
#[tokio::test]
async fn test_flush_chunks_commands_at_batch_size() {
// per_flush_batch_size=2, 5 keys × 2 commands each = 10 commands → 5 chunks.
let config = BillingAggregatorConfig {
per_flush_batch_size: 2,
..test_config()
};
let (redis, agg) = new_test_aggregator(config);
record_n_decide_with_library(&agg, 5);
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
let hincrby_count = hincrby_calls(&redis).len();
assert_eq!(
hincrby_count, 10,
"all commands should be flushed across chunks"
);
}
#[rstest]
// At cap, a new key (team 3) is dropped — the original two entries survive
// unchanged, and team 3 never appears in pending.
#[case::new_key_dropped_at_cap(3, &[(1, 1), (2, 1)])]
// At cap, an existing key (team 1) still increments — pending_len is
// unchanged but the count for team 1 grows.
#[case::existing_key_increments_at_cap(1, &[(1, 2), (2, 1)])]
fn test_cap_third_record(#[case] third_team: i32, #[case] expected_counts: &[(i32, u64)]) {
let config = BillingAggregatorConfig {
max_pending_entries: 2,
..test_config()
};
let (_, agg) = new_test_aggregator(config);
agg.record(1, FlagRequestType::Decide, None);
agg.record(2, FlagRequestType::Decide, None);
assert_eq!(agg.pending_len(), 2);
agg.record(third_team, FlagRequestType::Decide, None);
// Length-equals-expected implicitly verifies that no extra team
// (e.g. a capped-out team 3) snuck in.
assert_eq!(
agg.pending_len(),
expected_counts.len(),
"cap must hold pending_len at max_pending_entries",
);
let bucket = current_bucket();
let pending = agg.inner.pending.lock().unwrap();
for &(team_id, expected) in expected_counts {
let key = AggregationKey {
team_id,
request_type: FlagRequestType::Decide,
library: None,
bucket,
};
assert_eq!(
pending.get(&key).copied(),
Some(expected),
"team {team_id} count mismatch",
);
}
}
#[tokio::test(start_paused = true)]
async fn test_flusher_swallows_initial_tick_then_flushes_on_interval() {
// The only test exercising `run_flusher`'s timing path — every other
// test calls `flush_once` directly. Pins two invariants that the
// direct-flush tests can't reach:
//
// 1. The first immediate tick from `tokio::time::interval` is
// swallowed deliberately; no flush fires within the jitter
// window or at interval creation.
// 2. Subsequent ticks drive flushes on the configured cadence.
//
// Paused-time timing model. Jitter is pinned to exactly 100ms via
// `jitter_override` (production picks a random value in [0, 100ms]
// for flush_interval = 1s). We yield *before* advancing so the flusher
// parks on its `sleep(jitter)` at virtual t=0 — otherwise the
// sleep deadline would be set relative to whatever virtual time
// we'd already advanced to. Advancing past max_jitter wakes the
// sleep; after the flusher resumes (via yield), it creates its
// interval at virtual time ≈ 101ms, so the first *real* tick
// fires at ≈ 1101ms.
let flush_interval = Duration::from_secs(1);
let redis = Arc::new(MockRedisClient::new());
let agg = BillingAggregator::start(
redis.clone(),
BillingAggregatorConfig {
flush_interval,
// Pin jitter to 100ms so the interval is created at t=101ms
// and the first real tick fires at t=1101ms deterministically.
jitter_override: Some(Duration::from_millis(100)),
..test_config()
},
);
agg.record(1, FlagRequestType::Decide, None);
// Park the flusher on its jitter sleep before any advance.
tokio::task::yield_now().await;
// Advance past the pinned jitter (100ms) so the sleep wakes; the
// flusher then installs its interval and swallows the immediate
// first tick.
tokio::time::advance(Duration::from_millis(101)).await;
for _ in 0..16 {
tokio::task::yield_now().await;
}
assert_eq!(
hincrby_calls(&redis).len(),
0,
"the first immediate interval tick must be swallowed — a flush \
here would mean the flusher fired at t ≈ jitter, before any \
real tick had a chance to elapse"
);
// Just before the first real tick (at t=1101ms). We're at t=101ms,
// advance 998ms to t=1099ms < 1101ms.
tokio::time::advance(Duration::from_millis(998)).await;
for _ in 0..16 {
tokio::task::yield_now().await;
}
assert_eq!(
hincrby_calls(&redis).len(),
0,
"no flush before flush_interval elapses from interval creation"
);
// Cross the first real tick.
tokio::time::advance(Duration::from_millis(10)).await;
yield_until(|| !hincrby_calls(&redis).is_empty()).await;
assert_eq!(
hincrby_calls(&redis).len(),
1,
"first real interval tick should drive exactly one flush"
);
// Pin the cadence: a second record + another full interval should
// produce a second flush. Catches a regression that fired once and
// then stopped, or lost its interval state between ticks.
agg.record(2, FlagRequestType::Decide, None);
tokio::time::advance(flush_interval).await;
yield_until(|| hincrby_calls(&redis).len() >= 2).await;
assert_eq!(
hincrby_calls(&redis).len(),
2,
"a second flush should fire on the next interval tick"
);
}
#[tokio::test]
async fn test_shutdown_flushes_then_stops() {
let redis = Arc::new(MockRedisClient::new());
let agg = BillingAggregator::start(
redis.clone(),
BillingAggregatorConfig {
// Slow tick so shutdown has to drive the flush itself.
flush_interval: Duration::from_secs(60),
..test_config()
},
);
agg.record(1, FlagRequestType::Decide, None);
agg.record(1, FlagRequestType::Decide, None);
let bucket = current_bucket();
agg.shutdown().await;
let calls = hincrby_calls(&redis);
assert_eq!(calls.len(), 1, "shutdown should trigger one HIncrBy");
let expected_key = format!(
"{}:{bucket}",
get_team_request_key(1, FlagRequestType::Decide)
);
assert_eq!(calls[0], (expected_key, 2));
}
#[tokio::test]
async fn test_shutdown_is_idempotent_sequentially() {
let redis = Arc::new(MockRedisClient::new());
let agg = BillingAggregator::start(
redis.clone(),
BillingAggregatorConfig {
flush_interval: Duration::from_secs(60),
..test_config()
},
);
agg.record(1, FlagRequestType::Decide, None);
// First call performs the flush. Second call must not panic and
// must not double-flush.
agg.shutdown().await;
let calls_after_first = hincrby_calls(&redis).len();
agg.shutdown().await;
let calls_after_second = hincrby_calls(&redis).len();
assert_eq!(calls_after_first, calls_after_second);
assert_eq!(calls_after_first, 1);
}
#[tokio::test]
async fn test_shutdown_timeout_does_not_hang() {
// Redis pipeline blocks longer than `shutdown_flush_timeout`. Shutdown
// must return within roughly the timeout, not the block duration.
let mut mock = MockRedisClient::new();
mock.pipeline_block(Duration::from_secs(5));
let redis = Arc::new(mock);
let agg = BillingAggregator::start(
redis.clone(),
BillingAggregatorConfig {
flush_interval: Duration::from_secs(60),
shutdown_flush_timeout: Duration::from_millis(100),
..test_config()
},
);
agg.record(1, FlagRequestType::Decide, None);
let start = std::time::Instant::now();
agg.shutdown().await;
let elapsed = start.elapsed();
assert!(
elapsed < Duration::from_secs(2),
"shutdown should honour timeout, took {:?}",
elapsed
);
}
#[tokio::test]
async fn test_flush_bail_mode_breaks_on_first_chunk_error() {
// 5 keys with libraries = 10 commands. batch_size=2 → 5 chunks.
// With an always-fail mock, the first chunk must be attempted and the
// rest must NOT be attempted (bail-on-first-error — used by the
// flusher's normal tick path).
let mut mock = MockRedisClient::new();
mock.pipeline_error(CustomRedisError::Timeout);
let config = BillingAggregatorConfig {
per_flush_batch_size: 2,
..test_config()
};
let (redis, agg) = new_test_aggregator_with_redis(mock, config);
record_n_decide_with_library(&agg, 5);
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
// Mock records commands before returning the connection error, so
// exactly the failing first chunk's 2 commands appear in calls.
let attempted = hincrby_calls(&redis).len();
assert_eq!(
attempted, 2,
"only the first failing chunk should be attempted"
);
}
#[tokio::test]
async fn test_flush_bail_mode_requeues_failed_and_unattempted_entries() {
// 5 keys × 2 cmds = 10 cmds at batch_size=2 → 5 chunks. Always-fail
// mock: first chunk fails, remaining chunks are unattempted. Under
// BailOnError, every drained entry must be re-queued into `pending`
// so a later tick can retry — nothing should be silently dropped.
//
// Each team gets a distinct count (team T → T records) so the
// assertion checks per-key identity, not just totals. A bug that
// requeued the wrong 5 entries (e.g., duplicates of one chunk
// rather than the actual leftover iterator) would otherwise pass.
let mut mock = MockRedisClient::new();
mock.pipeline_error(CustomRedisError::Timeout);
let config = BillingAggregatorConfig {
per_flush_batch_size: 2,
..test_config()
};
let (_, agg) = new_test_aggregator_with_redis(mock, config);
for team_id in 1..=5 {
for _ in 0..team_id {
agg.record(team_id, FlagRequestType::Decide, Some(Library::PosthogJs));
}
}
assert_eq!(agg.pending_len(), 5);
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
assert_eq!(
agg.pending_len(),
5,
"all failed+unattempted entries must be re-queued"
);
let pending = agg.inner.pending.lock().unwrap();
let total: u64 = pending.values().sum();
assert_eq!(
total,
1 + 2 + 3 + 4 + 5,
"requeued counts must be preserved"
);
for (key, &count) in pending.iter() {
assert_eq!(
count, key.team_id as u64,
"team {} requeued with wrong count {count}",
key.team_id,
);
}
// in-flight is cleared — requeued counts live in `pending`, not in
// `in_flight_uncredited`, so shutdown drops can't double-count them.
assert_eq!(agg.inner.in_flight_uncredited.load(Ordering::Relaxed), 0);
}
#[tokio::test]
async fn test_flush_bail_mode_rebuckets_requeued_entries_to_current_bucket() {
// Multiple stale-bucket entries for the same (team, request_type,
// library) collapse into a single current-bucket entry on requeue.
// Without rebucket-on-requeue, a sustained Redis outage would
// accumulate one new key per CACHE_BUCKET_SIZE rollover, blowing
// through `max_pending_entries` purely from the wall clock advancing.
// After requeue, the count is conserved (3 + 5 + 7 = 15) and
// cardinality is bounded by (team × request_type × library).
let mut mock = MockRedisClient::new();
mock.pipeline_error(CustomRedisError::Timeout);
let (_, agg) = new_test_aggregator_with_redis(mock, test_config());
// Seed three distinct old buckets for the same tuple. Pick buckets
// far below `current_bucket()` so the test doesn't race a real
// bucket boundary.
let old_buckets = [10u64, 11, 12];
let counts = [3u64, 5, 7];
agg.seed_pending(old_buckets.iter().zip(counts.iter()).map(|(&b, &c)| {
(
AggregationKey {
team_id: 1,
request_type: FlagRequestType::Decide,
library: Some(Library::PosthogJs),
bucket: b,
},
c,
)
}));
assert_eq!(agg.pending_len(), 3, "seeded three distinct-bucket keys");
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
let pending = agg.inner.pending.lock().unwrap();
assert_eq!(
pending.len(),
1,
"rebucket must collapse stale-bucket entries for the same tuple into one"
);
let (key, &count) = pending.iter().next().unwrap();
assert_eq!(count, 3 + 5 + 7, "counts must be conserved across rebucket");
assert_eq!(
key.bucket,
current_bucket(),
"rebucketed key must carry the current bucket, not any stale bucket"
);
assert!(
!old_buckets.contains(&key.bucket),
"rebucketed key must not retain any stale bucket value"
);
}
#[tokio::test]
async fn test_flush_bail_mode_requeue_merges_with_concurrent_records() {
// A record() arriving during an in-flight flush lands in the fresh
// `pending` map the flusher just drained from. If that flush then
// errors and requeues, the requeued entries must be *merged* into
// the fresh map (not overwrite concurrent records, not lose them).
let mut mock = MockRedisClient::new();
mock.pipeline_block(Duration::from_millis(50));
mock.pipeline_error(CustomRedisError::Timeout);
let (_, agg) = new_test_aggregator_with_redis(mock, test_config());
// One pre-flush record for team 1 — will be drained and requeued.
agg.record(1, FlagRequestType::Decide, None);
let agg_for_flush = agg.clone();
let flush = tokio::spawn(async move {
flush_once(&agg_for_flush.inner, FlushPolicy::BailOnError).await;
});
// Race a concurrent record for team 1 into the post-drain map
// while the flusher is parked on the Redis error. Poll on
// `in_flight_uncredited > 0` — the flusher writes that *before*
// its first await, so observing it non-zero proves pending was
// drained and the flusher is now parked on Redis. Sleeping a
// fixed duration would race scheduler latency on slow CI.
wait_until(|| agg.in_flight_uncredited() > 0).await;
agg.record(1, FlagRequestType::Decide, None);
flush.await.unwrap();
// Both counts survive: the requeued pre-flush record + the racing
// record merged under the same AggregationKey = 2.
let pending = agg.inner.pending.lock().unwrap();
let total: u64 = pending.values().sum();
assert_eq!(total, 2, "requeue must merge, not overwrite or drop");
}
#[tokio::test]
async fn test_flush_bail_mode_recovery_flushes_requeued() {
// Fail only the first pipeline call. The first flush requeues its
// drained entries; the second flush should find them in `pending`
// and land them in Redis.
let mut mock = MockRedisClient::new();
mock.pipeline_error_at_call(0, CustomRedisError::Timeout);
let (redis, agg) = new_test_aggregator_with_redis(mock, test_config());
agg.record(1, FlagRequestType::Decide, None);
agg.record(2, FlagRequestType::Decide, None);
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
// First flush errored; entries should be waiting in `pending`.
assert_eq!(agg.pending_len(), 2, "first flush should requeue on error");
let calls_after_first = hincrby_calls(&redis);
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
assert_eq!(agg.pending_len(), 0, "second flush should drain pending");
// The mock records every attempted command even on errors, so we
// must look at the calls added by the second flush only. That's
// what actually reached Redis.
let all_calls = hincrby_calls(&redis);
let second_flush_calls: Vec<_> = all_calls[calls_after_first.len()..].to_vec();
let total: i64 = second_flush_calls.iter().map(|(_, v)| *v).sum();
assert_eq!(
total, 2,
"total recorded requests must reach Redis after recovery, got {second_flush_calls:?}"
);
}
#[tokio::test]
async fn test_flush_bail_mode_middle_chunk_failure_credits_successful_and_requeues_rest() {
// 3 keys × 2 cmds = 6 cmds at batch_size=2 → 3 chunks. Fail only
// chunk 2 (call index 1). Chunk 1's HINCRBYs landed in Redis and
// its counts must NOT be requeued (would double-credit next tick).
// Chunks 2 and 3+ go into `requeue` under BailOnError.
let mut mock = MockRedisClient::new();
mock.pipeline_error_at_call(1, CustomRedisError::Timeout);
let config = BillingAggregatorConfig {
per_flush_batch_size: 2,
..test_config()
};
let (_, agg) = new_test_aggregator_with_redis(mock, config);
record_n_decide_with_library(&agg, 3);
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
// Chunk 1 succeeded; its key is gone from `pending`. Chunks 2 & 3
// are requeued = 2 keys. A regression that also requeued chunk 1's
// already-credited entries would land 3 here.
assert_eq!(
agg.pending_len(),
2,
"successful chunk's entries must NOT be requeued",
);
let pending_total: u64 = agg.inner.pending.lock().unwrap().values().sum();
assert_eq!(pending_total, 2);
assert_eq!(agg.inner.in_flight_uncredited.load(Ordering::Relaxed), 0);
}
/// `FlushOutcome::Applied` chunks (e.g. ParseError on the response) must
/// be credited to `entries_flushed_total` and NOT requeued — Redis
/// already processed the HINCRBYs, so a retry would double-count. Since
/// every record landed in Redis (just via the error path), the flush
/// is treated as successful for staleness purposes and stamps
/// `last_successful_flush_epoch_ms` — otherwise a benign ParseError
/// storm would page on-call even though billing is whole.
#[tokio::test]
async fn test_flush_bail_applied_error_credits_chunk_without_requeue() {
let mut mock = MockRedisClient::new();
mock.pipeline_error(CustomRedisError::ParseError("bad reply".into()));
let (_, agg) = new_test_aggregator_with_redis(mock, test_config());
agg.record(1, FlagRequestType::Decide, None);
agg.record(2, FlagRequestType::Decide, None);
assert_eq!(agg.pending_total(), 2);
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
// Pending is empty (chunk was credited, not requeued) and the
// residual `in_flight_uncredited` is zero (the credit-bump deducted
// it instead of leaving it for the epilogue to bill against
// `flush_dropped_on_error`).
assert_eq!(
agg.pending_len(),
0,
"Applied chunks must not be requeued — retry would double-count"
);
assert_eq!(agg.pending_total(), 0);
assert_eq!(agg.inner.in_flight_uncredited.load(Ordering::Relaxed), 0);
assert!(
agg.inner
.last_successful_flush_epoch_ms
.load(Ordering::Relaxed)
> 0,
"Applied chunks landed in Redis, so the staleness epoch must stamp",
);
}
/// Counterpart for the shutdown path: an Applied error during a
/// `BestEffort` flush must also credit the chunk (Redis applied it) and
/// not double-bill it under `shutdown_drop` via the residual.
#[tokio::test]
async fn test_flush_best_effort_applied_error_credits_chunk_without_residual() {
let mut mock = MockRedisClient::new();
mock.pipeline_error(CustomRedisError::ParseError("bad reply".into()));
let (_, agg) = new_test_aggregator_with_redis(mock, test_config());
agg.record(1, FlagRequestType::Decide, None);
agg.record(2, FlagRequestType::Decide, None);
flush_once(&agg.inner, FlushPolicy::BestEffort).await;
// No residual to be credited as `flush_dropped_on_error`; the chunk
// is fully accounted for via `entries_flushed_total`.
assert_eq!(agg.inner.in_flight_uncredited.load(Ordering::Relaxed), 0);
assert_eq!(agg.pending_len(), 0);
assert!(
agg.inner
.last_successful_flush_epoch_ms
.load(Ordering::Relaxed)
> 0,
"every record landed via Applied, so the staleness epoch must stamp",
);
}
#[tokio::test]
async fn test_flush_best_effort_mode_attempts_all_chunks_despite_errors() {
// Same setup as the bail-mode test but with FlushPolicy::BestEffort.
// Every chunk fails, yet every chunk must still be attempted —
// shutdown's last-chance semantics.
let mut mock = MockRedisClient::new();
mock.pipeline_error(CustomRedisError::Timeout);
let config = BillingAggregatorConfig {
per_flush_batch_size: 2,
..test_config()
};
let (redis, agg) = new_test_aggregator_with_redis(mock, config);
record_n_decide_with_library(&agg, 5);
flush_once(&agg.inner, FlushPolicy::BestEffort).await;
let attempted = hincrby_calls(&redis).len();
assert_eq!(
attempted, 10,
"all chunks should be attempted in best-effort mode"
);
// Accounting closed out: in-flight cleared.
assert_eq!(agg.inner.in_flight_uncredited.load(Ordering::Relaxed), 0);
}
#[tokio::test]
async fn test_flush_best_effort_credits_successful_chunks_around_error() {
// Fail only the second pipeline call. Surrounding chunks should land
// and credit their counts (exercises the partial-success path that's
// the whole point of best-effort mode).
let mut mock = MockRedisClient::new();
mock.pipeline_error_at_call(1, CustomRedisError::Timeout);
let config = BillingAggregatorConfig {
per_flush_batch_size: 2,
..test_config()
};
let (redis, agg) = new_test_aggregator_with_redis(mock, config);
record_n_decide_with_library(&agg, 5);
flush_once(&agg.inner, FlushPolicy::BestEffort).await;
// All 5 chunks were attempted (the mock records commands even on
// injected errors), proving we continued past the failing chunk.
let attempted = hincrby_calls(&redis).len();
assert_eq!(attempted, 10);
// Accounting closed out.
assert_eq!(agg.inner.in_flight_uncredited.load(Ordering::Relaxed), 0);
}
#[tokio::test]
async fn test_flush_best_effort_flushes_trailing_partial_after_error() {
// Best-effort mode attempts the trailing partial chunk even if earlier
// chunks errored. 3 keys w/ libraries (6 cmds) + 1 key w/o library
// (1 cmd) = 7 cmds at batch_size=3 → 2 full chunks + 1 trailing
// 1-cmd chunk. Fail only the first pipeline call; assert the
// trailing partial still got attempted.
let mut mock = MockRedisClient::new();
mock.pipeline_error_at_call(0, CustomRedisError::Timeout);
let config = BillingAggregatorConfig {
per_flush_batch_size: 3,
..test_config()
};
let (redis, agg) = new_test_aggregator_with_redis(mock, config);
for team_id in 1..=3 {
agg.record(team_id, FlagRequestType::Decide, Some(Library::PosthogJs));
}
agg.record(99, FlagRequestType::Decide, None);
flush_once(&agg.inner, FlushPolicy::BestEffort).await;
// All 7 commands attempted despite the first chunk's error — proves
// both that we continued past the error AND that the trailing partial
// still ran.
let attempted = hincrby_calls(&redis).len();
assert_eq!(attempted, 7);
}
#[tokio::test]
async fn test_shutdown_uses_best_effort_flush() {
// When shutdown drives the flush, all chunks must be attempted even
// if earlier ones error. With bail-on-error, only chunk 1 would land.
let mut mock = MockRedisClient::new();
mock.pipeline_error(CustomRedisError::Timeout);
let redis = Arc::new(mock);
let agg = BillingAggregator::start(
redis.clone(),
BillingAggregatorConfig {
// Long interval so shutdown drives the flush, not a tick.
flush_interval: Duration::from_secs(60),
per_flush_batch_size: 2,
..test_config()
},
);
record_n_decide_with_library(&agg, 5);
agg.shutdown().await;
let attempted = hincrby_calls(&redis).len();
assert_eq!(
attempted, 10,
"shutdown should attempt all chunks in best-effort mode"
);
}
#[tokio::test]
async fn test_flush_clears_in_flight_uncredited_on_success() {
let (_, agg) = new_test_aggregator(test_config());
agg.record(1, FlagRequestType::Decide, None);
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
assert_eq!(agg.inner.in_flight_uncredited.load(Ordering::Relaxed), 0);
}
#[tokio::test]
async fn test_flush_clears_in_flight_uncredited_on_error() {
let mut mock = MockRedisClient::new();
mock.pipeline_error(CustomRedisError::Timeout);
let (_, agg) = new_test_aggregator_with_redis(mock, test_config());
agg.record(1, FlagRequestType::Decide, None);
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
// On completed (even errored) flush, in-flight must be cleared so a
// later shutdown drop doesn't double-count this batch.
assert_eq!(agg.inner.in_flight_uncredited.load(Ordering::Relaxed), 0);
}
#[tokio::test]
async fn test_empty_flush_does_no_work() {
let (redis, agg) = new_test_aggregator(test_config());
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
let calls = redis.get_calls();
assert!(
calls.is_empty(),
"flushing an empty map should make zero calls"
);
}
#[tokio::test]
async fn test_flush_preserves_distinct_buckets_per_key() {
// Two records for the same (team, request_type, library) but in
// different buckets must produce two separate HINCRBYs (distinct
// `field` values), preserving late-flushed records in their original
// bucket. Inject the keys directly so the test isn't dependent on
// wall-clock crossing a bucket boundary.
let (redis, agg) = new_test_aggregator(test_config());
agg.seed_pending([
(
AggregationKey {
team_id: 1,
request_type: FlagRequestType::Decide,
library: None,
bucket: 100,
},
3,
),
(
AggregationKey {
team_id: 1,
request_type: FlagRequestType::Decide,
library: None,
bucket: 101,
},
5,
),
]);
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
// The mock records calls as "{key}:{field}", so a per-bucket HINCRBY
// shows up under "{team_key}:{bucket}".
let team_key = get_team_request_key(1, FlagRequestType::Decide);
let calls = hincrby_calls(&redis);
let bucket_100_key = format!("{team_key}:100");
let bucket_101_key = format!("{team_key}:101");
assert!(
calls.iter().any(|(k, v)| k == &bucket_100_key && *v == 3),
"bucket 100 must produce HINCRBY count=3, got {:?}",
calls
);
assert!(
calls.iter().any(|(k, v)| k == &bucket_101_key && *v == 5),
"bucket 101 must produce HINCRBY count=5, got {:?}",
calls
);
}
#[tokio::test]
async fn test_flush_conserves_total_count() {
// Conservation invariant: the sum of HINCRBY values written to Redis
// equals the total number of recorded requests. Catches drift in
// the per-chunk accumulators and the in_flight_uncredited bookkeeping.
let (redis, agg) = new_test_aggregator(test_config());
let mut expected_total: i64 = 0;
for team_id in 1..=10 {
for _ in 0..(team_id as usize) {
agg.record(team_id, FlagRequestType::Decide, None);
expected_total += 1;
}
}
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
// No library = one HINCRBY per key, so summing values gives the total.
let actual_total: i64 = hincrby_calls(&redis).iter().map(|(_, v)| *v).sum();
assert_eq!(
actual_total, expected_total,
"sum of HINCRBYs must equal recorded request count"
);
assert_eq!(agg.pending_len(), 0, "pending must be drained on success");
assert_eq!(
agg.in_flight_uncredited(),
0,
"in-flight must be cleared on success"
);
}
#[tokio::test]
async fn test_shutdown_timeout_credits_residual_to_in_flight() {
// When the flush hangs and the shutdown timeout fires, the recorded
// count must be fully accounted for: pending is drained into the
// flush-local map (so pending_len == 0), and the count is parked in
// `in_flight_uncredited` so record_shutdown_drops can credit it as a
// drop. This is the invariant that
// `flags_billing_unflushed_requests_total{cause="shutdown_drop"}` depends on.
let mut mock = MockRedisClient::new();
mock.pipeline_block(Duration::from_secs(60)); // outlasts the timeout
let redis: Arc<dyn RedisClient + Send + Sync> = Arc::new(mock);
let config = BillingAggregatorConfig {
flush_interval: Duration::from_millis(20),
shutdown_flush_timeout: Duration::from_millis(50),
..test_config()
};
let agg = BillingAggregator::start(redis, config);
for _ in 0..7 {
agg.record(1, FlagRequestType::Decide, None);
}
// Wait for the flusher to drain `pending` and park the count in
// `in_flight_uncredited` (which `flush_once` writes before its first
// `.await`). Polling on `in_flight_uncredited` rather than sleeping
// a fixed duration keeps the test deterministic on slow CI.
for _ in 0..200 {
if agg.in_flight_uncredited() == 7 {
break;
}
tokio::time::sleep(Duration::from_millis(5)).await;
}
assert_eq!(
agg.pending_len(),
0,
"pending must be drained into the in-flight flush before shutdown"
);
assert_eq!(
agg.in_flight_uncredited(),
7,
"drained count must be parked in in_flight_uncredited so the timeout path can credit it as a drop"
);
agg.shutdown().await;
// After shutdown returns, all bookkeeping must close out: the
// residual count was credited via `record_shutdown_drops`, and both
// accumulators are zero. Without this assertion, a regression that
// skipped `record_shutdown_drops` on the timeout path would still
// pass — the residual would silently leak with no signal.
assert_eq!(
agg.pending_len(),
0,
"pending must remain empty after shutdown"
);
assert_eq!(
agg.pending_total(),
0,
"pending_total must remain zero after shutdown"
);
assert_eq!(
agg.in_flight_uncredited(),
7,
"in_flight_uncredited stays at the residual after shutdown — \
record_shutdown_drops reads it but does not zero it. Any future \
refactor that resets this on shutdown must also retain a way to \
prove the drops were emitted; otherwise this test pin breaks."
);
}
#[tokio::test]
async fn test_record_during_in_flight_flush_lands_in_next_batch() {
// The flusher releases the `pending` lock right after `mem::take`, so
// a `record()` arriving while Redis is still in flight must land in
// a fresh empty map and wait for the next flush. Regression guard
// for any future change that inadvertently holds the lock across
// `.await` — that would block `record()` for the duration of the
// Redis round-trip on the request hot path.
let mut mock = MockRedisClient::new();
mock.pipeline_block(Duration::from_millis(100));
let (redis, agg) = new_test_aggregator_with_redis(mock, test_config());
agg.record(1, FlagRequestType::Decide, None);
// Spawn the flush so we can race a `record()` against the in-flight
// pipeline. Use a separate `Arc` clone for the task.
let agg_for_flush = agg.clone();
let flush = tokio::spawn(async move {
flush_once(&agg_for_flush.inner, FlushPolicy::BailOnError).await;
});
// Wait until the flusher has drained `pending` and is parked on the
// Redis await — `in_flight_uncredited` is set just before the first
// `.await`, so observing it non-zero is the deterministic proof.
// Polling beats sleeping a fixed duration: the latter would race
// scheduler latency on slow CI.
wait_until(|| agg.in_flight_uncredited() > 0).await;
// The hot-path contract is that `record()` returns in microseconds
// even while a flush is in flight. If a future change holds the
// `pending` lock across the await, this call would block for the
// remainder of the 100ms `pipeline_block` window. 25ms is generous
// for a HashMap insert (~hundreds of ns) and tight enough to fail
// hard if the lock is held.
let record_start = std::time::Instant::now();
agg.record(2, FlagRequestType::Decide, None);
let record_elapsed = record_start.elapsed();
assert!(
record_elapsed < Duration::from_millis(25),
"record() blocked for {record_elapsed:?} while a flush was in flight — \
a regression that holds the pending lock across .await would block \
the hot path for the duration of the Redis round-trip"
);
flush.await.unwrap();
assert_eq!(
agg.pending_len(),
1,
"record(2) must land in the post-drain map and wait for next flush"
);
let calls = hincrby_calls(&redis);
assert_eq!(
calls.len(),
1,
"only the in-flight chunk (record(1)) should have flushed, got {calls:?}"
);
assert_eq!(calls[0].1, 1, "the flushed HINCRBY count must be 1");
}
#[tokio::test]
async fn test_empty_drain_stamps_last_successful_epoch() {
// An idle pod runs the flusher tick with nothing to drain. That is
// still evidence the loop is alive, so the staleness gauge must
// reset — otherwise a low-traffic pod that flushed at boot would
// show monotonically rising `seconds_since_successful_flush` and
// trip the wedged-flusher alarm despite being healthy.
let (_, agg) = new_test_aggregator(test_config());
let before = now_epoch_ms();
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
let after = now_epoch_ms();
let stamped = agg
.inner
.last_successful_flush_epoch_ms
.load(Ordering::Relaxed);
assert!(
(before..=after).contains(&stamped),
"empty drain stamp {stamped} must be in [{before}, {after}]",
);
}
#[tokio::test]
async fn test_successful_flush_stamps_last_successful_epoch() {
let (_, agg) = new_test_aggregator(test_config());
assert_eq!(
agg.inner
.last_successful_flush_epoch_ms
.load(Ordering::Relaxed),
0,
"no flush yet → epoch stays zero"
);
let before = now_epoch_ms();
agg.record(1, FlagRequestType::Decide, None);
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
let after = now_epoch_ms();
let stamped = agg
.inner
.last_successful_flush_epoch_ms
.load(Ordering::Relaxed);
// `stamped > 0` would be true on any clock past 1970 even if the
// code stored a stale or constant value. Bracketing pins the stamp
// to the actual flush moment.
assert!(
(before..=after).contains(&stamped),
"stamp {stamped} must be in [{before}, {after}]",
);
}
#[tokio::test]
async fn test_failed_flush_does_not_overwrite_prior_successful_epoch() {
// Prime with a successful flush so the epoch holds a known
// non-zero value, then run a failing flush. The failure must
// leave the prior stamp intact (not zero it, not overwrite it).
// A test that only verifies "still 0" can't distinguish "field
// never written" from "field correctly preserved."
let mut mock = MockRedisClient::new();
mock.pipeline_error_at_call(1, CustomRedisError::Timeout);
let (_, agg) = new_test_aggregator_with_redis(mock, test_config());
agg.record(1, FlagRequestType::Decide, None);
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
let primed = agg
.inner
.last_successful_flush_epoch_ms
.load(Ordering::Relaxed);
assert!(primed > 0, "priming flush must stamp the epoch");
agg.record(2, FlagRequestType::Decide, None);
flush_once(&agg.inner, FlushPolicy::BailOnError).await;
let after = agg
.inner
.last_successful_flush_epoch_ms
.load(Ordering::Relaxed);
assert_eq!(
after, primed,
"failed flush must not disturb the prior success stamp",
);
}
#[tokio::test]
async fn test_best_effort_partial_success_does_not_stamp_epoch() {
// A BestEffort (shutdown) flush that succeeds on some chunks but
// errors on others is not a "successful flush" — dropped_counts > 0
// and the staleness gauge should keep ticking.
let mut mock = MockRedisClient::new();
mock.pipeline_error_at_call(1, CustomRedisError::Timeout);
let config = BillingAggregatorConfig {
per_flush_batch_size: 2,
..test_config()
};
let (_, agg) = new_test_aggregator_with_redis(mock, config);
record_n_decide_with_library(&agg, 5);
flush_once(&agg.inner, FlushPolicy::BestEffort).await;
assert_eq!(
agg.inner
.last_successful_flush_epoch_ms
.load(Ordering::Relaxed),
0,
"partial-success flush must not stamp the epoch"
);
}
#[test]
fn test_sample_metrics_reads_live_pending_and_in_flight() {
// Smoke test: sample_metrics doesn't panic on any inner state
// (zero epoch, non-empty pending, live in-flight). The staleness
// math is asserted directly via `compute_staleness_seconds` below.
let (_, agg) = new_test_aggregator(test_config());
agg.record(1, FlagRequestType::Decide, None);
agg.record(2, FlagRequestType::Decide, None);
agg.inner.in_flight_uncredited.store(42, Ordering::Relaxed);
sample_metrics(&agg.inner);
agg.inner
.last_successful_flush_epoch_ms
.store(now_epoch_ms(), Ordering::Relaxed);
sample_metrics(&agg.inner);
}
#[test]
fn test_staleness_is_zero_before_first_flush() {
// A freshly booted pod with no traffic must not trip the
// stale-flush alarm. `last_ms == 0` is the sentinel for "no
// successful flush yet" and must collapse to 0 seconds, not
// "decades since 1970".
assert_eq!(compute_staleness_seconds(0, 1_700_000_000_000), 0.0);
assert_eq!(compute_staleness_seconds(0, 0), 0.0);
}
#[test]
fn test_staleness_reports_seconds_since_last_flush() {
assert_eq!(
compute_staleness_seconds(1_700_000_000_000, 1_700_000_005_500),
5.5,
);
assert_eq!(
compute_staleness_seconds(1_700_000_000_000, 1_700_000_000_001),
0.001,
);
}
#[test]
fn test_staleness_saturates_on_backward_clock_skew() {
// Backward NTP step: now < last. Saturating subtract clamps to 0
// rather than wrapping into a huge positive value that would
// misfire the staleness alarm.
assert_eq!(
compute_staleness_seconds(1_700_000_005_000, 1_700_000_000_000),
0.0,
);
}
#[test]
fn test_config_validate_rejects_zero_flush_interval() {
let config = BillingAggregatorConfig {
flush_interval: Duration::ZERO,
..test_config()
};
let err = config.validate().unwrap_err();
assert!(
err.contains("flush_interval"),
"error should name the bad knob, got: {err}"
);
}
#[test]
fn test_config_validate_rejects_zero_max_pending_entries() {
let config = BillingAggregatorConfig {
max_pending_entries: 0,
..test_config()
};
let err = config.validate().unwrap_err();
assert!(err.contains("max_pending_entries"), "got: {err}");
}
#[test]
fn test_config_validate_rejects_zero_per_flush_batch_size() {
let config = BillingAggregatorConfig {
per_flush_batch_size: 0,
..test_config()
};
let err = config.validate().unwrap_err();
assert!(err.contains("per_flush_batch_size"), "got: {err}");
}
#[test]
fn test_config_validate_rejects_zero_shutdown_flush_timeout() {
let config = BillingAggregatorConfig {
shutdown_flush_timeout: Duration::ZERO,
..test_config()
};
let err = config.validate().unwrap_err();
assert!(err.contains("shutdown_flush_timeout"), "got: {err}");
}
#[test]
fn test_config_validate_accepts_default() {
assert!(BillingAggregatorConfig::default().validate().is_ok());
}
#[tokio::test]
#[should_panic(expected = "invalid BillingAggregatorConfig")]
async fn test_start_panics_on_invalid_config() {
let redis = Arc::new(MockRedisClient::new());
BillingAggregator::start(
redis,
BillingAggregatorConfig {
flush_interval: Duration::ZERO,
..test_config()
},
);
}
}