Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

23.4 KB
Raw Blame
//! Interaction matrix for the four mechanisms that decide an analytics event's
//! person processing and routing: the customer's `$process_person_profile`
//! property, a `SkipPersonProcessing` restriction, a `ForceOverflow`
//! restriction, and the global rate limiter.
//!
//! Each mechanism has unit coverage on its own. What no other suite covers is
//! how they compose, and composition is where they are easy to get wrong: three
//! of the four can disable person processing, and two of them decide the
//! partition key through the same `person_ordering` rule.
//!
//! The assertions are the four facts downstream ingestion actually reads off the
//! wire, plus whether capture paid for the limiter at all:
//!
//! - the topic the record landed on
//! - whether it carries a partition key
//! - the `force_disable_person_processing` header
//! - how many times the limiter was consulted
//!
//! The `personless` column is the reason this file exists. `$process_person_profile`
//! and a skip-person restriction mean the same thing to the consumer, so it would
//! be easy to derive one from the other in capture. Doing so would silently stop
//! the limiter from running on personless traffic, which is a large share of
//! volume and exactly the traffic the limiter still has a job on. Every case is
//! run twice, once with the property and once without, and both rows carry the
//! same expectation: the property must change nothing here.
//!
//! Every case runs against both analytics endpoints, `/capture` (v0) and
//! `/i/v1/analytics/events` (v1), and asserts each against the same explicit
//! expectation rather than against the other, so two paths that drift together
//! still fail. The two express the personless choice differently: v0 reads the
//! `$process_person_profile` property, v1 reads `options.process_person_profile`.

#[path = "common/integration_utils.rs"]
mod integration_utils;

use axum::http::StatusCode;
use axum_test_helper::TestClient;
use capture::config::CaptureMode;
use capture::event_restrictions::{
    EventRestrictionService, Pipeline, Restriction, RestrictionFilters, RestrictionManager,
    RestrictionScope, RestrictionType,
};
use capture::global_rate_limiter::GlobalRateLimiter;
use capture::outputs::OutputRegistry;
use capture::quota_limiters::CaptureQuotaLimiter;
use capture::router::router;
use capture::sinks::kafka::KafkaSinkBase;
use capture::sinks::producer::MockKafkaProducer;
use capture::sinks::registry::OutputTable;
use capture::time::TimeSource;
use capture::v1::router::{router as v1_router, RouterConfig as V1RouterConfig};
use capture::v1::test_utils::TestStateBuilder;
use chrono::{DateTime, Utc};
use common_redis::MockRedisClient;
use integration_utils::{test_lifecycle_handlers, DEFAULT_CONFIG, DEFAULT_TEST_TIME};
use limiters::token_dropper::TokenDropper;
use metrics_util::debugging::{DebugValue, DebuggingRecorder};
use rstest::rstest;
use serde_json::json;
use std::collections::HashSet;
use std::sync::Arc;
use std::time::Duration;

const TOKEN: &str = "phc_matrix_token";
const DISTINCT_ID: &str = "matrix-user";
/// A key the filtered restriction covers, and one it does not.
const RESTRICTED_DISTINCT_ID: &str = "matrix-user-restricted";
const OPEN_DISTINCT_ID: &str = "matrix-user-open";

/// The limiter evaluates the local cache, so the first event for a key is always
/// a miss and passes. A threshold of 0 makes the second event for that key
/// exceed the window deterministically, with no Redis round trip and no clock.
const THRESHOLD_ALWAYS_LIMITS: u64 = 0;
const THRESHOLD_NEVER_LIMITS: u64 = 1_000_000;

struct FixedTime {
    time: DateTime<Utc>,
}

impl TimeSource for FixedTime {
    fn current_time(&self) -> DateTime<Utc> {
        self.time
    }
}

#[derive(Clone, Copy, Debug, Default)]
struct Inputs {
    /// The event carries `$process_person_profile: false`.
    personless: bool,
    /// A `SkipPersonProcessing` restriction covers the token.
    skip_person_restriction: bool,
    /// A `ForceOverflow` restriction covers the token.
    overflow_restriction: bool,
    /// The limiter's threshold is low enough to limit the observed event.
    over_rate_limit: bool,
    /// Narrow the restrictions to one distinct id instead of the whole token.
    /// Restrictions are keyed by token and may carry filters, so a token can
    /// have some of its keys restricted and the rest untouched.
    restricted_distinct_id: Option<&'static str>,
}

/// The lane an event landed on, independent of each path's topic names.
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
enum Lane {
    Main,
    Overflow,
    Other(&'static str),
}

/// The wire facts of one produced record.
#[derive(Debug, PartialEq, Eq, Clone, Copy)]
struct Record {
    lane: Lane,
    has_key: bool,
    force_disable_person_processing: Option<bool>,
}

/// One record's facts plus the limiter cost for the batch that produced it.
/// Consultations are a batch total, not per record, because the Redis work this
/// counts is paid once per consulted event and read back from one counter.
#[derive(Debug, PartialEq, Eq)]
struct Observed {
    record: Record,
    limiter_consultations: u64,
}

#[derive(Debug, PartialEq, Eq)]
struct Batch {
    records: Vec<Record>,
    limiter_consultations: u64,
}

impl Batch {
    /// The facts of one record, paired with the batch's limiter cost.
    fn at(&self, index: usize) -> Observed {
        Observed {
            record: self.records[index],
            limiter_consultations: self.limiter_consultations,
        }
    }
}

/// Count limiter evaluations on the recorder the caller installed. Every event
/// that reaches the stage charges the limiter, restrictions included.
fn consultations(snapshotter: &metrics_util::debugging::Snapshotter) -> u64 {
    snapshotter
        .snapshot()
        .into_vec()
        .iter()
        .filter(|(key, _, _, _)| key.key().name() == "global_rate_limiter_eval_counts_total")
        .filter_map(|(_, _, _, value)| match value {
            DebugValue::Counter(v) => Some(*v),
            _ => None,
        })
        .sum()
}

fn build_limiter(inputs: Inputs, redis: Arc<MockRedisClient>) -> GlobalRateLimiter {
    let mut cfg = DEFAULT_CONFIG.clone();
    cfg.capture_mode = CaptureMode::Events;
    cfg.global_rate_limit_enabled = true;
    cfg.global_rate_limit_token_distinctid_threshold = if inputs.over_rate_limit {
        THRESHOLD_ALWAYS_LIMITS
    } else {
        THRESHOLD_NEVER_LIMITS
    };
    // Park the background sync well beyond the request so the limiter's verdict
    // comes from the local cache alone and no Redis tick can race the batch.
    cfg.global_rate_limit_tick_interval_ms = 600_000;
    GlobalRateLimiter::new_token_distinct_id(&cfg, vec![redis])
        .expect("failed to build the global rate limiter")
}

async fn build_restrictions(inputs: Inputs) -> EventRestrictionService {
    let scope = match inputs.restricted_distinct_id {
        Some(distinct_id) => RestrictionScope::Filtered(RestrictionFilters {
            distinct_ids: HashSet::from([distinct_id.to_string()]),
            ..Default::default()
        }),
        None => RestrictionScope::AllEvents,
    };

    let mut restrictions = Vec::new();
    if inputs.skip_person_restriction {
        restrictions.push(Restriction {
            restriction_type: RestrictionType::SkipPersonProcessing,
            scope: scope.clone(),
            args: None,
        });
    }
    if inputs.overflow_restriction {
        restrictions.push(Restriction {
            restriction_type: RestrictionType::ForceOverflow,
            scope,
            args: None,
        });
    }
    let service = EventRestrictionService::new(
        Pipeline::for_capture_mode(CaptureMode::Events),
        Duration::from_secs(300),
    );
    let mut manager = RestrictionManager::new();
    manager.insert_restrictions(Pipeline::Analytics, TOKEN, restrictions);
    service.update(manager).await;
    service
}

/// Drive the v0 endpoint (`/capture`) into a real `KafkaSinkBase` and read every
/// record off the mock producer, in request order.
///
/// `distinct_ids` gives one event per entry. Repeating an id is how a case
/// reaches the limiter's limiting branch: the limiter evaluates the local cache,
/// so the first event for a key is always a miss that passes, and only a later
/// one can exceed a zero threshold. That keeps the limiting case free of Redis
/// and of the clock.
async fn run_v0(inputs: Inputs, distinct_ids: &[&str]) -> Batch {
    // The handler runs on this thread: `#[tokio::test]` is a current-thread
    // runtime, so a thread-local recorder sees the limiter's counters.
    let recorder = DebuggingRecorder::new();
    let snapshotter = recorder.snapshotter();
    let _guard = metrics::set_default_local_recorder(&recorder);

    let (readiness, liveness, _monitor) = test_lifecycle_handlers();
    let redis = Arc::new(MockRedisClient::new());
    let cfg = DEFAULT_CONFIG.clone();
    let limiter = build_limiter(inputs, redis.clone());
    let service = build_restrictions(inputs).await;

    let producer = MockKafkaProducer::new();
    let sink = KafkaSinkBase::with_producer(
        producer.clone(),
        OutputTable::new(&cfg.outputs, cfg.capture_mode.has_ai_overflow_lane()),
    );
    let quota_limiter =
        CaptureQuotaLimiter::new(&cfg, redis.clone(), Duration::from_secs(60 * 60 * 24 * 7));

    let router = router(
        FixedTime {
            time: DateTime::parse_from_rfc3339(DEFAULT_TEST_TIME)
                .expect("invalid fixed time")
                .with_timezone(&Utc),
        },
        readiness,
        liveness,
        Arc::new(OutputRegistry::single(sink)),
        redis,
        Some(Arc::new(limiter)),
        quota_limiter,
        TokenDropper::default(),
        Some(service),
        None,
        CaptureMode::Events,
        None,
        25 * 1024 * 1024,
        false,
        1_i64,
        false,
        0.0_f32,
        26_214_400,
        983_040, // ai_max_event_bytes (960KB)
        None,
        256,
        10 * 1024 * 1024,
        50 * 1024 * 1024,
        None,
        None,
        None, // ai_byte_rate_limiter
        None,
        8,
        None,
        None,
    );

    // v0 reads the customer's choice from the event property.
    let mut properties = json!({});
    if inputs.personless {
        properties["$process_person_profile"] = json!(false);
    }
    let batch: Vec<_> = distinct_ids
        .iter()
        .map(|distinct_id| {
            json!({
                "event": "$pageview",
                "distinct_id": distinct_id,
                "properties": properties,
            })
        })
        .collect();
    let payload = json!({ "api_key": TOKEN, "batch": batch });

    let response = TestClient::new(router)
        .post("/capture")
        .header("Content-Type", "application/json")
        .header("X-Forwarded-For", "127.0.0.1")
        .body(payload.to_string())
        .send()
        .await;
    assert_eq!(response.status(), StatusCode::OK, "v0 rejected the batch");

    let produced = producer.get_records();
    assert_eq!(
        produced.len(),
        distinct_ids.len(),
        "v0 must produce every event"
    );

    Batch {
        records: produced
            .iter()
            .map(|r| Record {
                lane: v0_lane(&r.topic),
                has_key: r.key.is_some(),
                force_disable_person_processing: r.headers.force_disable_person_processing,
            })
            .collect(),
        limiter_consultations: consultations(&snapshotter),
    }
}

/// Drive the v1 endpoint (`/i/v1/analytics/events`) through its router and the
/// shared outputs, reading every record off the mock producer, in request order.
async fn run_v1(inputs: Inputs, distinct_ids: &[&str]) -> Batch {
    let recorder = DebuggingRecorder::new();
    let snapshotter = recorder.snapshotter();
    let _guard = metrics::set_default_local_recorder(&recorder);

    let redis = Arc::new(MockRedisClient::new());
    let limiter = build_limiter(inputs, redis);
    let service = build_restrictions(inputs).await;

    let ts = TestStateBuilder::new()
        .with_restriction_service(service)
        .with_global_rate_limiter(Arc::new(limiter))
        .build();

    let router = v1_router(V1RouterConfig {
        concurrency_limit: None,
        max_compressed_body_bytes: 10 * 1024 * 1024,
        serves_analytics: true,
        serves_ai_events: false,
    })
    .with_state(ts.state.clone());

    // v1 reads the customer's choice from the event options, not the property.
    let options = if inputs.personless {
        json!({ "process_person_profile": false })
    } else {
        json!({})
    };
    // v1 requires a distinct uuid per event, so index them.
    let batch: Vec<_> = distinct_ids
        .iter()
        .enumerate()
        .map(|(i, distinct_id)| {
            json!({
                "event": "$pageview",
                "uuid": format!("0198f0a0-0000-7000-8000-00000000000{i}"),
                "distinct_id": distinct_id,
                "timestamp": "2026-03-19T14:29:58.123Z",
                "options": options,
                "properties": {},
            })
        })
        .collect();
    let payload = json!({
        "created_at": "2026-03-19T14:30:00.000Z",
        "batch": batch,
    });

    let response = TestClient::new(router)
        .post("/i/v1/analytics/events")
        .header("authorization", format!("Bearer {TOKEN}"))
        .header("content-type", "application/json")
        .header("X-Forwarded-For", "127.0.0.1")
        .header("PostHog-Sdk-Info", "posthog-rs/1.0.0")
        .header("PostHog-Attempt", "1")
        .header("PostHog-Request-Id", "00000000-0000-0000-0000-000000000001")
        .header("PostHog-Request-Timestamp", "2026-03-19T14:30:00.000Z")
        .header("user-agent", "test-client/1.0")
        .body(payload.to_string())
        .send()
        .await;
    let status = response.status();
    let body = response.text().await;
    assert_eq!(status, StatusCode::OK, "v1 rejected the batch: {body}");

    let produced = ts.mock_producer.get_records();
    assert_eq!(
        produced.len(),
        distinct_ids.len(),
        "v1 must produce every event"
    );

    Batch {
        records: produced
            .iter()
            .map(|r| Record {
                lane: v1_lane(&r.topic),
                has_key: r.key.is_some(),
                force_disable_person_processing: r.headers.force_disable_person_processing,
            })
            .collect(),
        limiter_consultations: consultations(&snapshotter),
    }
}

// Each path names its lanes differently; the matrix compares lanes, not strings.
fn v0_lane(topic: &str) -> Lane {
    match topic {
        "events_plugin_ingestion" => Lane::Main,
        "events_plugin_ingestion_overflow" => Lane::Overflow,
        other => Lane::Other(Box::leak(other.to_string().into_boxed_str())),
    }
}

fn v1_lane(topic: &str) -> Lane {
    match topic {
        "events_main" => Lane::Main,
        "events_overflow" => Lane::Overflow,
        other => Lane::Other(Box::leak(other.to_string().into_boxed_str())),
    }
}

// Each pair of cases differs only in `personless` and shares one expectation.
#[rstest]
// Nothing applies: the limiter runs on both events and changes nothing.
#[case::plain(
    Inputs::default(),
    Observed { record: Record { lane: Lane::Main, has_key: true, force_disable_person_processing: None }, limiter_consultations: 2 }
)]
#[case::plain_personless(
    Inputs { personless: true, ..Inputs::default() },
    Observed { record: Record { lane: Lane::Main, has_key: true, force_disable_person_processing: None }, limiter_consultations: 2 }
)]
// The limiter alone: it takes person processing away and spreads the key.
#[case::rate_limited(
    Inputs { over_rate_limit: true, ..Inputs::default() },
    Observed { record: Record { lane: Lane::Overflow, has_key: false, force_disable_person_processing: Some(true) }, limiter_consultations: 2 }
)]
#[case::rate_limited_personless(
    Inputs { over_rate_limit: true, personless: true, ..Inputs::default() },
    Observed { record: Record { lane: Lane::Overflow, has_key: false, force_disable_person_processing: Some(true) }, limiter_consultations: 2 }
)]
// The skip-person restriction alone: person processing off and the key dropped,
// on the main lane. The limiter is still charged, but it stamps nothing.
#[case::skip_person(
    Inputs { skip_person_restriction: true, ..Inputs::default() },
    Observed { record: Record { lane: Lane::Main, has_key: false, force_disable_person_processing: Some(true) }, limiter_consultations: 2 }
)]
#[case::skip_person_personless(
    Inputs { skip_person_restriction: true, personless: true, ..Inputs::default() },
    Observed { record: Record { lane: Lane::Main, has_key: false, force_disable_person_processing: Some(true) }, limiter_consultations: 2 }
)]
// The restriction shadows the limiter's stamps: the key is over its window, but
// the restriction already took person processing away, so nothing reroutes it.
// The charge still lands.
#[case::skip_person_and_rate_limited(
    Inputs { skip_person_restriction: true, over_rate_limit: true, ..Inputs::default() },
    Observed { record: Record { lane: Lane::Main, has_key: false, force_disable_person_processing: Some(true) }, limiter_consultations: 2 }
)]
#[case::skip_person_and_rate_limited_personless(
    Inputs { skip_person_restriction: true, over_rate_limit: true, personless: true, ..Inputs::default() },
    Observed { record: Record { lane: Lane::Main, has_key: false, force_disable_person_processing: Some(true) }, limiter_consultations: 2 }
)]
// The overflow restriction alone moves the lane but leaves person processing on,
// so the key survives.
#[case::force_overflow(
    Inputs { overflow_restriction: true, ..Inputs::default() },
    Observed { record: Record { lane: Lane::Overflow, has_key: true, force_disable_person_processing: None }, limiter_consultations: 2 }
)]
#[case::force_overflow_personless(
    Inputs { overflow_restriction: true, personless: true, ..Inputs::default() },
    Observed { record: Record { lane: Lane::Overflow, has_key: true, force_disable_person_processing: None }, limiter_consultations: 2 }
)]
// Overflow restriction plus the limiter: already on the overflow lane, and the
// limiter's verdict drops the key.
#[case::force_overflow_and_rate_limited(
    Inputs { overflow_restriction: true, over_rate_limit: true, ..Inputs::default() },
    Observed { record: Record { lane: Lane::Overflow, has_key: false, force_disable_person_processing: Some(true) }, limiter_consultations: 2 }
)]
#[case::force_overflow_and_rate_limited_personless(
    Inputs { overflow_restriction: true, over_rate_limit: true, personless: true, ..Inputs::default() },
    Observed { record: Record { lane: Lane::Overflow, has_key: false, force_disable_person_processing: Some(true) }, limiter_consultations: 2 }
)]
// Both restrictions: the overflow lane from one, the dropped key from the other.
#[case::both_restrictions(
    Inputs { skip_person_restriction: true, overflow_restriction: true, ..Inputs::default() },
    Observed { record: Record { lane: Lane::Overflow, has_key: false, force_disable_person_processing: Some(true) }, limiter_consultations: 2 }
)]
#[case::both_restrictions_personless(
    Inputs { skip_person_restriction: true, overflow_restriction: true, personless: true, ..Inputs::default() },
    Observed { record: Record { lane: Lane::Overflow, has_key: false, force_disable_person_processing: Some(true) }, limiter_consultations: 2 }
)]
// All three: the limiter stamps nothing, and the restrictions alone produce the
// same wire outcome the limiter would have.
#[case::everything(
    Inputs { skip_person_restriction: true, overflow_restriction: true, over_rate_limit: true, ..Inputs::default() },
    Observed { record: Record { lane: Lane::Overflow, has_key: false, force_disable_person_processing: Some(true) }, limiter_consultations: 2 }
)]
#[case::everything_personless(
    Inputs { skip_person_restriction: true, overflow_restriction: true, over_rate_limit: true, personless: true, restricted_distinct_id: None },
    Observed { record: Record { lane: Lane::Overflow, has_key: false, force_disable_person_processing: Some(true) }, limiter_consultations: 2 }
)]
#[tokio::test]
async fn person_processing_and_routing_matrix(#[case] inputs: Inputs, #[case] expected: Observed) {
    // Two events for one key: the second is the one a zero threshold can limit.
    let batch = [DISTINCT_ID, DISTINCT_ID];

    assert_eq!(
        run_v0(inputs, &batch).await.at(1),
        expected,
        "v0 endpoint, inputs: {inputs:?}"
    );
    assert_eq!(
        run_v1(inputs, &batch).await.at(1),
        expected,
        "v1 endpoint, inputs: {inputs:?}"
    );
}

/// A restriction filtered to one distinct id must not shield the token's other
/// keys from the limiter.
///
/// The skip is a per-event decision, so a batch mixing a restricted key with an
/// unrestricted one has to split: the restricted events keep the main lane and
/// take no stamps, while the unrestricted ones are still rerouted once they
/// exceed the window. Hoisting the check to the batch, or keying it on the token
/// rather than the event, would silently stop rate limiting every other key
/// belonging to a token that has any restriction at all.
///
/// Every event charges the limiter, restricted or not, so the consultation count
/// covers the whole batch.
#[rstest]
#[case::skip_person(false)]
#[case::skip_person_and_force_overflow(true)]
#[tokio::test]
async fn restriction_filtered_to_one_distinct_id_leaves_other_keys_rate_limited(
    #[case] overflow_restriction: bool,
) {
    let inputs = Inputs {
        skip_person_restriction: true,
        overflow_restriction,
        over_rate_limit: true,
        restricted_distinct_id: Some(RESTRICTED_DISTINCT_ID),
        personless: false,
    };
    // Two events per key, so each key reaches the limiter's limiting branch on
    // its second event if it is consulted at all.
    let batch = [
        RESTRICTED_DISTINCT_ID,
        RESTRICTED_DISTINCT_ID,
        OPEN_DISTINCT_ID,
        OPEN_DISTINCT_ID,
    ];
    let restricted_lane = if overflow_restriction {
        Lane::Overflow
    } else {
        Lane::Main
    };

    for (path, observed) in [
        ("v0", run_v0(inputs, &batch).await),
        ("v1", run_v1(inputs, &batch).await),
    ] {
        assert_eq!(
            observed.limiter_consultations, 4,
            "{path}: every event charges the limiter, so a restricted key's volume \
             stays in its own fleet count"
        );

        for i in 0..2 {
            assert_eq!(
                observed.records[i],
                Record {
                    lane: restricted_lane,
                    has_key: false,
                    force_disable_person_processing: Some(true),
                },
                "{path}: restricted event {i} must carry the restriction's outcome"
            );
        }

        // The unrestricted key is untouched by the restriction, so its first
        // event passes with person processing on and its key intact.
        assert_eq!(
            observed.records[2],
            Record {
                lane: Lane::Main,
                has_key: true,
                force_disable_person_processing: None,
            },
            "{path}: the unrestricted key's first event must pass untouched"
        );
        // Its second event exceeds the window, so the limiter acts on it.
        assert_eq!(
            observed.records[3],
            Record {
                lane: Lane::Overflow,
                has_key: false,
                force_disable_person_processing: Some(true),
            },
            "{path}: the unrestricted key must still be rate limited"
        );
    }
}