Skip to content

:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.

feat(cohorts): seed flag calls from flag_evaluations when a run pins it (#124588)

Phil Haackdb59f2c9bcf5
diff --git a/docs/internal/realtime-cohorts/backfill-overview.md b/docs/internal/realtime-cohorts/backfill-overview.md
index d4528ad108f1..eb775d18c935 100644
--- a/docs/internal/realtime-cohorts/backfill-overview.md
+++ b/docs/internal/realtime-cohorts/backfill-overview.md
@@ -140,6 +140,9 @@ So every event in the tile that live will ever count is already counted, and `ma
Without the fence, a tile could apply while an event it already counted is still waiting in the live topic.
When that event arrived, live would add 1 on top of the tile and double count it.
A shuffler stall longer than the margin can still cause that.
+This applies to both shufflers.
+When a run reads flag calls from `flag_evaluations`, a tile counts flag calls that reach the live path through `cohort-event-shuffler-realtime-only`.
+A stall of that shuffler alone opens the fence early in the same way.
One residue remains.
A late event that reaches ClickHouse after `S_chunk` is not in the tile.
diff --git a/docs/internal/realtime-cohorts/seeder.md b/docs/internal/realtime-cohorts/seeder.md
index 8c46bbba98a5..d7be5a15646a 100644
--- a/docs/internal/realtime-cohorts/seeder.md
+++ b/docs/internal/realtime-cohorts/seeder.md
@@ -107,7 +107,7 @@ Max-merge makes that harmless.
### Scanning
-One chunk is one streaming ClickHouse query:
+One chunk is one streaming ClickHouse query per table it reads:
```text
SELECT <projected columns>, canonical person_id
@@ -121,6 +121,7 @@ WHERE team_id = <team>
[AND hash(person) % bands = band]
```
+- **Flag calls table.** The run's pinned payload can name `events` or `flag_evaluations` in `flag_calls_table`. A missing key reads as `events`, and an unknown value fails the run. A chunk of a run pinned to `flag_evaluations` reads `$feature_flag_called` from `flag_evaluations` and its other event names from `events`, in two queries that fold into one set of tiles. Each query projects and filters only for the conditions on its own event names. The `flag_evaluations` query selects an empty `elements_chain`, because the table has none, and bounds arrival on `inserted_at` alone. The backfill in `posthog/dags/flag_evaluations_backfill.py` copies rows from `events` without `inserted_at`, so the column's `DEFAULT timestamp` sets it to the row's `timestamp`. A copied row therefore passes the `S_chunk` bound even when the backfill writes it after the chunk's claim. While the backfill runs, a chunk can miss rows that are not yet copied, and two queries over the same chunk can read different rows. Django must therefore pin `flag_evaluations` for a team only after the backfill covers the run's whole lookback. A chunk that queries `flag_evaluations` also fails once its day reaches the table's TTL, checked before its queries and again after them, because the TTL may have dropped some of the day's rows.
- **Canonical persons.** The join maps each distinct id through the latest person override, so events of merged persons count toward the survivor.
- **Timezone days.** The day's bounds are computed in the run's pinned timezone.
- **Arrival bound.** Only events that reached ClickHouse before `S_chunk` count.
@@ -382,13 +383,14 @@ Ingestion writes event properties with `JSON.stringify` of a parsed object, so a
`tests/ch_event_row_filter.rs` checks all of this against a live ClickHouse, with the VM as the oracle.
Where possible, the test reads a materialized column instead of the blob.
-Before each chunk with a row filter or [column-backed keys](#column-backed-keys), the seeder looks in `system.columns` for a `String` column of `sharded_events` whose default expression is exactly that extraction for the key, and that also exists on `events`.
+Before each chunk with a row filter or [column-backed keys](#column-backed-keys), the seeder looks in `system.columns` for a `String` column of the scanned table's data table (`sharded_events` or `sharded_flag_evaluations`) whose default expression is exactly that extraction for the key, and that also exists on the table the scan reads.
+On `flag_evaluations` that finds `flag_key` for `$feature_flag`.
The lookup serves both together, split into queries that each fit a GET.
Only such a column is used, because a column with another expression, such as a typed `JSONExtract`, reads different values for the same row.
The column holds '' for a blob ClickHouse cannot parse, so the test admits '' there instead of reading the blob.
Without one, the test extracts from the blob: ClickHouse still decompresses the blob for the event's rows, but only the matching rows are transferred and evaluated.
The filter is dropped when the query would grow past the 8192 bytes the client sends by GET, since the `cohort_seeder` profile refuses the POST form.
-`seeder_scan_row_filter_total{outcome}` reports per chunk whether the filter read materialized columns, read the blob, did not apply, or was dropped for length.
+`seeder_scan_row_filter_total{outcome,table}` reports, once for each table a scanned chunk reads, whether the filter read materialized columns, read the blob, did not apply, or was dropped for length.
### Column-backed keys
@@ -436,7 +438,7 @@ ClickHouse computes a column added after the data was written from `properties`
The materializer backfills only `MATERIALIZE_COLUMNS_BACKFILL_PERIOD_DAYS`, which defaults to 0.
So check `system.parts_columns` for the lookback before a long run: an old partition without the column reads the blob as before.
-`seeder_scan_properties_source_total{source}` reports per chunk where `properties` came from.
+`seeder_scan_properties_source_total{source,table}` reports, once for each table a scanned chunk reads, where `properties` came from.
`columns` and `empty` keep ClickHouse off the blob.
`rebuilt_inexact_key`, `rebuilt_no_column`, `rebuilt_ambiguous_value` and `rebuilt_too_long` name why a chunk kept the rebuild, and `rebuilt_no_column` includes a failed or timed-out lookup.
A chunk that reads a key holding the boolean `false`, such as a flag response, counts under `rebuilt_ambiguous_value`.
diff --git a/rust/cohort-seeder/src/app/newest_day.rs b/rust/cohort-seeder/src/app/newest_day.rs
index eab30fe4e743..5b8584d5da16 100644
--- a/rust/cohort-seeder/src/app/newest_day.rs
+++ b/rust/cohort-seeder/src/app/newest_day.rs
@@ -156,8 +156,8 @@ mod tests {
use super::*;
use crate::domain::row_filter::test_catalog::{catalog, event_is, hash, Leaf};
use crate::domain::{
- Boundary, ClaimEpoch, EventNameSet, Lookback, PinnedCondition, SChunkMs, TriggerKind,
- UtcMillis,
+ Boundary, ClaimEpoch, EventNameSet, Lookback, PinnedCondition, SChunkMs, ScanTable,
+ TriggerKind, UtcMillis,
};
const SLIDING: &str = "aaaaaaaaaaaaaaaa";
@@ -193,6 +193,7 @@ mod tests {
hash: SLIDING,
body: event_is("$pageview"),
}]),
+ flag_calls_table: ScanTable::Events,
}
}
diff --git a/rust/cohort-seeder/src/clickhouse/materialized.rs b/rust/cohort-seeder/src/clickhouse/materialized.rs
index dd2687741a18..e529b5b12fff 100644
--- a/rust/cohort-seeder/src/clickhouse/materialized.rs
+++ b/rust/cohort-seeder/src/clickhouse/materialized.rs
@@ -4,7 +4,7 @@
//! A column qualifies only when its stored default expression on the data table is exactly the
//! trim-quotes extraction the filter would otherwise evaluate. A typed `JSONExtract` column reads
//! different values for the same row, such as an empty value for a boolean that HogVM equality can
-//! match. The column must also exist on the distributed `events` table the scan reads.
+//! match. The column must also exist on the distributed table the scan reads.
use std::collections::{BTreeSet, HashMap};
use std::iter;
@@ -14,14 +14,12 @@ use serde::Deserialize;
use super::client::ClickHouseClient;
use super::sql::{clickhouse_string_literal, fits_client_get};
-use crate::domain::MaterializedColumns;
-
-/// Only the data table's columns carry the default expressions.
-const EVENTS_DATA_TABLE: &str = "sharded_events";
+use crate::domain::{MaterializedColumns, ScanTable};
impl MaterializedColumns {
pub async fn lookup(
client: &ClickHouseClient,
+ table: ScanTable,
keys: &BTreeSet<&str>,
) -> Result<Self, clickhouse::error::Error> {
let key_by_expression: HashMap<String, &str> = keys
@@ -34,10 +32,10 @@ impl MaterializedColumns {
.collect::<Vec<_>>();
expressions.sort_unstable();
let mut rows = Vec::new();
- for batch in lookup_batches(&expressions) {
+ for batch in lookup_batches(table, &expressions) {
rows.extend(
client
- .query(&lookup_sql(&batch))
+ .query(&lookup_sql(table, &batch))
.fetch_all::<ColumnRow>()
.await?,
);
@@ -65,28 +63,33 @@ fn trim_quotes_extraction(key: &str) -> String {
}
/// Each batch fits a GET, because the `cohort_seeder` profile refuses the POST form.
-fn lookup_batches<'a>(expressions: &'a [&'a str]) -> impl Iterator<Item = Vec<&'a str>> + 'a {
+fn lookup_batches<'a>(
+ table: ScanTable,
+ expressions: &'a [&'a str],
+) -> impl Iterator<Item = Vec<&'a str>> + 'a {
let mut pending = expressions.iter().copied().peekable();
iter::from_fn(move || {
let mut batch = vec![pending.next()?];
- while let Some(expression) = pending
- .next_if(|next| fits_client_get(&lookup_sql(&[batch.as_slice(), &[*next]].concat())))
- {
+ while let Some(expression) = pending.next_if(|next| {
+ fits_client_get(&lookup_sql(table, &[batch.as_slice(), &[*next]].concat()))
+ }) {
batch.push(expression);
}
Some(batch)
})
}
-fn lookup_sql(expressions: &[&str]) -> String {
+fn lookup_sql(table: ScanTable, expressions: &[&str]) -> String {
let mut literals = expressions
.iter()
.map(|expression| clickhouse_string_literal(expression))
.collect::<Vec<_>>();
literals.sort();
format!(
- "SELECT name, default_expression AS expression\nFROM system.columns\nWHERE database = currentDatabase() AND table = '{EVENTS_DATA_TABLE}'\n AND type = 'String' AND default_kind IN ('DEFAULT', 'MATERIALIZED')\n AND default_expression IN ({})\n AND name IN (SELECT name FROM system.columns WHERE database = currentDatabase() AND table = 'events' AND type = 'String')\nORDER BY name",
+ "SELECT name, default_expression AS expression\nFROM system.columns\nWHERE database = currentDatabase() AND table = '{}'\n AND type = 'String' AND default_kind IN ('DEFAULT', 'MATERIALIZED')\n AND default_expression IN ({})\n AND name IN (SELECT name FROM system.columns WHERE database = currentDatabase() AND table = '{}' AND type = 'String')\nORDER BY name",
+ table.data_table(),
literals.join(", "),
+ table.as_str(),
)
}
@@ -94,6 +97,8 @@ fn lookup_sql(expressions: &[&str]) -> String {
mod tests {
use super::*;
+ /// Checks both tables because their names differ in length. A batch packed for one table can
+ /// overflow a GET for the other.
#[test]
fn a_long_lookup_is_split_into_queries_that_each_fit_a_get() {
let expressions = (0..300)
@@ -102,15 +107,17 @@ mod tests {
let mut sorted = expressions.iter().map(String::as_str).collect::<Vec<_>>();
sorted.sort_unstable();
- let batches = lookup_batches(&sorted).collect::<Vec<_>>();
- assert!(
- batches.len() > 1,
- "the keys fit one query, so nothing was split"
- );
- for batch in &batches {
- assert!(fits_client_get(&lookup_sql(batch)));
+ for table in [ScanTable::Events, ScanTable::FlagEvaluations] {
+ let batches = lookup_batches(table, &sorted).collect::<Vec<_>>();
+ assert!(
+ batches.len() > 1,
+ "{table:?}: the keys fit one query, so nothing was split"
+ );
+ for batch in &batches {
+ assert!(fits_client_get(&lookup_sql(table, batch)), "{table:?}");
+ }
+ assert_eq!(batches.concat(), sorted, "{table:?}");
}
- assert_eq!(batches.concat(), sorted);
}
#[test]
diff --git a/rust/cohort-seeder/src/clickhouse/scanner.rs b/rust/cohort-seeder/src/clickhouse/scanner.rs
index 04c5e638d6c5..53c2f13730d0 100644
--- a/rust/cohort-seeder/src/clickhouse/scanner.rs
+++ b/rust/cohort-seeder/src/clickhouse/scanner.rs
@@ -28,12 +28,12 @@ use super::sql::{
ScanSpec,
};
use crate::domain::{
- conditions_active_on, diff_tiles, ActiveConditions, AggregateError, CancelCause,
+ conditions_active_on, diff_tiles, table_scans, ActiveConditions, AggregateError, CancelCause,
ChunkAccumulator, ChunkDomainError, ChunkProjection, ChunkSpec, ClaimedChunk, ColumnExactKeys,
- ColumnUpgrade, ConditionAnalyses, DayIdx, EventNameSet, Halted, MaterializedColumns,
- PinnedCondition, PinnedRun, ProjectedKeys, PropertiesSourcing, RecordOutcome, RecordStats,
- ScanRowFilter, ScanVolume, ScannedChunk, SeedDomain, SeedTile, SourcedProjection, TileDiff,
- UtcMillis,
+ ColumnUpgrade, ConditionAnalyses, DayIdx, Halted, MaterializedColumns, PinnedCondition,
+ PinnedRun, ProjectedKeys, PropertiesSourcing, RecordOutcome, RecordStats, ScanRowFilter,
+ ScanTable, ScanVolume, ScannedChunk, SeedDomain, SeedTile, SourcedProjection, TableScan,
+ TileDiff, UtcMillis,
};
use crate::observability::metrics::{
team_label, MetricTimer, AGGREGATE_ENTRIES, CHUNKS_PROJECTED, CHUNKS_VACUOUS,
@@ -84,7 +84,7 @@ impl ChunkScanner {
chunk,
run,
analyses,
- Utc::now().timestamp_millis(),
+ || Utc::now().timestamp_millis(),
lease_cancel,
shutdown,
)
@@ -96,12 +96,12 @@ impl ChunkScanner {
chunk: ClaimedChunk,
run: &PinnedRun,
analyses: &ConditionAnalyses,
- now_ms: i64,
+ clock: fn() -> i64,
lease_cancel: &CancellationToken,
shutdown: &CancellationToken,
) -> Result<ScannedChunk, Halted<ClaimedChunk, ScanError>> {
match self
- .scan_tiles(&chunk, run, analyses, now_ms, lease_cancel, shutdown)
+ .scan_tiles(&chunk, run, analyses, clock, lease_cancel, shutdown)
.await
{
Ok((tiles, volume)) => Ok(chunk.into_scanned(tiles, volume)),
@@ -115,11 +115,12 @@ impl ChunkScanner {
chunk: &ClaimedChunk,
run: &PinnedRun,
analyses: &ConditionAnalyses,
- now_ms: i64,
+ clock: fn() -> i64,
lease_cancel: &CancellationToken,
shutdown: &CancellationToken,
) -> Result<(Vec<SeedTile>, ScanVolume), ScanHalt> {
let timer = MetricTimer::start(CHUNK_SCAN_DURATION_SECONDS);
+ let now_ms = clock();
let spec = chunk.spec();
let domain = run.domain_for(&spec).map_err(ScanError::from)?;
let active = active_conditions_at(spec.day, run.tz, now_ms, &run.conditions);
@@ -132,55 +133,67 @@ impl ChunkScanner {
counter!(CHUNKS_VACUOUS, "reason" => "window_expired").increment(1);
return Ok((Vec::new(), ScanVolume::default()));
}
- let event_names = active_event_names(run, &active);
- let scan_spec = match plan_scan(spec.team_id, &domain, &event_names, spec.band) {
- ScanPlan::Scan(scan_spec) => scan_spec,
- ScanPlan::Vacuous => {
- counter!(CHUNKS_VACUOUS, "reason" => "empty_scan").increment(1);
- return Ok((Vec::new(), ScanVolume::default()));
- }
- };
- let projection = analyses.projection(&active);
- let exact = analyses.column_exact_keys(&active);
- let row_filter = analyses.row_filter(&event_names, &run.filters, &active);
- let columns = self
- .lookup_columns(&column_lookup_keys(&projection, &exact, &row_filter))
- .await;
+ let planned = table_scans(run, &active)
+ .into_iter()
+ .filter_map(|(table, scan)| {
+ match plan_scan(spec.team_id, &domain, &scan.event_names, spec.band, table) {
+ ScanPlan::Scan(scan_spec) => Some((table, scan, scan_spec)),
+ ScanPlan::Vacuous => None,
+ }
+ })
+ .collect::<Vec<_>>();
+ if planned.is_empty() {
+ counter!(CHUNKS_VACUOUS, "reason" => "empty_scan").increment(1);
+ return Ok((Vec::new(), ScanVolume::default()));
+ }
+ // The TTL may already have dropped some of this day's rows, so the tiles would undercount
+ // and the run would still complete.
+ let tables = planned.iter().map(|(table, ..)| *table).collect::<Vec<_>>();
+ check_retention(&tables, &domain, spec.day, now_ms)?;
let comment = ScanLogComment::BehavioralChunk {
spec,
cohort_id: run.sole_cohort_id(),
};
- let sourcing = match projection.source_properties(&exact, &columns) {
- PropertiesSourcing::Upgradable(upgrade) => {
- self.probe_ambiguous_values(&scan_spec, upgrade, comment, lease_cancel, shutdown)
- .await?
- }
- decided @ PropertiesSourcing::Decided(_) => decided,
- };
- let narrowed = narrow_scan(scan_spec, sourcing, &row_filter, &columns);
- self.record_narrowing(run.team_id, &narrowed);
- let scan_spec = narrowed.filtered.spec;
- let projection = narrowed.sourced.into_projection();
+ let mut queries = Vec::with_capacity(planned.len());
+ for (table, scan, scan_spec) in planned {
+ queries.push(
+ self.narrow_table_query(
+ run,
+ analyses,
+ table,
+ &scan,
+ scan_spec,
+ comment,
+ lease_cancel,
+ shutdown,
+ )
+ .await?,
+ );
+ }
let (tiles, volume, projected_fold) = self
- .scan_once(
+ .scan_tables(
spec,
run,
&domain,
&active,
- &scan_spec,
- &projection,
+ &queries,
ScanKind::Behavioral,
comment,
lease_cancel,
shutdown,
)
.await?;
+ // A query can run for hours, so the TTL can reach the day while the chunk scans.
+ check_retention(&tables, &domain, spec.day, clock())?;
// Closed before the diagnostic arm, which must not lengthen the chunk's reported scan.
drop(timer);
if self.shadow_compare {
- match CompareSkip::of(&projection, projected_fold) {
+ match CompareSkip::of(
+ queries.iter().map(|query| &query.projection),
+ projected_fold,
+ ) {
Some(skip) => {
let team = team_label(&self.allowlist, run.team_id);
counter!(SHADOW_COMPARE, "result" => skip.as_str(), "team_id" => team)
@@ -192,7 +205,7 @@ impl ChunkScanner {
run,
&domain,
&active,
- &scan_spec,
+ &queries,
&tiles,
projected_fold,
lease_cancel,
@@ -205,49 +218,88 @@ impl ChunkScanner {
Ok((tiles, volume))
}
- /// One rendering of this chunk's scan: build the cursor from [`scan_sql`], fold it through a
- /// fresh accumulator, meter the moved bytes under `kind`, and return the sorted tiles.
+ /// Narrow one table's scan to the conditions its rows can reach: its projection, its row
+ /// filter, and the materialized columns of its table.
+ #[allow(clippy::too_many_arguments)]
+ async fn narrow_table_query(
+ &self,
+ run: &PinnedRun,
+ analyses: &ConditionAnalyses,
+ table: ScanTable,
+ scan: &TableScan,
+ scan_spec: ScanSpec,
+ comment: ScanLogComment,
+ lease_cancel: &CancellationToken,
+ shutdown: &CancellationToken,
+ ) -> Result<TableQuery, ScanHalt> {
+ let projection = analyses.projection(&scan.active);
+ let exact = analyses.column_exact_keys(&scan.active);
+ let row_filter = analyses.row_filter(&scan.event_names, &run.filters, &scan.active);
+ let columns = self
+ .lookup_columns(table, &column_lookup_keys(&projection, &exact, &row_filter))
+ .await;
+ let sourcing = match projection.source_properties(&exact, &columns) {
+ PropertiesSourcing::Upgradable(upgrade) => {
+ self.probe_ambiguous_values(&scan_spec, upgrade, comment, lease_cancel, shutdown)
+ .await?
+ }
+ decided @ PropertiesSourcing::Decided(_) => decided,
+ };
+ let narrowed = narrow_scan(scan_spec, sourcing, &row_filter, &columns);
+ self.record_narrowing(run.team_id, table, &narrowed);
+ Ok(TableQuery {
+ spec: narrowed.filtered.spec,
+ projection: narrowed.sourced.into_projection(),
+ })
+ }
+
+ /// One rendering of this chunk's scan: build a cursor from [`scan_sql`] for each table, fold
+ /// them all through one fresh accumulator, meter the moved bytes under `kind`, and return the
+ /// sorted tiles. One accumulator keeps a tile key from splitting across two tile sets.
///
/// `kind` and `comment` co-vary at the two call sites. The per-row and per-chunk fold metrics
/// are emitted for the authoritative [`ScanKind::Behavioral`] arm only, so a diagnostic
/// re-scan of the same rows never doubles a throughput series.
#[allow(clippy::too_many_arguments)]
- async fn scan_once(
+ async fn scan_tables(
&self,
spec: ChunkSpec,
run: &PinnedRun,
domain: &SeedDomain,
active: &ActiveConditions,
- scan_spec: &ScanSpec,
- projection: &ChunkProjection,
+ queries: &[TableQuery],
kind: ScanKind,
comment: ScanLogComment,
lease_cancel: &CancellationToken,
shutdown: &CancellationToken,
) -> Result<(Vec<SeedTile>, ScanVolume, FoldSummary), ScanHalt> {
- let mut cursor = self
- .client
- .query(&scan_sql(scan_spec, projection))
- .with_option(LOG_COMMENT_OPTION, comment.to_string())
- .fetch::<EventRow>()
- .map_err(ScanError::Query)?;
let mut accumulator =
ChunkAccumulator::new(run.team_id, &run.filters, active).map_err(ScanError::from)?;
-
- // Every way out of the fold funnels back here, so the volume is metered once whether the
- // scan finished, was cancelled, or failed mid-stream.
- let folded = fold_cursor(
- &mut cursor,
- &mut accumulator,
- domain,
- run.team_id,
- kind,
- lease_cancel,
- shutdown,
- )
- .await;
- let volume = scan_volume::observe(kind, &cursor);
- let summary = folded?;
+ let mut volume = ScanVolume::default();
+ let mut summary = FoldSummary::default();
+ for query in queries {
+ let mut cursor = self
+ .client
+ .query(&scan_sql(&query.spec, &query.projection))
+ .with_option(LOG_COMMENT_OPTION, comment.to_string())
+ .fetch::<EventRow>()
+ .map_err(ScanError::Query)?;
+ // Every way out of the fold funnels back here, so the volume is metered once whether
+ // the scan finished, was cancelled, or failed mid-stream.
+ let folded = fold_cursor(
+ &mut cursor,
+ &mut accumulator,
+ &mut summary,
+ domain,
+ run.team_id,
+ kind,
+ lease_cancel,
+ shutdown,
+ )
+ .await;
+ volume = volume + scan_volume::observe(kind, &cursor);
+ folded?;
+ }
if kind == ScanKind::Behavioral {
if summary.rows == RowsSeen::None {
counter!(CHUNKS_VACUOUS, "reason" => "no_rows").increment(1);
@@ -279,7 +331,7 @@ impl ChunkScanner {
run: &PinnedRun,
domain: &SeedDomain,
active: &ActiveConditions,
- scan_spec: &ScanSpec,
+ queries: &[TableQuery],
projected_tiles: &[SeedTile],
projected_fold: FoldSummary,
lease_cancel: &CancellationToken,
@@ -289,14 +341,20 @@ impl ChunkScanner {
// Spans the re-scan, its fold, and the diff. Records on every exit, so a cancelled or
// failed compare still reports the time it held the chunk's slot and lease.
let _timer = MetricTimer::start(SHADOW_COMPARE_DURATION_SECONDS);
+ let wide: Vec<TableQuery> = queries
+ .iter()
+ .map(|query| TableQuery {
+ spec: query.spec.clone(),
+ projection: ChunkProjection::FullColumns,
+ })
+ .collect();
let (legacy_tiles, legacy_fold) = match self
- .scan_once(
+ .scan_tables(
spec,
run,
domain,
active,
- scan_spec,
- &ChunkProjection::FullColumns,
+ &wide,
ScanKind::BehavioralCompare,
ScanLogComment::BehavioralCompareChunk {
spec,
@@ -392,20 +450,23 @@ impl ChunkScanner {
}
/// A failed or timed-out lookup returns no columns, so the scan reads the blob.
- async fn lookup_columns(&self, keys: &BTreeSet<&str>) -> MaterializedColumns {
+ async fn lookup_columns(&self, table: ScanTable, keys: &BTreeSet<&str>) -> MaterializedColumns {
match tokio::time::timeout(
MATERIALIZED_LOOKUP_TIMEOUT,
- MaterializedColumns::lookup(&self.client, keys),
+ MaterializedColumns::lookup(&self.client, table, keys),
)
.await
{
Ok(Ok(columns)) => columns,
Ok(Err(error)) => {
- warn!(error = ?error, "materialized column lookup failed; the scan reads the properties blob");
+ warn!(table = table.as_str(), error = ?error, "materialized column lookup failed; the scan reads the properties blob");
MaterializedColumns::default()
}
Err(_) => {
- warn!("materialized column lookup timed out; the scan reads the properties blob");
+ warn!(
+ table = table.as_str(),
+ "materialized column lookup timed out; the scan reads the properties blob"
+ );
MaterializedColumns::default()
}
}
@@ -413,24 +474,28 @@ impl ChunkScanner {
/// Publish what this chunk's scan narrowed to, so a team that stops projecting is visible
/// before its scan cost is.
- fn record_narrowing(&self, team_id: TeamId, narrowed: &NarrowedScan) {
+ fn record_narrowing(&self, team_id: TeamId, table: ScanTable, narrowed: &NarrowedScan) {
let team = team_label(&self.allowlist, team_id);
+ let table = table.as_str();
let projection = narrowed.sourced.projection();
counter!(
CHUNKS_PROJECTED,
"outcome" => projection.outcome(),
+ "table" => table,
"team_id" => team.clone(),
)
.increment(1);
counter!(
SCAN_PROPERTIES_SOURCE,
"source" => narrowed.sourced.outcome().as_str(),
+ "table" => table,
"team_id" => team.clone(),
)
.increment(1);
counter!(
SCAN_ROW_FILTER,
"outcome" => narrowed.filtered.outcome.as_str(),
+ "table" => table,
"team_id" => team.clone(),
)
.increment(1);
@@ -442,13 +507,25 @@ impl ChunkScanner {
("person_properties", plan.person_properties.key_count()),
] {
if let Some(keys) = keys {
- histogram!(PROJECTION_KEYS, "blob" => blob, "team_id" => team.clone())
- .record(keys as f64);
+ histogram!(
+ PROJECTION_KEYS,
+ "blob" => blob,
+ "table" => table,
+ "team_id" => team.clone(),
+ )
+ .record(keys as f64);
}
}
}
}
+/// One table's narrowed scan within a chunk.
+#[derive(Debug)]
+struct TableQuery {
+ spec: ScanSpec,
+ projection: ChunkProjection,
+}
+
#[derive(Debug)]
struct NarrowedScan {
sourced: SourcedProjection,
@@ -568,19 +645,24 @@ enum CompareSkip {
/// nothing to skip. Only the unfenced overrides join could put rows on one side, which is a
/// divergence with no projection defect behind it.
NoRows,
- /// The authoritative scan was already wide, so `scan_sql` renders both arms identically and
- /// the verdict is `match` by construction.
+ /// The authoritative scan was already wide on every table, so `scan_sql` renders both arms
+ /// identically and the verdict is `match` by construction.
NotProjected,
}
impl CompareSkip {
- /// `None` when the compare is worth running.
- fn of(projection: &ChunkProjection, fold: FoldSummary) -> Option<Self> {
- match (fold.rows, projection) {
- (RowsSeen::None, _) => Some(Self::NoRows),
- (RowsSeen::Some, ChunkProjection::FullColumns) => Some(Self::NotProjected),
- (RowsSeen::Some, ChunkProjection::Projected(_)) => None,
+ /// `None` when the compare is worth running. `projections` holds one per table the chunk read.
+ fn of<'a>(
+ projections: impl IntoIterator<Item = &'a ChunkProjection>,
+ fold: FoldSummary,
+ ) -> Option<Self> {
+ if fold.rows == RowsSeen::None {
+ return Some(Self::NoRows);
}
+ let projected = projections
+ .into_iter()
+ .any(|projection| matches!(projection, ChunkProjection::Projected(_)));
+ (!projected).then_some(Self::NotProjected)
}
const fn as_str(self) -> &'static str {
@@ -654,17 +736,18 @@ impl FoldSummary {
/// Drive the cursor into the accumulator until it is exhausted, cancelled, or fails. Returns rather
/// than metering, so the caller owns the single recording site.
+#[allow(clippy::too_many_arguments)]
async fn fold_cursor(
cursor: &mut RowCursor<EventRow>,
accumulator: &mut ChunkAccumulator,
+ summary: &mut FoldSummary,
domain: &SeedDomain,
team_id: TeamId,
kind: ScanKind,
lease_cancel: &CancellationToken,
shutdown: &CancellationToken,
-) -> Result<FoldSummary, ScanHalt> {
+) -> Result<(), ScanHalt> {
let metered = kind == ScanKind::Behavioral;
- let mut summary = FoldSummary::default();
loop {
let row = tokio::select! {
biased;
@@ -673,7 +756,7 @@ async fn fold_cursor(
row = cursor.next() => row.map_err(ScanError::Cursor)?,
};
let Some(row) = row else {
- return Ok(summary);
+ return Ok(());
};
summary.rows = RowsSeen::Some;
// The per-row series describe the authoritative scan. A diagnostic re-scan walks the same
@@ -695,6 +778,24 @@ async fn fold_cursor(
}
}
+fn check_retention(
+ tables: &[ScanTable],
+ domain: &SeedDomain,
+ day: DayIdx,
+ now_ms: i64,
+) -> Result<(), ScanError> {
+ match tables
+ .iter()
+ .find(|table| table.past_retention(domain, now_ms))
+ {
+ Some(expired) => Err(ScanError::PastRetention {
+ table: expired.as_str(),
+ day,
+ }),
+ None => Ok(()),
+ }
+}
+
/// The scanner's internal stop signal, lifted to a [`Halted`] by `scan_at`: a cancellation cause or
/// a terminal [`ScanError`].
enum ScanHalt {
@@ -729,25 +830,6 @@ fn active_conditions_at(
conditions_active_on(day, day_idx_in_tz(now_ms, tz), conditions)
}
-fn active_event_names(run: &PinnedRun, active: &ActiveConditions) -> EventNameSet {
- EventNameSet::new(
- run.event_names
- .iter()
- .filter(|event_name| {
- run.filters
- .behavioral_by_event_name
- .get(*event_name)
- .is_some_and(|bucket| {
- bucket
- .conditions
- .iter()
- .any(|hash| active.get(hash).is_some())
- })
- })
- .cloned(),
- )
-}
-
fn fold_event(
domain: &SeedDomain,
accumulator: &mut ChunkAccumulator,
@@ -830,6 +912,8 @@ pub enum ScanError {
Cursor(#[source] clickhouse::error::Error),
#[error("aggregating ClickHouse scan row")]
Aggregate(#[from] AggregateError),
+ #[error("day {day} is past the retention of {table}, so the scan could miss rows")]
+ PastRetention { table: &'static str, day: DayIdx },
}
#[cfg(test)]
@@ -844,9 +928,11 @@ mod tests {
use super::super::sql::MAX_GET_QUERY_BYTES;
use super::*;
+ use crate::domain::pinned::test_runs::{run_with, RunLeaf};
use crate::domain::{
- plan_days, BandSpec, BlobSource, Boundary, ClaimEpoch, ColumnPlan, ConditionHash, Lookback,
- PlanCaps, PropertiesOutcome, PropertiesSource, RunId, SChunkMs, ScalarColumn,
+ plan_days, BandSpec, BlobSource, Boundary, ClaimEpoch, ColumnPlan, ConditionHash,
+ EventNameSet, Lookback, PlanCaps, PropertiesOutcome, PropertiesSource, RunId, SChunkMs,
+ ScalarColumn,
};
const HASH: &str = "aaaaaaaaaaaaaaaa";
@@ -979,6 +1065,7 @@ mod tests {
&domain(),
&EventNameSet::new(["$pageview".to_owned()]),
BandSpec::new(0, 1).unwrap(),
+ ScanTable::Events,
) else {
panic!("one event name on a one-day domain is a scan");
};
@@ -1026,7 +1113,8 @@ mod tests {
/// The projection metrics are the only report of what a chunk narrowed to, and a dashboard
/// reads them by label. This pins the three sampling rules the recorder applies, which no test
/// over the projection types themselves can see: a full blob takes no key sample at all, an
- /// unread one takes a `0`, and the outcome label follows the arm.
+ /// unread one takes a `0`, and the outcome label follows the arm. Each series also names the
+ /// table its query scanned, which is how a rollout tells the two tables apart.
///
/// The recorder here is a plain one, not the configured one — which ladder
/// [`PROJECTION_KEYS`] renders under is `observability::metrics`'s question, and answering it
@@ -1077,10 +1165,12 @@ mod tests {
metrics::with_local_recorder(&recorder, || {
scanner.record_narrowing(
TeamId(2),
+ ScanTable::Events,
&narrow(two_keys, &ColumnExactKeys::default(), &none),
);
scanner.record_narrowing(
TeamId(2),
+ ScanTable::Events,
&narrow(
ChunkProjection::FullColumns,
&ColumnExactKeys::default(),
@@ -1089,28 +1179,38 @@ mod tests {
);
scanner.record_narrowing(
TeamId(3),
+ ScanTable::Events,
&narrow(url, &ColumnExactKeys::exact(["$current_url"]), &url_column),
);
+ scanner.record_narrowing(
+ TeamId(2),
+ ScanTable::FlagEvaluations,
+ &narrow(
+ ChunkProjection::FullColumns,
+ &ColumnExactKeys::default(),
+ &none,
+ ),
+ );
});
let rendered = handle.render();
for (team, source) in [(2, "rebuilt_inexact_key"), (2, "whole"), (3, "columns")] {
assert!(
rendered.contains(&format!(
- "{SCAN_PROPERTIES_SOURCE}{{source=\"{source}\",team_id=\"{team}\"}} 1"
+ "{SCAN_PROPERTIES_SOURCE}{{source=\"{source}\",table=\"events\",team_id=\"{team}\"}} 1"
)),
"team {team} did not count its chunk under {source}:\n{rendered}"
);
}
assert!(
rendered.contains(&format!(
- "{SCAN_ROW_FILTER}{{outcome=\"none\",team_id=\"2\"}} 2"
+ "{SCAN_ROW_FILTER}{{outcome=\"none\",table=\"events\",team_id=\"2\"}} 2"
)),
"a chunk without a row filter was not counted:\n{rendered}"
);
assert!(
rendered.contains(&format!(
- "{PROJECTION_KEYS}_sum{{blob=\"properties\",team_id=\"3\"}} 1"
+ "{PROJECTION_KEYS}_sum{{blob=\"properties\",table=\"events\",team_id=\"3\"}} 1"
)),
"a column-backed key was not counted as a read key:\n{rendered}"
);
@@ -1118,7 +1218,7 @@ mod tests {
for outcome in ["projected", "full_columns"] {
assert!(
rendered.contains(&format!(
- "{CHUNKS_PROJECTED}{{outcome=\"{outcome}\",team_id=\"2\"}} 1"
+ "{CHUNKS_PROJECTED}{{outcome=\"{outcome}\",table=\"events\",team_id=\"2\"}} 1"
)),
"{outcome} chunk was not counted under its own label:\n{rendered}"
);
@@ -1126,13 +1226,13 @@ mod tests {
// Two keys read, and one blob read at nothing — the reading the whole change exists for.
assert!(
rendered.contains(&format!(
- "{PROJECTION_KEYS}_sum{{blob=\"properties\",team_id=\"2\"}} 2"
+ "{PROJECTION_KEYS}_sum{{blob=\"properties\",table=\"events\",team_id=\"2\"}} 2"
)),
"the kept-key count is not the number of keys:\n{rendered}"
);
assert!(
rendered.contains(&format!(
- "{PROJECTION_KEYS}_sum{{blob=\"person_properties\",team_id=\"2\"}} 0"
+ "{PROJECTION_KEYS}_sum{{blob=\"person_properties\",table=\"events\",team_id=\"2\"}} 0"
)),
"an unread blob did not record a zero:\n{rendered}"
);
@@ -1141,11 +1241,20 @@ mod tests {
for blob in ["properties", "person_properties"] {
assert!(
rendered.contains(&format!(
- "{PROJECTION_KEYS}_count{{blob=\"{blob}\",team_id=\"2\"}} 1"
+ "{PROJECTION_KEYS}_count{{blob=\"{blob}\",table=\"events\",team_id=\"2\"}} 1"
)),
"{blob} took a sample from the full-columns chunk:\n{rendered}"
);
}
+ // The `flag_evaluations` query's increments carry `table="flag_evaluations"`, so they leave
+ // the events series above unchanged.
+ for series in [
+ format!("{CHUNKS_PROJECTED}{{outcome=\"full_columns\",table=\"flag_evaluations\",team_id=\"2\"}} 1"),
+ format!("{SCAN_PROPERTIES_SOURCE}{{source=\"whole\",table=\"flag_evaluations\",team_id=\"2\"}} 1"),
+ format!("{SCAN_ROW_FILTER}{{outcome=\"none\",table=\"flag_evaluations\",team_id=\"2\"}} 1"),
+ ] {
+ assert!(rendered.contains(&series), "missing {series}:\n{rendered}");
+ }
}
/// The one number that separates a projection defect from the malformed-blob over-count
@@ -1209,30 +1318,43 @@ mod tests {
#[test]
fn the_compare_is_issued_only_where_the_two_arms_can_disagree() {
let projected = ChunkProjection::Projected(ColumnPlan::full());
+ let wide = ChunkProjection::FullColumns;
+ // A chunk that reads two tables projects each on its own, so one wide table must not skip
+ // the compare of the other.
let cases = [
(
fold(RowsSeen::None, 0),
- &projected,
+ vec![&projected],
Some(CompareSkip::NoRows),
),
(
fold(RowsSeen::None, 0),
- &ChunkProjection::FullColumns,
+ vec![&wide],
Some(CompareSkip::NoRows),
),
(
fold(RowsSeen::Some, 0),
- &ChunkProjection::FullColumns,
+ vec![&wide],
Some(CompareSkip::NotProjected),
),
- (fold(RowsSeen::Some, 0), &projected, None),
+ (fold(RowsSeen::Some, 0), vec![&projected], None),
+ (
+ fold(RowsSeen::Some, 0),
+ vec![&wide, &wide],
+ Some(CompareSkip::NotProjected),
+ ),
+ (fold(RowsSeen::Some, 0), vec![&wide, &projected], None),
+ (fold(RowsSeen::Some, 0), vec![&projected, &wide], None),
];
- for (summary, projection, expected) in cases {
+ for (summary, projections, expected) in cases {
assert_eq!(
- CompareSkip::of(projection, summary),
+ CompareSkip::of(projections.iter().copied(), summary),
expected,
- "{summary:?} on {}",
- projection.outcome()
+ "{summary:?} on {:?}",
+ projections
+ .iter()
+ .map(|projection| projection.outcome())
+ .collect::<Vec<_>>()
);
}
}
@@ -1411,4 +1533,346 @@ mod tests {
// window, so the seed correctly has nothing left to contribute.
assert_eq!(admitted_at(108), Vec::<DayIdx>::new());
}
+
+ const PAGEVIEW_HASH: &str = "pageview00000000";
+ const FLAG_CALL_HASH: &str = "flagcall00000000";
+ /// This instant falls on day 101, so a day-1 chunk sits past the `flag_evaluations` TTL.
+ const PAST_TTL_MS: i64 = 101 * 86_400_000;
+
+ fn pageview_leaf() -> RunLeaf<'static> {
+ use crate::domain::row_filter::test_catalog::event_is;
+
+ RunLeaf {
+ cohort_id: 1,
+ hash: PAGEVIEW_HASH,
+ event: "$pageview",
+ body: event_is("$pageview"),
+ }
+ }
+
+ fn mixed_run(window_days: u32) -> PinnedRun {
+ use crate::domain::row_filter::test_catalog::event_and_property;
+
+ run_with(
+ ScanTable::FlagEvaluations,
+ window_days,
+ &[
+ pageview_leaf(),
+ RunLeaf {
+ cohort_id: 1,
+ hash: FLAG_CALL_HASH,
+ event: "$feature_flag_called",
+ body: event_and_property("$feature_flag_called", "$feature_flag", "my-flag"),
+ },
+ ],
+ )
+ }
+
+ async fn scan_day_one(
+ scanner: &ChunkScanner,
+ run: &PinnedRun,
+ clock: fn() -> i64,
+ ) -> Result<ScannedChunk, Halted<ClaimedChunk, ScanError>> {
+ use crate::domain::{AttemptCount, ChunkId, ChunkLease};
+
+ let chunk = ClaimedChunk::new(ChunkSpec {
+ lease: ChunkLease::new(ChunkId(Uuid::nil()), run.run_id, ClaimEpoch(1)),
+ team_id: run.team_id,
+ day: 1,
+ band: BandSpec::new(0, 1).unwrap(),
+ s_chunk: SChunkMs(200_000_000),
+ person_range: None,
+ attempt: AttemptCount::new(1),
+ });
+ let never = CancellationToken::new();
+ scanner
+ .scan_at(
+ chunk,
+ run,
+ &ConditionAnalyses::build(&run.conditions, &run.filters),
+ clock,
+ &never,
+ &never,
+ )
+ .await
+ }
+
+ fn failure_of(result: Result<ScannedChunk, Halted<ClaimedChunk, ScanError>>) -> ScanError {
+ use crate::domain::HaltReason;
+
+ match result {
+ Ok(scanned) => panic!("the chunk scanned {} tiles", scanned.tiles().len()),
+ Err(Halted {
+ reason: HaltReason::Cancelled(cause),
+ ..
+ }) => panic!("the chunk was cancelled: {cause:?}"),
+ Err(Halted {
+ reason: HaltReason::Failed(error),
+ ..
+ }) => error,
+ }
+ }
+
+ /// The `flag_evaluations` query comes second, so a check of the first query alone passes the
+ /// chunk. The refusal comes before any query, so the client needs no server.
+ #[tokio::test]
+ async fn a_chunk_whose_flag_calls_are_past_retention_fails_rather_than_scanning() {
+ let scanner = ChunkScanner::new(
+ ClickHouseClient::new(clickhouse::Client::default(), Default::default()),
+ TeamAllowlist::All,
+ false,
+ );
+ let run = mixed_run(120);
+
+ let error = failure_of(scan_day_one(&scanner, &run, || PAST_TTL_MS).await);
+
+ assert!(
+ matches!(error, ScanError::PastRetention { .. }),
+ "the chunk failed for another reason: {error:?}"
+ );
+ }
+
+ /// The fold reads rows only from a ClickHouse cursor, so only a live server shows which rows
+ /// reach the chunk's tiles.
+ #[cfg(feature = "ch-test-support")]
+ mod live {
+ use std::collections::HashSet;
+ use std::sync::atomic::{AtomicUsize, Ordering};
+
+ use envconfig::Envconfig;
+
+ use super::*;
+ use crate::clickhouse::client::build_client;
+ use crate::config::Config;
+
+ const FLAG_KEY_EXPRESSION: &str =
+ r#"replaceRegexpAll(JSONExtractRaw(properties, '$feature_flag'), '^"|"$', '')"#;
+
+ const FLAG_EVALUATIONS_DDL: &str =
+ include_str!("../../tests/fixtures/flag_evaluations.sql");
+
+ fn connect(database: Option<&str>) -> ClickHouseClient {
+ let mut config =
+ Config::init_from_env().expect("the seeder config falls back to its defaults");
+ if let Some(database) = database {
+ config.clickhouse_database = database.to_owned();
+ }
+ build_client(&config).expect("the default ClickHouse client builds")
+ }
+
+ /// This instant falls on day 2, so a day-1 chunk sits inside a window of a week or more.
+ const NOW_MS: i64 = 200_000_000;
+
+ fn person(id: u128) -> Uuid {
+ Uuid::from_u128(id)
+ }
+
+ fn scanned_of(
+ result: Result<ScannedChunk, Halted<ClaimedChunk, ScanError>>,
+ ) -> ScannedChunk {
+ match result {
+ Ok(scanned) => scanned,
+ failed => panic!("{:?}", failure_of(failed)),
+ }
+ }
+
+ fn tile_counts(scanned: &ScannedChunk) -> Vec<(Uuid, ConditionHash, u32)> {
+ scanned
+ .tiles()
+ .iter()
+ .map(|tile| (tile.person_id(), tile.condition_hash(), tile.count()))
+ .collect()
+ }
+
+ /// The flag call by person 4 in `events` must not count in a run pinned to
+ /// `flag_evaluations`. Each table pair backs `$feature_flag` with a column of its own name,
+ /// so a query that reads the other pair's column fails. The flag call by person 5 holds a
+ /// blob that is not JSON, so its `flag_key` is empty. The row filter admits an empty
+ /// column value, so every scan reads the row. The row matches no condition. A scan that
+ /// builds `properties` from `flag_key` never parses the blob. A scan that selects the
+ /// whole blob fails to parse it and skips the row.
+ async fn mixed_chunk_database(database: &str) -> ClickHouseClient {
+ let events_columns = "uuid UUID, event String, properties String, \
+ timestamp DateTime64(6, 'UTC'), distinct_id String, person_id UUID, \
+ person_properties String, elements_chain String, team_id Int64, \
+ inserted_at Nullable(DateTime64(6, 'UTC')), _timestamp DateTime";
+ let event_row = |uuid: u128, event: &str, properties: &str, person_id: u128| {
+ format!(
+ "(toUUID('{}'), '{event}', '{properties}', toDateTime64(100000, 6, 'UTC'), \
+ 'distinct-{person_id}', toUUID('{}'), '{{}}', '', 2, toDateTime(100000, 'UTC'))",
+ person(uuid),
+ person(person_id),
+ )
+ };
+ let flag_row = |uuid: u128, properties: &str, person_id: u128| {
+ format!(
+ "(toUUID('{}'), '$feature_flag_called', '{properties}', \
+ toDateTime64(100000, 6, 'UTC'), 2, 'distinct-{person_id}', toUUID('{}'), \
+ toDateTime(100000, 'UTC'))",
+ person(uuid),
+ person(person_id),
+ )
+ };
+ let my_flag = r#"{"$feature_flag":"my-flag"}"#;
+ let admin = connect(None);
+ let flag_evaluations_tables = FLAG_EVALUATIONS_DDL
+ .split(';')
+ .map(str::trim)
+ .filter(|statement| !statement.is_empty())
+ .map(|statement| statement.replace("{database}", database));
+ let statements = [
+ format!("DROP DATABASE IF EXISTS {database}"),
+ format!("CREATE DATABASE {database}"),
+ format!(
+ "CREATE TABLE {database}.sharded_events ({events_columns}, \
+ `mat_$feature_flag` String DEFAULT {FLAG_KEY_EXPRESSION}) \
+ ENGINE = MergeTree ORDER BY (team_id, toDate(timestamp), event)"
+ ),
+ format!(
+ "CREATE TABLE {database}.events ({events_columns}, `mat_$feature_flag` String) \
+ ENGINE = Merge('{database}', '^sharded_events$')"
+ ),
+ format!(
+ "CREATE TABLE {database}.person_distinct_id_overrides (\
+ team_id Int64, distinct_id String, person_id UUID, is_deleted Int8, version Int64\
+ ) ENGINE = MergeTree ORDER BY (team_id, distinct_id)"
+ ),
+ ]
+ .into_iter()
+ .chain(flag_evaluations_tables)
+ .chain([
+ format!(
+ "INSERT INTO {database}.sharded_events (uuid, event, properties, timestamp, \
+ distinct_id, person_id, person_properties, elements_chain, team_id, _timestamp) \
+ VALUES {}, {}",
+ event_row(10, "$pageview", "{}", 2),
+ event_row(11, "$feature_flag_called", my_flag, 4),
+ ),
+ format!(
+ "INSERT INTO {database}.sharded_flag_evaluations (uuid, event, properties, \
+ timestamp, team_id, distinct_id, person_id, _timestamp) VALUES {}, {}, {}, {}, {}",
+ flag_row(20, my_flag, 1),
+ flag_row(21, my_flag, 3),
+ flag_row(22, my_flag, 3),
+ flag_row(23, r#"{"$feature_flag":"other-flag"}"#, 1),
+ flag_row(24, "not json", 5),
+ ),
+ ]);
+ for statement in statements {
+ admin
+ .query(&statement)
+ .execute()
+ .await
+ .unwrap_or_else(|error| panic!("{error}\n{statement}"));
+ }
+ connect(Some(database))
+ }
+
+ fn scanner(client: ClickHouseClient, shadow_compare: bool) -> ChunkScanner {
+ ChunkScanner::new(
+ client,
+ TeamAllowlist::Only(HashSet::from([2])),
+ shadow_compare,
+ )
+ }
+
+ /// Tiles sort by person. The expected order interleaves the tables. Two tile sets joined
+ /// end to end would put person 2 out of place whichever table came first.
+ #[tokio::test]
+ async fn a_mixed_chunk_folds_rows_from_both_tables_into_one_tile_set() {
+ let client = mixed_chunk_database("seeder_scanner_mixed_chunk").await;
+ let run = mixed_run(7);
+
+ let scanned = scanned_of(scan_day_one(&scanner(client, false), &run, || NOW_MS).await);
+
+ let flag_call = ConditionHash::parse(FLAG_CALL_HASH).unwrap();
+ let pageview = ConditionHash::parse(PAGEVIEW_HASH).unwrap();
+ assert_eq!(
+ tile_counts(&scanned),
+ vec![
+ (person(1), flag_call, 1),
+ (person(2), pageview, 1),
+ (person(3), flag_call, 2),
+ ]
+ );
+ }
+
+ #[tokio::test]
+ async fn a_flag_evaluations_run_scans_an_events_only_chunk_past_the_ttl() {
+ let client = mixed_chunk_database("seeder_scanner_events_past_ttl").await;
+ let run = run_with(ScanTable::FlagEvaluations, 120, &[pageview_leaf()]);
+
+ let scanned =
+ scanned_of(scan_day_one(&scanner(client, false), &run, || PAST_TTL_MS).await);
+
+ let pageview = ConditionHash::parse(PAGEVIEW_HASH).unwrap();
+ assert_eq!(tile_counts(&scanned), vec![(person(2), pageview, 1)]);
+ }
+
+ #[tokio::test]
+ async fn a_chunk_that_crosses_the_ttl_while_it_scans_fails() {
+ static CALLS: AtomicUsize = AtomicUsize::new(0);
+ fn crosses_the_ttl() -> i64 {
+ match CALLS.fetch_add(1, Ordering::SeqCst) {
+ 0 => NOW_MS,
+ _ => PAST_TTL_MS,
+ }
+ }
+ let recorder = metrics_exporter_prometheus::PrometheusBuilder::new().build_recorder();
+ let handle = recorder.handle();
+ let _recorder = metrics::set_default_local_recorder(&recorder);
+ let client = mixed_chunk_database("seeder_scanner_crosses_ttl").await;
+ let run = mixed_run(120);
+
+ let error =
+ failure_of(scan_day_one(&scanner(client, false), &run, crosses_the_ttl).await);
+
+ assert!(
+ matches!(error, ScanError::PastRetention { .. }),
+ "the chunk failed for another reason: {error:?}"
+ );
+ assert!(
+ handle.render().contains(ROWS_SCANNED),
+ "the chunk failed before its queries read a row"
+ );
+ }
+
+ /// The backfill gate reads this verdict. A compare that re-scanned one table, or read flag
+ /// calls from `events`, would report a diff on every mixed chunk. A compare that sent the
+ /// narrow projection would match as well, because both arms would then send the same
+ /// query. The projected arm reads person 5's flag call through `flag_key` and keeps it.
+ /// The wide arm skips it. The skip count is therefore 1 only when the compare sends the
+ /// wide query.
+ #[tokio::test]
+ async fn the_shadow_compare_of_a_mixed_chunk_rescans_both_tables() {
+ let recorder = metrics_exporter_prometheus::PrometheusBuilder::new().build_recorder();
+ let handle = recorder.handle();
+ let _recorder = metrics::set_default_local_recorder(&recorder);
+ let client = mixed_chunk_database("seeder_scanner_mixed_compare").await;
+ let run = mixed_run(7);
+
+ scanned_of(scan_day_one(&scanner(client, true), &run, || NOW_MS).await);
+ let rendered = handle.render();
+
+ assert!(
+ rendered.contains(&format!(
+ "{SHADOW_COMPARE}{{result=\"match\",team_id=\"2\"}} 1"
+ )),
+ "the mixed chunk's compare did not match:\n{rendered}"
+ );
+ assert!(
+ rendered.contains(&format!(
+ "{SCAN_PROPERTIES_SOURCE}{{source=\"columns\",table=\"flag_evaluations\",team_id=\"2\"}} 1"
+ )),
+ "the projected arm did not read `properties` from `flag_key`, so it parses person \
+ 5's blob too:\n{rendered}"
+ );
+ assert!(
+ rendered.contains(&format!(
+ "{SHADOW_COMPARE_LEGACY_SKIPPED}{{team_id=\"2\"}} 1"
+ )),
+ "the compare did not re-scan the whole blob:\n{rendered}"
+ );
+ }
+ }
}
diff --git a/rust/cohort-seeder/src/clickhouse/sql.rs b/rust/cohort-seeder/src/clickhouse/sql.rs
index 4b418dc4e4de..4f37c3680f1c 100644
--- a/rust/cohort-seeder/src/clickhouse/sql.rs
+++ b/rust/cohort-seeder/src/clickhouse/sql.rs
@@ -9,9 +9,29 @@ use cohort_core::hogvm::analysis::PropertyAlternatives;
use crate::domain::{
BandSpec, BlobSource, ChunkProjection, ColumnBackedKeys, ColumnName, ColumnPlan, EventNameSet,
- MaterializedColumns, ProjectedKeys, PropertiesSource, ScalarColumn, ScanRowFilter, SeedDomain,
+ MaterializedColumns, ProjectedKeys, PropertiesSource, ScalarColumn, ScanRowFilter, ScanTable,
+ SeedDomain,
};
+impl ScanTable {
+ /// The sharded table under the Distributed one. Only its columns carry default expressions.
+ pub(crate) const fn data_table(self) -> &'static str {
+ match self {
+ Self::Events => "sharded_events",
+ Self::FlagEvaluations => "sharded_flag_evaluations",
+ }
+ }
+
+ /// `events.inserted_at` is nullable. `flag_evaluations.inserted_at` is not. The
+ /// `flag_evaluations` minmax index is on the bare column.
+ const fn arrival_sql(self) -> &'static str {
+ match self {
+ Self::Events => "coalesce(e.inserted_at, e._timestamp)",
+ Self::FlagEvaluations => "e.inserted_at",
+ }
+ }
+}
+
/// The crate sends a longer query by POST with `readonly=1`, which the `cohort_seeder` profile's
/// `readonly=2` refuses with code 164, because the seeder sends settings with every query. The
/// profile lives in the infrastructure repository, so re-check this against the profile rather than
@@ -25,6 +45,7 @@ const FETCH_FORMAT_CLAUSE: &str = " FORMAT RowBinary";
/// types, so [`scan_sql`] never re-validates. Fields stay private — the SQL text is the only output.
#[derive(Debug, Clone, PartialEq, Eq)]
pub struct ScanSpec {
+ table: ScanTable,
team_id: TeamId,
day_start_ms: i64,
day_end_ms: i64,
@@ -58,19 +79,22 @@ pub enum ScanPlan {
Scan(ScanSpec),
}
-/// Parse a claimed chunk into a scan plan. Takes the resolved `EventNameSet` by reference (already
-/// sorted and deduplicated), so no set is rebuilt. An empty domain or empty event set is vacuous.
+/// Parse a claimed chunk into a scan plan of `table`, which holds every one of `event_names`. Takes
+/// the resolved `EventNameSet` by reference (already sorted and deduplicated), so no set is rebuilt.
+/// An empty domain or empty event set is vacuous.
pub fn plan_scan(
team_id: TeamId,
domain: &SeedDomain,
event_names: &EventNameSet,
band: BandSpec,
+ table: ScanTable,
) -> ScanPlan {
if domain.is_empty() || event_names.is_empty() {
return ScanPlan::Vacuous;
}
let range = domain.utc_range();
ScanPlan::Scan(ScanSpec {
+ table,
team_id,
day_start_ms: range.start().as_i64(),
day_end_ms: range.end().as_i64(),
@@ -93,8 +117,8 @@ pub fn plan_scan(
/// decoder's positional read never has to know which arm ran.
pub fn scan_sql(spec: &ScanSpec, projection: &ChunkProjection) -> String {
let select_list = match projection {
- ChunkProjection::FullColumns => render_select_list(&ColumnPlan::full()),
- ChunkProjection::Projected(plan) => render_select_list(plan),
+ ChunkProjection::FullColumns => render_select_list(&ColumnPlan::full(), spec.table),
+ ChunkProjection::Projected(plan) => render_select_list(plan, spec.table),
};
let row_filter = spec
.row_filter
@@ -111,8 +135,9 @@ pub fn scan_sql(spec: &ScanSpec, projection: &ChunkProjection) -> String {
};
format!(
- "SELECT {}\nFROM events AS e\nLEFT JOIN (\n SELECT distinct_id, argMax(person_id, version) AS person_id\n FROM person_distinct_id_overrides\n WHERE team_id = {}\n GROUP BY distinct_id\n HAVING argMax(is_deleted, version) = 0\n) AS ov ON e.distinct_id = ov.distinct_id\nWHERE {}{}{}",
+ "SELECT {}\nFROM {} AS e\nLEFT JOIN (\n SELECT distinct_id, argMax(person_id, version) AS person_id\n FROM person_distinct_id_overrides\n WHERE team_id = {}\n GROUP BY distinct_id\n HAVING argMax(is_deleted, version) = 0\n) AS ov ON e.distinct_id = ov.distinct_id\nWHERE {}{}{}",
select_list,
+ spec.table.as_str(),
spec.team_id.0,
chunk_rows_sql(spec),
row_filter,
@@ -125,7 +150,8 @@ pub fn scan_sql(spec: &ScanSpec, projection: &ChunkProjection) -> String {
/// scan can read.
pub fn ambiguous_value_probe_sql(spec: &ScanSpec, columns: &ColumnBackedKeys) -> String {
format!(
- "SELECT 1\nFROM events AS e\nWHERE {}\n AND {}\nLIMIT 1",
+ "SELECT 1\nFROM {} AS e\nWHERE {}\n AND {}\nLIMIT 1",
+ spec.table.as_str(),
chunk_rows_sql(spec),
ambiguous_values_sql(columns),
)
@@ -152,8 +178,13 @@ fn chunk_rows_sql(spec: &ScanSpec) -> String {
.collect::<Vec<_>>()
.join(", ");
format!(
- "e.team_id = {}\n AND e.timestamp >= fromUnixTimestamp64Milli({})\n AND e.timestamp < fromUnixTimestamp64Milli({})\n AND e.event IN ({})\n AND coalesce(e.inserted_at, e._timestamp) < fromUnixTimestamp64Milli({})",
- spec.team_id.0, spec.day_start_ms, spec.day_end_ms, event_names, spec.s_chunk_ms,
+ "e.team_id = {}\n AND e.timestamp >= fromUnixTimestamp64Milli({})\n AND e.timestamp < fromUnixTimestamp64Milli({})\n AND e.event IN ({})\n AND {} < fromUnixTimestamp64Milli({})",
+ spec.team_id.0,
+ spec.day_start_ms,
+ spec.day_end_ms,
+ event_names,
+ spec.table.arrival_sql(),
+ spec.s_chunk_ms,
)
}
@@ -278,7 +309,10 @@ fn clickhouse_identifier(name: &str) -> String {
/// The eight-column select list. `event`, `timestamp`, `distinct_id`, and the override-resolved
/// `person_id` are always selected: the fold places the row in its day by timestamp, the accumulator
/// keys on the person, and the event name decides which conditions run.
-fn render_select_list(plan: &ColumnPlan) -> String {
+///
+/// `flag_evaluations` has no `elements_chain`. A flag call never carries elements, so the live
+/// path also evaluates it with an empty chain.
+fn render_select_list(plan: &ColumnPlan, table: ScanTable) -> String {
format!(
"{}, e.event, {}, toString(e.timestamp) AS timestamp,\n e.distinct_id,\n toString(if(notEmpty(ov.distinct_id), ov.person_id, e.person_id)) AS person_id,\n {}, {}",
match plan.uuid {
@@ -291,9 +325,11 @@ fn render_select_list(plan: &ColumnPlan) -> String {
"person_properties",
&plan.person_properties
),
- match plan.elements_chain {
- ScalarColumn::Keep => "e.elements_chain".to_string(),
- ScalarColumn::Empty => "'' AS elements_chain".to_string(),
+ match (plan.elements_chain, table) {
+ (ScalarColumn::Keep, ScanTable::Events) => "e.elements_chain".to_string(),
+ (ScalarColumn::Empty, _) | (_, ScanTable::FlagEvaluations) => {
+ "'' AS elements_chain".to_string()
+ }
},
)
}
@@ -442,7 +478,7 @@ mod tests {
/// past by matching on a substring the test built the same way the code did.
fn select_list_of(sql: &str) -> &str {
sql.strip_prefix("SELECT ")
- .and_then(|rest| rest.split_once("\nFROM events AS e"))
+ .and_then(|rest| rest.split_once("\nFROM "))
.map(|(select_list, _)| select_list)
.expect("every rendered scan opens with its select list")
}
@@ -455,13 +491,31 @@ mod tests {
SeedDomain::new(1, UTC, SChunkMs(200_000_000)).unwrap()
}
- fn spec(event_names: Vec<String>, band: BandSpec) -> ScanSpec {
- match plan_scan(TeamId(2), &domain(), &EventNameSet::new(event_names), band) {
+ fn spec_of(table: ScanTable, event_names: Vec<String>, band: BandSpec) -> ScanSpec {
+ match plan_scan(
+ TeamId(2),
+ &domain(),
+ &EventNameSet::new(event_names),
+ band,
+ table,
+ ) {
ScanPlan::Scan(spec) => spec,
ScanPlan::Vacuous => panic!("expected a scannable plan"),
}
}
+ fn spec(event_names: Vec<String>, band: BandSpec) -> ScanSpec {
+ spec_of(ScanTable::Events, event_names, band)
+ }
+
+ fn flag_evaluations_spec() -> ScanSpec {
+ spec_of(
+ ScanTable::FlagEvaluations,
+ vec!["$feature_flag_called".to_string()],
+ BandSpec::new(0, 1).unwrap(),
+ )
+ }
+
#[test]
fn unbanded_scan_sql_pins_tenant_time_cutoff_and_override_semantics() {
let spec = spec(
@@ -724,4 +778,45 @@ mod tests {
scan_sql(&spec, &ChunkProjection::Projected(ColumnPlan::full()))
);
}
+
+ #[test]
+ fn unbanded_flag_evaluations_scan_sql_reads_its_own_table_and_arrival_bound() {
+ assert_eq!(
+ full_sql(&flag_evaluations_spec()),
+ "SELECT toString(e.uuid) AS uuid, e.event, e.properties, toString(e.timestamp) AS timestamp,\n e.distinct_id,\n toString(if(notEmpty(ov.distinct_id), ov.person_id, e.person_id)) AS person_id,\n e.person_properties, '' AS elements_chain\nFROM flag_evaluations AS e\nLEFT JOIN (\n SELECT distinct_id, argMax(person_id, version) AS person_id\n FROM person_distinct_id_overrides\n WHERE team_id = 2\n GROUP BY distinct_id\n HAVING argMax(is_deleted, version) = 0\n) AS ov ON e.distinct_id = ov.distinct_id\nWHERE e.team_id = 2\n AND e.timestamp >= fromUnixTimestamp64Milli(86400000)\n AND e.timestamp < fromUnixTimestamp64Milli(172800000)\n AND e.event IN ('$feature_flag_called')\n AND e.inserted_at < fromUnixTimestamp64Milli(200000000)"
+ );
+ }
+
+ /// A scan that named `elements_chain` on `flag_evaluations` would fail every attempt of its
+ /// chunk. A condition that reads elements keeps the column on a projected plan.
+ #[test]
+ fn a_flag_evaluations_scan_selects_an_empty_elements_chain_under_every_projection() {
+ let keeps_elements = ChunkProjection::Projected(ColumnPlan {
+ uuid: ScalarColumn::Empty,
+ elements_chain: ScalarColumn::Keep,
+ properties: PropertiesSource::Blob(BlobSource::Empty),
+ person_properties: BlobSource::Empty,
+ });
+ for projection in [ChunkProjection::FullColumns, keeps_elements] {
+ let sql = scan_sql(&flag_evaluations_spec(), &projection);
+ assert!(
+ select_list_of(&sql).ends_with("'' AS elements_chain"),
+ "{sql}"
+ );
+ assert!(!sql.contains("e.elements_chain"), "{sql}");
+ }
+ }
+
+ /// The probe has to read the rows its scan reads. On `events` it would name a column that table
+ /// lacks, or find no flag call to probe.
+ #[test]
+ fn the_flag_evaluations_probe_reads_its_own_table_and_arrival_bound() {
+ assert_eq!(
+ ambiguous_value_probe_sql(
+ &flag_evaluations_spec(),
+ &backed(&[("$feature_flag", "flag_key")])
+ ),
+ "SELECT 1\nFROM flag_evaluations AS e\nWHERE e.team_id = 2\n AND e.timestamp >= fromUnixTimestamp64Milli(86400000)\n AND e.timestamp < fromUnixTimestamp64Milli(172800000)\n AND e.event IN ('$feature_flag_called')\n AND e.inserted_at < fromUnixTimestamp64Milli(200000000)\n AND trim(BOTH ' ' FROM e.`flag_key`) = 'false'\nLIMIT 1"
+ );
+ }
}
diff --git a/rust/cohort-seeder/src/domain/chunk.rs b/rust/cohort-seeder/src/domain/chunk.rs
index c2394cffbbce..7c4999e902f7 100644
--- a/rust/cohort-seeder/src/domain/chunk.rs
+++ b/rust/cohort-seeder/src/domain/chunk.rs
@@ -205,6 +205,17 @@ impl ScanVolume {
}
}
+impl std::ops::Add for ScanVolume {
+ type Output = Self;
+
+ fn add(self, other: Self) -> Self {
+ Self::new(
+ self.received_bytes + other.received_bytes,
+ self.decoded_bytes + other.decoded_bytes,
+ )
+ }
+}
+
/// Freshly claimed and locked; minted only by `store.claim_next` (or `test_support`).
#[derive(Debug)]
pub struct ClaimedChunk {
diff --git a/rust/cohort-seeder/src/domain/mod.rs b/rust/cohort-seeder/src/domain/mod.rs
index b32175b7bc88..3ccb385ff2aa 100644
--- a/rust/cohort-seeder/src/domain/mod.rs
+++ b/rust/cohort-seeder/src/domain/mod.rs
@@ -2,7 +2,7 @@
//!
//! No module here reaches into PostgreSQL, ClickHouse, or Kafka. The internal
//! dependency order is strictly downward:
-//! `ids` ← {`condition`, `window`} ← `chunk` ← {`plan`, `pinned`, `aggregate`} ←
+//! `ids` ← {`condition`, `window`} ← `chunk` ← `pinned` ← `plan` ← `aggregate` ←
//! {`person_relevance`, `person_analysis`} ← `person`,
//! except that `chunk` reaches back into `person` for the `PersonRange` vocabulary its spec
//! carries. The seed wire contract (`SeedTile`/`PersonSeed` and the ids that ride them) lives in
@@ -75,10 +75,12 @@ pub use person::{
pub use person_analysis::{AlwaysEvaluateReason, PersonAnalysisCensus};
pub use pinned::{
PinnedDropReason, PinnedError, PinnedParticipation, PinnedParticipationState, PinnedRun,
- PinnedRunSnapshot, PinnedWarning, TriggerKind, UncoveredCohort, UncoveredParticipations,
- UncoveredReason, UnknownTriggerKind, ValidatedPinnedRun,
+ PinnedRunSnapshot, PinnedWarning, ScanTable, TriggerKind, UncoveredCohort,
+ UncoveredParticipations, UncoveredReason, UnknownTriggerKind, ValidatedPinnedRun,
+};
+pub use plan::{
+ bands_for_day, conditions_active_on, plan_days, table_scans, ActiveConditions, TableScan,
};
-pub use plan::{bands_for_day, conditions_active_on, plan_days, ActiveConditions};
pub use projection::{
BlobSource, ChunkProjection, ColumnBackedKeys, ColumnExactKeys, ColumnPlan, ColumnUpgrade,
ProjectedKeys, PropertiesOutcome, PropertiesSource, PropertiesSourcing, RebuildReason,
diff --git a/rust/cohort-seeder/src/domain/pinned.rs b/rust/cohort-seeder/src/domain/pinned.rs
index b1e57c5e165f..8c33e9780464 100644
--- a/rust/cohort-seeder/src/domain/pinned.rs
+++ b/rust/cohort-seeder/src/domain/pinned.rs
@@ -5,7 +5,8 @@ use std::collections::{HashMap, HashSet};
use std::fmt;
use std::str::FromStr;
-use chrono_tz::Tz;
+use chrono_tz::{Tz, UTC};
+use cohort_core::day_idx_in_tz;
use cohort_core::eligibility::{CohortEligibility, ExcludedReason};
use cohort_core::filters::tree::{BehavioralLeafConfig, BehavioralValue};
use cohort_core::filters::{CohortId, FilterError, TeamFilters, TeamFiltersBuilder, TeamId};
@@ -18,7 +19,7 @@ use serde_json::Value;
use super::chunk::{ChunkDomainError, ChunkSpec};
use super::condition::{EventNameSet, Lookback, PinnedCondition};
-use super::ids::{ConditionHash, ConditionHashError, RunId, UtcMillis};
+use super::ids::{ConditionHash, ConditionHashError, DayIdx, RunId, UtcMillis};
use super::window::{Boundary, SeedDomain};
#[derive(Debug, Clone, Copy, PartialEq, Eq)]
@@ -58,6 +59,61 @@ impl FromStr for TriggerKind {
#[error("unknown backfill trigger {0:?}")]
pub struct UnknownTriggerKind(pub String);
+/// The only event `flag_evaluations` stores.
+const FLAG_CALLED_EVENT: &str = "$feature_flag_called";
+
+/// Mirrors `FLAG_EVALUATIONS_TTL_DAYS` in `posthog/models/flag_evaluations/sql.py`. The Node fork
+/// step in `fork-flag-evaluations-step.ts` mirrors it too. Change all three together.
+const FLAG_EVALUATIONS_TTL_DAYS: DayIdx = 92;
+
+/// The ClickHouse table a scan reads.
+#[derive(Debug, Clone, Copy, Default, PartialEq, Eq, PartialOrd, Ord, Deserialize)]
+#[serde(try_from = "Value")]
+pub enum ScanTable {
+ /// Also what a payload without `flag_calls_table` reads as.
+ #[default]
+ Events,
+ FlagEvaluations,
+}
+
+impl TryFrom<Value> for ScanTable {
+ type Error = String;
+
+ /// A derived enum would also accept a one-key object such as `{"flag_evaluations": null}`.
+ fn try_from(value: Value) -> Result<Self, Self::Error> {
+ match value.as_str() {
+ Some("events") => Ok(Self::Events),
+ Some("flag_evaluations") => Ok(Self::FlagEvaluations),
+ _ => Err(format!("unknown flag_calls_table {value}")),
+ }
+ }
+}
+
+impl ScanTable {
+ /// The table name, which is also the `table` metric label.
+ pub const fn as_str(self) -> &'static str {
+ match self {
+ Self::Events => "events",
+ Self::FlagEvaluations => "flag_evaluations",
+ }
+ }
+
+ /// Whether the TTL may already have dropped rows of `domain`'s day at `now_ms`. The TTL is
+ /// `toDate(timestamp)` plus [`FLAG_EVALUATIONS_TTL_DAYS`] days, so the day's first rows expire
+ /// that many days after the UTC date of the day's start. A team east of UTC starts its day on the UTC date before. Django pins
+ /// `flag_evaluations` only for windows inside the TTL. A chunk can still run or retry days
+ /// after Django created its run.
+ pub fn past_retention(self, domain: &SeedDomain, now_ms: i64) -> bool {
+ match self {
+ Self::Events => false,
+ Self::FlagEvaluations => {
+ day_idx_in_tz(domain.utc_range().start().as_i64(), UTC) + FLAG_EVALUATIONS_TTL_DAYS
+ <= day_idx_in_tz(now_ms, UTC)
+ }
+ }
+ }
+}
+
#[derive(Debug)]
pub struct PinnedRun {
pub run_id: RunId,
@@ -68,6 +124,10 @@ pub struct PinnedRun {
pub conditions: Vec<PinnedCondition>,
pub event_names: EventNameSet,
pub filters: TeamFilters,
+ /// The table the run reads `$feature_flag_called` from. Django pins it per run, so every chunk
+ /// and every retry reads the same table even if the organization changes where its flag calls
+ /// go.
+ pub flag_calls_table: ScanTable,
}
/// Proven coverable: [`PinnedRun::validate`] refuses a run with an uncovered participation.
@@ -293,6 +353,8 @@ pub enum PinnedError {
struct PinnedPayload {
schema_version: u32,
conditions: Vec<RawPinnedCondition>,
+ #[serde(default)]
+ flag_calls_table: ScanTable,
}
#[derive(Debug, Deserialize)]
@@ -367,6 +429,7 @@ impl PinnedRun {
conditions,
event_names,
filters: participation.into_filters(),
+ flag_calls_table: payload.flag_calls_table,
},
warnings,
})
@@ -385,6 +448,15 @@ impl PinnedRun {
.then_some(first)
}
+ /// The table that holds this run's rows of `event_name`.
+ pub fn table_for(&self, event_name: &str) -> ScanTable {
+ if event_name == FLAG_CALLED_EVENT {
+ self.flag_calls_table
+ } else {
+ ScanTable::Events
+ }
+ }
+
/// The seed domain for a claimed chunk, after proving the chunk belongs to this run/team. Named
/// here (not in the store) so the store never references a scan/ClickHouse type — cycle break.
pub fn domain_for(&self, spec: &ChunkSpec) -> Result<SeedDomain, ChunkDomainError> {
@@ -673,23 +745,34 @@ fn derive_lookback(
}
#[cfg(test)]
-mod tests {
- use super::super::condition_analysis::{ConditionAnalyses, ConditionClass};
- use chrono::NaiveDate;
- use chrono_tz::UTC;
- use cohort_core::day_idx_of_naive_date;
- use serde_json::json;
- use uuid::Uuid;
+pub(crate) mod test_runs {
+ use std::collections::BTreeMap;
- use super::*;
+ use cohort_core::filters::{CohortId, TeamId};
+ use serde_json::{json, Value};
+ use uuid::Uuid;
- const BYTECODE: &[Value] = &[];
+ use super::{
+ PinnedParticipation, PinnedParticipationState, PinnedRun, PinnedRunSnapshot, ScanTable,
+ TriggerKind,
+ };
+ use crate::domain::ids::RunId;
+ use crate::domain::row_filter::test_catalog::event_is;
+
+ /// One `performed_event` condition of a [`run_with`] run. `body` is its bytecode without the
+ /// `_H` header.
+ pub(crate) struct RunLeaf<'a> {
+ pub(crate) cohort_id: i32,
+ pub(crate) hash: &'a str,
+ pub(crate) event: &'a str,
+ pub(crate) body: Vec<Value>,
+ }
- fn bytecode(event: &str) -> Value {
+ pub(crate) fn bytecode(event: &str) -> Value {
json!(["_H", 1, 32, event, 32, "event", 1, 1, 11])
}
- fn condition(
+ pub(crate) fn condition(
cohort_id: i32,
hash: &str,
value: &str,
@@ -712,7 +795,10 @@ mod tests {
})
}
- fn snapshot(pinned: Value, participations: Vec<PinnedParticipation>) -> PinnedRunSnapshot {
+ pub(crate) fn snapshot(
+ pinned: Value,
+ participations: Vec<PinnedParticipation>,
+ ) -> PinnedRunSnapshot {
PinnedRunSnapshot {
run_id: RunId(Uuid::nil()),
team_id: TeamId(2),
@@ -724,6 +810,99 @@ mod tests {
}
}
+ /// A validated run of team 2 in UTC, pinned to `flag_calls_table`, with every leaf over the
+ /// last `window_days` days and every named cohort active.
+ pub(crate) fn run_with(
+ flag_calls_table: ScanTable,
+ window_days: u32,
+ leaves: &[RunLeaf<'_>],
+ ) -> PinnedRun {
+ let mut payload_conditions = Vec::new();
+ let mut values_by_cohort: BTreeMap<i32, Vec<Value>> = BTreeMap::new();
+ for leaf in leaves {
+ let mut raw = condition(
+ leaf.cohort_id,
+ leaf.hash,
+ "performed_event",
+ Some(leaf.event),
+ window_days,
+ );
+ raw["time_value"] = json!(window_days);
+ raw["time_interval"] = json!("day");
+ payload_conditions.push(raw);
+ let mut bytecode = vec![json!("_H"), json!(1)];
+ bytecode.extend(leaf.body.iter().cloned());
+ values_by_cohort
+ .entry(leaf.cohort_id)
+ .or_default()
+ .push(json!({
+ "type": "behavioral",
+ "value": "performed_event",
+ "key": leaf.event,
+ "conditionHash": leaf.hash,
+ "time_value": window_days,
+ "time_interval": "day",
+ "bytecode": bytecode,
+ }));
+ }
+ let participations = values_by_cohort
+ .into_iter()
+ .map(|(cohort_id, values)| PinnedParticipation {
+ cohort_id: CohortId(cohort_id),
+ pinned_filters: json!({ "properties": { "type": "AND", "values": values } }),
+ state: PinnedParticipationState::Active,
+ })
+ .collect();
+ let event_names = leaves.iter().map(|leaf| leaf.event).collect::<Vec<_>>();
+ PinnedRun::validate(snapshot(
+ json!({
+ "schema_version": 1,
+ "flag_calls_table": flag_calls_table.as_str(),
+ "conditions": payload_conditions,
+ "event_names": event_names,
+ }),
+ participations,
+ ))
+ .expect("the fixture run validates")
+ .run
+ }
+
+ /// A validated `events` run carrying one 7-day `performed_event` condition per
+ /// `(cohort, event)` pair. Each hash is the event name padded to 16 characters.
+ pub(crate) fn run_over(conditions: &[(i32, &str)]) -> PinnedRun {
+ let hashes = conditions
+ .iter()
+ .map(|(_, event_name)| format!("{event_name:0<16}"))
+ .collect::<Vec<_>>();
+ let leaves = conditions
+ .iter()
+ .zip(&hashes)
+ .map(|((cohort_id, event), hash)| RunLeaf {
+ cohort_id: *cohort_id,
+ hash,
+ event,
+ body: event_is(event),
+ })
+ .collect::<Vec<_>>();
+ run_with(ScanTable::Events, 7, &leaves)
+ }
+}
+
+#[cfg(test)]
+mod tests {
+ use super::super::condition_analysis::{ConditionAnalyses, ConditionClass};
+ use chrono::NaiveDate;
+ use chrono_tz::Australia::Sydney;
+ use chrono_tz::UTC;
+ use cohort_core::{day_idx_of_naive_date, start_of_day_ms_in_tz};
+ use serde_json::json;
+
+ use super::super::ids::SChunkMs;
+ use super::test_runs::{bytecode, condition, run_over, snapshot};
+ use super::*;
+
+ const BYTECODE: &[Value] = &[];
+
#[test]
fn validation_rejects_unknown_schema_and_malformed_hashes() {
let filters = json!({ "properties": { "type": "AND", "values": [] } });
@@ -774,6 +953,74 @@ mod tests {
);
}
+ fn pinned_with_flag_calls_table(value: Option<Value>) -> PinnedRunSnapshot {
+ let mut pinned = json!({ "schema_version": 1, "conditions": [], "event_names": [] });
+ if let Some(value) = value {
+ pinned["flag_calls_table"] = value;
+ }
+ snapshot(pinned, vec![])
+ }
+
+ #[test]
+ fn the_flag_calls_table_reads_events_when_absent_and_each_known_name_as_itself() {
+ for (value, expected) in [
+ (None, ScanTable::Events),
+ (Some("events"), ScanTable::Events),
+ (Some("flag_evaluations"), ScanTable::FlagEvaluations),
+ ] {
+ let validated =
+ PinnedRun::validate(pinned_with_flag_calls_table(value.map(Value::from)))
+ .unwrap_or_else(|error| panic!("{value:?} failed: {error}"));
+ assert_eq!(validated.run.flag_calls_table, expected, "{value:?}");
+ }
+ }
+
+ /// Scanning `events` for a value the seeder cannot read would give a cohort in a
+ /// flag-evaluations-only organization an empty history that still counts as backfilled.
+ #[test]
+ fn a_flag_calls_table_the_seeder_cannot_read_fails_the_run() {
+ for value in [
+ json!("FLAG_EVALUATIONS"),
+ json!(" events"),
+ json!(""),
+ json!(null),
+ json!(1),
+ json!(["flag_evaluations"]),
+ json!({ "table": "flag_evaluations" }),
+ json!({ "flag_evaluations": null }),
+ ] {
+ let error = PinnedRun::validate(pinned_with_flag_calls_table(Some(value.clone())))
+ .map(|validated| validated.run.flag_calls_table)
+ .expect_err(&format!("{value} validated"));
+ assert!(
+ error.to_string().contains("flag_calls_table"),
+ "{value}: the error does not name the key: {error}"
+ );
+ }
+ }
+
+ #[test]
+ fn a_flag_evaluations_day_is_past_retention_once_the_ttl_reaches_its_first_utc_date() {
+ // Sydney's day 1 starts on 1970-01-01 in UTC, the date the TTL counts its first rows from.
+ for (tz, expires_ms) in [
+ (UTC, start_of_day_ms_in_tz(93, UTC)),
+ (Sydney, start_of_day_ms_in_tz(92, UTC)),
+ ] {
+ let domain = SeedDomain::new(1, tz, SChunkMs(200_000_000)).unwrap();
+ for (now_ms, expired) in [(expires_ms - 1, false), (expires_ms, true)] {
+ assert_eq!(
+ ScanTable::FlagEvaluations.past_retention(&domain, now_ms),
+ expired,
+ "{tz} at {now_ms}"
+ );
+ assert!(
+ !ScanTable::Events.past_retention(&domain, now_ms),
+ "{tz} at {now_ms}"
+ );
+ }
+ }
+ }
+
#[test]
fn lookbacks_come_from_the_frozen_catalog_across_all_participations() {
let hashes = [
@@ -1044,51 +1291,6 @@ mod tests {
);
}
- /// A validated run carrying one 7-day `performed_event` condition per `(cohort, event)` pair,
- /// every named cohort active.
- fn run_over(conditions: &[(i32, &str)]) -> PinnedRun {
- let mut payload_conditions = Vec::new();
- let mut values_by_cohort: HashMap<i32, Vec<Value>> = HashMap::new();
- for (cohort_id, event_name) in conditions {
- let hash = format!("{event_name:0<16}");
- let mut raw = condition(*cohort_id, &hash, "performed_event", Some(event_name), 7);
- raw["time_value"] = json!(7);
- raw["time_interval"] = json!("day");
- payload_conditions.push(raw);
- values_by_cohort.entry(*cohort_id).or_default().push(json!({
- "type": "behavioral",
- "value": "performed_event",
- "key": event_name,
- "conditionHash": hash,
- "time_value": 7,
- "time_interval": "day",
- "bytecode": bytecode(event_name),
- }));
- }
- let participations = values_by_cohort
- .into_iter()
- .map(|(cohort_id, values)| PinnedParticipation {
- cohort_id: CohortId(cohort_id),
- pinned_filters: json!({ "properties": { "type": "AND", "values": values } }),
- state: PinnedParticipationState::Active,
- })
- .collect();
- let event_names = conditions
- .iter()
- .map(|(_, event_name)| *event_name)
- .collect::<Vec<_>>();
- PinnedRun::validate(snapshot(
- json!({
- "schema_version": 1,
- "conditions": payload_conditions,
- "event_names": event_names,
- }),
- participations,
- ))
- .expect("the fixture run validates")
- .run
- }
-
/// `sole_cohort_id` picks the cohort a scan's `log_comment` is charged to, so it must name one
/// only when every surviving condition agrees. Naming a cohort on a run that spans two would
/// charge one cohort for the other's scan cost in `query_log_archive`.
diff --git a/rust/cohort-seeder/src/domain/plan.rs b/rust/cohort-seeder/src/domain/plan.rs
index d671708ee7d0..b412f9f8979d 100644
--- a/rust/cohort-seeder/src/domain/plan.rs
+++ b/rust/cohort-seeder/src/domain/plan.rs
@@ -1,13 +1,15 @@
-//! Domain layer: pure day/band planning — `plan_days`, `conditions_active_on`, `bands_for_day`.
-//! Depends on `condition`, `window`, `ids`, and `cohort-core`.
+//! Domain layer: pure day/band planning (`plan_days`, `conditions_active_on`, `bands_for_day`)
+//! and the split of one chunk's scan by table (`table_scans`). Depends on `pinned`, `condition`,
+//! `window`, `ids`, and `cohort-core`.
-use std::collections::BTreeSet;
+use std::collections::{BTreeMap, BTreeSet};
use std::num::NonZeroU16;
use cohort_core::bucket_tz::window_start_for_now;
-use super::condition::{Lookback, PinnedCondition};
+use super::condition::{EventNameSet, Lookback, PinnedCondition};
use super::ids::{Band, ConditionHash, DayIdx};
+use super::pinned::{PinnedRun, ScanTable};
use super::window::{Boundary, PlanCaps};
#[derive(Debug, Clone, PartialEq, Eq, Default)]
@@ -111,6 +113,62 @@ pub fn bands_for_day(_day: DayIdx, bands_per_day: NonZeroU16) -> Vec<Band> {
.collect()
}
+/// One table's share of a chunk's scan.
+#[derive(Debug, Clone, PartialEq, Eq)]
+pub struct TableScan {
+ pub event_names: EventNameSet,
+ pub active: ActiveConditions,
+}
+
+/// The chunk's active event names and active conditions, grouped by the table that holds each
+/// event. A table appears only when it has an event name to scan.
+pub fn table_scans(run: &PinnedRun, active: &ActiveConditions) -> BTreeMap<ScanTable, TableScan> {
+ let mut event_names = BTreeMap::<ScanTable, Vec<String>>::new();
+ for event_name in active_event_names(run, active) {
+ event_names
+ .entry(run.table_for(&event_name))
+ .or_default()
+ .push(event_name);
+ }
+ event_names
+ .into_iter()
+ .map(|(table, names)| {
+ let conditions = run
+ .conditions
+ .iter()
+ .filter(|condition| {
+ active.contains(&condition.hash)
+ && run.table_for(&condition.event_name) == table
+ })
+ .map(|condition| condition.hash);
+ let scan = TableScan {
+ event_names: EventNameSet::new(names),
+ active: ActiveConditions::new(conditions),
+ };
+ (table, scan)
+ })
+ .collect()
+}
+
+fn active_event_names(run: &PinnedRun, active: &ActiveConditions) -> EventNameSet {
+ EventNameSet::new(
+ run.event_names
+ .iter()
+ .filter(|event_name| {
+ run.filters
+ .behavioral_by_event_name
+ .get(*event_name)
+ .is_some_and(|bucket| {
+ bucket
+ .conditions
+ .iter()
+ .any(|hash| active.get(hash).is_some())
+ })
+ })
+ .cloned(),
+ )
+}
+
#[cfg(test)]
mod tests {
use std::time::Duration;
@@ -120,6 +178,8 @@ mod tests {
use proptest::prelude::*;
use super::super::ids::UtcMillis;
+ use super::super::pinned::test_runs::{run_with, RunLeaf};
+ use super::super::row_filter::test_catalog::event_is;
use super::*;
fn condition(hash: &str, lookback: Lookback) -> PinnedCondition {
@@ -217,6 +277,96 @@ mod tests {
);
}
+ /// Each table's scan narrows to its own conditions, so a condition filed under the wrong table
+ /// widens or narrows the other table's projection and row filter.
+ #[test]
+ fn a_chunk_splits_its_event_names_and_conditions_by_the_table_that_holds_each_event() {
+ const PAGEVIEW: &str = "pageview00000000";
+ const FLAG_CALL: &str = "flagcall00000000";
+ const LOOKALIKE: &str = "lookalike0000000";
+ const UNBUCKETED: &str = "unbucketed000000";
+ let leaves = || {
+ ["$pageview", "$feature_flag_called", "$Feature_Flag_Called"]
+ .into_iter()
+ .zip([PAGEVIEW, FLAG_CALL, LOOKALIKE])
+ .map(|(event, hash)| RunLeaf {
+ cohort_id: 1,
+ hash,
+ event,
+ body: event_is(event),
+ })
+ .collect::<Vec<_>>()
+ };
+ let hashes = |hashes: &[&str]| {
+ hashes
+ .iter()
+ .map(|hash| ConditionHash::parse(hash).unwrap())
+ .collect::<Vec<_>>()
+ };
+ let scan = |names: &[&str], active: &[&str]| TableScan {
+ event_names: EventNameSet::new(names.iter().map(|name| (*name).to_owned())),
+ active: ActiveConditions::new(hashes(active)),
+ };
+ let active_except = |run: &PinnedRun, inactive: &[&str]| {
+ let inactive = hashes(inactive);
+ ActiveConditions::new(
+ run.conditions
+ .iter()
+ .map(|condition| condition.hash)
+ .filter(|hash| !inactive.contains(hash)),
+ )
+ };
+
+ let flags = run_with(ScanTable::FlagEvaluations, 7, &leaves());
+ assert_eq!(
+ table_scans(&flags, &active_except(&flags, &[])),
+ BTreeMap::from([
+ (
+ ScanTable::Events,
+ scan(
+ &["$Feature_Flag_Called", "$pageview"],
+ &[LOOKALIKE, PAGEVIEW]
+ ),
+ ),
+ (
+ ScanTable::FlagEvaluations,
+ scan(&["$feature_flag_called"], &[FLAG_CALL]),
+ ),
+ ])
+ );
+ assert_eq!(
+ table_scans(&flags, &active_except(&flags, &[FLAG_CALL])),
+ BTreeMap::from([(
+ ScanTable::Events,
+ scan(
+ &["$Feature_Flag_Called", "$pageview"],
+ &[LOOKALIKE, PAGEVIEW]
+ ),
+ )]),
+ "a table with no active event name is not scanned"
+ );
+
+ // An `events` run scans one table with every active condition, even one whose event name
+ // no active bucket holds.
+ let mut events = run_with(ScanTable::Events, 7, &leaves());
+ events.conditions.push(PinnedCondition {
+ cohort_id: CohortId(1),
+ hash: ConditionHash::parse(UNBUCKETED).unwrap(),
+ event_name: "unbucketed".to_owned(),
+ lookback: Lookback::SlidingDays(7),
+ });
+ assert_eq!(
+ table_scans(&events, &active_except(&events, &[])),
+ BTreeMap::from([(
+ ScanTable::Events,
+ scan(
+ &["$Feature_Flag_Called", "$feature_flag_called", "$pageview"],
+ &[FLAG_CALL, LOOKALIKE, PAGEVIEW, UNBUCKETED],
+ ),
+ )])
+ );
+ }
+
#[test]
fn active_conditions_expire_sliding_days_and_keep_inclusive_fixed_bounds() {
let sliding = condition("sliding000000000", Lookback::SlidingDays(2));
diff --git a/rust/cohort-seeder/src/observability/metrics.rs b/rust/cohort-seeder/src/observability/metrics.rs
index 1b724df3bd76..a9d362d1607e 100644
--- a/rust/cohort-seeder/src/observability/metrics.rs
+++ b/rust/cohort-seeder/src/observability/metrics.rs
@@ -45,11 +45,12 @@ pub const CONDITIONS_CLASSIFIED: &str = "seeder_conditions_classified_total";
/// `unsupported_op` bucket.
pub const CONDITIONS_UNANALYZABLE: &str = "seeder_conditions_unanalyzable_total";
/// Behavioral chunks by whether their scan narrowed, labelled by `outcome`
-/// (`projected`/`full_columns`) and `team_id` (counter). One increment per scanned chunk.
+/// (`projected`/`full_columns`), `table` (`events`/`flag_evaluations`) and `team_id` (counter). One
+/// increment per table a scanned chunk reads.
///
-/// `full_columns` is the fail-closed arm: at least one condition active on that chunk reads a whole
-/// object or escaped the static analysis, so the scan selects every column. A team sitting at
-/// `full_columns` is the signal to read the run's census for which event names block it.
+/// `full_columns` is the fail-closed arm: at least one condition active on that chunk's table
+/// reads a whole object or escaped the static analysis, so the scan selects every column. A team
+/// sitting at `full_columns` is the signal to read the run's census for which event names block it.
pub const CHUNKS_PROJECTED: &str = "seeder_chunks_projected_total";
/// Shadow-compare verdicts per chunk, labelled by closed `result`
/// (`match`/`diff`/`error`/`no_rows`/`not_projected`) and `team_id` (counter). Emitted only while
@@ -82,19 +83,21 @@ pub const SHADOW_COMPARE_DURATION_SECONDS: &str = "seeder_shadow_compare_duratio
/// over-count `sql::render_blob` documents, which lands in `result="diff"` indistinguishably.
pub const SHADOW_COMPARE_LEGACY_SKIPPED: &str = "seeder_shadow_compare_legacy_skipped_total";
/// Top-level JSON keys a narrowed scan rebuilds one blob from, labelled by `blob`
-/// (`properties`/`person_properties`) and `team_id` (histogram). Zero means the blob is not read at
-/// all and the scan selects an empty literal for it.
+/// (`properties`/`person_properties`), `table` and `team_id` (histogram). Zero means the blob is
+/// not read at all and the scan selects an empty literal for it.
///
/// A blob some condition needs whole takes no sample: there is no key count to report for it, and a
/// sentinel would be indistinguishable from a real reading. Those show up as decoded bytes that do
/// not fall — read this against `seeder_scan_decoded_bytes_total{kind="behavioral"}`.
pub const PROJECTION_KEYS: &str = "seeder_projection_keys";
-/// Behavioral chunks by row filter `outcome` (`none`/`materialized`/`properties_blob`/`too_long`)
-/// and `team_id` (counter). Only `materialized` saves ClickHouse reads; `properties_blob` saves the
-/// transfer and evaluation of rows no condition matches.
+/// Behavioral chunks by row filter `outcome` (`none`/`materialized`/`properties_blob`/`too_long`),
+/// `table` and `team_id` (counter), one increment per table a scanned chunk reads. Only
+/// `materialized` saves ClickHouse reads. `properties_blob` saves the transfer and evaluation of
+/// rows no condition matches.
pub const SCAN_ROW_FILTER: &str = "seeder_scan_row_filter_total";
/// Behavioral chunks by where the scan reads `properties` from, labelled by `source`
-/// (`whole`/`empty`/`columns`/`rebuilt_*`) and `team_id` (counter).
+/// (`whole`/`empty`/`columns`/`rebuilt_*`), `table` and `team_id` (counter), one increment per
+/// table a scanned chunk reads.
pub const SCAN_PROPERTIES_SOURCE: &str = "seeder_scan_properties_source_total";
pub const CHUNKS_PLANNED: &str = "seeder_chunks_planned_total";
pub const CHUNKS_CLAIMED: &str = "seeder_chunks_claimed_total";
diff --git a/rust/cohort-seeder/tests/ch_event_row_filter.rs b/rust/cohort-seeder/tests/ch_event_row_filter.rs
index b0cfbdbd24cc..e7ab9e943598 100644
--- a/rust/cohort-seeder/tests/ch_event_row_filter.rs
+++ b/rust/cohort-seeder/tests/ch_event_row_filter.rs
@@ -1,11 +1,11 @@
//! Checks against a live ClickHouse that a behavioral scan's row filter admits every row the
//! condition's program matches, with the VM as the oracle, that the materialized-column lookup
-//! accepts only columns holding the same values, and that the scan streams events through the
-//! overrides join.
+//! accepts only columns holding the same values on the scanned table, and that the scan streams
+//! events and flag calls through the overrides join.
#![cfg(feature = "ch-test-support")]
-use std::collections::BTreeSet;
+use std::collections::{BTreeMap, BTreeSet};
use chrono_tz::UTC;
use clickhouse::Row;
@@ -14,16 +14,18 @@ use cohort_core::filters::{CohortId, TeamFilters, TeamFiltersBuilder, TeamId};
use cohort_core::hogvm::analysis::GlobalsPlan;
use cohort_core::hogvm::{build_behavioral_globals, evaluate_detailed, EvalOutcome, GlobalsBuild};
use cohort_seeder::clickhouse::client::build_client;
+use cohort_seeder::clickhouse::row::EventRow;
use cohort_seeder::clickhouse::sql::{plan_scan, row_filter_sql, scan_sql, ScanPlan};
use cohort_seeder::clickhouse::ClickHouseClient;
use cohort_seeder::config::Config;
use cohort_seeder::domain::{
- ActiveConditions, BandSpec, ChunkProjection, ColumnName, ConditionAnalyses, ConditionHash,
- EventNameSet, Lookback, MaterializedColumns, PinnedCondition, SChunkMs, SeedDomain,
+ ActiveConditions, BandSpec, ChunkProjection, ConditionAnalyses, ConditionHash, EventNameSet,
+ Lookback, MaterializedColumns, PinnedCondition, SChunkMs, ScanTable, SeedDomain,
};
use envconfig::Envconfig;
use serde::Deserialize;
use serde_json::{json, Value};
+use uuid::Uuid;
const HASH: &str = "aaaaaaaaaaaaaaaa";
const EVENT: &str = "$feature_flag_called";
@@ -141,12 +143,15 @@ async fn the_row_filter_drops_rows_no_program_can_match() {
assert_eq!(admitted, vec![false; dropped.len()], "{predicate}");
}
-/// Only a live server shows how ClickHouse formats the stored default expression.
+/// Only a live server shows how ClickHouse formats the stored default expression. Each lookup
+/// reads only the table pair of its scan. `$cross_to_flags` has its expression on `sharded_events`
+/// and its name on `flag_evaluations`. `$cross_to_events` has the reverse. A lookup that mixed the
+/// two pairs would accept one of them.
#[tokio::test]
-async fn the_lookup_accepts_only_trim_quotes_columns_on_both_tables() {
+async fn the_lookup_accepts_only_trim_quotes_columns_on_both_tables_of_the_scanned_pair() {
const DATABASE: &str = "seeder_materialized_lookup";
let admin = connect(None);
- for statement in [
+ let statements = [
format!("DROP DATABASE IF EXISTS {DATABASE}"),
format!("CREATE DATABASE {DATABASE}"),
format!(
@@ -155,7 +160,8 @@ async fn the_lookup_accepts_only_trim_quotes_columns_on_both_tables() {
`mat_$feature_flag` String DEFAULT replaceRegexpAll(JSONExtractRaw(properties, '$feature_flag'), '^\"|\"$', ''), \
mat_typed String DEFAULT JSONExtract(properties, 'typed', 'String'), \
mat_nullable Nullable(String) DEFAULT JSONExtract(properties, 'nullable', 'Nullable(String)'), \
- `mat_$data_only` String DEFAULT replaceRegexpAll(JSONExtractRaw(properties, '$data_only'), '^\"|\"$', '')\
+ `mat_$data_only` String DEFAULT replaceRegexpAll(JSONExtractRaw(properties, '$data_only'), '^\"|\"$', ''), \
+ cross_to_flags String DEFAULT replaceRegexpAll(JSONExtractRaw(properties, '$cross_to_flags'), '^\"|\"$', '')\
) ENGINE = MergeTree ORDER BY tuple()"
),
format!(
@@ -163,10 +169,21 @@ async fn the_lookup_accepts_only_trim_quotes_columns_on_both_tables() {
properties String, \
`mat_$feature_flag` String COMMENT 'column_materializer::properties::$feature_flag', \
mat_typed String, \
- mat_nullable Nullable(String)\
+ mat_nullable Nullable(String), \
+ cross_to_events String\
) ENGINE = Memory"
),
- ] {
+ ]
+ .into_iter()
+ .chain(flag_evaluations_tables(DATABASE))
+ .chain([
+ format!(
+ "ALTER TABLE {DATABASE}.sharded_flag_evaluations ADD COLUMN cross_to_events String \
+ DEFAULT replaceRegexpAll(JSONExtractRaw(properties, '$cross_to_events'), '^\"|\"$', '')"
+ ),
+ format!("ALTER TABLE {DATABASE}.flag_evaluations ADD COLUMN cross_to_flags String"),
+ ]);
+ for statement in statements {
admin
.query(&statement)
.execute()
@@ -175,16 +192,30 @@ async fn the_lookup_accepts_only_trim_quotes_columns_on_both_tables() {
}
let client = connect(Some(DATABASE));
- let keys = BTreeSet::from(["$feature_flag", "typed", "nullable", "$data_only", "absent"]);
- let columns = MaterializedColumns::lookup(&client, &keys)
- .await
- .expect("the lookup reads only system.columns");
- assert_eq!(
- columns.column_for("$feature_flag").map(ColumnName::as_str),
- Some("mat_$feature_flag")
- );
- for refused in ["typed", "nullable", "$data_only", "absent"] {
- assert_eq!(columns.column_for(refused), None, "{refused}");
+ let keys = BTreeSet::from([
+ "$feature_flag",
+ "$feature_flag_response",
+ "typed",
+ "nullable",
+ "$data_only",
+ "$cross_to_flags",
+ "$cross_to_events",
+ "absent",
+ ]);
+ // `response` holds `$feature_flag_response` with the same expression. Its type is
+ // LowCardinality(String), so the lookup refuses it.
+ for (table, expected) in [
+ (ScanTable::Events, ("$feature_flag", "mat_$feature_flag")),
+ (ScanTable::FlagEvaluations, ("$feature_flag", "flag_key")),
+ ] {
+ let columns = MaterializedColumns::lookup(&client, table, &keys)
+ .await
+ .expect("the lookup reads only system.columns");
+ let found = keys
+ .iter()
+ .filter_map(|key| Some((*key, columns.column_for(key)?.as_str())))
+ .collect::<BTreeMap<_, _>>();
+ assert_eq!(found, BTreeMap::from([expected]), "{table:?}");
}
}
@@ -228,16 +259,7 @@ async fn the_scan_builds_the_join_from_the_overrides_and_streams_the_events() {
.unwrap_or_else(|error| panic!("{error}\n{statement}"));
}
- let domain = SeedDomain::new(1, UTC, SChunkMs(200_000_000)).unwrap();
- let ScanPlan::Scan(spec) = plan_scan(
- TeamId(2),
- &domain,
- &EventNameSet::new(["purchase".to_owned()]),
- BandSpec::new(0, 1).unwrap(),
- ) else {
- panic!("a day with an event name is scannable");
- };
- let scan = scan_sql(&spec, &ChunkProjection::FullColumns);
+ let scan = day_one_scan("purchase", ScanTable::Events);
let client = connect(Some(DATABASE));
assert_eq!(
join_type(
@@ -251,6 +273,121 @@ async fn the_scan_builds_the_join_from_the_overrides_and_streams_the_events() {
assert_eq!(join_type(&client, &scan).await, "LEFT");
}
+/// A row that the 90-day import copied carries `inserted_at = timestamp`, which reads as arrived.
+#[tokio::test]
+async fn the_flag_evaluations_scan_resolves_overrides_and_reads_only_rows_inserted_before_the_claim(
+) {
+ const DATABASE: &str = "seeder_flag_evaluations_scan";
+ let id = |id: u128| Uuid::from_u128(id).to_string();
+ let flag_call = |uuid: u128,
+ distinct_id: &str,
+ person_id: u128,
+ inserted_at_ms: Option<i64>| {
+ let inserted_at = inserted_at_ms
+ .map(|ms| format!(", fromUnixTimestamp64Milli(toInt64({ms}), 'UTC')"))
+ .unwrap_or_default();
+ format!(
+ "(toUUID('{}'), '{EVENT}', '{{\"{KEY}\":\"my-flag\"}}', toDateTime64(100000, 6, 'UTC'), 2, \
+ '{distinct_id}', toUUID('{}'), toDateTime(100000, 'UTC'){inserted_at})",
+ id(uuid),
+ id(person_id),
+ )
+ };
+ let columns = "uuid, event, properties, timestamp, team_id, distinct_id, person_id, _timestamp";
+ let admin = connect(None);
+ let statements = [
+ format!("DROP DATABASE IF EXISTS {DATABASE}"),
+ format!("CREATE DATABASE {DATABASE}"),
+ format!(
+ "CREATE TABLE {DATABASE}.person_distinct_id_overrides (\
+ team_id Int64, distinct_id String, person_id UUID, is_deleted Int8, version Int64\
+ ) ENGINE = MergeTree ORDER BY (team_id, distinct_id)"
+ ),
+ ]
+ .into_iter()
+ .chain(flag_evaluations_tables(DATABASE))
+ .chain([
+ format!(
+ "INSERT INTO {DATABASE}.sharded_flag_evaluations ({columns}, inserted_at) VALUES {}",
+ [
+ flag_call(1, "merged", 101, Some(S_CHUNK_MS - 1)),
+ flag_call(2, "plain", 102, Some(S_CHUNK_MS - 3_600_000)),
+ flag_call(3, "unmerged", 103, Some(100_000_000)),
+ flag_call(5, "plain", 105, Some(S_CHUNK_MS)),
+ flag_call(6, "plain", 106, Some(S_CHUNK_MS + 86_400_000)),
+ ]
+ .join(", ")
+ ),
+ format!(
+ "INSERT INTO {DATABASE}.sharded_flag_evaluations ({columns}) VALUES {}",
+ flag_call(4, "imported", 104, None)
+ ),
+ // `plain` has an override only in team 3. A later version deletes the override of
+ // `unmerged`.
+ format!(
+ "INSERT INTO {DATABASE}.person_distinct_id_overrides VALUES \
+ (2, 'merged', toUUID('{}'), 0, 1), \
+ (2, 'unmerged', toUUID('{}'), 0, 1), \
+ (2, 'unmerged', toUUID('{}'), 1, 2), \
+ (3, 'plain', toUUID('{}'), 0, 1)",
+ id(201),
+ id(202),
+ id(202),
+ id(203),
+ ),
+ ]);
+ for statement in statements {
+ admin
+ .query(&statement)
+ .execute()
+ .await
+ .unwrap_or_else(|error| panic!("{error}\n{statement}"));
+ }
+
+ let scan = day_one_scan(EVENT, ScanTable::FlagEvaluations);
+ let persons = connect(Some(DATABASE))
+ .query(&scan)
+ .fetch_all::<EventRow>()
+ .await
+ .unwrap_or_else(|error| panic!("{error}\n{scan}"))
+ .into_iter()
+ .map(|row| (row.uuid, row.person_id))
+ .collect::<BTreeMap<_, _>>();
+
+ assert_eq!(
+ persons,
+ BTreeMap::from([
+ (id(1), id(201)),
+ (id(2), id(102)),
+ (id(3), id(103)),
+ (id(4), id(104)),
+ ])
+ );
+}
+
+const S_CHUNK_MS: i64 = 200_000_000;
+
+fn day_one_scan(event_name: &str, table: ScanTable) -> String {
+ let ScanPlan::Scan(spec) = plan_scan(
+ TeamId(2),
+ &SeedDomain::new(1, UTC, SChunkMs(S_CHUNK_MS)).unwrap(),
+ &EventNameSet::new([event_name.to_owned()]),
+ BandSpec::new(0, 1).unwrap(),
+ table,
+ ) else {
+ panic!("a day with an event name is scannable");
+ };
+ scan_sql(&spec, &ChunkProjection::FullColumns)
+}
+
+fn flag_evaluations_tables(database: &str) -> impl Iterator<Item = String> + '_ {
+ include_str!("fixtures/flag_evaluations.sql")
+ .split(';')
+ .map(str::trim)
+ .filter(|statement| !statement.is_empty())
+ .map(move |statement| statement.replace("{database}", database))
+}
+
async fn join_type(client: &ClickHouseClient, sql: &str) -> String {
let plan = client
.query(&format!("EXPLAIN actions = 1 {sql}"))
diff --git a/rust/cohort-seeder/tests/fixtures/flag_evaluations.sql b/rust/cohort-seeder/tests/fixtures/flag_evaluations.sql
new file mode 100644
index 000000000000..42891dd51d70
--- /dev/null
+++ b/rust/cohort-seeder/tests/fixtures/flag_evaluations.sql
@@ -0,0 +1,43 @@
+-- The `flag_evaluations` tables of posthog/models/flag_evaluations/sql.py, cut down to the columns
+-- a scan reads plus `flag_key` and `response`. The materialized-column lookup must accept `flag_key`
+-- and refuse `response`, which shares the trim-quotes default but is a LowCardinality(String).
+-- `flag_evaluations` stands in for the Distributed table. It declares the typed columns without
+-- their default expressions, as the Distributed table does.
+--
+-- Nothing checks this file against Django. Re-derive it by hand when the ClickHouse schema of
+-- `flag_evaluations` changes a column listed here.
+--
+-- Callers replace `{database}` with the test database, split the file at each semicolon, and send
+-- each statement as its own query, because ClickHouse takes one statement per query. A comment
+-- here must not hold a semicolon.
+
+CREATE TABLE {database}.sharded_flag_evaluations (
+ uuid UUID,
+ event LowCardinality(String),
+ properties String,
+ timestamp DateTime64(6, 'UTC'),
+ team_id Int64,
+ distinct_id String,
+ person_id UUID,
+ person_properties String DEFAULT '{}',
+ inserted_at DateTime64(6, 'UTC') DEFAULT timestamp,
+ flag_key String DEFAULT replaceRegexpAll(JSONExtractRaw(properties, '$feature_flag'), '^"|"$', ''),
+ response LowCardinality(String) DEFAULT replaceRegexpAll(JSONExtractRaw(properties, '$feature_flag_response'), '^"|"$', ''),
+ _timestamp DateTime
+) ENGINE = ReplacingMergeTree(_timestamp)
+ORDER BY (team_id, flag_key, toDate(timestamp), cityHash64(distinct_id), cityHash64(uuid));
+
+CREATE TABLE {database}.flag_evaluations (
+ uuid UUID,
+ event LowCardinality(String),
+ properties String,
+ timestamp DateTime64(6, 'UTC'),
+ team_id Int64,
+ distinct_id String,
+ person_id UUID,
+ person_properties String,
+ inserted_at DateTime64(6, 'UTC'),
+ flag_key String,
+ response LowCardinality(String),
+ _timestamp DateTime
+) ENGINE = Merge('{database}', '^sharded_flag_evaluations$');