:hedgehog: PostHog is the leading platform for building self-driving products. Our developer tools – AI observability, analytics, session replay, flags, experiments, error tracking, logs, and more – capture all the context agents need to diagnose problems, uncover opportunities, and ship fixes. Steer it all from Slack, web, desktop, or the MCP.
Python ab-testingai-analyticsanalyticscdpdata-warehouseexperimentsfeature-flagsjavascriptproduct-analyticspythonreactsession-replaysurveystypescriptweb-analytics
fix(cohorts): compare booleans with strings as text in realtime cohorts (#129579)
Matheus Batistaf01cb2ee3149
diff --git a/docs/internal/realtime-cohorts/definitions-and-eligibility.md b/docs/internal/realtime-cohorts/definitions-and-eligibility.mdindex 5b53f3661d3d..01f9f82cb280 100644--- a/docs/internal/realtime-cohorts/definitions-and-eligibility.md+++ b/docs/internal/realtime-cohorts/definitions-and-eligibility.md@@ -298,7 +298,7 @@ It goes from an event to the few map entries that matter.| Composable cohorts | LSK → cohorts that contain this leaf | Which cohorts Stage 2 recomposes when the leaf flips || Referencing cohorts | cohort → cohorts that reference it | Cascades || Person conditions, ordered | sorted list of person condition hashes | The person evaluation order |-| Catalog fingerprint | digest of the sorted person condition hashes | Skipping person re-evaluation while nothing changed |+| Catalog fingerprint | digest of the evaluator version and person condition hashes | Skipping person re-evaluation while nothing changed || Eligibility and trees | cohort → class, cohort → parsed tree | Stage 2 composition || Shape hashes | cohort → behavioral and person shape hash | Rejecting reconcile work for a stale definition || Timezone | the team's IANA timezone | Every calendar-day computation |@@ -314,6 +314,10 @@ Leaves are evaluated with the Rust HogVM, configured the same way in every proceEach program is decoded once per catalog build.- The cohort evaluator enables type-coercing comparisons, as the Python and TypeScript HogVMs do, and orders dates by their instant, as ClickHouse does.A string compares as a number only when the other side is a number, and `1 == 1.0` is true.+- A boolean and a string compare as text in `==`, `!=`, `in`, `not in`, `has`, `indexOf` and `in()`: the boolean becomes `"true"` or `"false"`, and the comparison is case-sensitive.+ The batch calculation reads an untyped or String-typed property as its JSON text, so `$feature/<flag> = 'variant'` skips events where the flag was off on both paths.+ It reads a Boolean-typed property through a case-sensitive `transform` of that text, so a stored string `"True"` does not equal the constant `true` on either path.+ A bare read and an ordering of a boolean keep boolean semantics.- A program that fails at runtime, calls an unknown function, or returns a non-boolean counts as `false`.Failures are counted by reason.- An event with malformed `properties` JSON skips only the behavioral side.diff --git a/docs/internal/realtime-cohorts/live-evaluation.md b/docs/internal/realtime-cohorts/live-evaluation.mdindex a6b18b4cbe8a..72f29878f5c4 100644--- a/docs/internal/realtime-cohorts/live-evaluation.md+++ b/docs/internal/realtime-cohorts/live-evaluation.md@@ -132,7 +132,7 @@ Each person has one **person record**.It stores the set of person condition hashes that currently match, plus:- a **properties fingerprint**, a digest of the raw `person_properties` string last evaluated,-- a **catalog fingerprint**, a digest of the team's distinct person condition hashes when last evaluated,+- a **catalog fingerprint**, a digest of the evaluator version and the team's distinct person condition hashes when last evaluated,- a **stamp**, the event time and source offset of the newest event applied.For each event with person properties, the worker makes one of four decisions.@@ -245,6 +245,7 @@ Every change applies lazily, person by person, on their next relevant event, fli- **A composition edit** keeps every leaf's state.A person is recomposed under the new tree when one of their leaves next flips, or, with cascades on, when a cohort it references flips for them.- **A change to the set of distinct person conditions** changes the catalog fingerprint, so every person in the team is re-evaluated on their next event with person properties.+ A new evaluator version does the same for every team at once.A new cohort that reuses a person condition the team already has does not change the fingerprint, so the reused condition makes no transition.Its current matchers enter a single-leaf cohort only through the backfill, and a composed cohort through the backfill or the next flip of one of its other leaves.- **A cohort that is deleted, stops being realtime, or becomes excluded** stops producing output.diff --git a/docs/internal/realtime-cohorts/seeder.md b/docs/internal/realtime-cohorts/seeder.mdindex 884a6b5486b1..43887d63d06a 100644--- a/docs/internal/realtime-cohorts/seeder.md+++ b/docs/internal/realtime-cohorts/seeder.md@@ -380,7 +380,8 @@ The accumulator evaluates a row only against the active conditions of the row'sHogVM equality coerces across types, so the property test is a superset of it.The test reads `replaceRegexpAll(JSONExtractRaw(properties, key), '^"|"$', '')`, the value's raw JSON text without its outer quotes.-It admits the value itself, `true` and `false` (the VM equates a JSON boolean with a string), text holding a backslash (an escaped string), and text starting with `{` (the VM compares a temporal object by epoch).+It admits the value itself, text holding a backslash (an escaped string), and text starting with `{` (the VM compares a temporal object by epoch).+The value itself also covers a JSON boolean, because the cohort VM compares a boolean with a string as its text, `true` or `false`.A value that parses as a number is never filtered on, because the VM also equates it with a JSON number.The test also admits a blob that ClickHouse cannot parse, such as one holding an integer past 64 bits: ClickHouse then extracts '' for every key, while `serde_json` reads the blob.Ingestion writes event properties with `JSON.stringify` of a parsed object, so a key appears once and the first-value and last-value readings of ClickHouse and `serde_json` agree.@@ -411,7 +412,7 @@ concat('{', '"$current_url":', if(isValidJSON(e.`mat_$current_url`), e.`mat_$curThat gives back every value except two kinds.A string whose text is JSON comes back typed (`"5"` becomes `5`), and an absent key comes back as `""`.-HogVM equality gives the same answer for both only when the condition compares the key with a literal that is not empty, does not parse as a number, is not JSON text, and is not `true` or `false` in any case.+HogVM equality gives the same answer for both only when the condition compares the key with a literal that is not empty, does not parse as a number, and is not JSON text.1. **Deciding per key.**The static pass that recognizes row filters also lists the keys each condition reads only through `properties.<key> == <literal>` with such a literal.@@ -419,11 +420,7 @@ HogVM equality gives the same answer for both only when the condition compares t2. **Deciding per chunk.**The scan uses columns only when every projected key qualifies and has a column.One key left on the blob would read the whole blob again.- A column value that reads as `false` can be the boolean or the string `"false"`, and the VM equates the boolean with every literal that is not `true`.- An over-counted tile stays, because the processor applies the larger of the live and seeded counts.- So before it uses columns, the scan asks ClickHouse for one row of the chunk where such a column trims to `false`, which reads only the columns.- If one exists, the chunk keeps the rebuild.- The scan also keeps the rebuild when the column form makes the query too long to send by GET.+ The scan keeps the rebuild when the column form makes the query too long to send by GET.When the column form and the row filter do not both fit, the row filter is dropped, because the columns keep ClickHouse off the blob for every row.The shadow compare's wide arm still reads the blob, so the compare checks the columns against it.@@ -444,8 +441,7 @@ So check `system.parts_columns` for the lookback before a long run: an old parti`seeder_scan_properties_source_total{source,table}` reports, once for each table a scanned chunk reads, where `properties` came from.`columns` and `empty` keep ClickHouse off the blob.-`rebuilt_inexact_key`, `rebuilt_no_column`, `rebuilt_ambiguous_value` and `rebuilt_too_long` name why a chunk kept the rebuild, and `rebuilt_no_column` includes a failed or timed-out lookup.-A chunk that reads a key holding the boolean `false`, such as a flag response, counts under `rebuilt_ambiguous_value`.+`rebuilt_inexact_key`, `rebuilt_no_column` and `rebuilt_too_long` name why a chunk kept the rebuild, and `rebuilt_no_column` includes a failed or timed-out lookup.### ClickHouse settingsdiff --git a/rust/cohort-core/proptest-regressions/hogvm/analysis/row_filter.txt b/rust/cohort-core/proptest-regressions/hogvm/analysis/row_filter.txtnew file mode 100644index 000000000000..e32998471dff--- /dev/null+++ b/rust/cohort-core/proptest-regressions/hogvm/analysis/row_filter.txt@@ -0,0 +1,8 @@+# Seeds for failure cases proptest has generated in the past. It is+# automatically read and these particular cases re-run before any+# novel cases are generated.+#+# It is recommended to check this file in to source control so that+# everyone who runs the test benefits from these saved cases.+cc eb970ca426eb9751f4f6b63c4854b76f1a7268ac291c38b0e156f4097304bdb6 # shrinks to actual = Bool(false), expected = ""+cc 6010d4ad367f4d798a2da4c6ba3547241939a40cd4a08dea3b9d9467626dd9c6 # shrinks to actual = Some(String(" false ")), literal = "my-flag"diff --git a/rust/cohort-core/src/fingerprint.rs b/rust/cohort-core/src/fingerprint.rsindex 9eb33072a0cc..9aedc8e54f35 100644--- a/rust/cohort-core/src/fingerprint.rs+++ b/rust/cohort-core/src/fingerprint.rs@@ -1,6 +1,10 @@use sha2::{Digest, Sha256};-/// 128 bits of SHA-256 over the concatenation of a team's sorted person condition hashes.+/// Raise with any change to `cohort_execution_context` that can flip a verdict, so that every stored+/// person record is evaluated again instead of keeping a verdict of the old rules.+const EVALUATOR_VERSION: u8 = 1;++/// 128 bits of SHA-256 over [`EVALUATOR_VERSION`] and a team's sorted person condition hashes.#[derive(Clone, Copy, Debug, PartialEq, Eq)]pub struct CatalogFingerprint(pub u128);@@ -9,6 +13,7 @@ impl CatalogFingerprint {/// permutation of the same conditions yields one value.pub fn of_sorted(conditions: &[[u8; 16]]) -> Self {let mut hasher = Sha256::new();+ hasher.update([EVALUATOR_VERSION]);for condition in conditions {hasher.update(condition);}diff --git a/rust/cohort-core/src/hogvm/analysis/exactness.rs b/rust/cohort-core/src/hogvm/analysis/exactness.rsindex faa4e1ba2bf9..8aa70268aab7 100644--- a/rust/cohort-core/src/hogvm/analysis/exactness.rs+++ b/rust/cohort-core/src/hogvm/analysis/exactness.rs@@ -17,12 +17,9 @@ impl Exactness {/// The column retypes JSON-looking strings and reads absent keys as `""`, so only a literal/// neither can equal is exact.pub fn of_literal(literal: &str) -> Self {- let lowercase = literal.to_lowercase();let retyped = literal.is_empty()|| Num::from_str(literal).is_ok()- || serde_json::from_str::<Value>(literal).is_ok()- || lowercase == "true"- || lowercase == "false";+ || serde_json::from_str::<Value>(literal).is_ok();if retyped {Self::Inexact} else {diff --git a/rust/cohort-core/src/hogvm/analysis/row_filter.rs b/rust/cohort-core/src/hogvm/analysis/row_filter.rsindex e5a787263387..b3797e835fa8 100644--- a/rust/cohort-core/src/hogvm/analysis/row_filter.rs+++ b/rust/cohort-core/src/hogvm/analysis/row_filter.rs@@ -26,10 +26,10 @@ pub struct EventEqualities {/// A necessary condition for the program to return true: the row's `event` is [`Self::event`], and/// every conjunct has some `(key, value)` with `properties[key] == value` under HogVM equality.///-/// HogVM equality coerces, so besides the string `value` it also holds for any JSON boolean-/// (compared with `value.to_lowercase() == "true"`) and for an object carrying a temporal marker-/// (compared by epoch). A value that parses as a number is never kept, because it would also equal a-/// JSON number. A consumer that renders a row filter must admit all of these.+/// HogVM equality coerces, so besides the string `value` it also holds for the JSON boolean whose+/// text is `value` and for an object carrying a temporal marker (compared by epoch). A value that+/// parses as a number is never kept, because it would also equal a JSON number. A consumer that+/// renders a row filter must admit all of these.#[derive(Debug, Clone, PartialEq, Eq)]pub struct EventRowFilter {pub event: String,@@ -651,7 +651,7 @@ mod tests {AND,vec![event_is("e"),- fold(OR, vec![property_is("k", "a"), property_is("k", "TRUE")]),+ fold(OR, vec![property_is("k", "a"), property_is("k", "true")]),property_is("j", "b"),],),@@ -730,8 +730,7 @@ mod tests {}for literal in [- "", "5", "-1", "1e3", "inf", "true", "TRUE", "False", "null", "{}", "[1]", "\"q\"",- " 5",+ "", "5", "-1", "1e3", "inf", "true", "null", "{}", "[1]", "\"q\"", " 5",] {let body = fold(AND, vec![event_is("e"), property_is("k", literal)]);assert_eq!(exact_keys(body), BTreeSet::new(), "{literal:?}");@@ -760,6 +759,8 @@ mod tests {Just("a\"b\\c\n".to_owned()),Just("é😀".to_owned()),// Exact, but close to retyped text.+ Just("True".to_owned()),+ Just("FALSE".to_owned()),Just("True ".to_owned()),Just("5a".to_owned()),Just("nul".to_owned()),@@ -792,6 +793,7 @@ mod tests {Just("[1]".to_owned()),Just("{}".to_owned()),Just(" 5".to_owned()),+ Just(" false ".to_owned()),"\\PC{0,6}",].prop_map(Value::String),@@ -841,23 +843,19 @@ mod tests {if let EvalOutcome::Matched(true) = evaluate_detailed(&bytecode, globals) {let admitted = match &actual {Value::String(value) => *value == expected,- Value::Bool(_) | Value::Object(_) => true,+ Value::Bool(flag) => flag.to_string() == expected,+ Value::Object(_) => true,Value::Null | Value::Number(_) | Value::Array(_) => false,};prop_assert!(admitted, "the VM matched {actual} against {expected:?}");}}- /// The string `"false"` is excluded: the column returns it as the boolean.#[test]fn an_exact_literal_answers_a_column_rebuilt_value_like_the_stored_one(actual in prop::option::of(property_value()),literal in exact_literal(),) {- let reads_as_false = |text: &str| {- serde_json::from_str::<Value>(text).is_ok_and(|value| value == Value::Bool(false))- };- prop_assume!(!matches!(&actual, Some(Value::String(text)) if reads_as_false(text)));let bytecode = program(fold(AND, vec![event_is("e"), property_is("k", &literal)]));prop_assert_eq!(event_equalities(&bytecode).column_exactness.and_then(|keys| keys.get("k")),diff --git a/rust/cohort-core/src/hogvm/executor.rs b/rust/cohort-core/src/hogvm/executor.rsindex 5a4f5288d389..5e145f3aa24b 100644--- a/rust/cohort-core/src/hogvm/executor.rs+++ b/rust/cohort-core/src/hogvm/executor.rs@@ -34,9 +34,14 @@ const COHORT_HOGVM_MAX_STEPS: usize = 131_072;/// Build the cohort evaluator's [`ExecutionContext`]: coercing comparisons plus the raised/// stack/step ceilings. Both construction sites route through this so production and the parity path/// share identical limits — a large `IN`-list must not overflow in one and succeed in the other.+///+/// Boolean text equality matches legacy, which reads an untyped or String-typed property such as+/// `$feature/<flag>` as text. A change here that can flip a verdict must raise `EVALUATOR_VERSION`+/// in [`crate::fingerprint`].fn cohort_execution_context(program: Program) -> ExecutionContext {ExecutionContext::with_defaults(program).with_coercing_comparisons()+ .with_boolean_text_equality().with_max_stack_depth(COHORT_HOGVM_MAX_STACK_DEPTH).with_max_steps(COHORT_HOGVM_MAX_STEPS)}@@ -257,11 +262,13 @@ mod tests {const OP_GET_GLOBAL: i64 = 1;const OP_CALL_GLOBAL: i64 = 2;const OP_EQ: i64 = 11;+ const OP_NOT_EQ: i64 = 12;const OP_GT: i64 = 13;const OP_LT: i64 = 15;const OP_TRUE: i64 = 29;const OP_FALSE: i64 = 30;const OP_IN: i64 = 21;+ const OP_NOT_IN: i64 = 22;const OP_INTEGER: i64 = 33;const OP_STRING: i64 = 32;// Fixtures terminate explicitly with RETURN, mirroring what the catalog loader stores.@@ -357,6 +364,17 @@ mod tests {assert!(!evaluate(&[], json!({})));}+ fn push_event_property(key: &str) -> Vec<Value> {+ vec![+ json!(OP_STRING),+ json!(key),+ json!(OP_STRING),+ json!("properties"),+ json!(OP_GET_GLOBAL),+ json!(2),+ ]+ }+/// Build opcodes to read `person.properties.<key>`. GET_GLOBAL consumes path segments in/// reverse push order.fn push_person_property(key: &str) -> Vec<Value> {@@ -540,6 +558,90 @@ mod tests {assert!(!evaluate(&bc, globals(json!("05"))));}+ /// The expected answers are legacy ClickHouse's: it reads `$feature/<flag>` as text, and a+ /// Boolean-typed property through a case-sensitive `transform` of that text.+ #[test]+ fn a_boolean_property_equals_a_string_only_as_its_text() {+ let flag = push_event_property("$feature/flag");+ let verdict = |bytecode: &[Value], value: Value| match evaluate_detailed(+ bytecode,+ json!({ "properties": { "$feature/flag": value } }),+ ) {+ EvalOutcome::Matched(matched) => matched,+ other => panic!("{other:?}"),+ };+ let string = |value: &str| vec![json!(OP_STRING), json!(value)];+ let control_or_test = [string("control"), string("test")].concat();+ let tuple = [control_or_test, vec![json!(OP_TUPLE), json!(2)]].concat();+ let true_or_control = [+ string("true"),+ string("control"),+ vec![json!(OP_TUPLE), json!(2)],+ ]+ .concat();+ let true_constant = vec![json!(OP_TRUE)];+ let cases = [+ (+ "off == 'variant'",+ json!(false),+ string("variant"),+ OP_EQ,+ false,+ ),+ (+ "off != 'variant'",+ json!(false),+ string("variant"),+ OP_NOT_EQ,+ true,+ ),+ (+ "off in (control, test)",+ json!(false),+ tuple.clone(),+ OP_IN,+ false,+ ),+ (+ "off not in (control, test)",+ json!(false),+ tuple,+ OP_NOT_IN,+ true,+ ),+ (+ "on in (true, control)",+ json!(true),+ true_or_control,+ OP_IN,+ true,+ ),+ ("on == 'true'", json!(true), string("true"), OP_EQ, true),+ ("on == 'True'", json!(true), string("True"), OP_EQ, false),+ (+ "'true' == true",+ json!("true"),+ true_constant.clone(),+ OP_EQ,+ true,+ ),+ ("'True' == true", json!("True"), true_constant, OP_EQ, false),+ ];+ for (label, value, right, op, expected) in cases {+ // The compiler emits a comparison as `right, left, op`.+ let bytecode = [+ header(),+ right,+ flag.clone(),+ vec![json!(op), json!(OP_RETURN)],+ ]+ .concat();+ assert_eq!(verdict(&bytecode, value), expected, "{label}");+ }+ let bare_read = [header(), flag, vec![json!(OP_RETURN)]].concat();+ assert!(verdict(&bare_read, json!(true)));+ }+/// `header + N×(OP_INTEGER, i) + OP_TUPLE N` — the exact shape a `person.properties.X IN (...)`/// leaf compiles its list to. Building the tuple pushes all N elements before they are popped, so/// peak stack depth ≈ N.diff --git a/rust/cohort-seeder/src/clickhouse/scanner.rs b/rust/cohort-seeder/src/clickhouse/scanner.rsindex 53c2f13730d0..5a231a002602 100644--- a/rust/cohort-seeder/src/clickhouse/scanner.rs+++ b/rust/cohort-seeder/src/clickhouse/scanner.rs@@ -23,17 +23,14 @@ use super::client::ClickHouseClient;use super::log_comment::{ScanLogComment, LOG_COMMENT_OPTION};use super::row::{row_to_event, EventRow};use super::scan_volume::{self, ScanKind};-use super::sql::{- ambiguous_value_probe_sql, fits_client_get, plan_scan, row_filter_sql, scan_sql, ScanPlan,- ScanSpec,-};+use super::sql::{fits_client_get, plan_scan, row_filter_sql, scan_sql, ScanPlan, ScanSpec};use crate::domain::{conditions_active_on, diff_tiles, table_scans, ActiveConditions, AggregateError, CancelCause,ChunkAccumulator, ChunkDomainError, ChunkProjection, ChunkSpec, ClaimedChunk, ColumnExactKeys,- ColumnUpgrade, ConditionAnalyses, DayIdx, Halted, MaterializedColumns, PinnedCondition,- PinnedRun, ProjectedKeys, PropertiesSourcing, RecordOutcome, RecordStats, ScanRowFilter,- ScanTable, ScanVolume, ScannedChunk, SeedDomain, SeedTile, SourcedProjection, TableScan,- TileDiff, UtcMillis,+ ConditionAnalyses, DayIdx, Halted, MaterializedColumns, PinnedCondition, PinnedRun,+ ProjectedKeys, PropertiesSourcing, RecordOutcome, RecordStats, ScanRowFilter, ScanTable,+ ScanVolume, ScannedChunk, SeedDomain, SeedTile, SourcedProjection, TableScan, TileDiff,+ UtcMillis,};use crate::observability::metrics::{team_label, MetricTimer, AGGREGATE_ENTRIES, CHUNKS_PROJECTED, CHUNKS_VACUOUS,@@ -157,17 +154,8 @@ impl ChunkScanner {let mut queries = Vec::with_capacity(planned.len());for (table, scan, scan_spec) in planned {queries.push(- self.narrow_table_query(- run,- analyses,- table,- &scan,- scan_spec,- comment,- lease_cancel,- shutdown,- )- .await?,+ self.narrow_table_query(run, analyses, table, &scan, scan_spec)+ .await,);}@@ -220,7 +208,6 @@ impl ChunkScanner {/// Narrow one table's scan to the conditions its rows can reach: its projection, its row/// filter, and the materialized columns of its table.- #[allow(clippy::too_many_arguments)]async fn narrow_table_query(&self,run: &PinnedRun,@@ -228,29 +215,20 @@ impl ChunkScanner {table: ScanTable,scan: &TableScan,scan_spec: ScanSpec,- comment: ScanLogComment,- lease_cancel: &CancellationToken,- shutdown: &CancellationToken,- ) -> Result<TableQuery, ScanHalt> {+ ) -> TableQuery {let projection = analyses.projection(&scan.active);let exact = analyses.column_exact_keys(&scan.active);let row_filter = analyses.row_filter(&scan.event_names, &run.filters, &scan.active);let columns = self.lookup_columns(table, &column_lookup_keys(&projection, &exact, &row_filter)).await;- let sourcing = match projection.source_properties(&exact, &columns) {- PropertiesSourcing::Upgradable(upgrade) => {- self.probe_ambiguous_values(&scan_spec, upgrade, comment, lease_cancel, shutdown)- .await?- }- decided @ PropertiesSourcing::Decided(_) => decided,- };+ let sourcing = projection.source_properties(&exact, &columns);let narrowed = narrow_scan(scan_spec, sourcing, &row_filter, &columns);self.record_narrowing(run.team_id, table, &narrowed);- Ok(TableQuery {+ TableQuery {spec: narrowed.filtered.spec,projection: narrowed.sourced.into_projection(),- })+ }}/// One rendering of this chunk's scan: build a cursor from [`scan_sql`] for each table, fold@@ -416,39 +394,6 @@ impl ChunkScanner {Ok(())}- /// The column form reads a column value that trims to `false` as the boolean, which the VM- /// equates with every exact literal, while the string `"false"` equals none. Only the blob holds- /// the type, so a chunk with such a value keeps the rebuild.- async fn probe_ambiguous_values(- &self,- spec: &ScanSpec,- upgrade: ColumnUpgrade,- comment: ScanLogComment,- lease_cancel: &CancellationToken,- shutdown: &CancellationToken,- ) -> Result<PropertiesSourcing, ScanHalt> {- let sql = ambiguous_value_probe_sql(spec, upgrade.backed());- // The column form's scan is longer than the probe, so it would not fit either.- if !fits_client_get(&sql) {- return Ok(PropertiesSourcing::Decided(upgrade.decline_too_long()));- }- let probe = self- .client- .query(&sql)- .with_option(LOG_COMMENT_OPTION, comment.to_string())- .fetch_optional::<u8>();- let found = tokio::select! {- biased;- _ = shutdown.cancelled() => return Err(ScanHalt::Cancelled(CancelCause::Shutdown)),- _ = lease_cancel.cancelled() => return Err(ScanHalt::Cancelled(CancelCause::LeaseLost)),- found = probe => found.map_err(ScanError::Probe)?,- };- Ok(match found {- Some(_) => PropertiesSourcing::Decided(upgrade.decline_ambiguous_value()),- None => PropertiesSourcing::Upgradable(upgrade),- })- }-/// A failed or timed-out lookup returns no columns, so the scan reads the blob.async fn lookup_columns(&self, table: ScanTable, keys: &BTreeSet<&str>) -> MaterializedColumns {match tokio::time::timeout(@@ -906,8 +851,6 @@ pub enum ScanError {Domain(#[from] ChunkDomainError),#[error("building ClickHouse scan cursor")]Query(#[source] clickhouse::error::Error),- #[error("probing ClickHouse for ambiguous column values")]- Probe(#[source] clickhouse::error::Error),#[error("streaming ClickHouse scan cursor")]Cursor(#[source] clickhouse::error::Error),#[error("aggregating ClickHouse scan row")]diff --git a/rust/cohort-seeder/src/clickhouse/sql.rs b/rust/cohort-seeder/src/clickhouse/sql.rsindex 4f37c3680f1c..ca2fe96e242c 100644--- a/rust/cohort-seeder/src/clickhouse/sql.rs+++ b/rust/cohort-seeder/src/clickhouse/sql.rs@@ -145,29 +145,6 @@ pub fn scan_sql(spec: &ScanSpec, projection: &ChunkProjection) -> String {)}-/// Finds a row of the chunk where a column the scan would read cannot tell the boolean `false` from-/// the string `"false"`. It leaves out only the row filter and the band, so it reads every row the-/// scan can read.-pub fn ambiguous_value_probe_sql(spec: &ScanSpec, columns: &ColumnBackedKeys) -> String {- format!(- "SELECT 1\nFROM {} AS e\nWHERE {}\n AND {}\nLIMIT 1",- spec.table.as_str(),- chunk_rows_sql(spec),- ambiguous_values_sql(columns),- )-}--/// True where [`columns_object_expr`] would build the boolean `false`. A string keeps its spaces in-/// the column, and JSON reads ` false ` as the boolean. Tabs and line breaks stay escaped.-pub fn ambiguous_values_sql(columns: &ColumnBackedKeys) -> String {- join_terms(- columns.iter().map(|(_, column)| {- format!("trim(BOTH ' ' FROM {}) = 'false'", column_reference(column))- }),- "OR",- )-}-/// One team, one day, the chunk's event names, and only rows inserted before the chunk's claim./// The claim is in the past, so every query over the chunk reads the same rows.fn chunk_rows_sql(spec: &ScanSpec) -> String {@@ -190,9 +167,9 @@ fn chunk_rows_sql(spec: &ScanSpec) -> String {/// Renders a superset of HogVM equality (see [`cohort_core::hogvm::analysis::EventRowFilter`]) over/// the value's raw JSON text without its outer quotes. The string `value` reads as itself, or holds a-/// backslash when JSON escapes one of its characters; a boolean reads as `true` or `false`; an object-/// starts with `{`. Assumes a blob holds each key once, since ClickHouse reads the first value and-/// `serde_json` the last.+/// backslash when JSON escapes one of its characters; a boolean reads as its text, which is `value`+/// whenever the boolean equals it; an object starts with `{`. Assumes a blob holds each key once,+/// since ClickHouse reads the first value and `serde_json` the last.pub fn row_filter_sql(filter: &ScanRowFilter, columns: &MaterializedColumns) -> String {let filtered_events = filter.events()@@ -264,9 +241,6 @@ fn property_value_sql<'a>(),};let literals = values- .into_iter()- .chain(["true", "false"])- .collect::<BTreeSet<_>>().into_iter().map(clickhouse_string_literal).collect::<Vec<_>>()@@ -669,10 +643,6 @@ mod tests {("$current_url", "mat_$current_url"),("$pathname", "mat_$pathname"),]);- assert_eq!(- ambiguous_value_probe_sql(&unbanded_spec(), &columns),- "SELECT 1\nFROM events AS e\nWHERE e.team_id = 2\n AND e.timestamp >= fromUnixTimestamp64Milli(86400000)\n AND e.timestamp < fromUnixTimestamp64Milli(172800000)\n AND e.event IN ('purchase')\n AND coalesce(e.inserted_at, e._timestamp) < fromUnixTimestamp64Milli(200000000)\n AND (trim(BOTH ' ' FROM e.`mat_$current_url`) = 'false' OR trim(BOTH ' ' FROM e.`mat_$pathname`) = 'false')\nLIMIT 1"- );let projection = ChunkProjection::Projected(ColumnPlan {uuid: ScalarColumn::Empty,elements_chain: ScalarColumn::Empty,@@ -746,7 +716,7 @@ mod tests {let rendered = row_filter_sql(&row_filter, &columns);assert_eq!(rendered,- "(e.event NOT IN ('$feature_flag_called', '$pageview') OR (e.event = '$feature_flag_called' AND arrayExists(v -> v IN ('a', 'b', 'false', 'true') OR position(v, '\\\\') > 0 OR startsWith(v, '{') OR v = '', [e.`mat_$feature_flag`])) OR (e.event = '$pageview' AND arrayExists(v -> v IN ('false', 'https://example.com/', 'true') OR position(v, '\\\\') > 0 OR startsWith(v, '{') OR JSONType(e.properties) != 'Object', [replaceRegexpAll(JSONExtractRaw(e.properties, '$current_url'), '^\"|\"$', '')])))"+ "(e.event NOT IN ('$feature_flag_called', '$pageview') OR (e.event = '$feature_flag_called' AND arrayExists(v -> v IN ('a', 'b') OR position(v, '\\\\') > 0 OR startsWith(v, '{') OR v = '', [e.`mat_$feature_flag`])) OR (e.event = '$pageview' AND arrayExists(v -> v IN ('https://example.com/') OR position(v, '\\\\') > 0 OR startsWith(v, '{') OR JSONType(e.properties) != 'Object', [replaceRegexpAll(JSONExtractRaw(e.properties, '$current_url'), '^\"|\"$', '')])))");let banded = spec(event_names.into_vec(), BandSpec::new(1, 2).unwrap());@@ -806,17 +776,4 @@ mod tests {assert!(!sql.contains("e.elements_chain"), "{sql}");}}-- /// The probe has to read the rows its scan reads. On `events` it would name a column that table- /// lacks, or find no flag call to probe.- #[test]- fn the_flag_evaluations_probe_reads_its_own_table_and_arrival_bound() {- assert_eq!(- ambiguous_value_probe_sql(- &flag_evaluations_spec(),- &backed(&[("$feature_flag", "flag_key")])- ),- "SELECT 1\nFROM flag_evaluations AS e\nWHERE e.team_id = 2\n AND e.timestamp >= fromUnixTimestamp64Milli(86400000)\n AND e.timestamp < fromUnixTimestamp64Milli(172800000)\n AND e.event IN ('$feature_flag_called')\n AND e.inserted_at < fromUnixTimestamp64Milli(200000000)\n AND trim(BOTH ' ' FROM e.`flag_key`) = 'false'\nLIMIT 1"- );- }}diff --git a/rust/cohort-seeder/src/domain/condition_analysis.rs b/rust/cohort-seeder/src/domain/condition_analysis.rsindex 8734050a154f..5adf8efdd952 100644--- a/rust/cohort-seeder/src/domain/condition_analysis.rs+++ b/rust/cohort-seeder/src/domain/condition_analysis.rs@@ -834,7 +834,7 @@ mod tests {("inexactplan00000","purchase",- property_equality("plan", "TRUE"),+ property_equality("plan", "true"),),("exacturl00000000",diff --git a/rust/cohort-seeder/src/domain/projection.rs b/rust/cohort-seeder/src/domain/projection.rsindex a9abc1447dd4..34947807c36f 100644--- a/rust/cohort-seeder/src/domain/projection.rs+++ b/rust/cohort-seeder/src/domain/projection.rs@@ -164,9 +164,6 @@ pub enum Unbacked {pub enum RebuildReason {Unbacked(Unbacked),TooLong,- /// A column of the chunk holds a value that reads as `false`. It can be the boolean or the- /// string `"false"`, and the VM answers an exact equality on the two oppositely.- AmbiguousValue,}/// The `source` label of `seeder_scan_properties_source_total`.@@ -187,7 +184,6 @@ impl PropertiesOutcome {Self::Rebuilt(RebuildReason::Unbacked(Unbacked::InexactKey)) => "rebuilt_inexact_key",Self::Rebuilt(RebuildReason::Unbacked(Unbacked::NoColumn)) => "rebuilt_no_column",Self::Rebuilt(RebuildReason::TooLong) => "rebuilt_too_long",- Self::Rebuilt(RebuildReason::AmbiguousValue) => "rebuilt_ambiguous_value",}}}@@ -195,8 +191,7 @@ impl PropertiesOutcome {#[derive(Debug)]pub enum PropertiesSourcing {Decided(SourcedProjection),- /// The caller decides whether the column form fits and whether the chunk's values are- /// unambiguous.+ /// The caller decides whether the column form fits.Upgradable(ColumnUpgrade),}@@ -207,10 +202,6 @@ pub struct ColumnUpgrade {}impl ColumnUpgrade {- pub fn backed(&self) -> &ColumnBackedKeys {- &self.backed- }-pub fn column_form(&self) -> ChunkProjection {ChunkProjection::Projected(ColumnPlan {properties: PropertiesSource::Columns(self.backed.clone()),@@ -226,17 +217,9 @@ impl ColumnUpgrade {}pub fn decline_too_long(self) -> SourcedProjection {- self.decline(RebuildReason::TooLong)- }-- pub fn decline_ambiguous_value(self) -> SourcedProjection {- self.decline(RebuildReason::AmbiguousValue)- }-- fn decline(self, reason: RebuildReason) -> SourcedProjection {SourcedProjection {projection: ChunkProjection::Projected(self.rebuilt),- outcome: PropertiesOutcome::Rebuilt(reason),+ outcome: PropertiesOutcome::Rebuilt(RebuildReason::TooLong),}}}@@ -766,21 +749,14 @@ mod tests {assert_eq!(plan.person_properties, BlobSource::Full);assert_eq!(plan.elements_chain, ScalarColumn::Keep);- for (declined, reason) in [- (- upgrade(both.clone()).decline_too_long(),- RebuildReason::TooLong,- ),+ let declined = upgrade(both.clone()).decline_too_long();+ assert_eq!(+ (declined.outcome(), declined.into_projection()),(- upgrade(both.clone()).decline_ambiguous_value(),- RebuildReason::AmbiguousValue,- ),- ] {- assert_eq!(- (declined.outcome(), declined.into_projection()),- (PropertiesOutcome::Rebuilt(reason), both.clone())- );- }+ PropertiesOutcome::Rebuilt(RebuildReason::TooLong),+ both.clone()+ )+ );let inexact = rebuilt(&["$current_url", "plan"]);assert_eq!(inexact.column_candidates(&exact), None);diff --git a/rust/cohort-seeder/tests/ch_event_row_filter.rs b/rust/cohort-seeder/tests/ch_event_row_filter.rsindex e7ab9e943598..66e34df829a3 100644--- a/rust/cohort-seeder/tests/ch_event_row_filter.rs+++ b/rust/cohort-seeder/tests/ch_event_row_filter.rs@@ -135,6 +135,8 @@ async fn the_row_filter_drops_rows_no_program_can_match() {r#"{"$feature_flag":"another-flag"}"#,r#"{"$feature_flag":"MY-FLAG"}"#,r#"{"$feature_flag":null}"#,+ r#"{"$feature_flag":true}"#,+ r#"{"$feature_flag":false}"#,r#"{"$feature_flag":5}"#,r#"{"$feature_flag":["my-flag"]}"#,r#"{"nested":{"$feature_flag":"my-flag"}}"#,diff --git a/rust/cohort-seeder/tests/ch_materialized_properties.rs b/rust/cohort-seeder/tests/ch_materialized_properties.rsindex f6ccec3d17af..326dbacef1cc 100644--- a/rust/cohort-seeder/tests/ch_materialized_properties.rs+++ b/rust/cohort-seeder/tests/ch_materialized_properties.rs@@ -1,6 +1,5 @@//! Checks against a live ClickHouse, with the VM as the oracle, that `properties` built from-//! trim-quotes columns answers exact equalities like the blob, except for the named residuals and-//! the rows the ambiguity probe sends back to the blob.+//! trim-quotes columns answers exact equalities like the blob, except for the named residuals.#![cfg(feature = "ch-test-support")]@@ -13,7 +12,7 @@ use cohort_core::filters::{CohortId, TeamFilters, TeamFiltersBuilder, TeamId};use cohort_core::hogvm::analysis::GlobalsPlan;use cohort_core::hogvm::{build_behavioral_globals, evaluate_detailed, EvalOutcome, GlobalsBuild};use cohort_seeder::clickhouse::client::build_client;-use cohort_seeder::clickhouse::sql::{ambiguous_values_sql, columns_object_expr};+use cohort_seeder::clickhouse::sql::columns_object_expr;use cohort_seeder::clickhouse::ClickHouseClient;use cohort_seeder::config::Config;use cohort_seeder::domain::{@@ -85,8 +84,6 @@ impl Divergence {#[derive(Debug, Clone, Copy, PartialEq, Eq)]enum Expect {Same,- /// A column reads as `false`, so the scan keeps the blob for the whole chunk.- Ambiguous,Residual(Residual),}@@ -103,13 +100,6 @@ impl Case {}}- fn ambiguous(blob: impl Into<String>) -> Self {- Self {- blob: blob.into(),- expect: Expect::Ambiguous,- }- }-fn residual(blob: impl Into<String>, residual: Residual) -> Self {Self {blob: blob.into(),@@ -170,10 +160,10 @@ fn corpus() -> Vec<Case> {]);}cases.extend([- Case::ambiguous(r#"{"$current_url":false}"#),- Case::ambiguous(r#"{"$current_url":"false"}"#),- Case::ambiguous(r#"{"$current_url":" false "}"#),- Case::ambiguous(json!({ URL: "https://example.com/", ODD: "false" }).to_string()),+ Case::exact(r#"{"$current_url":false}"#),+ Case::exact(r#"{"$current_url":"false"}"#),+ Case::exact(r#"{"$current_url":" false "}"#),+ Case::exact(json!({ URL: "https://example.com/", ODD: "false" }).to_string()),Case::residual(r#"{"$current_url":"elsewhere","$current_url":"https://example.com/"}"#,Residual::RepeatedKey,@@ -195,7 +185,6 @@ fn corpus() -> Vec<Case> {struct Rebuilt {seq: u32,properties: String,- ambiguous: bool,}#[tokio::test]@@ -216,21 +205,6 @@ async fn columns_answer_every_exact_equality_like_the_blob_but_the_named_residualet corpus = corpus();let rebuilt = rebuild_all(&corpus, &columns).await;- assert_eq!(- corpus- .iter()- .zip(&rebuilt)- .filter(|(_, row)| row.ambiguous)- .map(|(case, _)| case.blob.as_str())- .collect::<Vec<_>>(),- corpus- .iter()- .filter(|case| case.expect == Expect::Ambiguous)- .map(|case| case.blob.as_str())- .collect::<Vec<_>>(),- "the probe flagged other rows than those whose columns read as false"- );-for (case,Rebuilt {@@ -260,7 +234,6 @@ async fn columns_answer_every_exact_equality_like_the_blob_but_the_named_residua.iter().zip(&rebuilt).enumerate()- .filter(|(_, (_, row))| !row.ambiguous).filter_map(|(index, (case, row))| {let found = programs.iter()@@ -440,9 +413,8 @@ async fn rebuild_all(corpus: &[Case], columns: &ColumnBackedKeys) -> Vec<Rebuilt.expect("the corpus inserts");let sql = format!(- "SELECT seq, {} AS properties, {} AS ambiguous FROM events AS e ORDER BY seq",+ "SELECT seq, {} AS properties FROM events AS e ORDER BY seq",columns_object_expr(columns),- ambiguous_values_sql(columns),);assert!(!sql.contains("e.properties"),diff --git a/rust/common/hogvm/README.md b/rust/common/hogvm/README.mdindex 1eb11f798dfc..e602e36c84e6 100644--- a/rust/common/hogvm/README.md+++ b/rust/common/hogvm/README.md@@ -8,7 +8,8 @@ program corpus, diffed against committed Node output) and `tests/stl_parity.rs`- **`sync_execute`** — synchronous, boolean/value-returning evaluation. This is what the productionconsumers use (`cymbal` error-tracking rule evaluation, `cohort-stream-processor`), and it is at- parity with the reference VM.+ parity with the reference VM, except under `with_boolean_text_equality`, which the cohort+ evaluator turns on to match ClickHouse.- **`execute_resumable` / `resume`** — the suspendable-coroutine model for CDP hog functions(destinations): on a registered async function (`fetch`, `sleep`, …) the VM suspends and returns aserializable `VmSnapshot`; the host performs the side effect and resumes.diff --git a/rust/common/hogvm/src/context.rs b/rust/common/hogvm/src/context.rsindex efb13d3807c3..a8fd5858c929 100644--- a/rust/common/hogvm/src/context.rs+++ b/rust/common/hogvm/src/context.rs@@ -50,6 +50,7 @@ pub struct ExecutionContext {/// by epoch, and `Eq` compare two temporals by epoch — the ClickHouse/Python-TS-aligned semantics/// the realtime-cohort evaluator needs. Set via [`ExecutionContext::with_coercing_comparisons`].pub(crate) coerce_comparisons: bool,+ pub(crate) boolean_text_equality: bool,// `Arc`-shared so cloning a default context is a refcount bump, not a deep copy; mutators// copy-on-write via `Arc::make_mut`.native_fns: Arc<HashMap<String, NativeFunction>>,@@ -90,6 +91,7 @@ impl ExecutionContext {max_heap_size,max_steps,coerce_comparisons: false,+ boolean_text_equality: false,native_fns: Arc::new(native_fns),symbol_table: Arc::new(flatten_modules(&modules)),async_fns: HashSet::new(),@@ -111,6 +113,7 @@ impl ExecutionContext {max_heap_size: 1024 * 1024,max_steps: 10_000,coerce_comparisons: false,+ boolean_text_equality: false,native_fns: DEFAULT_NATIVE_FNS.clone(),symbol_table: DEFAULT_SYMBOL_TABLE.clone(),async_fns: HashSet::new(),@@ -150,13 +153,21 @@ impl ExecutionContext {}/// Opt into coercing comparison semantics (cross-type ordering coercion + epoch ordering/equality- /// of temporals). Off by default; only the realtime-cohort evaluator opts in, so consumers like- /// `cymbal` keep strict behavior. See [`Self::coerce_comparisons`].+ /// of temporals). Off by default; the realtime-cohort evaluator and the Node shadow binding opt+ /// in, so consumers like `cymbal` keep strict behavior. See [`Self::coerce_comparisons`].pub fn with_coercing_comparisons(mut self) -> Self {self.coerce_comparisons = true;self}+ /// Compare a boolean with a string as the boolean's text, case-sensitively, the way ClickHouse+ /// compares a property it reads as text. Off by default because the Node shadow binding must+ /// match the TS VM, where `false == "variant"` is true.+ pub fn with_boolean_text_equality(mut self) -> Self {+ self.boolean_text_equality = true;+ self+ }+/// Register the global function names that suspend the VM (the reference `asyncFunctions`). Pair/// with [`Self::with_max_async_steps`] to allow at least one suspension, otherwise the VM errors/// the moment it calls one. Accumulates across calls, like [`Self::with_ext_fns`].diff --git a/rust/common/hogvm/src/stl.rs b/rust/common/hogvm/src/stl.rsindex 412aaf425e69..a80484125361 100644--- a/rust/common/hogvm/src/stl.rs+++ b/rust/common/hogvm/src/stl.rs@@ -297,7 +297,7 @@ pub fn stl() -> Vec<(String, NativeFunction)> {match args[0].deref(&vm.heap)? {HogLiteral::Array(vals) => {for val in vals.iter() {- if *needle.equals(val, &vm.heap)?.try_as()? {+ if vm.values_equal(needle, val)? {return Ok(HogLiteral::Boolean(true).into());}}@@ -316,7 +316,7 @@ pub fn stl() -> Vec<(String, NativeFunction)> {match haystack {HogLiteral::Array(vals) => {for (i, val) in vals.iter().enumerate() {- if *needle.equals(val, &vm.heap)?.try_as()? {+ if vm.values_equal(needle, val)? {return Ok((i as i64).saturating_add(1).into());}}@@ -1461,7 +1461,7 @@ pub fn stl() -> Vec<(String, NativeFunction)> {match args[1].deref(&vm.heap)? {HogLiteral::Array(vals) | HogLiteral::Tuple(vals) => {for val in vals.iter() {- if *needle.equals(val, &vm.heap)?.try_as()? {+ if vm.values_equal(needle, val)? {return Ok(HogLiteral::Boolean(true).into());}}diff --git a/rust/common/hogvm/src/values.rs b/rust/common/hogvm/src/values.rsindex 4c87361b8034..9f3c9cdcd356 100644--- a/rust/common/hogvm/src/values.rs+++ b/rust/common/hogvm/src/values.rs@@ -281,8 +281,7 @@ impl HogValue {// `1 == 1.0` is true here because both reference VMs treat all numbers as one domain (see// `PartialEq for Num`). Only the two operands are dereferenced, so a nested array or object// compares by heap reference, which matches the TS bytecode VM. The cohort evaluator's- // temporal epoch-equality is the only opt-in part, and it lives behind the flag in- // `HogVM::eq_op`, not here.+ // opt-ins live in `HogVM::eq_op` and `HogVM::values_equal`, not here.let (lhs, rhs) = (self.deref(heap)?, rhs.deref(heap)?);lhs.equals(rhs)}@@ -541,6 +540,17 @@ pub(crate) fn temporal_seconds_pair(}}+/// How ClickHouse compares a boolean with a property it reads as text.+pub(crate) fn boolean_text_equals(a: &HogLiteral, b: &HogLiteral) -> Option<bool> {+ match (a, b) {+ (HogLiteral::Boolean(flag), HogLiteral::String(text))+ | (HogLiteral::String(text), HogLiteral::Boolean(flag)) => {+ Some(text.as_str() == if *flag { "true" } else { "false" })+ }+ _ => None,+ }+}+/// Ordering comparison (`Gt`/`Lt`/`GtEq`/`LtEq`) for two literals, two concerns in order:////// TWO ENTRY POINTS, and only one of them is opt-in, so weigh both before you edit this. Thediff --git a/rust/common/hogvm/src/vm.rs b/rust/common/hogvm/src/vm.rsindex ec740f94e9a7..8e319a6eadb7 100644--- a/rust/common/hogvm/src/vm.rs+++ b/rust/common/hogvm/src/vm.rs@@ -18,8 +18,9 @@ use crate::{},util::{get_json_nested, like, regex_match},values::{- compare_values, temporal_seconds_pair, Callable, Closure, FromHogLiteral, HogLiteral,- HogMap, HogStr, HogValue, LocalCallable, Num, NumOp, Upvalue, UpvalueCell,+ boolean_text_equals, compare_values, temporal_seconds_pair, Callable, Closure,+ FromHogLiteral, HogLiteral, HogMap, HogStr, HogValue, LocalCallable, Num, NumOp, Upvalue,+ UpvalueCell,},};@@ -330,11 +331,13 @@ impl<'a> HogVM<'a> {}Operation::In => {let (needle, haystack) = (self.pop_stack()?, self.pop_stack()?);- self.push_stack(haystack.contains(&needle, &self.heap)?)?;+ let result = self.in_op(&haystack, &needle)?;+ self.push_stack(result)?;}Operation::NotIn => {let (needle, haystack) = (self.pop_stack()?, self.pop_stack()?);- self.push_stack(haystack.contains(&needle, &self.heap)?.not()?)?;+ let result = self.in_op(&haystack, &needle)?.not()?;+ self.push_stack(result)?;}Operation::Regex => self.regex_op(true, false)?,Operation::NotRegex => self.regex_op(true, true)?,@@ -1030,7 +1033,7 @@ impl<'a> HogVM<'a> {/// `1 == 1.0` holds for every shared-crate consumer (see [`PartialEq for Num`](crate::Num))./// The flag gates only what sits on top: when the context opts into coercing comparisons (the/// realtime-cohort evaluator), a temporal pair compares by epoch seconds to match ClickHouse- /// (`is_date_exact`); every non-temporal pair is unchanged either way.+ /// (`is_date_exact`); every non-temporal pair compares through [`Self::values_equal`].////// "Temporal pair" is [`temporal_seconds_pair`] — the same predicate [`compare_values`] uses for/// ordering, so `timestamp == toDateTime(...)` and `timestamp > toDateTime(...)` agree on which@@ -1044,7 +1047,33 @@ impl<'a> HogVM<'a> {return Ok((x == y).into());}}- a.equals(b, &self.heap)+ Ok(self.values_equal(a, b)?.into())+ }++ /// Shared by `==` and every membership test (`in`, `has`, `indexOf`, `in()`) so that they agree.+ pub(crate) fn values_equal(&self, a: &HogValue, b: &HogValue) -> Result<bool, VmError> {+ if self.context.boolean_text_equality {+ if let Some(equal) = boolean_text_equals(a.deref(&self.heap)?, b.deref(&self.heap)?) {+ return Ok(equal);+ }+ }+ Ok(*a.equals(b, &self.heap)?.try_as::<bool>()?)+ }++ fn in_op(&self, haystack: &HogValue, needle: &HogValue) -> Result<HogLiteral, VmError> {+ if !self.context.boolean_text_equality {+ return haystack.contains(needle, &self.heap);+ }+ let (HogLiteral::Array(items) | HogLiteral::Tuple(items)) = haystack.deref(&self.heap)?+ else {+ return haystack.contains(needle, &self.heap);+ };+ for item in items {+ if self.values_equal(item, needle)? {+ return Ok(true.into());+ }+ }+ Ok(false.into())}// Move a value from the stack onto the heap, replacing it with a reference to the heap value,diff --git a/rust/common/hogvm/tests/js_coercion.rs b/rust/common/hogvm/tests/js_coercion.rsindex 250ea7df127f..123da48fb769 100644--- a/rust/common/hogvm/tests/js_coercion.rs+++ b/rust/common/hogvm/tests/js_coercion.rs@@ -7,11 +7,14 @@ use serde_json::{json, Value};const OP_CALL_GLOBAL: i64 = 2;const OP_MINUS: i64 = 7;+const OP_EQ: i64 = 11;+const OP_FALSE: i64 = 30;const OP_NULL: i64 = 31;const OP_STRING: i64 = 32;const OP_INTEGER: i64 = 33;const OP_RETURN: i64 = 38;const OP_DICT: i64 = 42;+const OP_ARRAY: i64 = 43;const OP_GET_PROPERTY: i64 = 45;fn run(bytecode: Vec<Value>) -> Value {@@ -149,3 +152,67 @@ fn to_unix_timestamp_non_string_zone_is_null() {Value::Null);}++/// The Node shadow binding needs the TS answer by default, and the cohort evaluator opts into+/// ClickHouse's.+#[test]+fn a_boolean_equals_a_string_as_text_only_under_the_opt_in() {+ let control_or_test = vec![+ json!(OP_STRING),+ json!("control"),+ json!(OP_STRING),+ json!("test"),+ json!(OP_ARRAY),+ json!(2),+ ];+ let call = |name: &str| vec![json!(OP_CALL_GLOBAL), json!(name), json!(2)];+ let off = || vec![json!(OP_FALSE)];+ let cases = [+ (+ "==",+ vec![+ json!(OP_STRING),+ json!("variant"),+ json!(OP_FALSE),+ json!(OP_EQ),+ ],+ json!(true),+ json!(false),+ ),+ (+ "has",+ [control_or_test.clone(), off(), call("has")].concat(),+ json!(true),+ json!(false),+ ),+ (+ "indexOf",+ [control_or_test.clone(), off(), call("indexOf")].concat(),+ json!(1),+ json!(0),+ ),+ (+ "in()",+ [off(), control_or_test, call("in")].concat(),+ json!(true),+ json!(false),+ ),+ ];+ for (label, body, by_default, as_text) in cases {+ let context = || {+ let bytecode = [+ vec![json!("_H"), json!(1)],+ body.clone(),+ vec![json!(OP_RETURN)],+ ]+ .concat();+ ExecutionContext::with_defaults(Program::new(bytecode).expect("valid program"))+ .with_coercing_comparisons()+ };+ let result = sync_execute(&context(), false).expect("execution succeeds");+ assert_eq!(result, by_default, "{label}");+ let result = sync_execute(&context().with_boolean_text_equality(), false)+ .expect("execution succeeds");+ assert_eq!(result, as_text, "{label}");+ }+}