Apache DataFusion SQL Query Engine
bench(predicate_eval): add expensive-first and wide-column shapes (#25032)
Adrian Garcia Badaracco2a421405a775
diff --git a/benchmarks/bench.sh b/benchmarks/bench.shindex 419fd5be3a..df33d2aec7 100755--- a/benchmarks/bench.sh+++ b/benchmarks/bench.sh@@ -106,7 +106,7 @@ wide_schema: Small-projection queries on a wide synthetic dataset (10(runs both 'wide' and 'narrow' subgroups: narrow is an internal baseline; the wide-vs-narrow ratio is the signal)predicate_eval: Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how anadaptive predicate-ordering system behaves across them (see https://github.com/apache/datafusion/issues/11262)- (subgroups via BENCH_SUBGROUP: costsel, cost, selectivity, cardinality, width, scale, neutral, correlation, drift)+ (subgroups via BENCH_SUBGROUP: costsel, cost, selectivity, cardinality, width, scale, neutral, correlation, drift, nulls)(toggle a system under test with its native DATAFUSION_* env var; size data with PRED_ROWS, string width with PRED_FILL)parquet_row_filter_skip: Per-RG fully-matched RowFilter skip on Parquet (apache/datafusion#23696); clustered string key + low-selectivityrange filter + pushdown, so most row groups are fully matched and the per-row RowFilter is skipped on them@@ -876,7 +876,11 @@ run_push_down_topk() {# micro-benchmarks where each subgroup is a different predicate pattern, used to# test how an adaptive predicate-ordering system behaves across them (see# https://github.com/apache/datafusion/issues/11262). Data is generated inline-# by the suite's load SQL, so there is no data step.+# by the suite's load SQL, so there is no data step (drift q82 and q83 share 16+# small Parquet files written into sql_benchmarks/predicate_eval/scratch/, which+# is gitignored; q82 reads them with target_partitions=1, so the selectivity flip+# lands halfway through one stream, and q83 with 16, so each stream gets one whole+# file and a fixed profile).## By default the suite measures DataFusion's built-in left-deep AND short-circuit# and sets no engine config of its own. To evaluate a system under test, export@@ -885,7 +889,7 @@ run_push_down_topk() {# DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING=true ./bench.sh run predicate_eval# Suite-specific knobs (string-substituted into the load SQL, not engine config):# BENCH_SUBGROUP run one subgroup (costsel, cost, selectivity, cardinality,-# width, scale, neutral, correlation, drift)+# width, scale, neutral, correlation, drift, nulls)# PRED_ROWS synthetic row count (default 1_000_000; the scale subgroup# overrides this per query)# PRED_FILL filler chars per marker = string-column width knobdiff --git a/benchmarks/sql_benchmarks/README.md b/benchmarks/sql_benchmarks/README.mdindex 1ce1fa488c..b8bf8adede 100644--- a/benchmarks/sql_benchmarks/README.md+++ b/benchmarks/sql_benchmarks/README.md@@ -43,7 +43,7 @@ in the community:| `tpcds` | TPC‑DS queries || `tpch` | TPC‑H queries || `wide_schema` | Small-projection queries on a wide (1024-col, 256-file) synthetic dataset; runs `wide` + `narrow` subgroups for comparison |-| `predicate_eval` | Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an adaptive predicate-ordering system behaves across them ([#11262](https://github.com/apache/datafusion/issues/11262)). Subgroups (`--subgroup`): `costsel`, `cost`, `selectivity`, `cardinality`, `width`, `scale`, `neutral`, `correlation`, `drift`. Configure the system under test through its DataFusion settings. |+| `predicate_eval` | Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an adaptive predicate-ordering system behaves across them ([#11262](https://github.com/apache/datafusion/issues/11262)). Subgroups (`--subgroup`): `costsel`, `cost`, `selectivity`, `cardinality`, `width`, `scale`, `neutral`, `correlation`, `drift`, `nulls`. The suite sets no engine config of its own, so by default it measures DataFusion's built-in left-deep `AND` short-circuit; point it at a system under test by exporting that system's own DataFusion setting (the harness builds its `SessionConfig` with `SessionConfig::from_env`). Every query is a `count(*)`, and the counts are checked in under `predicate_eval/results/`, so `--result-mode validate` also checks that a reordering under test still returns the same rows; the checked-in counts were persisted at the suite defaults (`PRED_ROWS=1000000`, `PRED_FILL=30`), so validation assumes those (the `scale` and `width` subgroups pin their own values per query and validate at any setting). || `parquet_row_filter_skip` | Micro-benchmark for the per-row-group fully-matched RowFilter skip on Parquet scans ([#23696](https://github.com/apache/datafusion/issues/23696)). Subgroups (`--subgroup`): `skip` (clustered key, most row groups fully matched by statistics so the per-row filter is skipped), `control` (scrambled key, no row group is ever fully matched). Size the data with `PRED_ROWS` and the row-group size with `RG_SIZE`. |# Running Benchmarksdiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q30.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q30.benchmarkindex 760ea2ca90..d404a193d4 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q30.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q30.benchmark@@ -1,7 +1,6 @@subgroup cardinalitytemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=cardinality-QPAD=30+QUERY=cardinality/q30DATASET=intsNAME=cardinality_q30_k2diff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q31.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q31.benchmarkindex 74f22715d1..54a25337bd 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q31.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q31.benchmark@@ -1,7 +1,6 @@subgroup cardinalitytemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=cardinality-QPAD=31+QUERY=cardinality/q31DATASET=intsNAME=cardinality_q31_k4diff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q32.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q32.benchmarkindex b6b69c3852..aff0173e3d 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q32.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q32.benchmark@@ -1,7 +1,6 @@subgroup cardinalitytemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=cardinality-QPAD=32+QUERY=cardinality/q32DATASET=intsNAME=cardinality_q32_k8diff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q33.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q33.benchmarkindex 1260e68137..bb1464d200 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q33.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q33.benchmark@@ -1,7 +1,6 @@subgroup cardinalitytemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=cardinality-QPAD=33+QUERY=cardinality/q33DATASET=intsNAME=cardinality_q33_k16diff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q34.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q34.benchmarknew file mode 100644index 0000000000..c1851e34ad--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cardinality/q34.benchmark@@ -0,0 +1,6 @@+subgroup cardinality++template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template+QUERY=cardinality/q34+DATASET=ints_wide+NAME=cardinality_q34_k8_wide64diff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q70.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q70.benchmarkindex ef20f7dc49..2084ddab97 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q70.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q70.benchmark@@ -1,7 +1,6 @@subgroup correlationtemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=correlation-QPAD=70+QUERY=correlation/q70DATASET=corrNAME=correlation_q70_independentdiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q71.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q71.benchmarkindex 8875f6c44e..cd5e63aa95 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q71.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q71.benchmark@@ -1,7 +1,6 @@subgroup correlationtemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=correlation-QPAD=71+QUERY=correlation/q71DATASET=corrNAME=correlation_q71_positivediff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q72.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q72.benchmarkindex 8109f1439a..e1dfc3fdaf 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q72.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q72.benchmark@@ -1,7 +1,6 @@subgroup correlationtemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=correlation-QPAD=72+QUERY=correlation/q72DATASET=corrNAME=correlation_q72_antidiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q73.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q73.benchmarkindex cc3f7bcf54..c9a89b1da8 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q73.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/correlation/q73.benchmark@@ -1,7 +1,6 @@subgroup correlationtemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=correlation-QPAD=73+QUERY=correlation/q73DATASET=corrproxyNAME=correlation_q73_redundant_proxydiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cost/q10.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cost/q10.benchmarkindex 9b864b8594..57c7034573 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cost/q10.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cost/q10.benchmark@@ -1,7 +1,6 @@subgroup costtemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=cost-QPAD=10+QUERY=cost/q10DATASET=mixedNAME=cost_q10_expensive_firstdiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cost/q11.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cost/q11.benchmarkindex 296ea443b3..30398925cb 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cost/q11.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/cost/q11.benchmark@@ -1,7 +1,6 @@subgroup costtemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=cost-QPAD=11+QUERY=cost/q11DATASET=mixedNAME=cost_q11_cheap_firstdiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/costsel/q01.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/costsel/q01.benchmarkindex abedd1d580..123839a2de 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/costsel/q01.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/costsel/q01.benchmark@@ -1,7 +1,6 @@subgroup costseltemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=costsel-QPAD=01+QUERY=costsel/q01DATASET=markersNAME=costsel_q01_regexp_selective_lastdiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/costsel/q02.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/costsel/q02.benchmarkindex f50aab6642..304e67189d 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/costsel/q02.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/costsel/q02.benchmark@@ -1,7 +1,6 @@subgroup costseltemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=costsel-QPAD=02+QUERY=costsel/q02DATASET=markersNAME=costsel_q02_regexp_selective_firstdiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/costsel/q03.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/costsel/q03.benchmarkindex 10c4ce184e..d039e6e28d 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/costsel/q03.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/costsel/q03.benchmark@@ -1,7 +1,6 @@subgroup costseltemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=costsel-QPAD=03+QUERY=costsel/q03DATASET=mixedNAME=costsel_q03_cheap_unselective_then_expensive_selectivediff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/costsel/q04.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/costsel/q04.benchmarknew file mode 100644index 0000000000..811e5d642a--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/costsel/q04.benchmark@@ -0,0 +1,6 @@+subgroup costsel++template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template+QUERY=costsel/q04+DATASET=mixed+NAME=costsel_q04_expensive_selective_then_cheap_unselectivediff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/drift/q80.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/drift/q80.benchmarkindex 970adc53f8..fcbd32b0bf 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/drift/q80.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/drift/q80.benchmark@@ -1,7 +1,6 @@subgroup drifttemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=drift-QPAD=80+QUERY=drift/q80DATASET=driftNAME=drift_q80_a_then_bdiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/drift/q81.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/drift/q81.benchmarkdeleted file mode 100644index 93cde75ffe..0000000000--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/drift/q81.benchmark+++ /dev/null@@ -1,7 +0,0 @@-subgroup drift--template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=drift-QPAD=81-DATASET=drift-NAME=drift_q81_b_then_adiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/drift/q82.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/drift/q82.benchmarknew file mode 100644index 0000000000..8ec68bd5c1--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/drift/q82.benchmark@@ -0,0 +1,10 @@+subgroup drift++template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template+QUERY=drift/q82+DATASET=drift_split+NAME=drift_q82_late_flip++# One stream over the whole directory: the flip lands mid-scan.+init+set datafusion.execution.target_partitions = 1;diff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/drift/q83.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/drift/q83.benchmarknew file mode 100644index 0000000000..be25d56612--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/drift/q83.benchmark@@ -0,0 +1,10 @@+subgroup drift++template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template+QUERY=drift/q83+DATASET=drift_split+NAME=drift_q83_per_partition_skew++# One file per stream: each stream sees a single profile.+init+set datafusion.execution.target_partitions = 16;diff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/neutral/q60.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/neutral/q60.benchmarkindex 039fee622b..5d39993e98 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/neutral/q60.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/neutral/q60.benchmark@@ -1,7 +1,6 @@subgroup neutraltemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=neutral-QPAD=60+QUERY=neutral/q60DATASET=intsNAME=neutral_q60_cheap_uniformdiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/neutral/q61.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/neutral/q61.benchmarkindex edaf89b471..ec66cdfc39 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/neutral/q61.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/neutral/q61.benchmark@@ -1,7 +1,6 @@subgroup neutraltemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=neutral-QPAD=61+QUERY=neutral/q61DATASET=markersNAME=neutral_q61_expensive_uniformdiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/nulls/q90.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/nulls/q90.benchmarknew file mode 100644index 0000000000..4e82661a56--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/nulls/q90.benchmark@@ -0,0 +1,6 @@+subgroup nulls++template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template+QUERY=nulls/q90+DATASET=ints+NAME=nulls_q90_nullable_selective_firstdiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/nulls/q91.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/nulls/q91.benchmarknew file mode 100644index 0000000000..377b139bf0--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/nulls/q91.benchmark@@ -0,0 +1,6 @@+subgroup nulls++template sql_benchmarks/predicate_eval/predicate_eval.benchmark.template+QUERY=nulls/q91+DATASET=ints+NAME=nulls_q91_nullable_selective_lastdiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/scale/q50.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/scale/q50.benchmarkindex 0bef31e14f..1c8f2c148f 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/scale/q50.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/scale/q50.benchmark@@ -1,8 +1,7 @@subgroup scaletemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=scale-QPAD=50+QUERY=costsel/q03DATASET=mixedPRED_ROWS=5000NAME=scale_q50_5kdiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/scale/q51.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/scale/q51.benchmarkindex 8f1315fb11..52bd2a0fd0 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/scale/q51.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/scale/q51.benchmark@@ -1,8 +1,7 @@subgroup scaletemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=scale-QPAD=51+QUERY=costsel/q03DATASET=mixedPRED_ROWS=100000NAME=scale_q51_100kdiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/scale/q52.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/scale/q52.benchmarkindex 7ddbfc19b4..9b37a48627 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/scale/q52.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/scale/q52.benchmark@@ -1,8 +1,7 @@subgroup scaletemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=scale-QPAD=52+QUERY=costsel/q03DATASET=mixedPRED_ROWS=5000000NAME=scale_q52_5mdiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/scale/q53.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/scale/q53.benchmarkindex 6cea5c44a1..9839289113 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/scale/q53.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/scale/q53.benchmark@@ -1,8 +1,7 @@subgroup scaletemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=scale-QPAD=53+QUERY=costsel/q03DATASET=mixedPRED_ROWS=50000000NAME=scale_q53_50mdiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/selectivity/q20.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/selectivity/q20.benchmarkindex 077a62650d..f587dc50d9 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/selectivity/q20.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/selectivity/q20.benchmark@@ -1,7 +1,6 @@subgroup selectivitytemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=selectivity-QPAD=20+QUERY=selectivity/q20DATASET=intsNAME=selectivity_q20_unselective_firstdiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/selectivity/q21.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/selectivity/q21.benchmarkindex 24fc6ef4cd..655b067211 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/selectivity/q21.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/selectivity/q21.benchmark@@ -1,7 +1,6 @@subgroup selectivitytemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=selectivity-QPAD=21+QUERY=selectivity/q21DATASET=intsNAME=selectivity_q21_selective_firstdiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/width/q40.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/width/q40.benchmarkindex df66cf16a3..186fafc40b 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/width/q40.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/width/q40.benchmark@@ -1,8 +1,7 @@subgroup widthtemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=width-QPAD=40+QUERY=costsel/q01DATASET=markersPRED_FILL=2NAME=width_q40_narrowdiff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/width/q41.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/width/q41.benchmarkindex c260dc9985..16967c4d87 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/width/q41.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/width/q41.benchmark@@ -1,8 +1,7 @@subgroup widthtemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=width-QPAD=41+QUERY=costsel/q01DATASET=markersPRED_FILL=30NAME=width_q41_widediff --git a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/width/q42.benchmark b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/width/q42.benchmarkindex 988ff59c70..44ebe3137d 100644--- a/benchmarks/sql_benchmarks/predicate_eval/benchmarks/width/q42.benchmark+++ b/benchmarks/sql_benchmarks/predicate_eval/benchmarks/width/q42.benchmark@@ -1,8 +1,7 @@subgroup widthtemplate sql_benchmarks/predicate_eval/predicate_eval.benchmark.template-SUBGROUP=width-QPAD=42+QUERY=costsel/q01DATASET=markersPRED_FILL=170NAME=width_q42_xwidediff --git a/benchmarks/sql_benchmarks/predicate_eval/load/corr.sql b/benchmarks/sql_benchmarks/predicate_eval/load/corr.sqlindex 2d7ceb73e6..946a999ceb 100644--- a/benchmarks/sql_benchmarks/predicate_eval/load/corr.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/load/corr.sql@@ -1,15 +1,6 @@--- Correlation dataset: a base column plus derived columns that control the--- *conditional* selectivity of one predicate given another (its selectivity--- among the rows that already passed the other).------ x uniform [0,100)--- x_pos = x (perfectly positively correlated: `x<k AND x_pos<k`--- passes ~k%, not the ~k%^2 an independence assumption--- predicts)--- x_anti = 99 - x (anti-correlated: `x<k AND x_anti<k` is empty for k<=50)--- ind independent control column, uniform [0,100)------ PRED_ROWS sizes the table.+-- Correlation dataset: `x` uniform on [0,100); `x_pos` = x (perfectly positively+-- correlated); `x_anti` = 99 - x (anti-correlated); `ind` uniform on [0,100) and+-- independent of x. PRED_ROWS sizes the table.CREATE TABLE t ASSELECT(value * 7) % 100 AS x,diff --git a/benchmarks/sql_benchmarks/predicate_eval/load/corrproxy.sql b/benchmarks/sql_benchmarks/predicate_eval/load/corrproxy.sqlindex f06e68d38c..e17d0d4925 100644--- a/benchmarks/sql_benchmarks/predicate_eval/load/corrproxy.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/load/corrproxy.sql@@ -1,30 +1,20 @@--- Correlated-proxy dataset: a cheap integer predicate that is a perfect proxy--- for three string predicates, plus one independent string predicate.---+-- Correlated-proxy dataset:-- c0 = 1 for ~30% of rows (cheap proxy)--- s1, s2, s3 each contain a marker exactly where c0 = 1 (correlated)--- s4 contains a marker for an independent ~30% (independent)------ The four string columns are deliberately *identical in shape*: same width,--- the same single marker at the same offset, each matched by an equally cheap--- regex with the same ~30% marginal selectivity. Marginally the four regex--- predicates are therefore indistinguishable -- same cost, same selectivity, in--- every position -- so neither a marginal cost/selectivity estimator nor--- runtime timing can prefer one over another. Only their *conditional*--- behaviour behind the proxy differs: after `c0 = 1`, the s1/s2/s3 regexes keep--- every survivor (each re-tests the proxy's condition) while the s4 regex still--- discards ~70%. Only joint statistics can see that; an independence assumption--- prices all four regexes identically in every position.+-- s1, s2, s3 each carry a marker exactly where c0 = 1 (correlated)+-- s4 carries a marker for an independent ~30% (independent)+-- PRED_FILL sets the filler width each side of the marker, PRED_ROWS the row count.----- PRED_FILL sets the filler width on each side of the marker (a non-matching--- `regexp_like` must scan the whole value), and PRED_ROWS sizes the table.+-- The four string columns are identical in shape -- same width, same marker+-- offset, same regex cost, same ~30% marginal selectivity -- so marginally they+-- are indistinguishable in every position. Only their joint distribution with the+-- proxy differs: after `c0 = 1` the s1/s2/s3 regexes keep every survivor while s4+-- still discards ~70%. Ranking them therefore takes joint statistics; an+-- independence assumption prices all four regexes identically everywhere.CREATE TABLE t ASWITH base AS (SELECT- -- The cheap proxy and the independent control share one definition each, so- -- the perfect-proxy / independence invariants can't drift apart silently.- (value * 7) % 100 < 30 AS proxy, -- ~30%, drives c0 and s1/s2/s3- (value * 13) % 100 < 30 AS indep -- ~30%, independent of proxy, drives s4+ (value * 7) % 100 < 30 AS proxy,+ (value * 13) % 100 < 30 AS indepFROM generate_series(1, ${PRED_ROWS:-1000000}))SELECTdiff --git a/benchmarks/sql_benchmarks/predicate_eval/load/drift.sql b/benchmarks/sql_benchmarks/predicate_eval/load/drift.sqlindex d5635d91dd..0345abc3f2 100644--- a/benchmarks/sql_benchmarks/predicate_eval/load/drift.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/load/drift.sql@@ -1,16 +1,11 @@--- Drift dataset: two predicates whose *relative* selectivity flips partway--- through the scan, so whole-table selectivity differs from per-batch--- selectivity. Rows are emitted in `seq` order, so batches observe the drift in--- order.------ a_sel = 0 is selective (~0.1%) in the first 10% of rows, unselective--- (~50%) afterwards.--- b_sel = 0 is the mirror: unselective early, selective late.------ PRED_ROWS sizes the table.+-- Drift dataset: rows are emitted in `seq` order and which predicate is more+-- selective flips partway through, so whole-table selectivity differs from+-- per-batch selectivity. `a_sel = 0` matches ~0.1% over the first 2% of rows and+-- ~50% after; `b_sel = 0` is the mirror. At the default PRED_ROWS, which sizes the+-- table, that flip lands inside the pooled 8-batch warm-up (~65k rows).CREATE TABLE t ASSELECTvalue AS seq,- CASE WHEN value < ${PRED_ROWS:-1000000} / 10 THEN value % 1000 ELSE value % 2 END AS a_sel,- CASE WHEN value < ${PRED_ROWS:-1000000} / 10 THEN value % 2 ELSE value % 1000 END AS b_sel+ CASE WHEN value < ${PRED_ROWS:-1000000} / 50 THEN value % 1000 ELSE value % 2 END AS a_sel,+ CASE WHEN value < ${PRED_ROWS:-1000000} / 50 THEN value % 2 ELSE value % 1000 END AS b_selFROM generate_series(1, ${PRED_ROWS:-1000000});diff --git a/benchmarks/sql_benchmarks/predicate_eval/load/drift_split.sql b/benchmarks/sql_benchmarks/predicate_eval/load/drift_split.sqlnew file mode 100644index 0000000000..b36a0d122f--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/load/drift_split.sql@@ -0,0 +1,96 @@+-- Split drift dataset, shared by q82 and q83: 16 Parquet files of PRED_ROWS / 16+-- rows each, at the q80 rates -- f00..f07 carry profile A (`a_sel = 0` selective+-- at ~0.1%, `b_sel = 0` unselective at ~50%) and f08..f15 the mirror.+--+-- One `COPY` per file, because a directory-target `COPY` names its output with a+-- random write id and Parquet has no overwrite, so a second load would add files;+-- the names are zero-padded so they sort numerically. `FileGroup::split_files`+-- sorts a group's files by path, so `target_partitions = 1` (q82) reads f00..f15+-- in order and `= 16` (q83) gives each stream one whole file; each query sets it.+-- `repartition_file_scans` stays off, or both profiles land in one stream again.++COPY (SELECT value AS seq, value % 1000 AS a_sel, value % 2 AS b_sel+ FROM generate_series(1, ${PRED_ROWS:-1000000} / 16) ORDER BY value)+TO 'sql_benchmarks/predicate_eval/scratch/drift_split/f00.parquet'+STORED AS PARQUET;++COPY (SELECT value AS seq, value % 1000 AS a_sel, value % 2 AS b_sel+ FROM generate_series(1, ${PRED_ROWS:-1000000} / 16) ORDER BY value)+TO 'sql_benchmarks/predicate_eval/scratch/drift_split/f01.parquet'+STORED AS PARQUET;++COPY (SELECT value AS seq, value % 1000 AS a_sel, value % 2 AS b_sel+ FROM generate_series(1, ${PRED_ROWS:-1000000} / 16) ORDER BY value)+TO 'sql_benchmarks/predicate_eval/scratch/drift_split/f02.parquet'+STORED AS PARQUET;++COPY (SELECT value AS seq, value % 1000 AS a_sel, value % 2 AS b_sel+ FROM generate_series(1, ${PRED_ROWS:-1000000} / 16) ORDER BY value)+TO 'sql_benchmarks/predicate_eval/scratch/drift_split/f03.parquet'+STORED AS PARQUET;++COPY (SELECT value AS seq, value % 1000 AS a_sel, value % 2 AS b_sel+ FROM generate_series(1, ${PRED_ROWS:-1000000} / 16) ORDER BY value)+TO 'sql_benchmarks/predicate_eval/scratch/drift_split/f04.parquet'+STORED AS PARQUET;++COPY (SELECT value AS seq, value % 1000 AS a_sel, value % 2 AS b_sel+ FROM generate_series(1, ${PRED_ROWS:-1000000} / 16) ORDER BY value)+TO 'sql_benchmarks/predicate_eval/scratch/drift_split/f05.parquet'+STORED AS PARQUET;++COPY (SELECT value AS seq, value % 1000 AS a_sel, value % 2 AS b_sel+ FROM generate_series(1, ${PRED_ROWS:-1000000} / 16) ORDER BY value)+TO 'sql_benchmarks/predicate_eval/scratch/drift_split/f06.parquet'+STORED AS PARQUET;++COPY (SELECT value AS seq, value % 1000 AS a_sel, value % 2 AS b_sel+ FROM generate_series(1, ${PRED_ROWS:-1000000} / 16) ORDER BY value)+TO 'sql_benchmarks/predicate_eval/scratch/drift_split/f07.parquet'+STORED AS PARQUET;++COPY (SELECT value AS seq, value % 2 AS a_sel, value % 1000 AS b_sel+ FROM generate_series(1, ${PRED_ROWS:-1000000} / 16) ORDER BY value)+TO 'sql_benchmarks/predicate_eval/scratch/drift_split/f08.parquet'+STORED AS PARQUET;++COPY (SELECT value AS seq, value % 2 AS a_sel, value % 1000 AS b_sel+ FROM generate_series(1, ${PRED_ROWS:-1000000} / 16) ORDER BY value)+TO 'sql_benchmarks/predicate_eval/scratch/drift_split/f09.parquet'+STORED AS PARQUET;++COPY (SELECT value AS seq, value % 2 AS a_sel, value % 1000 AS b_sel+ FROM generate_series(1, ${PRED_ROWS:-1000000} / 16) ORDER BY value)+TO 'sql_benchmarks/predicate_eval/scratch/drift_split/f10.parquet'+STORED AS PARQUET;++COPY (SELECT value AS seq, value % 2 AS a_sel, value % 1000 AS b_sel+ FROM generate_series(1, ${PRED_ROWS:-1000000} / 16) ORDER BY value)+TO 'sql_benchmarks/predicate_eval/scratch/drift_split/f11.parquet'+STORED AS PARQUET;++COPY (SELECT value AS seq, value % 2 AS a_sel, value % 1000 AS b_sel+ FROM generate_series(1, ${PRED_ROWS:-1000000} / 16) ORDER BY value)+TO 'sql_benchmarks/predicate_eval/scratch/drift_split/f12.parquet'+STORED AS PARQUET;++COPY (SELECT value AS seq, value % 2 AS a_sel, value % 1000 AS b_sel+ FROM generate_series(1, ${PRED_ROWS:-1000000} / 16) ORDER BY value)+TO 'sql_benchmarks/predicate_eval/scratch/drift_split/f13.parquet'+STORED AS PARQUET;++COPY (SELECT value AS seq, value % 2 AS a_sel, value % 1000 AS b_sel+ FROM generate_series(1, ${PRED_ROWS:-1000000} / 16) ORDER BY value)+TO 'sql_benchmarks/predicate_eval/scratch/drift_split/f14.parquet'+STORED AS PARQUET;++COPY (SELECT value AS seq, value % 2 AS a_sel, value % 1000 AS b_sel+ FROM generate_series(1, ${PRED_ROWS:-1000000} / 16) ORDER BY value)+TO 'sql_benchmarks/predicate_eval/scratch/drift_split/f15.parquet'+STORED AS PARQUET;++set datafusion.optimizer.repartition_file_scans = false;++CREATE EXTERNAL TABLE t+STORED AS PARQUET+LOCATION 'sql_benchmarks/predicate_eval/scratch/drift_split/';diff --git a/benchmarks/sql_benchmarks/predicate_eval/load/ints.sql b/benchmarks/sql_benchmarks/predicate_eval/load/ints.sqlindex 1d3f030915..f36b283c2a 100644--- a/benchmarks/sql_benchmarks/predicate_eval/load/ints.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/load/ints.sql@@ -1,8 +1,8 @@--- Sixteen independent integer columns, each uniform on [0,100). The predicate--- `cN < k` therefore has selectivity ~k%. All columns are equally cheap to--- evaluate, so only selectivity (not cost) distinguishes orderings here. The--- multipliers are all coprime to 100, which keeps the residues uniform and the--- columns mutually decorrelated. PRED_ROWS sizes the table.+-- Seventeen integer columns: c0..c15 uniform on [0,100), so `cN < k` has+-- selectivity ~k%, all equally cheap and mutually decorrelated (the multipliers+-- are coprime to 100); plus `c_sel`, NULL on 10% of rows and uniform on [0,100)+-- elsewhere, so `c_sel < 5` is true on 4% of rows and NULL on 10% -- a selective+-- conjunct that is not NULL-free. PRED_ROWS sizes the table.CREATE TABLE t ASSELECT(value * 1) % 100 AS c0,@@ -20,5 +20,6 @@ SELECT(value * 31) % 100 AS c12,(value * 33) % 100 AS c13,(value * 37) % 100 AS c14,- (value * 39) % 100 AS c15+ (value * 39) % 100 AS c15,+ CASE WHEN value % 10 = 0 THEN NULL ELSE (value * 11) % 100 END AS c_selFROM generate_series(1, ${PRED_ROWS:-1000000});diff --git a/benchmarks/sql_benchmarks/predicate_eval/load/ints_wide.sql b/benchmarks/sql_benchmarks/predicate_eval/load/ints_wide.sqlnew file mode 100644index 0000000000..cddce6f8e5--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/load/ints_wide.sql@@ -0,0 +1,71 @@+-- `ints.sql` widened to 64 integer columns, each uniform on [0,100), so `cN < k`+-- again has selectivity ~k%. c0..c15 reuse the same multipliers, so a predicate+-- over them has exactly the same selectivities as on `ints` and only the width of+-- the batches flowing through the filter changes. PRED_ROWS sizes the table.+CREATE TABLE t AS+SELECT+ (value * 1) % 100 AS c0,+ (value * 3) % 100 AS c1,+ (value * 7) % 100 AS c2,+ (value * 9) % 100 AS c3,+ (value * 11) % 100 AS c4,+ (value * 13) % 100 AS c5,+ (value * 17) % 100 AS c6,+ (value * 19) % 100 AS c7,+ (value * 21) % 100 AS c8,+ (value * 23) % 100 AS c9,+ (value * 27) % 100 AS c10,+ (value * 29) % 100 AS c11,+ (value * 31) % 100 AS c12,+ (value * 33) % 100 AS c13,+ (value * 37) % 100 AS c14,+ (value * 39) % 100 AS c15,+ (value * 41) % 100 AS c16,+ (value * 43) % 100 AS c17,+ (value * 47) % 100 AS c18,+ (value * 49) % 100 AS c19,+ (value * 51) % 100 AS c20,+ (value * 53) % 100 AS c21,+ (value * 57) % 100 AS c22,+ (value * 59) % 100 AS c23,+ (value * 61) % 100 AS c24,+ (value * 63) % 100 AS c25,+ (value * 67) % 100 AS c26,+ (value * 69) % 100 AS c27,+ (value * 71) % 100 AS c28,+ (value * 73) % 100 AS c29,+ (value * 77) % 100 AS c30,+ (value * 79) % 100 AS c31,+ (value * 81) % 100 AS c32,+ (value * 83) % 100 AS c33,+ (value * 87) % 100 AS c34,+ (value * 89) % 100 AS c35,+ (value * 91) % 100 AS c36,+ (value * 93) % 100 AS c37,+ (value * 97) % 100 AS c38,+ (value * 99) % 100 AS c39,+ (value * 101) % 100 AS c40,+ (value * 103) % 100 AS c41,+ (value * 107) % 100 AS c42,+ (value * 109) % 100 AS c43,+ (value * 111) % 100 AS c44,+ (value * 113) % 100 AS c45,+ (value * 117) % 100 AS c46,+ (value * 119) % 100 AS c47,+ (value * 121) % 100 AS c48,+ (value * 123) % 100 AS c49,+ (value * 127) % 100 AS c50,+ (value * 129) % 100 AS c51,+ (value * 131) % 100 AS c52,+ (value * 133) % 100 AS c53,+ (value * 137) % 100 AS c54,+ (value * 139) % 100 AS c55,+ (value * 141) % 100 AS c56,+ (value * 143) % 100 AS c57,+ (value * 147) % 100 AS c58,+ (value * 149) % 100 AS c59,+ (value * 151) % 100 AS c60,+ (value * 153) % 100 AS c61,+ (value * 157) % 100 AS c62,+ (value * 159) % 100 AS c63+FROM generate_series(1, ${PRED_ROWS:-1000000});diff --git a/benchmarks/sql_benchmarks/predicate_eval/load/markers.sql b/benchmarks/sql_benchmarks/predicate_eval/load/markers.sqlindex 6b8fff9385..d1c818fe0f 100644--- a/benchmarks/sql_benchmarks/predicate_eval/load/markers.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/load/markers.sql@@ -1,15 +1,8 @@--- Wide-string dataset: five markers embedded in `PRED_FILL`-wide filler so that a--- non-matching `regexp_like` must scan the whole value (every string predicate--- is "expensive"). Selectivities are coprime so the predicates are independent:------ 'aaa' present in ~90% of rows (value % 10 <> 0)--- 'bbb' present in ~86% of rows (value % 7 <> 0)--- 'ccc' present in ~80% of rows (value % 5 <> 0)--- 'ddd' present in ~75% of rows (value % 4 <> 0)--- 'rare' present in ~0.1% of rows (value % 1009 = 5) <- the selective one------ PRED_FILL sets the filler width per marker (the string-column width knob: ~6*PRED_FILL--- chars per row), and PRED_ROWS sizes the table.+-- Wide-string dataset: one column `s` holding five markers in PRED_FILL-wide+-- filler, so a non-matching `regexp_like` must scan the whole value. 'aaa' ~90%,+-- 'bbb' ~86%, 'ccc' ~80%, 'ddd' ~75%, 'rare' ~0.1%; the moduli are coprime, so the+-- markers are independent. PRED_FILL is the string-width knob (~6*PRED_FILL chars+-- per row) and PRED_ROWS sizes the table.CREATE TABLE t ASSELECTrepeat('q', ${PRED_FILL:-30})diff --git a/benchmarks/sql_benchmarks/predicate_eval/load/mixed.sql b/benchmarks/sql_benchmarks/predicate_eval/load/mixed.sqlindex a51c1040da..bc931e8a40 100644--- a/benchmarks/sql_benchmarks/predicate_eval/load/mixed.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/load/mixed.sql@@ -1,15 +1,7 @@--- Mixed-cost dataset: cheap integer columns (`cN < k` ~ k% selectivity)--- alongside one wide string column carrying three markers matched by expensive--- `regexp_like`:------ 'rare' present in ~0.1% of rows (value % 1009 = 5)--- 'ten' present in ~10% of rows (value % 10 = 0)--- 'aaa' present in ~90% of rows (value % 10 <> 0)------ This lets a single table mix cheap integer compares with expensive regexp--- scans at independently chosen selectivities (e.g. a cheap, unselective compare--- next to an expensive, selective regexp). PRED_FILL is the string-width knob;--- PRED_ROWS sizes the table.+-- Mixed-cost dataset: cheap integer columns c0..c3 uniform on [0,100) (`cN < k`+-- ~k%) alongside one wide string column `s` carrying three markers matched by an+-- expensive `regexp_like`: 'rare' ~0.1%, 'ten' ~10%, 'aaa' ~90%. PRED_FILL is the+-- string-width knob and PRED_ROWS sizes the table.CREATE TABLE t ASSELECT(value * 1) % 100 AS c0,diff --git a/benchmarks/sql_benchmarks/predicate_eval/predicate_eval.benchmark.template b/benchmarks/sql_benchmarks/predicate_eval/predicate_eval.benchmark.templateindex e2e3d6175e..766bd20270 100644--- a/benchmarks/sql_benchmarks/predicate_eval/predicate_eval.benchmark.template+++ b/benchmarks/sql_benchmarks/predicate_eval/predicate_eval.benchmark.template@@ -1,35 +1,22 @@-# Shared template for every predicate_eval benchmark. Each qNN.benchmark sets-# its `subgroup` directive and then includes this template with parameters:-# SUBGROUP subgroup name, also the query sub-directory (e.g. costsel)-# QPAD zero-padded query id, also the query file stem (e.g. 01)-# DATASET load script stem under load/ (e.g. markers)-# NAME criterion display name (e.g. costsel_q01_regexp_selective_last)-# Optional (consumed by the load scripts via ${...:-default}):-# PRED_ROWS synthetic row count (default 1_000_000)-# PRED_FILL filler chars per marker = string-column width knob (default 30)-#-# The run SQL lives in queries/${SUBGROUP}/q${QPAD}.sql so the WHERE clause is-# readable on its own. The table is always named `t`, so the assert and cleanup-# are uniform across datasets.-#-# The suite is implementation-agnostic and sets no engine config of its own: it-# measures DataFusion's built-in left-deep `AND` short-circuit by default. To-# evaluate a predicate-ordering system under test, set its native config via the-# environment (the bench harness builds its SessionContext with-# SessionConfig::from_env), e.g.-# DATAFUSION_EXECUTION_ADAPTIVE_FILTER_REORDERING=true+# Shared template for every predicate_eval benchmark. Each qNN.benchmark sets its+# `subgroup` directive and includes this template with parameters:+# QUERY query file stem under queries/ (e.g. costsel/q01)+# DATASET load script stem under load/ (e.g. markers)+# NAME criterion display name (e.g. costsel_q01_regexp_selective_last)+# Optional, read by the load scripts as ${...:-default}:+# PRED_ROWS synthetic row count (default 1000000)+# PRED_FILL filler chars per marker, the string-width knob (default 30)+# The knob sweeps reuse another subgroup's query file through QUERY, and the table+# is always named `t`, so cleanup is uniform across datasets.load sql_benchmarks/predicate_eval/load/${DATASET}.sqlname ${NAME}group predicate_eval-assert I-SELECT count(*) > 0 FROM t;------true+run sql_benchmarks/predicate_eval/queries/${QUERY}.sql-run sql_benchmarks/predicate_eval/queries/${SUBGROUP}/q${QPAD}.sql+result sql_benchmarks/predicate_eval/results/${NAME}.csvresult sql_benchmarks/predicate_eval/results/${NAME}.csvdiff --git a/benchmarks/sql_benchmarks/predicate_eval/predicate_eval.suite b/benchmarks/sql_benchmarks/predicate_eval/predicate_eval.suiteindex af1a326cd8..4ee33994d3 100644--- a/benchmarks/sql_benchmarks/predicate_eval/predicate_eval.suite+++ b/benchmarks/sql_benchmarks/predicate_eval/predicate_eval.suite@@ -1,4 +1,4 @@-description = "Conjunctive filter evaluation micro-benchmarks covering predicate cost, selectivity, cardinality, width, scale, correlation, and drift"+description = "Conjunctive filter evaluation micro-benchmarks covering predicate cost, selectivity, cardinality, width, scale, correlation, drift, and nullable predicates"query_pattern = "q{QUERY_ID_PADDED}.benchmark"diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q30.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q30.sqlindex 3be840e917..dbd410f957 100644--- a/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q30.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q30.sql@@ -1,6 +1,4 @@--- Hidden: cheap integer compares; `c1 < 5` matches ~5%, the `c0 < 90` family--- ~90%. k = 2 here. q30..q33 sweep k = 2/4/8/16 with one ~5% predicate written--- last among ~90% ones.+-- k = 2: one ~90% compare then one ~5% compare (q30..q33 sweep k = 2/4/8/16).SELECT count(*) FROM tWHERE c0 < 90AND c1 < 5;diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q31.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q31.sqlindex 4ba84f8124..d21dd6273e 100644--- a/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q31.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q31.sql@@ -1,4 +1,4 @@--- k = 4: three ~90% compares followed by one ~5% compare. See q30.+-- k = 4: three ~90% compares then one ~5% compare.SELECT count(*) FROM tWHERE c0 < 90AND c1 < 90diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q32.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q32.sqlindex d9e920cc62..83037bdfc5 100644--- a/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q32.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q32.sql@@ -1,4 +1,4 @@--- k = 8: seven ~90% compares followed by one ~5% compare. See q30.+-- k = 8: seven ~90% compares then one ~5% compare.SELECT count(*) FROM tWHERE c0 < 90AND c1 < 90diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q33.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q33.sqlindex 2408427ab7..890ff4205d 100644--- a/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q33.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q33.sql@@ -1,4 +1,4 @@--- k = 16: fifteen ~90% compares followed by one ~5% compare. See q30.+-- k = 16: fifteen ~90% compares then one ~5% compare.SELECT count(*) FROM tWHERE c0 < 90AND c1 < 90diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q34.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q34.sqlnew file mode 100644index 0000000000..2c5b586e54--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/cardinality/q34.sql@@ -0,0 +1,10 @@+-- k = 8 as in q32, but over the 64-column `ints_wide` table.+SELECT count(*) FROM t+WHERE c0 < 90+ AND c1 < 90+ AND c2 < 90+ AND c3 < 90+ AND c4 < 90+ AND c5 < 90+ AND c6 < 90+ AND c7 < 5;diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/correlation/q70.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/correlation/q70.sqlindex 86e33534c7..f9d7017e61 100644--- a/benchmarks/sql_benchmarks/predicate_eval/queries/correlation/q70.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/correlation/q70.sql@@ -1,6 +1,4 @@--- Hidden: `x` and `ind` are independent, each ~20%, so the conjunction matches--- ~4% and the second predicate is just as selective among the first's survivors--- as on its own. Baseline for the correlation sweep. cf. q71, q72.+-- Independent: `x` and `ind` are uncorrelated, each ~20%, so the pair matches ~4%.SELECT count(*) FROM tWHERE x < 20AND ind < 20;diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/correlation/q71.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/correlation/q71.sqlindex eda61cc289..4b5925e80a 100644--- a/benchmarks/sql_benchmarks/predicate_eval/queries/correlation/q71.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/correlation/q71.sql@@ -1,6 +1,4 @@--- Hidden: `x_pos` is a copy of `x`, so `x < 20 AND x_pos < 20` still matches--- ~20% (not the ~4% independence would imply) -- the second predicate removes--- none of the first's survivors. cf. q70.+-- Positively correlated: `x_pos` is a copy of `x`, so the pair still matches ~20%.SELECT count(*) FROM tWHERE x < 20AND x_pos < 20;diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/correlation/q72.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/correlation/q72.sqlindex ff987524da..63ec8d57de 100644--- a/benchmarks/sql_benchmarks/predicate_eval/queries/correlation/q72.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/correlation/q72.sql@@ -1,6 +1,4 @@--- Hidden: `x_anti` is `99 - x`, so `x < 50 AND x_anti < 50` is empty -- the--- second predicate removes all of the first's survivors, though each matches--- ~50% alone. cf. q70.+-- Anti-correlated: `x_anti` is `99 - x`, so the pair is empty though each matches ~50%.SELECT count(*) FROM tWHERE x < 50AND x_anti < 50;diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/correlation/q73.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/correlation/q73.sqlindex 5e1e822e92..6dc76a8cab 100644--- a/benchmarks/sql_benchmarks/predicate_eval/queries/correlation/q73.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/correlation/q73.sql@@ -1,11 +1,5 @@--- Hidden: `c0 = 1` is a perfect proxy for the s1/s2/s3 regexes -- after the--- cheap proxy, each of those keeps every survivor while the equally selective--- (~30%) s4 regex still discards ~70%. The optimal order is [c0, s4, s1/s2/s3]--- (one informative regex on 30% of rows, the three redundant ones on 9%), but--- the four regexes are marginally identical -- same width, same marker offset,--- same cost, same selectivity -- so ranking them takes their *joint*--- distribution with the proxy. Written with the redundant regexes first,--- grouped with their proxy, as an author naturally would.+-- Redundant proxy: `c0 = 1` implies the s1/s2/s3 regexes, while the marginally+-- identical s4 regex is independent of it. Written proxy-first, regexes grouped.SELECT count(*) FROM tWHERE c0 = 1AND regexp_like(s1, 'a.a')diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/cost/q10.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/cost/q10.sqlindex b089ebc7a1..cbe0e937ef 100644--- a/benchmarks/sql_benchmarks/predicate_eval/queries/cost/q10.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/cost/q10.sql@@ -1,6 +1,4 @@--- Hidden: both predicates match ~10%, but `regexp_like(s, 'ten')` scans the--- string (expensive) while `c0 < 10` is a cheap compare. Equal selectivity,--- unequal cost; expensive one written first. cf. q11 (opposite order).+-- Equal selectivity (~10%), unequal cost: the expensive regexp written first. cf. q11.SELECT count(*) FROM tWHERE regexp_like(s, 'ten')AND c0 < 10;diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/cost/q11.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/cost/q11.sqlindex 82d748c93b..54167ff194 100644--- a/benchmarks/sql_benchmarks/predicate_eval/queries/cost/q11.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/cost/q11.sql@@ -1,5 +1,4 @@--- Same two predicates as q10 (both ~10%; regexp expensive, compare cheap),--- opposite written order. cf. q10.+-- q10 with the cheap compare written first.SELECT count(*) FROM tWHERE c0 < 10AND regexp_like(s, 'ten');diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/costsel/q01.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/costsel/q01.sqlindex bc029ed5d8..1618c54814 100644--- a/benchmarks/sql_benchmarks/predicate_eval/queries/costsel/q01.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/costsel/q01.sql@@ -1,7 +1,5 @@--- Hidden in the data: the five markers have very different selectivities ----- 'aaa' ~90%, 'bbb' ~86%, 'ccc' ~80%, 'ddd' ~75%, 'rare' ~0.1% -- while every--- regexp_like costs about the same. 'rare' (most selective) is written last.--- cf. q02 (most selective written first).+-- Five equally expensive regexps of very different selectivity ('rare' ~0.1%, the+-- rest 75-90%), the selective one written last. cf. q02.SELECT count(*) FROM tWHERE regexp_like(s, 'aaa')AND regexp_like(s, 'bbb')diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/costsel/q02.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/costsel/q02.sqlindex 7f7fc61831..dc57aa8d88 100644--- a/benchmarks/sql_benchmarks/predicate_eval/queries/costsel/q02.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/costsel/q02.sql@@ -1,5 +1,4 @@--- Same predicates and hidden selectivities as q01 ('rare' ~0.1% is the--- selective one, the rest 75-90%), but with 'rare' written first. cf. q01.+-- q01 with the selective regexp written first.SELECT count(*) FROM tWHERE regexp_like(s, 'rare')AND regexp_like(s, 'aaa')diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/costsel/q03.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/costsel/q03.sqlindex a583a498b2..a9d110c65a 100644--- a/benchmarks/sql_benchmarks/predicate_eval/queries/costsel/q03.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/costsel/q03.sql@@ -1,6 +1,4 @@--- Hidden: `c0 < 90` matches ~90% (cheap integer compare); `regexp_like(s,--- 'rare')` matches ~0.1% (scans the wide string). The cheaper predicate is the--- less selective one.+-- Cheap unselective compare (~90%) then expensive selective regexp (~0.1%). cf. q04.SELECT count(*) FROM tWHERE c0 < 90AND regexp_like(s, 'rare');diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/costsel/q04.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/costsel/q04.sqlnew file mode 100644index 0000000000..2c5a253664--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/costsel/q04.sql@@ -0,0 +1,4 @@+-- q03 the other way round, so the as-written order is already the best one.+SELECT count(*) FROM t+WHERE regexp_like(s, 'rare')+ AND c0 < 90;diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/drift/q80.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/drift/q80.sqlindex b8cb61e85a..63e55ce473 100644--- a/benchmarks/sql_benchmarks/predicate_eval/queries/drift/q80.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/drift/q80.sql@@ -1,7 +1,5 @@--- The non-obvious property: selectivity changes across the scan. Rows arrive in--- `seq` order; `a_sel = 0` matches ~0.1% in the first 10% of rows and ~50%--- after, `b_sel = 0` is the mirror -- so which predicate is more selective flips--- partway through. cf. q81 (opposite order).+-- Selectivity drifts across the scan: `a_sel = 0` matches ~0.1% over the first 2%+-- of rows and ~50% after, and `b_sel = 0` is the mirror.SELECT count(*) FROM tWHERE a_sel = 0AND b_sel = 0;diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/drift/q81.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/drift/q81.sqldeleted file mode 100644index d65ef475cc..0000000000--- a/benchmarks/sql_benchmarks/predicate_eval/queries/drift/q81.sql+++ /dev/null@@ -1,5 +0,0 @@--- Same drifting predicates as q80 (a_sel/b_sel flip which is more selective--- partway through the scan), opposite written order. cf. q80.-SELECT count(*) FROM t-WHERE b_sel = 0- AND a_sel = 0;diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/drift/q82.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/drift/q82.sqlnew file mode 100644index 0000000000..57bb65d216--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/drift/q82.sql@@ -0,0 +1,5 @@+-- Late flip: one stream reads f=00..f=15 in order, so the drift lands halfway+-- through the scan and a warm-up-and-freeze decision is wrong for half the rows.+SELECT count(*) FROM t+WHERE a_sel = 0+ AND b_sel = 0;diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/drift/q83.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/drift/q83.sqlnew file mode 100644index 0000000000..7ecc0fb1c5--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/drift/q83.sql@@ -0,0 +1,5 @@+-- Per-partition skew: one whole file per stream, so each stream sees a single fixed+-- profile and one pooled decision is backwards for half of them.+SELECT count(*) FROM t+WHERE a_sel = 0+ AND b_sel = 0;diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/neutral/q60.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/neutral/q60.sqlindex b217f56953..76f5a154da 100644--- a/benchmarks/sql_benchmarks/predicate_eval/queries/neutral/q60.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/neutral/q60.sql@@ -1,5 +1,4 @@--- Hidden: four integer compares of equal cost, each ~50% selective. Nothing is--- selective and the costs are equal, so the predicates are interchangeable.+-- Four equally cheap compares, each ~50%: nothing to reorder.SELECT count(*) FROM tWHERE c0 < 50AND c1 < 50diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/neutral/q61.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/neutral/q61.sqlindex 7029a3d9f8..43225f2e48 100644--- a/benchmarks/sql_benchmarks/predicate_eval/queries/neutral/q61.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/neutral/q61.sql@@ -1,6 +1,4 @@--- Hidden: four regexp scans of about equal cost, all unselective ('aaa' ~90%,--- 'bbb' ~86%, 'ccc' ~80%, 'ddd' ~75%). Like q60 the predicates are--- interchangeable, but here each one is expensive.+-- Four equally expensive regexps, all unselective (75-90%): nothing to reorder.SELECT count(*) FROM tWHERE regexp_like(s, 'aaa')AND regexp_like(s, 'bbb')diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/nulls/q90.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/nulls/q90.sqlnew file mode 100644index 0000000000..ce4e40d9ac--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/nulls/q90.sql@@ -0,0 +1,6 @@+-- The selective conjunct (~4%) is nullable, so AND pre-selection is disabled and+-- writing it first gates nothing. cf. q91.+SELECT count(*) FROM t+WHERE c_sel < 5+ AND c0 < 90+ AND c1 < 90;diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/nulls/q91.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/nulls/q91.sqlnew file mode 100644index 0000000000..abca8bb1f9--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/nulls/q91.sql@@ -0,0 +1,5 @@+-- q90 with the nullable selective conjunct written last, which is the same work.+SELECT count(*) FROM t+WHERE c0 < 90+ AND c1 < 90+ AND c_sel < 5;diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/scale/q50.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/scale/q50.sqldeleted file mode 100644index 03a0f1c0db..0000000000--- a/benchmarks/sql_benchmarks/predicate_eval/queries/scale/q50.sql+++ /dev/null@@ -1,6 +0,0 @@--- Same predicates as costsel/q03 (`c0 < 90` ~90% cheap, `regexp_like(s, 'rare')`--- ~0.1% expensive). q50..q53 sweep table size; here PRED_ROWS=5_000, roughly a--- single batch.-SELECT count(*) FROM t-WHERE c0 < 90- AND regexp_like(s, 'rare');diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/scale/q51.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/scale/q51.sqldeleted file mode 100644index 28174a5df4..0000000000--- a/benchmarks/sql_benchmarks/predicate_eval/queries/scale/q51.sql+++ /dev/null@@ -1,4 +0,0 @@--- q50 at PRED_ROWS=100_000 (~12 batches). See q50.-SELECT count(*) FROM t-WHERE c0 < 90- AND regexp_like(s, 'rare');diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/scale/q52.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/scale/q52.sqldeleted file mode 100644index 74938c4634..0000000000--- a/benchmarks/sql_benchmarks/predicate_eval/queries/scale/q52.sql+++ /dev/null@@ -1,4 +0,0 @@--- q50 at PRED_ROWS=5_000_000 (~610 batches). See q50.-SELECT count(*) FROM t-WHERE c0 < 90- AND regexp_like(s, 'rare');diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/scale/q53.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/scale/q53.sqldeleted file mode 100644index 8edb4d4a05..0000000000--- a/benchmarks/sql_benchmarks/predicate_eval/queries/scale/q53.sql+++ /dev/null@@ -1,4 +0,0 @@--- q50 at PRED_ROWS=50_000_000 (~6100 batches); builds a ~9 GB table. See q50.-SELECT count(*) FROM t-WHERE c0 < 90- AND regexp_like(s, 'rare');diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/selectivity/q20.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/selectivity/q20.sqlindex 3638f757a7..b834a4bd7b 100644--- a/benchmarks/sql_benchmarks/predicate_eval/queries/selectivity/q20.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/selectivity/q20.sql@@ -1,6 +1,5 @@--- Hidden: two equally cheap integer compares of unequal selectivity -- `c4 < 95`--- matches ~95%, `c0 < 5` matches ~5%. Less selective one written first.--- cf. q21 (opposite order).+-- Two equally cheap compares of unequal selectivity (~95%, ~5%), less selective+-- first. cf. q21.SELECT count(*) FROM tWHERE c4 < 95AND c0 < 5;diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/selectivity/q21.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/selectivity/q21.sqlindex 5181faf387..1f8531a510 100644--- a/benchmarks/sql_benchmarks/predicate_eval/queries/selectivity/q21.sql+++ b/benchmarks/sql_benchmarks/predicate_eval/queries/selectivity/q21.sql@@ -1,5 +1,4 @@--- Same two equally-cheap compares as q20 (`c4 < 95` ~95%, `c0 < 5` ~5%),--- opposite written order. cf. q20.+-- q20 with the selective compare written first.SELECT count(*) FROM tWHERE c0 < 5AND c4 < 95;diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/width/q40.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/width/q40.sqldeleted file mode 100644index 1b3df3e937..0000000000--- a/benchmarks/sql_benchmarks/predicate_eval/queries/width/q40.sql+++ /dev/null@@ -1,9 +0,0 @@--- Same predicate set and hidden selectivities as costsel/q01 ('rare' ~0.1%, the--- rest 75-90%); only the string-column width differs across q40/q41/q42. Narrow:--- PRED_FILL=2, ~12 chars/row.-SELECT count(*) FROM t-WHERE regexp_like(s, 'aaa')- AND regexp_like(s, 'bbb')- AND regexp_like(s, 'ccc')- AND regexp_like(s, 'ddd')- AND regexp_like(s, 'rare');diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/width/q41.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/width/q41.sqldeleted file mode 100644index a03b576d9c..0000000000--- a/benchmarks/sql_benchmarks/predicate_eval/queries/width/q41.sql+++ /dev/null@@ -1,7 +0,0 @@--- q40 with wide strings: PRED_FILL=30, ~186 chars/row. See q40.-SELECT count(*) FROM t-WHERE regexp_like(s, 'aaa')- AND regexp_like(s, 'bbb')- AND regexp_like(s, 'ccc')- AND regexp_like(s, 'ddd')- AND regexp_like(s, 'rare');diff --git a/benchmarks/sql_benchmarks/predicate_eval/queries/width/q42.sql b/benchmarks/sql_benchmarks/predicate_eval/queries/width/q42.sqldeleted file mode 100644index cb55d828e3..0000000000--- a/benchmarks/sql_benchmarks/predicate_eval/queries/width/q42.sql+++ /dev/null@@ -1,7 +0,0 @@--- q40 with extra-wide strings: PRED_FILL=170, ~1KB/row. See q40.-SELECT count(*) FROM t-WHERE regexp_like(s, 'aaa')- AND regexp_like(s, 'bbb')- AND regexp_like(s, 'ccc')- AND regexp_like(s, 'ddd')- AND regexp_like(s, 'rare');diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/cardinality_q30_k2.csv b/benchmarks/sql_benchmarks/predicate_eval/results/cardinality_q30_k2.csvnew file mode 100644index 0000000000..23d7344bc8--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/cardinality_q30_k2.csv@@ -0,0 +1,2 @@+count(*)+50000diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/cardinality_q31_k4.csv b/benchmarks/sql_benchmarks/predicate_eval/results/cardinality_q31_k4.csvnew file mode 100644index 0000000000..94f68af265--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/cardinality_q31_k4.csv@@ -0,0 +1,2 @@+count(*)+40000diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/cardinality_q32_k8.csv b/benchmarks/sql_benchmarks/predicate_eval/results/cardinality_q32_k8.csvnew file mode 100644index 0000000000..23d7344bc8--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/cardinality_q32_k8.csv@@ -0,0 +1,2 @@+count(*)+50000diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/cardinality_q33_k16.csv b/benchmarks/sql_benchmarks/predicate_eval/results/cardinality_q33_k16.csvnew file mode 100644index 0000000000..6291042419--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/cardinality_q33_k16.csv@@ -0,0 +1,2 @@+count(*)+10000diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/cardinality_q34_k8_wide64.csv b/benchmarks/sql_benchmarks/predicate_eval/results/cardinality_q34_k8_wide64.csvnew file mode 100644index 0000000000..23d7344bc8--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/cardinality_q34_k8_wide64.csv@@ -0,0 +1,2 @@+count(*)+50000diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/correlation_q70_independent.csv b/benchmarks/sql_benchmarks/predicate_eval/results/correlation_q70_independent.csvnew file mode 100644index 0000000000..23d7344bc8--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/correlation_q70_independent.csv@@ -0,0 +1,2 @@+count(*)+50000diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/correlation_q71_positive.csv b/benchmarks/sql_benchmarks/predicate_eval/results/correlation_q71_positive.csvnew file mode 100644index 0000000000..e3795c1d30--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/correlation_q71_positive.csv@@ -0,0 +1,2 @@+count(*)+200000diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/correlation_q72_anti.csv b/benchmarks/sql_benchmarks/predicate_eval/results/correlation_q72_anti.csvnew file mode 100644index 0000000000..4303b90458--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/correlation_q72_anti.csv@@ -0,0 +1,2 @@+count(*)+0diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/correlation_q73_redundant_proxy.csv b/benchmarks/sql_benchmarks/predicate_eval/results/correlation_q73_redundant_proxy.csvnew file mode 100644index 0000000000..1cabb81336--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/correlation_q73_redundant_proxy.csv@@ -0,0 +1,2 @@+count(*)+90000diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/cost_q10_expensive_first.csv b/benchmarks/sql_benchmarks/predicate_eval/results/cost_q10_expensive_first.csvnew file mode 100644index 0000000000..6291042419--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/cost_q10_expensive_first.csv@@ -0,0 +1,2 @@+count(*)+10000diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/cost_q11_cheap_first.csv b/benchmarks/sql_benchmarks/predicate_eval/results/cost_q11_cheap_first.csvnew file mode 100644index 0000000000..6291042419--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/cost_q11_cheap_first.csv@@ -0,0 +1,2 @@+count(*)+10000diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/costsel_q01_regexp_selective_last.csv b/benchmarks/sql_benchmarks/predicate_eval/results/costsel_q01_regexp_selective_last.csvnew file mode 100644index 0000000000..5eb0b761fa--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/costsel_q01_regexp_selective_last.csv@@ -0,0 +1,2 @@+count(*)+508diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/costsel_q02_regexp_selective_first.csv b/benchmarks/sql_benchmarks/predicate_eval/results/costsel_q02_regexp_selective_first.csvnew file mode 100644index 0000000000..5eb0b761fa--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/costsel_q02_regexp_selective_first.csv@@ -0,0 +1,2 @@+count(*)+508diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/costsel_q03_cheap_unselective_then_expensive_selective.csv b/benchmarks/sql_benchmarks/predicate_eval/results/costsel_q03_cheap_unselective_then_expensive_selective.csvnew file mode 100644index 0000000000..1c82d80b6f--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/costsel_q03_cheap_unselective_then_expensive_selective.csv@@ -0,0 +1,2 @@+count(*)+893diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/costsel_q04_expensive_selective_then_cheap_unselective.csv b/benchmarks/sql_benchmarks/predicate_eval/results/costsel_q04_expensive_selective_then_cheap_unselective.csvnew file mode 100644index 0000000000..1c82d80b6f--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/costsel_q04_expensive_selective_then_cheap_unselective.csv@@ -0,0 +1,2 @@+count(*)+893diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/drift_q80_a_then_b.csv b/benchmarks/sql_benchmarks/predicate_eval/results/drift_q80_a_then_b.csvnew file mode 100644index 0000000000..c543899cb1--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/drift_q80_a_then_b.csv@@ -0,0 +1,2 @@+count(*)+1000diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/drift_q82_late_flip.csv b/benchmarks/sql_benchmarks/predicate_eval/results/drift_q82_late_flip.csvnew file mode 100644index 0000000000..571ea61b27--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/drift_q82_late_flip.csv@@ -0,0 +1,2 @@+count(*)+992diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/drift_q83_per_partition_skew.csv b/benchmarks/sql_benchmarks/predicate_eval/results/drift_q83_per_partition_skew.csvnew file mode 100644index 0000000000..571ea61b27--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/drift_q83_per_partition_skew.csv@@ -0,0 +1,2 @@+count(*)+992diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/neutral_q60_cheap_uniform.csv b/benchmarks/sql_benchmarks/predicate_eval/results/neutral_q60_cheap_uniform.csvnew file mode 100644index 0000000000..a1c39233b5--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/neutral_q60_cheap_uniform.csv@@ -0,0 +1,2 @@+count(*)+150000diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/neutral_q61_expensive_uniform.csv b/benchmarks/sql_benchmarks/predicate_eval/results/neutral_q61_expensive_uniform.csvnew file mode 100644index 0000000000..5d7624926c--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/neutral_q61_expensive_uniform.csv@@ -0,0 +1,2 @@+count(*)+514286diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/nulls_q90_nullable_selective_first.csv b/benchmarks/sql_benchmarks/predicate_eval/results/nulls_q90_nullable_selective_first.csvnew file mode 100644index 0000000000..ce62ca3d04--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/nulls_q90_nullable_selective_first.csv@@ -0,0 +1,2 @@+count(*)+20000diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/nulls_q91_nullable_selective_last.csv b/benchmarks/sql_benchmarks/predicate_eval/results/nulls_q91_nullable_selective_last.csvnew file mode 100644index 0000000000..ce62ca3d04--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/nulls_q91_nullable_selective_last.csv@@ -0,0 +1,2 @@+count(*)+20000diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/scale_q50_5k.csv b/benchmarks/sql_benchmarks/predicate_eval/results/scale_q50_5k.csvnew file mode 100644index 0000000000..8cedd33498--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/scale_q50_5k.csv@@ -0,0 +1,2 @@+count(*)+5diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/scale_q51_100k.csv b/benchmarks/sql_benchmarks/predicate_eval/results/scale_q51_100k.csvnew file mode 100644index 0000000000..5066201ce8--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/scale_q51_100k.csv@@ -0,0 +1,2 @@+count(*)+90diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/scale_q52_5m.csv b/benchmarks/sql_benchmarks/predicate_eval/results/scale_q52_5m.csvnew file mode 100644index 0000000000..41c4f54ec1--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/scale_q52_5m.csv@@ -0,0 +1,2 @@+count(*)+4461diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/scale_q53_50m.csv b/benchmarks/sql_benchmarks/predicate_eval/results/scale_q53_50m.csvnew file mode 100644index 0000000000..defd47ee70--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/scale_q53_50m.csv@@ -0,0 +1,2 @@+count(*)+44600diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/selectivity_q20_unselective_first.csv b/benchmarks/sql_benchmarks/predicate_eval/results/selectivity_q20_unselective_first.csvnew file mode 100644index 0000000000..23d7344bc8--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/selectivity_q20_unselective_first.csv@@ -0,0 +1,2 @@+count(*)+50000diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/selectivity_q21_selective_first.csv b/benchmarks/sql_benchmarks/predicate_eval/results/selectivity_q21_selective_first.csvnew file mode 100644index 0000000000..23d7344bc8--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/selectivity_q21_selective_first.csv@@ -0,0 +1,2 @@+count(*)+50000diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/width_q40_narrow.csv b/benchmarks/sql_benchmarks/predicate_eval/results/width_q40_narrow.csvnew file mode 100644index 0000000000..5eb0b761fa--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/width_q40_narrow.csv@@ -0,0 +1,2 @@+count(*)+508diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/width_q41_wide.csv b/benchmarks/sql_benchmarks/predicate_eval/results/width_q41_wide.csvnew file mode 100644index 0000000000..5eb0b761fa--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/width_q41_wide.csv@@ -0,0 +1,2 @@+count(*)+508diff --git a/benchmarks/sql_benchmarks/predicate_eval/results/width_q42_xwide.csv b/benchmarks/sql_benchmarks/predicate_eval/results/width_q42_xwide.csvnew file mode 100644index 0000000000..5eb0b761fa--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/results/width_q42_xwide.csv@@ -0,0 +1,2 @@+count(*)+508diff --git a/benchmarks/sql_benchmarks/predicate_eval/scratch/.gitignore b/benchmarks/sql_benchmarks/predicate_eval/scratch/.gitignorenew file mode 100644index 0000000000..4bed5da93f--- /dev/null+++ b/benchmarks/sql_benchmarks/predicate_eval/scratch/.gitignore@@ -0,0 +1 @@+*.parquet