Skip to content

Apache DataFusion SQL Query Engine

bench: make the spill_views memory limits overridable from the environment (#25634)

Adrian Garcia Badaracco207d7f3c2721
diff --git a/benchmarks/sql_benchmarks/README.md b/benchmarks/sql_benchmarks/README.md
index e83c221ad7..340c05657e 100644
--- a/benchmarks/sql_benchmarks/README.md
+++ b/benchmarks/sql_benchmarks/README.md
@@ -47,7 +47,7 @@ in the community:
| `wide_schema` | Small-projection queries on a wide (1024-col, 256-file) synthetic dataset; runs `wide` + `narrow` subgroups for comparison |
| `predicate_eval` | Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an adaptive predicate-ordering system behaves across them ([#11262](https://github.com/apache/datafusion/issues/11262)). Subgroups (`--subgroup`): `costsel`, `cost`, `selectivity`, `cardinality`, `width`, `scale`, `neutral`, `correlation`, `drift`, `nulls`. The suite sets no engine config of its own, so by default it measures DataFusion's built-in left-deep `AND` short-circuit; point it at a system under test by exporting that system's own DataFusion setting (the harness builds its `SessionConfig` with `SessionConfig::from_env`). Every query is a `count(*)`, and the counts are checked in under `predicate_eval/results/`, so `--result-mode validate` also checks that a reordering under test still returns the same rows; the checked-in counts were persisted at the suite defaults (`PRED_ROWS=1000000`, `PRED_FILL=30`), so validation assumes those (the `scale` and `width` subgroups pin their own values per query and validate at any setting). |
| `parquet_row_filter_skip` | Micro-benchmark for the per-row-group fully-matched RowFilter skip on Parquet scans ([#23696](https://github.com/apache/datafusion/issues/23696)). Subgroups (`--subgroup`): `skip` (clustered key, most row groups fully matched by statistics so the per-row filter is skipped), `control` (scrambled key, no row group is ever fully matched). Size the data with `PRED_ROWS` and the row-group size with `RG_SIZE`. |
-| `spill_views` | Sort and `GROUP BY` queries that spill `StringView` and `BinaryView` columns read from Parquet ([#23564](https://github.com/apache/datafusion/issues/23564)). Each query sets its own memory limit and `target_partitions = 4`, so it spills in any environment. Subgroups (`--subgroup`): `repeated` (sort with a low-cardinality payload that shares one dictionary buffer, where spill compaction can remove data), `distinct` (sort and `GROUP BY` on all-distinct strings, where compaction cannot remove data). |
+| `spill_views` | Sort and `GROUP BY` queries that spill `StringView` and `BinaryView` columns read from Parquet ([#23564](https://github.com/apache/datafusion/issues/23564)). Each query sets its own memory limit and `target_partitions = 4`, so it spills in any environment. Subgroups (`--subgroup`): `repeated` (sort with a low-cardinality payload that shares one dictionary buffer, where spill compaction can remove data), `distinct` (sort and `GROUP BY` on all-distinct strings, where compaction cannot remove data). Set the memory limit of each subgroup with `SPILL_VIEWS_LIMIT_REPEATED` (default `40M`) and `SPILL_VIEWS_LIMIT_DISTINCT` (default `96M`), for example to let both sides of a comparison finish when one of them runs out of memory at the default. The queries must still spill, so check a raised limit with `EXPLAIN ANALYZE` and keep it well below the working set: `q05` stops spilling at about `192M`. Give a whole number of K, M or G units, because the run asserts the limit that the memory pool reports back. |
| `projection_subquery` | `IN`, `NOT IN` and `EXISTS` subqueries that sit in the `SELECT` list instead of a filter ([#25341](https://github.com/apache/datafusion/issues/25341)). Every query projects the boolean subquery result and aggregates it, so the measured cost is the decorrelation plan and not the size of the output. `q07` correlates on `<` instead of `=`, so it keeps the nested-loop plan and is the control that must not change. Size the two inline tables with `PSQ_ROWS` (default `30000`). The counts are checked in under `projection_subquery/results/`, so `--result-mode validate` also proves that a change to the decorrelation still returns the same rows; the checked-in counts were persisted at the default `PSQ_ROWS`, so validation assumes that value. |
# Running Benchmarks
@@ -177,6 +177,8 @@ Some benchmarks use custom environment variables as outlined below:
| PRED_FILL | Used in the predicate_eval benchmark as the string-column width knob (filler chars per marker). | `30` |
| RG_SIZE | Used in the parquet_row_filter_skip benchmark as the Parquet `max_row_group_size` for the generated datasets. | `1000000` |
| PSQ_ROWS | Used in the projection_subquery benchmark to size the two inline tables. | `30000` |
+| SPILL_VIEWS_LIMIT_REPEATED | Used in the spill_views benchmark as the memory limit of the `repeated` subgroup. | `40M` |
+| SPILL_VIEWS_LIMIT_DISTINCT | Used in the spill_views benchmark as the memory limit of the `distinct` subgroup. | `96M` |
## How it works
diff --git a/benchmarks/sql_benchmarks/spill_views/benchmarks/distinct/q04.benchmark b/benchmarks/sql_benchmarks/spill_views/benchmarks/distinct/q04.benchmark
index f705df06ee..d3bb3c5b43 100644
--- a/benchmarks/sql_benchmarks/spill_views/benchmarks/distinct/q04.benchmark
+++ b/benchmarks/sql_benchmarks/spill_views/benchmarks/distinct/q04.benchmark
@@ -4,6 +4,6 @@ template sql_benchmarks/spill_views/spill_views.benchmark.template
QPAD=04
DATASET=distinct
NAME=q04_sort_string_all_distinct
-SPILL_LIMIT=96M
+SPILL_LIMIT=${SPILL_VIEWS_LIMIT_DISTINCT:-96M}
VIEW_TYPE=Utf8View
DISTINCT_S=1000000
diff --git a/benchmarks/sql_benchmarks/spill_views/benchmarks/distinct/q05.benchmark b/benchmarks/sql_benchmarks/spill_views/benchmarks/distinct/q05.benchmark
index 6af904c2a7..24dad0366c 100644
--- a/benchmarks/sql_benchmarks/spill_views/benchmarks/distinct/q05.benchmark
+++ b/benchmarks/sql_benchmarks/spill_views/benchmarks/distinct/q05.benchmark
@@ -4,6 +4,6 @@ template sql_benchmarks/spill_views/spill_views.benchmark.template
QPAD=05
DATASET=distinct
NAME=q05_group_by_string_all_distinct
-SPILL_LIMIT=96M
+SPILL_LIMIT=${SPILL_VIEWS_LIMIT_DISTINCT:-96M}
VIEW_TYPE=Utf8View
DISTINCT_S=1000000
diff --git a/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q01.benchmark b/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q01.benchmark
index cdbfa55f6c..21d2463ea6 100644
--- a/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q01.benchmark
+++ b/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q01.benchmark
@@ -4,6 +4,6 @@ template sql_benchmarks/spill_views/spill_views.benchmark.template
QPAD=01
DATASET=low_card_1
NAME=q01_sort_string_1_distinct
-SPILL_LIMIT=40M
+SPILL_LIMIT=${SPILL_VIEWS_LIMIT_REPEATED:-40M}
VIEW_TYPE=Utf8View
DISTINCT_S=1
diff --git a/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q02.benchmark b/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q02.benchmark
index 7473cb3e58..365b1cb34b 100644
--- a/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q02.benchmark
+++ b/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q02.benchmark
@@ -4,6 +4,6 @@ template sql_benchmarks/spill_views/spill_views.benchmark.template
QPAD=02
DATASET=low_card_1000
NAME=q02_sort_string_1000_distinct
-SPILL_LIMIT=40M
+SPILL_LIMIT=${SPILL_VIEWS_LIMIT_REPEATED:-40M}
VIEW_TYPE=Utf8View
DISTINCT_S=1000
diff --git a/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q03.benchmark b/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q03.benchmark
index 659d492ba8..b92406933a 100644
--- a/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q03.benchmark
+++ b/benchmarks/sql_benchmarks/spill_views/benchmarks/repeated/q03.benchmark
@@ -4,6 +4,6 @@ template sql_benchmarks/spill_views/spill_views.benchmark.template
QPAD=03
DATASET=low_card_1000_binary
NAME=q03_sort_binary_1000_distinct
-SPILL_LIMIT=40M
+SPILL_LIMIT=${SPILL_VIEWS_LIMIT_REPEATED:-40M}
VIEW_TYPE=BinaryView
DISTINCT_S=1000
diff --git a/benchmarks/sql_benchmarks/spill_views/spill_views.benchmark.template b/benchmarks/sql_benchmarks/spill_views/spill_views.benchmark.template
index a58dfc9643..e8fdd2a57d 100644
--- a/benchmarks/sql_benchmarks/spill_views/spill_views.benchmark.template
+++ b/benchmarks/sql_benchmarks/spill_views/spill_views.benchmark.template
@@ -7,6 +7,10 @@
# VIEW_TYPE expected Arrow type of column `s` (Utf8View, BinaryView)
# DISTINCT_S expected number of distinct values of `s`
#
+# Each qNN.benchmark takes SPILL_LIMIT from the subgroup's environment
+# variable, SPILL_VIEWS_LIMIT_REPEATED or SPILL_VIEWS_LIMIT_DISTINCT, and
+# falls back to the value the suite metadata declares as the default.
+#
# The load script COPYs a 1M-row Parquet file into scratch/ and registers it
# as table `t` (id BIGINT in shuffled order, s = a 64-byte payload). Reading
# the data back from Parquet is load-bearing: the Parquet reader makes the
@@ -17,6 +21,12 @@
# limits were sized for 4 partitions so that each query spills on main
# without running out of memory. Do not remove the pin: with more partitions
# the per-partition sort reservations can exceed the limit.
+#
+# The assert below reads the limit back from information_schema, so it also
+# shows which limit an overridden run used. The memory pool reports the
+# limit in whole K, M or G units, so give an overridden limit in the same
+# form: 128M passes the assert, 1024M and 1.5G do not, because the pool
+# reports both as 1G.
load sql_benchmarks/spill_views/load/${DATASET}.sql
diff --git a/benchmarks/sql_benchmarks/spill_views/spill_views.suite b/benchmarks/sql_benchmarks/spill_views/spill_views.suite
index ce314862fe..d027ac269c 100644
--- a/benchmarks/sql_benchmarks/spill_views/spill_views.suite
+++ b/benchmarks/sql_benchmarks/spill_views/spill_views.suite
@@ -1,7 +1,21 @@
-description = "Sort and aggregation queries that spill StringView and BinaryView columns read from Parquet (https://github.com/apache/datafusion/issues/23564). Each query sets its own memory limit and target_partitions = 4, so it spills with any environment. The repeated subgroup sorts rows whose string payload has few distinct values, so the Parquet reader makes views into one shared dictionary buffer and the spill writer can compact that buffer. The distinct subgroup sorts and groups all-distinct strings, where compaction cannot remove data and only adds work."
+description = "Sort and aggregation queries that spill StringView and BinaryView columns read from Parquet (https://github.com/apache/datafusion/issues/23564). Each query sets its own memory limit and target_partitions = 4, so it spills with any environment. The repeated subgroup sorts rows whose string payload has few distinct values, so the Parquet reader makes views into one shared dictionary buffer and the spill writer can compact that buffer. The distinct subgroup sorts and groups all-distinct strings, where compaction cannot remove data and only adds work. Set the memory limit of each subgroup with SPILL_VIEWS_LIMIT_REPEATED and SPILL_VIEWS_LIMIT_DISTINCT, for example to let both sides of a comparison finish when one of them runs out of memory at the default."
query_pattern = "q{QUERY_ID_PADDED}.benchmark"
+[[options]]
+name = "limit-repeated"
+env = "SPILL_VIEWS_LIMIT_REPEATED"
+default = "40M"
+values = ["40M", "..."]
+help = "Sets the memory limit of the repeated subgroup. The queries must still spill, so keep the value well below the working set. Give a whole number of K, M or G units, for example 128M: the run asserts the limit that the memory pool reports back, and the pool reports 1024M as 1G."
+
+[[options]]
+name = "limit-distinct"
+env = "SPILL_VIEWS_LIMIT_DISTINCT"
+default = "96M"
+values = ["96M", "..."]
+help = "Sets the memory limit of the distinct subgroup. The queries must still spill, so keep the value well below the working set. Give a whole number of K, M or G units, for example 128M: the run asserts the limit that the memory pool reports back, and the pool reports 1024M as 1G."
+
[[examples]]
command = "cargo run --release --bin benchmark_runner -- spill_views"
description = "Run all spill_views queries."
@@ -9,3 +23,7 @@ description = "Run all spill_views queries."
[[examples]]
command = "cargo run --release --bin benchmark_runner -- spill_views --subgroup distinct"
description = "Run only the all-distinct (cost case) queries."
+
+[[examples]]
+command = "cargo run --release --bin benchmark_runner -- spill_views --subgroup distinct --limit-distinct 128M"
+description = "Raise the memory limit of the distinct subgroup, for example when a query runs out of memory at the default."