Apache DataFusion SQL Query Engine
chore: Add TPCDS benchmarks (#19138)
Oleks Ve8384fbbf591
diff --git a/benchmarks/README.md b/benchmarks/README.mdindex 13b73074e0..bf9b50028b 100644--- a/benchmarks/README.md+++ b/benchmarks/README.md@@ -119,7 +119,6 @@ You can also invoke the helper directly if you need to customise arguments furth./benchmarks/compile_profile.py --profiles dev release --data /path/to/tpch_sf1```-## Benchmark with modified configurations### Select join algorithm@@ -147,6 +146,19 @@ To verify that datafusion picked up your configuration, run the benchmarks with## Comparing performance of main and a branch+For TPC-H+```shell+./benchmarks/compare_tpch.sh main mybranch+```++For TPC-DS.+To get data in `DATA_DIR` for TPCDS, please follow instructions in `./benchmarks/bench.sh data tcpds`+```shell+DATA_DIR=../../datafusion-benchmarks/tpcds/data/sf1/ ./benchmarks/compare_tpcds.sh main mybranch+```++Alternatively you can compare manually followng the example velor+```shellgit checkout main@@ -299,7 +311,6 @@ This will produce output like:└──────────────┴──────────────┴──────────────┴───────────────┘```-# Benchmark RunnerThe `dfbench` program contains subcommands to run the various@@ -339,24 +350,28 @@ FLAGS:```# Profiling Memory Stats for each benchmark query+The `mem_profile` program wraps benchmark execution to measure memory usage statistics, such as peak RSS. It runs each benchmark query in a separate subprocess, capturing the child process’s stdout to print structured output.Subcommands supported by mem_profile are the subset of those in `dfbench`.-Currently supported benchmarks include: Clickbench, H2o, Imdb, SortTpch, Tpch+Currently supported benchmarks include: Clickbench, H2o, Imdb, SortTpch, Tpch, TPCDSBefore running benchmarks, `mem_profile` automatically compiles the benchmark binary (`dfbench`) using `cargo build`. Note that the build profile used for `dfbench` is not tied to the profile used for running `mem_profile` itself. We can explicitly specify the desired build profile using the `--bench-profile` option (e.g. release-nonlto). By prebuilding the binary and running each query in a separate process, we can ensure accurate memory statistics.Currently, `mem_profile` only supports `mimalloc` as the memory allocator, since it relies on `mimalloc`'s API to collect memory statistics.-Because it runs the compiled binary directly from the target directory, make sure your working directory is the top-level datafusion/ directory, where the target/ is also located.+Because it runs the compiled binary directly from the target directory, make sure your working directory is the top-level datafusion/ directory, where the target/ is also located.++The benchmark subcommand (e.g., `tpch`) and all following arguments are passed directly to `dfbench`. Be sure to specify `--bench-profile` before the benchmark subcommand.-The benchmark subcommand (e.g., `tpch`) and all following arguments are passed directly to `dfbench`. Be sure to specify `--bench-profile` before the benchmark subcommand.+Example:-Example:```shelldatafusion$ cargo run --profile release-nonlto --bin mem_profile -- --bench-profile release-nonlto tpch --path benchmarks/data/tpch_sf1 --partitions 4 --format parquet```+Example Output:+```Query Time (ms) Peak RSS Peak Commit Major Page Faults----------------------------------------------------------------@@ -385,19 +400,21 @@ Query Time (ms) Peak RSS Peak Commit Major Page Faults```## Reported Metrics+When running benchmarks, `mem_profile` collects several memory-related statistics using the mimalloc API:-- Peak RSS (Resident Set Size):-The maximum amount of physical memory used by the process.-This is a process-level metric collected via OS-specific mechanisms and is not mimalloc-specific.+- Peak RSS (Resident Set Size):+ The maximum amount of physical memory used by the process.+ This is a process-level metric collected via OS-specific mechanisms and is not mimalloc-specific.- Peak Commit:-The peak amount of memory committed by the allocator (i.e., total virtual memory reserved).-This is mimalloc-specific. It gives a more allocator-aware view of memory usage than RSS.+ The peak amount of memory committed by the allocator (i.e., total virtual memory reserved).+ This is mimalloc-specific. It gives a more allocator-aware view of memory usage than RSS.- Major Page Faults:-The number of major page faults triggered during execution.-This metric is obtained from the operating system and is not mimalloc-specific.+ The number of major page faults triggered during execution.+ This metric is obtained from the operating system and is not mimalloc-specific.+# Writing a new benchmark## Creating or downloading data outside of the benchmark@@ -586,6 +603,34 @@ This benchmarks is derived from the [TPC-H][1] version[2]: https://github.com/databricks/tpch-dbgen.git,[2.17.1]: https://www.tpc.org/tpc_documents_current_versions/pdf/tpc-h_v2.17.1.pdf+## TPCDS++Run the tpcds benchmark.++For data please clone `datafusion-benchmarks` repo which contains the predefined parquet data with SF1.++```shell+git clone https://github.com/apache/datafusion-benchmarks+```++Then run the benchmark with the following command:++```shell+DATA_DIR=../../datafusion-benchmarks/tpcds/data/sf1/ ./benchmarks/bench.sh run tpcds+```++Alternatively benchmark the specific query++```shell+DATA_DIR=../../datafusion-benchmarks/tpcds/data/sf1/ ./benchmarks/bench.sh run tpcds 30+```++More help++```shell+cargo run --release --bin dfbench -- tpcds --help+```+## External AggregationRun the benchmark for aggregations with limited memory.diff --git a/benchmarks/bench.sh b/benchmarks/bench.shindex 8c0a925232..6b94dadeb9 100755--- a/benchmarks/bench.sh+++ b/benchmarks/bench.sh@@ -87,6 +87,9 @@ tpch10: TPCH inspired benchmark on Scale Factor (SF) 10 (~10GB),tpch_csv10: TPCH inspired benchmark on Scale Factor (SF) 10 (~10GB), single csv file per table, hash jointpch_mem10: TPCH inspired benchmark on Scale Factor (SF) 10 (~10GB), query from memory+# TPC-DS Benchmarks+tpcds: TPCDS inspired benchmark on Scale Factor (SF) 1 (~1GB), single parquet file per table, hash join+# Extended TPC-H Benchmarkssort_tpch: Benchmark of sorting speed for end-to-end sort queries on TPC-H dataset (SF=1)sort_tpch10: Benchmark of sorting speed for end-to-end sort queries on TPC-H dataset (SF=10)@@ -220,6 +223,9 @@ main() {tpch_csv10)data_tpch "10" "csv";;+ tpcds)+ data_tpcds+ ;;clickbench_1)data_clickbench_1;;@@ -388,6 +394,7 @@ main() {run_external_aggrrun_nljrun_hj+ run_tpcds;;tpch)run_tpch "1" "parquet"@@ -407,6 +414,9 @@ main() {tpch_mem10)run_tpch_mem "10";;+ tpcds)+ run_tpcds+ ;;cancellation)run_cancellation;;@@ -601,6 +611,24 @@ data_tpch() {exit 1}+# Points to TPCDS data generation instructions+data_tpcds() {+ TPCDS_DIR="${DATA_DIR}"++ # Check if TPCDS data directory exists+ if [ ! -d "${TPCDS_DIR}" ]; then+ echo ""+ echo "For TPC-DS data generation, please clone the datafusion-benchmarks repository:"+ echo " git clone https://github.com/apache/datafusion-benchmarks"+ echo ""+ return 1+ fi++ echo ""+ echo "TPC-DS data already exists in ${TPCDS_DIR}"+ echo ""+}+# Runs the tpch benchmarkrun_tpch() {SCALE_FACTOR=$1@@ -634,6 +662,37 @@ run_tpch_mem() {debug_run $CARGO_COMMAND --bin dfbench -- tpch --iterations 5 --path "${TPCH_DIR}" --prefer_hash_join "${PREFER_HASH_JOIN}" -m --format parquet -o "${RESULTS_FILE}" ${QUERY_ARG}}+# Runs the tpcds benchmark+run_tpcds() {+ TPCDS_DIR="${DATA_DIR}"++ # Check if TPCDS data directory exists+ if [ ! -d "${TPCDS_DIR}" ]; then+ echo "Error: TPC-DS data directory does not exist: ${TPCDS_DIR}" >&2+ echo "" >&2+ echo "Please prepare TPC-DS data first by following instructions:" >&2+ echo " ./bench.sh data tpcds" >&2+ echo "" >&2+ exit 1+ fi++ # Check if directory contains parquet files+ if ! find "${TPCDS_DIR}" -name "*.parquet" -print -quit | grep -q .; then+ echo "Error: TPC-DS data directory exists but contains no parquet files: ${TPCDS_DIR}" >&2+ echo "" >&2+ echo "Please prepare TPC-DS data first by following instructions:" >&2+ echo " ./bench.sh data tpcds" >&2+ echo "" >&2+ exit 1+ fi++ RESULTS_FILE="${RESULTS_DIR}/tpcds_sf1.json"+ echo "RESULTS_FILE: ${RESULTS_FILE}"+ echo "Running tpcds benchmark..."++ debug_run $CARGO_COMMAND --bin dfbench -- tpcds --iterations 5 --path "${TPCDS_DIR}" --query_path "../datafusion/core/tests/tpc-ds" --prefer_hash_join "${PREFER_HASH_JOIN}" -o "${RESULTS_FILE}" ${QUERY_ARG}+}+# Runs the compile profile benchmark helperrun_compile_profile() {local profiles=("$@")diff --git a/benchmarks/compare_tpcds.sh b/benchmarks/compare_tpcds.shnew file mode 100755index 0000000000..48331a7c75--- /dev/null+++ b/benchmarks/compare_tpcds.sh@@ -0,0 +1,58 @@+#!/usr/bin/env bash+# Licensed to the Apache Software Foundation (ASF) under one+# or more contributor license agreements. See the NOTICE file+# distributed with this work for additional information+# regarding copyright ownership. The ASF licenses this file+# to you under the Apache License, Version 2.0 (the+# "License"); you may not use this file except in compliance+# with the License. You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing,+# software distributed under the License is distributed on an+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+# KIND, either express or implied. See the License for the+# specific language governing permissions and limitations+# under the License.++# Compare TPC-DS benchmarks between two branches++set -e++SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )++usage() {+ echo "Usage: $0 <branch1> <branch2>"+ echo ""+ echo "Example: $0 main dev2"+ echo ""+ echo "Note: TPC-DS benchmarks are not currently implemented in bench.sh"+ exit 1+}++BRANCH1=${1:-""}+BRANCH2=${2:-""}++if [ -z "$BRANCH1" ] || [ -z "$BRANCH2" ]; then+ usage+fi++# Store current branch+CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)++echo "Comparing TPC-DS benchmarks: ${BRANCH1} vs ${BRANCH2}"++# Run benchmark on first branch+git checkout "$BRANCH1"+./benchmarks/bench.sh run tpcds++# Run benchmark on second branch+git checkout "$BRANCH2"+./benchmarks/bench.sh run tpcds++# Compare results+./benchmarks/bench.sh compare "$BRANCH1" "$BRANCH2"++# Return to original branch+git checkout "$CURRENT_BRANCH"\ No newline at end of filediff --git a/benchmarks/compare_tpch.sh b/benchmarks/compare_tpch.shnew file mode 100755index 0000000000..85e8da29ce--- /dev/null+++ b/benchmarks/compare_tpch.sh@@ -0,0 +1,56 @@+#!/usr/bin/env bash+# Licensed to the Apache Software Foundation (ASF) under one+# or more contributor license agreements. See the NOTICE file+# distributed with this work for additional information+# regarding copyright ownership. The ASF licenses this file+# to you under the Apache License, Version 2.0 (the+# "License"); you may not use this file except in compliance+# with the License. You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing,+# software distributed under the License is distributed on an+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+# KIND, either express or implied. See the License for the+# specific language governing permissions and limitations+# under the License.++# Compare TPC-H benchmarks between two branches++set -e++SCRIPT_DIR=$( cd -- "$( dirname -- "${BASH_SOURCE[0]}" )" &> /dev/null && pwd )++usage() {+ echo "Usage: $0 <branch1> <branch2>"+ echo ""+ echo "Example: $0 main dev2"+ exit 1+}++BRANCH1=${1:-""}+BRANCH2=${2:-""}++if [ -z "$BRANCH1" ] || [ -z "$BRANCH2" ]; then+ usage+fi++# Store current branch+CURRENT_BRANCH=$(git rev-parse --abbrev-ref HEAD)++echo "Comparing TPC-H benchmarks: ${BRANCH1} vs ${BRANCH2}"++# Run benchmark on first branch+git checkout "$BRANCH1"+./benchmarks/bench.sh run tpch++# Run benchmark on second branch+git checkout "$BRANCH2"+./benchmarks/bench.sh run tpch++# Compare results+./benchmarks/bench.sh compare "$BRANCH1" "$BRANCH2"++# Return to original branch+git checkout "$CURRENT_BRANCH"\ No newline at end of filediff --git a/benchmarks/src/bin/dfbench.rs b/benchmarks/src/bin/dfbench.rsindex 68dea188dc..6fc382822d 100644--- a/benchmarks/src/bin/dfbench.rs+++ b/benchmarks/src/bin/dfbench.rs@@ -34,7 +34,7 @@ static ALLOC: snmalloc_rs::SnMalloc = snmalloc_rs::SnMalloc;static ALLOC: mimalloc::MiMalloc = mimalloc::MiMalloc;use datafusion_benchmarks::{- cancellation, clickbench, h2o, hj, imdb, nlj, sort_tpch, tpch,+ cancellation, clickbench, h2o, hj, imdb, nlj, sort_tpch, tpcds, tpch,};#[derive(Debug, StructOpt)]@@ -48,6 +48,7 @@ enum Options {Nlj(nlj::RunOpt),SortTpch(sort_tpch::RunOpt),Tpch(tpch::RunOpt),+ Tpcds(tpcds::RunOpt),}// Main benchmark runner entrypoint@@ -64,5 +65,6 @@ pub async fn main() -> Result<()> {Options::Nlj(opt) => opt.run().await,Options::SortTpch(opt) => opt.run().await,Options::Tpch(opt) => Box::pin(opt.run()).await,+ Options::Tpcds(opt) => Box::pin(opt.run()).await,}}diff --git a/benchmarks/src/lib.rs b/benchmarks/src/lib.rsindex 07cffa5ae4..d885ec94a3 100644--- a/benchmarks/src/lib.rs+++ b/benchmarks/src/lib.rs@@ -23,5 +23,6 @@ pub mod hj;pub mod imdb;pub mod nlj;pub mod sort_tpch;+pub mod tpcds;pub mod tpch;pub mod util;diff --git a/benchmarks/src/tpcds/mod.rs b/benchmarks/src/tpcds/mod.rsnew file mode 100644index 0000000000..4829eb9fd3--- /dev/null+++ b/benchmarks/src/tpcds/mod.rs@@ -0,0 +1,19 @@+// Licensed to the Apache Software Foundation (ASF) under one+// or more contributor license agreements. See the NOTICE file+// distributed with this work for additional information+// regarding copyright ownership. The ASF licenses this file+// to you under the Apache License, Version 2.0 (the+// "License"); you may not use this file except in compliance+// with the License. You may obtain a copy of the License at+//+// http://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing,+// software distributed under the License is distributed on an+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+// KIND, either express or implied. See the License for the+// specific language governing permissions and limitations+// under the License.++mod run;+pub use run::RunOpt;diff --git a/benchmarks/src/tpcds/run.rs b/benchmarks/src/tpcds/run.rsnew file mode 100644index 0000000000..3b7e4ad135--- /dev/null+++ b/benchmarks/src/tpcds/run.rs@@ -0,0 +1,356 @@+// Licensed to the Apache Software Foundation (ASF) under one+// or more contributor license agreements. See the NOTICE file+// distributed with this work for additional information+// regarding copyright ownership. The ASF licenses this file+// to you under the Apache License, Version 2.0 (the+// "License"); you may not use this file except in compliance+// with the License. You may obtain a copy of the License at+//+// http://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing,+// software distributed under the License is distributed on an+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+// KIND, either express or implied. See the License for the+// specific language governing permissions and limitations+// under the License.++use std::fs;+use std::path::PathBuf;+use std::sync::Arc;++use crate::util::{print_memory_stats, BenchmarkRun, CommonOpt, QueryResult};++use arrow::record_batch::RecordBatch;+use arrow::util::pretty::{self, pretty_format_batches};+use datafusion::datasource::file_format::parquet::ParquetFormat;+use datafusion::datasource::listing::{+ ListingOptions, ListingTable, ListingTableConfig, ListingTableUrl,+};+use datafusion::datasource::{MemTable, TableProvider};+use datafusion::error::Result;+use datafusion::physical_plan::display::DisplayableExecutionPlan;+use datafusion::physical_plan::{collect, displayable};+use datafusion::prelude::*;+use datafusion_common::instant::Instant;+use datafusion_common::utils::get_available_parallelism;+use datafusion_common::{plan_err, DEFAULT_PARQUET_EXTENSION};++use log::info;+use structopt::StructOpt;++// hack to avoid `default_value is meaningless for bool` errors+type BoolDefaultTrue = bool;+pub const TPCDS_QUERY_START_ID: usize = 1;+pub const TPCDS_QUERY_END_ID: usize = 99;++pub const TPCDS_TABLES: &[&str] = &[+ "call_center",+ "customer_address",+ "household_demographics",+ "promotion",+ "store_sales",+ "web_page",+ "catalog_page",+ "customer_demographics",+ "income_band",+ "reason",+ "store",+ "web_returns",+ "catalog_returns",+ "customer",+ "inventory",+ "ship_mode",+ "time_dim",+ "web_sales",+ "catalog_sales",+ "date_dim",+ "item",+ "store_returns",+ "warehouse",+ "web_site",+];++/// Get the SQL statements from the specified query file+pub fn get_query_sql(base_query_path: &str, query: usize) -> Result<Vec<String>> {+ if query > 0 && query < 100 {+ let filename = format!("{base_query_path}/{query}.sql");+ let mut errors = vec![];+ match fs::read_to_string(&filename) {+ Ok(contents) => {+ return Ok(contents+ .split(';')+ .map(|s| s.trim())+ .filter(|s| !s.is_empty())+ .map(|s| s.to_string())+ .collect());+ }+ Err(e) => errors.push(format!("{filename}: {e}")),+ };++ plan_err!("invalid query. Could not find query: {:?}", errors)+ } else {+ plan_err!("invalid query. Expected value between 1 and 99")+ }+}++/// Run the tpcds benchmark.+#[derive(Debug, StructOpt, Clone)]+#[structopt(verbatim_doc_comment)]+pub struct RunOpt {+ /// Query number. If not specified, runs all queries+ #[structopt(short, long)]+ pub query: Option<usize>,++ /// Common options+ #[structopt(flatten)]+ common: CommonOpt,++ /// Path to data files+ #[structopt(parse(from_os_str), required = true, short = "p", long = "path")]+ path: PathBuf,++ /// Path to query files+ #[structopt(parse(from_os_str), required = true, short = "Q", long = "query_path")]+ query_path: PathBuf,++ /// Load the data into a MemTable before executing the query+ #[structopt(short = "m", long = "mem-table")]+ mem_table: bool,++ /// Path to machine readable output file+ #[structopt(parse(from_os_str), short = "o", long = "output")]+ output_path: Option<PathBuf>,++ /// Whether to disable collection of statistics (and cost based optimizations) or not.+ #[structopt(short = "S", long = "disable-statistics")]+ disable_statistics: bool,++ /// If true then hash join used, if false then sort merge join+ /// True by default.+ #[structopt(short = "j", long = "prefer_hash_join", default_value = "true")]+ prefer_hash_join: BoolDefaultTrue,++ /// If true then Piecewise Merge Join can be used, if false then it will opt for Nested Loop Join+ /// False by default.+ #[structopt(+ short = "w",+ long = "enable_piecewise_merge_join",+ default_value = "false"+ )]+ enable_piecewise_merge_join: BoolDefaultTrue,++ /// Mark the first column of each table as sorted in ascending order.+ /// The tables should have been created with the `--sort` option for this to have any effect.+ #[structopt(short = "t", long = "sorted")]+ sorted: bool,+}++impl RunOpt {+ pub async fn run(self) -> Result<()> {+ println!("Running benchmarks with the following options: {self:?}");+ let query_range = match self.query {+ Some(query_id) => query_id..=query_id,+ None => TPCDS_QUERY_START_ID..=TPCDS_QUERY_END_ID,+ };++ let mut benchmark_run = BenchmarkRun::new();+ let mut config = self+ .common+ .config()?+ .with_collect_statistics(!self.disable_statistics);+ config.options_mut().optimizer.prefer_hash_join = self.prefer_hash_join;+ config.options_mut().optimizer.enable_piecewise_merge_join =+ self.enable_piecewise_merge_join;+ let rt_builder = self.common.runtime_env_builder()?;+ let ctx = SessionContext::new_with_config_rt(config, rt_builder.build_arc()?);+ // register tables+ self.register_tables(&ctx).await?;++ for query_id in query_range {+ benchmark_run.start_new_case(&format!("Query {query_id}"));+ let query_run = self.benchmark_query(query_id, &ctx).await;+ match query_run {+ Ok(query_results) => {+ for iter in query_results {+ benchmark_run.write_iter(iter.elapsed, iter.row_count);+ }+ }+ Err(e) => {+ benchmark_run.mark_failed();+ eprintln!("Query {query_id} failed: {e}");+ }+ }+ }+ benchmark_run.maybe_write_json(self.output_path.as_ref())?;+ benchmark_run.maybe_print_failures();+ Ok(())+ }++ async fn benchmark_query(+ &self,+ query_id: usize,+ ctx: &SessionContext,+ ) -> Result<Vec<QueryResult>> {+ let mut millis = vec![];+ // run benchmark+ let mut query_results = vec![];++ let sql = &get_query_sql(self.query_path.to_str().unwrap(), query_id)?;++ if self.common.debug {+ println!("=== SQL for query {query_id} ===\n{}\n", sql.join(";\n"));+ }++ for i in 0..self.iterations() {+ let start = Instant::now();++ // query 15 is special, with 3 statements. the second statement is the one from which we+ // want to capture the results+ let mut result = vec![];++ for query in sql {+ result = self.execute_query(ctx, query).await?;+ }++ let elapsed = start.elapsed();+ let ms = elapsed.as_secs_f64() * 1000.0;+ millis.push(ms);+ info!("output:\n\n{}\n\n", pretty_format_batches(&result)?);+ let row_count = result.iter().map(|b| b.num_rows()).sum();+ println!(+ "Query {query_id} iteration {i} took {ms:.1} ms and returned {row_count} rows"+ );+ query_results.push(QueryResult { elapsed, row_count });+ }++ let avg = millis.iter().sum::<f64>() / millis.len() as f64;+ println!("Query {query_id} avg time: {avg:.2} ms");++ // Print memory stats using mimalloc (only when compiled with --features mimalloc_extended)+ print_memory_stats();++ Ok(query_results)+ }++ async fn register_tables(&self, ctx: &SessionContext) -> Result<()> {+ for table in TPCDS_TABLES {+ let table_provider = { self.get_table(ctx, table).await? };++ if self.mem_table {+ println!("Loading table '{table}' into memory");+ let start = Instant::now();+ let memtable =+ MemTable::load(table_provider, Some(self.partitions()), &ctx.state())+ .await?;+ println!(+ "Loaded table '{}' into memory in {} ms",+ table,+ start.elapsed().as_millis()+ );+ ctx.register_table(*table, Arc::new(memtable))?;+ } else {+ ctx.register_table(*table, table_provider)?;+ }+ }+ Ok(())+ }++ async fn execute_query(+ &self,+ ctx: &SessionContext,+ sql: &str,+ ) -> Result<Vec<RecordBatch>> {+ let debug = self.common.debug;+ let plan = ctx.sql(sql).await?;+ let (state, plan) = plan.into_parts();++ if debug {+ println!("=== Logical plan ===\n{plan}\n");+ }++ let plan = state.optimize(&plan)?;+ if debug {+ println!("=== Optimized logical plan ===\n{plan}\n");+ }+ let physical_plan = state.create_physical_plan(&plan).await?;+ if debug {+ println!(+ "=== Physical plan ===\n{}\n",+ displayable(physical_plan.as_ref()).indent(true)+ );+ }+ let result = collect(physical_plan.clone(), state.task_ctx()).await?;+ if debug {+ println!(+ "=== Physical plan with metrics ===\n{}\n",+ DisplayableExecutionPlan::with_metrics(physical_plan.as_ref())+ .indent(true)+ );+ if !result.is_empty() {+ // do not call print_batches if there are no batches as the result is confusing+ // and makes it look like there is a batch with no columns+ pretty::print_batches(&result)?;+ }+ }+ Ok(result)+ }++ async fn get_table(+ &self,+ ctx: &SessionContext,+ table: &str,+ ) -> Result<Arc<dyn TableProvider>> {+ let path = self.path.to_str().unwrap();+ let target_partitions = self.partitions();++ // Obtain a snapshot of the SessionState+ let state = ctx.state();+ let path = format!("{path}/{table}.parquet");++ // Check if the file exists+ if !std::path::Path::new(&path).exists() {+ eprintln!("Warning registering {table}: Table file does not exist: {path}");+ }++ let format = ParquetFormat::default()+ .with_options(ctx.state().table_options().parquet.clone());++ let table_path = ListingTableUrl::parse(path)?;+ let options = ListingOptions::new(Arc::new(format))+ .with_file_extension(DEFAULT_PARQUET_EXTENSION)+ .with_target_partitions(target_partitions)+ .with_collect_stat(state.config().collect_statistics());+ let schema = options.infer_schema(&state, &table_path).await?;++ if self.common.debug {+ println!(+ "Inferred schema from {table_path} for table '{table}':\n{schema:#?}\n"+ );+ }++ let options = if self.sorted {+ let key_column_name = schema.fields()[0].name();+ options+ .with_file_sort_order(vec![vec![col(key_column_name).sort(true, false)]])+ } else {+ options+ };++ let config = ListingTableConfig::new(table_path)+ .with_listing_options(options)+ .with_schema(schema);++ Ok(Arc::new(ListingTable::try_new(config)?))+ }++ fn iterations(&self) -> usize {+ self.common.iterations+ }++ fn partitions(&self) -> usize {+ self.common+ .partitions+ .unwrap_or_else(get_available_parallelism)+ }+}diff --git a/benchmarks/src/tpch/run.rs b/benchmarks/src/tpch/run.rsindex 63a1cd58d4..84dfc53a5e 100644--- a/benchmarks/src/tpch/run.rs+++ b/benchmarks/src/tpch/run.rs@@ -94,7 +94,11 @@ pub struct RunOpt {/// If true then Piecewise Merge Join can be used, if false then it will opt for Nested Loop Join/// False by default.- #[structopt(long = "enable_piecewise_merge_join", default_value = "false")]+ #[structopt(+ short = "w",+ long = "enable_piecewise_merge_join",+ default_value = "false"+ )]enable_piecewise_merge_join: BoolDefaultTrue,/// Mark the first column of each table as sorted in ascending order.diff --git a/datafusion/core/tests/tpc-ds/30.sql b/datafusion/core/tests/tpc-ds/30.sqlindex 78f34b807e..80624f4900 100644--- a/datafusion/core/tests/tpc-ds/30.sql+++ b/datafusion/core/tests/tpc-ds/30.sql@@ -14,7 +14,7 @@ with customer_total_return as,ca_state)select c_customer_id,c_salutation,c_first_name,c_last_name,c_preferred_cust_flag,c_birth_day,c_birth_month,c_birth_year,c_birth_country,c_login,c_email_address- ,c_last_review_date_sk,ctr_total_return+ ,c_last_review_date,ctr_total_returnfrom customer_total_return ctr1,customer_address,customer@@ -26,7 +26,7 @@ with customer_total_return asand ctr1.ctr_customer_sk = c_customer_skorder by c_customer_id,c_salutation,c_first_name,c_last_name,c_preferred_cust_flag,c_birth_day,c_birth_month,c_birth_year,c_birth_country,c_login,c_email_address- ,c_last_review_date_sk,ctr_total_return+ ,c_last_review_date,ctr_total_returnlimit 100;diff --git a/test-utils/src/tpcds.rs b/test-utils/src/tpcds.rsindex ce5bac5bfd..a12ae8ceae 100644--- a/test-utils/src/tpcds.rs+++ b/test-utils/src/tpcds.rs@@ -299,7 +299,7 @@ pub fn tpcds_schemas() -> Vec<TableDef> {Field::new("c_birth_country", DataType::Utf8, false),Field::new("c_login", DataType::Utf8, false),Field::new("c_email_address", DataType::Utf8, false),- Field::new("c_last_review_date_sk", DataType::Utf8, false),+ Field::new("c_last_review_date", DataType::Utf8, false),]),),TableDef::new(