Apache DataFusion SQL Query Engine
feat: rewrite inner joins against a single constant row into a filter (#26404)
xudong.w2c2ecae08bd4
diff --git a/datafusion/core/src/optimizer_rule_reference.md b/datafusion/core/src/optimizer_rule_reference.mdindex 298a01edd6..13e7f62e72 100644--- a/datafusion/core/src/optimizer_rule_reference.md+++ b/datafusion/core/src/optimizer_rule_reference.md@@ -73,22 +73,23 @@ in multiple phases.| ----- | ------------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------ || 1 | `OutputRequirements` | add phase | Adds helper nodes so output requirements survive later physical rewrites. || 2 | `aggregate_statistics` | - | Uses exact source statistics to answer some aggregates without scanning data. |-| 3 | `join_selection` | - | Chooses join implementation, build side, and partition mode from statistics and stream properties. |-| 4 | `LimitedDistinctAggregation` | - | Pushes limit hints into grouped distinct-style aggregations when only a small result is needed. |-| 5 | `FilterPushdown` | pre-optimization phase | Pushes supported physical filters down toward data sources before distribution and sorting are enforced. |-| 6 | `WindowTopN` | - | Replaces eligible row-number window and filter patterns with per-partition TopK execution. |-| 7 | `EnsureRequirements` | - | Enforces both distribution and sorting requirements in a single idempotent rule. |-| 8 | `CombinePartialFinalAggregate` | - | Collapses adjacent partial and final aggregates when the distributed shape makes them redundant. |-| 9 | `OptimizeAggregateOrder` | - | Updates aggregate expressions to use the best ordering once sort requirements are known. |-| 10 | `ProjectionPushdown` | early pass | Pushes projections toward inputs before later physical rewrites add more limit and TopK structure. |-| 11 | `OutputRequirements` | remove phase | Removes the temporary output-requirement helper nodes after requirement-sensitive planning is done. |-| 12 | `LimitAggregation` | - | Passes a limit hint into eligible aggregations so they can keep fewer accumulator buckets. |-| 13 | `LimitPushPastWindows` | - | Pushes fetch limits through bounded window operators when doing so keeps the result correct. |-| 14 | `HashJoinBuffering` | - | Adds buffering on the probe side of hash joins so probing can start before build completion. |-| 15 | `LimitPushdown` | - | Moves physical limits into child operators or fetch-enabled variants to cut data early. |-| 16 | `TopKRepartition` | - | Pushes TopK below hash repartition when the partition key is a prefix of the sort key. |-| 17 | `ProjectionPushdown` | late pass | Runs projection pushdown again after limit and TopK rewrites expose new pruning opportunities. |-| 18 | `PushdownSort` | - | Pushes sort requirements into data sources that can already return sorted output. |-| 19 | `EnsureCooperative` | - | Wraps non-cooperative plan parts so long-running tasks yield fairly. |-| 20 | `FilterPushdown(Post)` | post-optimization phase | Pushes dynamic filters at the end of optimization, after plan references stop moving. |-| 21 | `SanityCheckPlan` | - | Validates that the final physical plan meets ordering, distribution, and infinite-input safety requirements. |+| 3 | `EliminateConstantJoin` | - | Replaces inner joins against a single row of literals with a filter that keeps the other input's order. |+| 4 | `join_selection` | - | Chooses join implementation, build side, and partition mode from statistics and stream properties. |+| 5 | `LimitedDistinctAggregation` | - | Pushes limit hints into grouped distinct-style aggregations when only a small result is needed. |+| 6 | `FilterPushdown` | pre-optimization phase | Pushes supported physical filters down toward data sources before distribution and sorting are enforced. |+| 7 | `WindowTopN` | - | Replaces eligible row-number window and filter patterns with per-partition TopK execution. |+| 8 | `EnsureRequirements` | - | Enforces both distribution and sorting requirements in a single idempotent rule. |+| 9 | `CombinePartialFinalAggregate` | - | Collapses adjacent partial and final aggregates when the distributed shape makes them redundant. |+| 10 | `OptimizeAggregateOrder` | - | Updates aggregate expressions to use the best ordering once sort requirements are known. |+| 11 | `ProjectionPushdown` | early pass | Pushes projections toward inputs before later physical rewrites add more limit and TopK structure. |+| 12 | `OutputRequirements` | remove phase | Removes the temporary output-requirement helper nodes after requirement-sensitive planning is done. |+| 13 | `LimitAggregation` | - | Passes a limit hint into eligible aggregations so they can keep fewer accumulator buckets. |+| 14 | `LimitPushPastWindows` | - | Pushes fetch limits through bounded window operators when doing so keeps the result correct. |+| 15 | `HashJoinBuffering` | - | Adds buffering on the probe side of hash joins so probing can start before build completion. |+| 16 | `LimitPushdown` | - | Moves physical limits into child operators or fetch-enabled variants to cut data early. |+| 17 | `TopKRepartition` | - | Pushes TopK below hash repartition when the partition key is a prefix of the sort key. |+| 18 | `ProjectionPushdown` | late pass | Runs projection pushdown again after limit and TopK rewrites expose new pruning opportunities. |+| 19 | `PushdownSort` | - | Pushes sort requirements into data sources that can already return sorted output. |+| 20 | `EnsureCooperative` | - | Wraps non-cooperative plan parts so long-running tasks yield fairly. |+| 21 | `FilterPushdown(Post)` | post-optimization phase | Pushes dynamic filters at the end of optimization, after plan references stop moving. |+| 22 | `SanityCheckPlan` | - | Validates that the final physical plan meets ordering, distribution, and infinite-input safety requirements. |diff --git a/datafusion/core/tests/physical_optimizer/eliminate_constant_join.rs b/datafusion/core/tests/physical_optimizer/eliminate_constant_join.rsnew file mode 100644index 0000000000..04b66ce7e4--- /dev/null+++ b/datafusion/core/tests/physical_optimizer/eliminate_constant_join.rs@@ -0,0 +1,201 @@+// Licensed to the Apache Software Foundation (ASF) under one+// or more contributor license agreements. See the NOTICE file+// distributed with this work for additional information+// regarding copyright ownership. The ASF licenses this file+// to you under the Apache License, Version 2.0 (the+// "License"); you may not use this file except in compliance+// with the License. You may obtain a copy of the License at+//+// http://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing,+// software distributed under the License is distributed on an+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+// KIND, either express or implied. See the License for the+// specific language governing permissions and limitations+// under the License.++//! Integration tests for the [`EliminateConstantJoin`] physical optimizer rule.+//! Cases that SQL can express are in `eliminate_constant_join.slt`.++use std::collections::HashMap;+use std::sync::Arc;++use arrow::array::{Int64Array, RecordBatch};+use arrow::compute::concat_batches;+use arrow::datatypes::{DataType, Field, Schema};+use arrow::util::pretty::pretty_format_batches;+use datafusion::datasource::MemTable;+use datafusion::datasource::memory::MemorySourceConfig;+use datafusion::execution::session_state::SessionStateBuilder;+use datafusion::prelude::{SessionConfig, SessionContext};+use datafusion_common::config::ConfigOptions;+use datafusion_common::{JoinSide, JoinType, Result};+use datafusion_expr::Operator;+use datafusion_physical_expr::expressions::{BinaryExpr, Column, lit};+use datafusion_physical_optimizer::PhysicalOptimizerRule;+use datafusion_physical_optimizer::eliminate_constant_join::EliminateConstantJoin;+use datafusion_physical_plan::joins::NestedLoopJoinExec;+use datafusion_physical_plan::joins::utils::{ColumnIndex, JoinFilter};+use datafusion_physical_plan::placeholder_row::PlaceholderRowExec;+use datafusion_physical_plan::projection::ProjectionExec;+use datafusion_physical_plan::{ExecutionPlan, collect};++/// A placeholder emits one row per partition, so a two-partition constant input+/// matches every row twice.+#[tokio::test]+async fn keeps_join_with_multi_row_constant() -> Result<()> {+ let schema = Arc::new(Schema::new(vec![Field::new("t", DataType::Int64, true)]));+ let batch = RecordBatch::try_new(+ Arc::clone(&schema),+ vec![Arc::new(Int64Array::from(vec![10, 30]))],+ )?;+ let events = MemorySourceConfig::try_new_exec(&[vec![batch]], schema, None)?;+ let constant = Arc::new(ProjectionExec::try_new(+ [(lit(25i64), "cutoff".to_string())],+ Arc::new(PlaceholderRowExec::new(Arc::new(Schema::empty())).with_partitions(2)),+ )?);+ let filter = JoinFilter::new(+ Arc::new(BinaryExpr::new(+ Arc::new(Column::new("t", 0)),+ Operator::Lt,+ Arc::new(Column::new("cutoff", 1)),+ )),+ vec![+ ColumnIndex {+ index: 0,+ side: JoinSide::Left,+ },+ ColumnIndex {+ index: 0,+ side: JoinSide::Right,+ },+ ],+ Arc::new(Schema::new(vec![+ Field::new("t", DataType::Int64, true),+ Field::new("cutoff", DataType::Int64, false),+ ])),+ );+ let join: Arc<dyn ExecutionPlan> = Arc::new(NestedLoopJoinExec::try_new(+ events,+ constant,+ Some(filter),+ &JoinType::Inner,+ None,+ )?);++ let optimized = EliminateConstantJoin::new()+ .optimize(Arc::clone(&join), &ConfigOptions::new())?;+ assert_eq!(rows(optimized).await?, rows(join).await?);+ Ok(())+}++/// Lambda variables are numbered after the join filter's own columns, so a filter+/// over the wider input would read the wrong columns.+#[tokio::test]+async fn keeps_join_with_lambda_filter() -> Result<()> {+ // Plan without optimizer rules: projection pushdown cannot yet plan lambdas in+ // nested-loop join filters.+ let config =+ SessionConfig::new().set_str("datafusion.sql_parser.dialect", "databricks");+ let ctx = SessionContext::new_with_state(+ SessionStateBuilder::new()+ .with_config(config)+ .with_default_features()+ .with_physical_optimizer_rules(vec![])+ .build(),+ );+ ctx.sql(+ "CREATE TABLE lists AS \+ SELECT * FROM (VALUES ([1, 30], 'a', 7), ([40], 'b', 8)) AS v(l, k, n)",+ )+ .await?+ .collect()+ .await?;+ let join = ctx+ .sql(+ "SELECT s.k, s.n FROM (SELECT 25 AS cutoff) c \+ JOIN lists s ON array_any_match(s.l, x -> x < c.cutoff)",+ )+ .await?+ .create_physical_plan()+ .await?;++ let optimized = EliminateConstantJoin::new()+ .optimize(Arc::clone(&join), &ConfigOptions::new())?;+ assert_eq!(rows(optimized).await?, rows(join).await?);+ Ok(())+}++/// The rewritten plan keeps the schema metadata that came from the constant input;+/// otherwise the optimizer's schema check fails the query.+#[tokio::test]+async fn keeps_metadata_of_constant_input() -> Result<()> {+ // With `*`, no projection above the join keeps the metadata.+ let sql = "SELECT * FROM (SELECT count(*) AS n FROM tagged) c \+ JOIN events e ON e.t >= c.n";+ let expected = rows(+ metadata_session(false)?+ .sql(sql)+ .await?+ .create_physical_plan()+ .await?,+ )+ .await?;+ let ctx = metadata_session(true)?;+ let plan = ctx.sql(sql).await?.create_physical_plan().await?;+ assert_eq!(rows(plan).await?, expected);+ Ok(())+}++/// Registers `events` and `tagged`, a table whose schema carries metadata, with the+/// default rules, without [`EliminateConstantJoin`] unless `eliminate` is set.+fn metadata_session(eliminate: bool) -> Result<SessionContext> {+ let mut state = SessionStateBuilder::new().with_default_features().build();+ if !eliminate {+ let rules = state+ .physical_optimizers()+ .iter()+ .filter(|rule| rule.name() != EliminateConstantJoin::new().name())+ .cloned()+ .collect();+ state = SessionStateBuilder::new_from_existing(state)+ .with_physical_optimizer_rules(rules)+ .build();+ }+ let ctx = SessionContext::new_with_state(state);+ for (name, metadata) in [+ ("events", HashMap::new()),+ (+ "tagged",+ HashMap::from([("source".to_string(), "tagged".to_string())]),+ ),+ ] {+ let schema = Arc::new(Schema::new_with_metadata(+ vec![Field::new("t", DataType::Int64, true)],+ metadata,+ ));+ let batch = RecordBatch::try_new(+ Arc::clone(&schema),+ vec![Arc::new(Int64Array::from(vec![1, 2, 3]))],+ )?;+ ctx.register_table(+ name,+ Arc::new(MemTable::try_new(schema, vec![vec![batch]])?),+ )?;+ }+ Ok(ctx)+}++/// Runs `plan` and formats its rows in sorted order.+async fn rows(plan: Arc<dyn ExecutionPlan>) -> Result<Vec<String>> {+ let schema = plan.schema();+ let batches = collect(plan, Arc::new(Default::default())).await?;+ let mut rows = pretty_format_batches(&[concat_batches(&schema, &batches)?])?+ .to_string()+ .lines()+ .map(str::to_string)+ .collect::<Vec<_>>();+ rows.sort_unstable();+ Ok(rows)+}diff --git a/datafusion/core/tests/physical_optimizer/mod.rs b/datafusion/core/tests/physical_optimizer/mod.rsindex f89d86074b..43b5771802 100644--- a/datafusion/core/tests/physical_optimizer/mod.rs+++ b/datafusion/core/tests/physical_optimizer/mod.rs@@ -20,6 +20,7 @@#[expect(clippy::needless_pass_by_value)]mod aggregate_statistics;mod combine_partial_final_agg;+mod eliminate_constant_join;#[expect(clippy::needless_pass_by_value)]mod enforce_distribution;mod enforce_sorting;diff --git a/datafusion/physical-optimizer/src/eliminate_constant_join.rs b/datafusion/physical-optimizer/src/eliminate_constant_join.rsnew file mode 100644index 0000000000..d445cb9395--- /dev/null+++ b/datafusion/physical-optimizer/src/eliminate_constant_join.rs@@ -0,0 +1,243 @@+// Licensed to the Apache Software Foundation (ASF) under one+// or more contributor license agreements. See the NOTICE file+// distributed with this work for additional information+// regarding copyright ownership. The ASF licenses this file+// to you under the Apache License, Version 2.0 (the+// "License"); you may not use this file except in compliance+// with the License. You may obtain a copy of the License at+//+// http://www.apache.org/licenses/LICENSE-2.0+//+// Unless required by applicable law or agreed to in writing,+// software distributed under the License is distributed on an+// "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+// KIND, either express or implied. See the License for the+// specific language governing permissions and limitations+// under the License.++//! Replace inner joins against a single row of literals with a filter.++use std::sync::Arc;++use arrow::datatypes::SchemaRef;+use datafusion_common::Result;+use datafusion_common::config::ConfigOptions;+use datafusion_common::tree_node::{Transformed, TransformedResult, TreeNode};+use datafusion_common::{JoinSide, JoinType, internal_datafusion_err};+use datafusion_physical_expr::expressions::{Column, LambdaExpr, Literal};+use datafusion_physical_expr::{PhysicalExpr, PhysicalExprSimplifier};+use datafusion_physical_plan::filter::FilterExec;+use datafusion_physical_plan::joins::utils::{ColumnIndex, JoinFilter};+use datafusion_physical_plan::joins::{CrossJoinExec, NestedLoopJoinExec};+use datafusion_physical_plan::placeholder_row::PlaceholderRowExec;+use datafusion_physical_plan::projection::{ProjectionExec, ProjectionExpr};+use datafusion_physical_plan::{ExecutionPlan, ExecutionPlanProperties};++use crate::PhysicalOptimizerRule;++/// Rewrites an inner [`NestedLoopJoinExec`] or a [`CrossJoinExec`] whose input+/// provably produces one row of literals into a [`FilterExec`] over the other+/// input, with the literals substituted into the join filter, under a+/// [`ProjectionExec`] that restores the join's output.+///+/// For example, [`AggregateStatistics`] folds the scalar subquery in+///+/// ```sql+/// SELECT e.* FROM events e CROSS JOIN (SELECT min(ts) AS first_ts FROM live) l+/// WHERE e.ts < l.first_ts+/// ```+///+/// into literals over a [`PlaceholderRowExec`], leaving a join that only filters+/// `events`. A nested-loop join preserves neither input's order, while the filter+/// and projection preserve the other input's ordering and partitioning. This lets+/// later operators, such as an aggregation grouped by a sort key of `events`,+/// use that ordering.+///+/// [`AggregateStatistics`]: crate::aggregate_statistics::AggregateStatistics+#[derive(Debug, Default)]+pub struct EliminateConstantJoin {}++impl EliminateConstantJoin {+ #[expect(missing_docs)]+ pub fn new() -> Self {+ Self {}+ }+}++impl PhysicalOptimizerRule for EliminateConstantJoin {+ fn optimize(+ &self,+ plan: Arc<dyn ExecutionPlan>,+ _config: &ConfigOptions,+ ) -> Result<Arc<dyn ExecutionPlan>> {+ plan.transform_up(|plan| {+ let rewritten = if let Some(join) = plan.downcast_ref::<NestedLoopJoinExec>()+ {+ eliminate_nested_loop_join(join)?+ } else if let Some(join) = plan.downcast_ref::<CrossJoinExec>() {+ eliminate(join.left(), join.right(), None, None, &join.schema())?+ } else {+ None+ };+ Ok(match rewritten {+ Some(rewritten) => Transformed::yes(rewritten),+ None => Transformed::no(plan),+ })+ })+ .data()+ }++ fn name(&self) -> &str {+ "EliminateConstantJoin"+ }++ fn schema_check(&self) -> bool {+ true+ }+}++/// Rewrites `join` if it is an inner join against a constant row and its filter+/// has no lambdas.+fn eliminate_nested_loop_join(+ join: &NestedLoopJoinExec,+) -> Result<Option<Arc<dyn ExecutionPlan>>> {+ if join.join_type() != &JoinType::Inner {+ return Ok(None);+ }+ if let Some(filter) = join.filter()+ // Lambda variables are numbered after the filter's own columns, not the+ // other input's.+ && filter+ .expression()+ .exists(|expr| Ok(expr.is::<LambdaExpr>()))?+ {+ return Ok(None);+ }+ eliminate(+ join.left(),+ join.right(),+ join.filter(),+ join.projection().as_deref(),+ &join.schema(),+ )+}++/// Rewrites an inner join of `left_input` and `right_input` over its non-constant+/// input, if the other input is a constant row.+fn eliminate(+ left_input: &Arc<dyn ExecutionPlan>,+ right_input: &Arc<dyn ExecutionPlan>,+ filter: Option<&JoinFilter>,+ projection: Option<&[usize]>,+ schema: &SchemaRef,+) -> Result<Option<Arc<dyn ExecutionPlan>>> {+ // Each input's columns as expressions over the non-constant input.+ let (left, right, input) = if let Some(row) = constant_row(left_input) {+ (row, columns(right_input), right_input)+ } else if let Some(row) = constant_row(right_input) {+ (columns(left_input), row, left_input)+ } else {+ return Ok(None);+ };++ let mut input = Arc::clone(input);+ if let Some(filter) = filter {+ let predicate = Arc::clone(filter.expression())+ .transform_up(|expr| {+ let Some(column) = expr.downcast_ref::<Column>() else {+ return Ok(Transformed::no(expr));+ };+ // Filter columns index the filter's own schema, which+ // `column_indices` maps back to the join inputs.+ let ColumnIndex { index, side } =+ filter.column_indices().get(column.index()).ok_or_else(|| {+ internal_datafusion_err!("Unmapped join filter column")+ })?;+ let value = match side {+ JoinSide::Left => left.get(*index),+ JoinSide::Right => right.get(*index),+ JoinSide::None => None,+ }+ .ok_or_else(|| {+ internal_datafusion_err!("Join filter column out of range")+ })?;+ Ok(Transformed::yes(Arc::clone(value)))+ })+ .data()?;+ input = Arc::new(FilterExec::try_new(predicate, input)?);+ }++ let mut outputs = left.into_iter().chain(right).collect::<Vec<_>>();+ if let Some(projection) = projection {+ outputs = projection+ .iter()+ .map(|&index| outputs.get(index).cloned())+ .collect::<Option<_>>()+ .ok_or_else(|| internal_datafusion_err!("Join projection out of range"))?;+ }+ let exprs = outputs+ .into_iter()+ .zip(schema.fields())+ .map(|(expr, field)| ProjectionExpr {+ expr,+ alias: field.name().clone(),+ });+ // Keep the join's metadata, which may come from the constant input.+ Ok(Some(Arc::new(+ ProjectionExec::try_new_with_schema_metadata(exprs, input, schema)?,+ )))+}++/// Returns the literal values of the one row `plan` produces, if it is a stack of+/// projections over a single-partition [`PlaceholderRowExec`] whose expressions+/// evaluate to literals at planning time.+fn constant_row(plan: &Arc<dyn ExecutionPlan>) -> Option<Vec<Arc<dyn PhysicalExpr>>> {+ let projection = plan.downcast_ref::<ProjectionExec>()?;+ let input = projection.input();+ let input_row = if input.is::<PlaceholderRowExec>() {+ // A placeholder emits one row per partition, and its columns are not+ // literals.+ if input.output_partitioning().partition_count() != 1 {+ return None;+ }+ Vec::new()+ } else {+ constant_row(input)?+ };+ // The logical simplifier already folds constant expressions, but those over a+ // value `AggregateStatistics` folded, such as `min(ts) + 1`, are only constant+ // now. Volatile expressions and ones that fail to evaluate are not folded, and+ // are rejected: the filter would evaluate them for every row instead of once.+ let schema = input.schema();+ let simplifier = PhysicalExprSimplifier::new(&schema);+ projection+ .expr()+ .iter()+ .map(|ProjectionExpr { expr, .. }| {+ let expr = Arc::clone(expr)+ .transform_up(|expr| {+ let value = expr+ .downcast_ref::<Column>()+ .and_then(|column| input_row.get(column.index()));+ Ok(match value {+ Some(value) => Transformed::yes(Arc::clone(value)),+ None => Transformed::no(expr),+ })+ })+ .data()+ .and_then(|expr| simplifier.simplify(expr))+ .ok()?;+ expr.is::<Literal>().then_some(expr)+ })+ .collect()+}++/// Returns a column reference for each field of `plan`.+fn columns(plan: &Arc<dyn ExecutionPlan>) -> Vec<Arc<dyn PhysicalExpr>> {+ plan.schema()+ .fields()+ .iter()+ .enumerate()+ .map(|(index, field)| Arc::new(Column::new(field.name(), index)) as _)+ .collect()+}diff --git a/datafusion/physical-optimizer/src/lib.rs b/datafusion/physical-optimizer/src/lib.rsindex b9eb248f6e..25c9452915 100644--- a/datafusion/physical-optimizer/src/lib.rs+++ b/datafusion/physical-optimizer/src/lib.rs@@ -27,6 +27,7 @@pub mod aggregate_statistics;pub mod combine_partial_final_agg;+pub mod eliminate_constant_join;pub mod ensure_coop;pub mod ensure_requirements;// `enforce_distribution` and `enforce_sorting` are now internal implementationdiff --git a/datafusion/physical-optimizer/src/optimizer.rs b/datafusion/physical-optimizer/src/optimizer.rsindex f81555fcc2..eb4b707a74 100644--- a/datafusion/physical-optimizer/src/optimizer.rs+++ b/datafusion/physical-optimizer/src/optimizer.rs@@ -24,6 +24,7 @@ use std::sync::Arc;use crate::aggregate_statistics::AggregateStatistics;use crate::combine_partial_final_agg::CombinePartialFinalAggregate;+use crate::eliminate_constant_join::EliminateConstantJoin;use crate::ensure_coop::EnsureCooperative;use crate::ensure_requirements::EnsureRequirements;use crate::filter_pushdown::FilterPushdown;@@ -200,6 +201,10 @@ impl PhysicalOptimizer {// this information is not lost across different rules during optimization.Arc::new(OutputRequirements::new_add_mode()),Arc::new(AggregateStatistics::new()),+ // Runs after AggregateStatistics, which can fold a scalar subquery into a+ // row of literals, and before EnsureRequirements, so that the filter+ // replacing a join against such a row keeps the other input's ordering.+ Arc::new(EliminateConstantJoin::new()),// Statistics-based join selection will change the Auto mode to a real join implementation,// like collect left, or hash join, or future sort merge join, which will influence the// EnsureRequirements rule as it decides whether to add additional repartitioning anddiff --git a/datafusion/sqllogictest/test_files/cte.slt b/datafusion/sqllogictest/test_files/cte.sltindex fe02e39e3b..e0f8a4d4da 100644--- a/datafusion/sqllogictest/test_files/cte.slt+++ b/datafusion/sqllogictest/test_files/cte.slt@@ -1155,10 +1155,8 @@ logical_plan04)----Projection: Int64(1)05)------EmptyRelation: rows=1physical_plan-01)CrossJoinExec+01)ProjectionExec: expr=[a@0 as a, 1 as Int64(1)]02)--DataSourceExec: partitions=1, partition_sizes=[0]-03)--ProjectionExec: expr=[1 as Int64(1)]-04)----PlaceholderRowExecstatement count 0drop table j1;diff --git a/datafusion/sqllogictest/test_files/eliminate_constant_join.slt b/datafusion/sqllogictest/test_files/eliminate_constant_join.sltnew file mode 100644index 0000000000..ac441794b6--- /dev/null+++ b/datafusion/sqllogictest/test_files/eliminate_constant_join.slt@@ -0,0 +1,178 @@+# Licensed to the Apache Software Foundation (ASF) under one+# or more contributor license agreements. See the NOTICE file+# distributed with this work for additional information+# regarding copyright ownership. The ASF licenses this file+# to you under the Apache License, Version 2.0 (the+# "License"); you may not use this file except in compliance+# with the License. You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing,+# software distributed under the License is distributed on an+# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+# KIND, either express or implied. See the License for the+# specific language governing permissions and limitations+# under the License.++# Inner joins against a single row of literals become a filter over the other+# input, which keeps that input's ordering (EliminateConstantJoin).++statement ok+set datafusion.execution.target_partitions = 1;++statement ok+set datafusion.explain.physical_plan_only = true;++# Events sorted by account+query I+COPY (SELECT * FROM (VALUES ('a', 10), ('a', 30), ('b', 20), ('b', NULL), ('c', 5)) AS v(acct, t))+TO 'test_files/scratch/eliminate_constant_join/events.parquet'+STORED AS PARQUET;+----+5++statement ok+CREATE EXTERNAL TABLE events+STORED AS PARQUET+WITH ORDER (acct ASC NULLS LAST)+LOCATION 'test_files/scratch/eliminate_constant_join/events.parquet';++query I+COPY (SELECT * FROM (VALUES (25), (40)) AS v(ts))+TO 'test_files/scratch/eliminate_constant_join/live.parquet'+STORED AS PARQUET;+----+2++statement ok+CREATE EXTERNAL TABLE live+STORED AS PARQUET+LOCATION 'test_files/scratch/eliminate_constant_join/live.parquet';++# min(ts) is answered from Parquet statistics, so the join only filters the+# events, and the aggregation keeps their account order+query TT+EXPLAIN SELECT e.acct, count(*) AS n+FROM events e CROSS JOIN (SELECT min(ts) AS first_ts FROM live) l+WHERE e.t < l.first_ts+GROUP BY e.acct;+----+physical_plan+01)ProjectionExec: expr=[acct@0 as acct, count(Int64(1))@1 as n]+02)--AggregateExec: mode=Single, gby=[acct@0 as acct], aggr=[count(Int64(1))], group_clustering_mode=Full+03)----FilterExec: 25 > t@1, projection=[acct@0]+04)------DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/eliminate_constant_join/events.parquet]]}, projection=[acct, t], output_ordering=[acct@0 ASC NULLS LAST], file_type=parquet, predicate=25 > t@1, pruning_predicate=t_null_count@1 != row_count@2 AND t_min@0 < 25, required_guarantees=[]++query TI+SELECT e.acct, count(*) AS n+FROM events e CROSS JOIN (SELECT min(ts) AS first_ts FROM live) l+WHERE e.t < l.first_ts+GROUP BY e.acct+ORDER BY e.acct;+----+a 1+b 1+c 1++# Expressions over a value folded from statistics are evaluated at planning time+query TT+EXPLAIN SELECT e.acct+FROM events e CROSS JOIN (SELECT min(ts) + 1 AS cutoff FROM live) l+WHERE e.t < l.cutoff;+----+physical_plan+01)FilterExec: 26 > t@1, projection=[acct@0]+02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/eliminate_constant_join/events.parquet]]}, projection=[acct, t], output_ordering=[acct@0 ASC NULLS LAST], file_type=parquet, predicate=26 > t@1, pruning_predicate=t_null_count@1 != row_count@2 AND t_min@0 < 26, required_guarantees=[]++query T rowsort+SELECT e.acct+FROM events e CROSS JOIN (SELECT min(ts) + 1 AS cutoff FROM live) l+WHERE e.t < l.cutoff;+----+a+b+c++# A cross join with a constant row keeps the other input's ordering+query TT+EXPLAIN SELECT e.acct, l.first_ts, count(*) AS n+FROM events e CROSS JOIN (SELECT min(ts) AS first_ts FROM live) l+GROUP BY e.acct, l.first_ts;+----+physical_plan+01)ProjectionExec: expr=[acct@0 as acct, first_ts@1 as first_ts, count(Int64(1))@2 as n]+02)--AggregateExec: mode=Single, gby=[acct@0 as acct, first_ts@1 as first_ts], aggr=[count(Int64(1))], group_clustering_mode=Full+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/eliminate_constant_join/events.parquet]]}, projection=[acct, 25 as first_ts], output_ordering=[acct@0 ASC NULLS LAST], file_type=parquet++query TII+SELECT e.acct, l.first_ts, count(*) AS n+FROM events e CROSS JOIN (SELECT min(ts) AS first_ts FROM live) l+GROUP BY e.acct, l.first_ts+ORDER BY e.acct;+----+a 25 2+b 25 2+c 25 1++# The constant row's columns stay in the output+query TT+EXPLAIN SELECT e.acct, e.t, c.cutoff+FROM (SELECT 25 AS cutoff) c JOIN events e ON e.t < c.cutoff;+----+physical_plan+01)ProjectionExec: expr=[acct@0 as acct, t@1 as t, 25 as cutoff]+02)--FilterExec: t@1 < 25+03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/eliminate_constant_join/events.parquet]]}, projection=[acct, t], output_ordering=[acct@0 ASC NULLS LAST], file_type=parquet, predicate=t@1 < 25, pruning_predicate=t_null_count@1 != row_count@2 AND t_min@0 < 25, required_guarantees=[]++query TII rowsort+SELECT e.acct, e.t, c.cutoff+FROM (SELECT 25 AS cutoff) c JOIN events e ON e.t < c.cutoff;+----+a 10 25+b 20 25+c 5 25++# A left join keeps the constant row when nothing matches, so it stays a join+query TT+EXPLAIN SELECT c.cutoff, e.acct+FROM (SELECT 25 AS cutoff) c LEFT JOIN events e ON e.t > c.cutoff + 100;+----+physical_plan+01)ProjectionExec: expr=[cutoff@0 as cutoff, acct@1 as acct]+02)--NestedLoopJoinExec: join_type=Left, filter=t@0 > join_proj_push_down_1@1, projection=[cutoff@0, acct@2, t@3]+03)----ProjectionExec: expr=[cutoff@0 as cutoff, cutoff@0 + 100 as join_proj_push_down_1]+04)------ProjectionExec: expr=[25 as cutoff]+05)--------PlaceholderRowExec+06)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/eliminate_constant_join/events.parquet]]}, projection=[acct, t], output_ordering=[acct@0 ASC NULLS LAST], file_type=parquet++query IT+SELECT c.cutoff, e.acct+FROM (SELECT 25 AS cutoff) c LEFT JOIN events e ON e.t > c.cutoff + 100;+----+25 NULL++# The join evaluates random() once; a filter would evaluate it for every row+query TT+EXPLAIN SELECT e.acct+FROM (SELECT random() AS r) c JOIN events e ON e.t < c.r * 100;+----+physical_plan+01)ProjectionExec: expr=[acct@1 as acct]+02)--NestedLoopJoinExec: join_type=Inner, filter=join_proj_push_down_2@1 < join_proj_push_down_1@0, projection=[r@0, acct@2, t@3]+03)----ProjectionExec: expr=[r@0 as r, r@0 * 100 as join_proj_push_down_1]+04)------ProjectionExec: expr=[random() as r]+05)--------PlaceholderRowExec+06)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/sqllogictest/test_files/scratch/eliminate_constant_join/events.parquet]]}, projection=[acct, t, CAST(t@1 AS Float64) as join_proj_push_down_2], output_ordering=[acct@0 ASC NULLS LAST], file_type=parquet++statement ok+DROP TABLE events;++statement ok+DROP TABLE live;++statement ok+set datafusion.execution.target_partitions = 4;++statement ok+RESET datafusion.explain.physical_plan_only;diff --git a/datafusion/sqllogictest/test_files/explain.slt b/datafusion/sqllogictest/test_files/explain.sltindex 6c910b9681..1cf1a29558 100644--- a/datafusion/sqllogictest/test_files/explain.slt+++ b/datafusion/sqllogictest/test_files/explain.slt@@ -236,6 +236,7 @@ physical_plan after OutputRequirements01)OutputRequirementExec: order_by=[], dist_by=Unspecified02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/example.csv]]}, projection=[a, b, c], file_type=csv, has_header=truephysical_plan after aggregate_statistics SAME TEXT AS ABOVE+physical_plan after EliminateConstantJoin SAME TEXT AS ABOVEphysical_plan after join_selection SAME TEXT AS ABOVEphysical_plan after LimitedDistinctAggregation SAME TEXT AS ABOVEphysical_plan after FilterPushdown SAME TEXT AS ABOVE@@ -351,6 +352,7 @@ physical_plan after OutputRequirements02)--GlobalLimitExec: skip=0, fetch=10, statistics=[Rows=Exact(8), Bytes=Absent, [(Col[0]: ScanBytes=Exact(32)),(Col[1]: ScanBytes=Inexact(24)),(Col[2]: ScanBytes=Exact(32)),(Col[3]: ScanBytes=Exact(32)),(Col[4]: ScanBytes=Exact(32)),(Col[5]: ScanBytes=Exact(64)),(Col[6]: ScanBytes=Exact(32)),(Col[7]: ScanBytes=Exact(64)),(Col[8]: ScanBytes=Inexact(88)),(Col[9]: ScanBytes=Inexact(49)),(Col[10]: ScanBytes=Exact(64))]]03)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/parquet-testing/data/alltypes_plain.parquet]]}, projection=[id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col], limit=10, file_type=parquet, statistics=[Rows=Exact(8), Bytes=Absent, [(Col[0]: ScanBytes=Exact(32)),(Col[1]: ScanBytes=Inexact(24)),(Col[2]: ScanBytes=Exact(32)),(Col[3]: ScanBytes=Exact(32)),(Col[4]: ScanBytes=Exact(32)),(Col[5]: ScanBytes=Exact(64)),(Col[6]: ScanBytes=Exact(32)),(Col[7]: ScanBytes=Exact(64)),(Col[8]: ScanBytes=Inexact(88)),(Col[9]: ScanBytes=Inexact(49)),(Col[10]: ScanBytes=Exact(64))]]physical_plan after aggregate_statistics SAME TEXT AS ABOVE+physical_plan after EliminateConstantJoin SAME TEXT AS ABOVEphysical_plan after join_selection SAME TEXT AS ABOVEphysical_plan after LimitedDistinctAggregation SAME TEXT AS ABOVEphysical_plan after FilterPushdown SAME TEXT AS ABOVE@@ -397,6 +399,7 @@ physical_plan after OutputRequirements02)--GlobalLimitExec: skip=0, fetch=1003)----DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/parquet-testing/data/alltypes_plain.parquet]]}, projection=[id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col], limit=10, file_type=parquetphysical_plan after aggregate_statistics SAME TEXT AS ABOVE+physical_plan after EliminateConstantJoin SAME TEXT AS ABOVEphysical_plan after join_selection SAME TEXT AS ABOVEphysical_plan after LimitedDistinctAggregation SAME TEXT AS ABOVEphysical_plan after FilterPushdown SAME TEXT AS ABOVE@@ -652,6 +655,7 @@ physical_plan after OutputRequirements01)OutputRequirementExec: order_by=[], dist_by=Unspecified02)--DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/example.csv]]}, projection=[a, b, c], file_type=csv, has_header=truephysical_plan after aggregate_statistics SAME TEXT AS ABOVE+physical_plan after EliminateConstantJoin SAME TEXT AS ABOVEphysical_plan after join_selection SAME TEXT AS ABOVEphysical_plan after LimitedDistinctAggregation SAME TEXT AS ABOVEphysical_plan after FilterPushdown SAME TEXT AS ABOVEdiff --git a/datafusion/sqllogictest/test_files/joins.slt b/datafusion/sqllogictest/test_files/joins.sltindex 6f498a9925..9d0541d1bb 100644--- a/datafusion/sqllogictest/test_files/joins.slt+++ b/datafusion/sqllogictest/test_files/joins.slt@@ -4843,10 +4843,8 @@ logical_plan04)----Projection: Int64(1)05)------EmptyRelation: rows=1physical_plan-01)CrossJoinExec+01)ProjectionExec: expr=[j1_string@0 as j1_string, j1_id@1 as j1_id, 1 as Int64(1)]02)--DataSourceExec: partitions=1, partition_sizes=[0]-03)--ProjectionExec: expr=[1 as Int64(1)]-04)----PlaceholderRowExecstatement count 0drop table j1;