Apache DataFusion SQL Query Engine
docs: Improve optimizer rule reference by consolidating into rustdocs (#26899)
Andrew Lamb306dbf5dc46f
diff --git a/datafusion/common/src/test_util.rs b/datafusion/common/src/test_util.rsindex 5c47e125ea..3107b8f329 100644--- a/datafusion/common/src/test_util.rs+++ b/datafusion/common/src/test_util.rs@@ -729,6 +729,47 @@ pub mod array_conversion {}}+/// Extracts names in the first column of a markdown table.+///+/// Used to extract names from optimizer rule tables and verify they match+///+/// # Example+///+/// Given this `source`:+///+/// ```text+/// /// # Built-in Analyzer Rules+/// ///+/// /// The rules are applied in the following order.+/// ///+/// /// | rule (name in EXPLAIN plan) | struct name | summary |+/// /// | --------------------------- | --------------------------- | -------------- |+/// /// | `resolve_grouping_function` | [`ResolveGroupingFunction`] | Rewrites ... |+/// /// | `type_coercion` | [`TypeCoercion`] | Adds casts ... |+/// #[derive(Clone, Debug)]+/// pub struct Analyzer {+/// ```+///+/// returns `["resolve_grouping_function", "type_coercion"]`.+pub fn documented_rule_names(source: &str, heading: &str) -> Vec<String> {+ let mut lines = source.lines().skip_while(|line| line.trim() != heading);+ assert!(lines.next().is_some(), "heading '{heading}' not found");++ lines+ .map(str::trim)+ // skip the text between the heading and the table+ .skip_while(|line| !line.starts_with("/// |"))+ .take_while(|line| line.starts_with("/// |"))+ // skip the header and separator rows+ .skip(2)+ .map(|line| {+ let first_column =+ line.trim_start_matches("/// |").split('|').next().unwrap();+ first_column.trim().trim_matches('`').to_string()+ })+ .collect()+}+#[cfg(test)]mod tests {use crate::cast::{as_float64_array, as_int32_array, as_string_array};diff --git a/datafusion/core/src/lib.rs b/datafusion/core/src/lib.rsindex b21c231a6f..2c3dfcac96 100644--- a/datafusion/core/src/lib.rs+++ b/datafusion/core/src/lib.rs@@ -277,18 +277,25 @@//! possible, significant effort is spent planning and//! optimizing, in the following manner://!-//! 1. The [`LogicalPlan`] is checked and rewritten to enforce-//! semantic rules, such as type coercion, by [`AnalyzerRule`]s+//! 1. The [`LogicalPlan`] is checked and rewritten by the [`Analyzer`], using+//! [`AnalyzerRule`]s, to enforce semantic rules such as type coercion.//!-//! 2. The [`LogicalPlan`] is rewritten by [`OptimizerRule`]s, such as-//! projection and filter pushdown, to improve its efficiency.+//! 2. The [`LogicalPlan`] is rewritten by the [`Optimizer`], using+//! [`OptimizerRule`]s such as projection and filter pushdown, to improve+//! its efficiency.//!//! 3. The [`LogicalPlan`] is converted to an [`ExecutionPlan`] by a-//! [`PhysicalPlanner`]+//! [`PhysicalPlanner`].//!-//! 4. The [`ExecutionPlan`] is rewritten by-//! [`PhysicalOptimizerRule`]s, such as sort and join selection, to-//! improve its efficiency.+//! 4. The [`ExecutionPlan`] is rewritten by the [`PhysicalOptimizer`], using+//! [`PhysicalOptimizerRule`]s such as join algorithm selection, to further+//! improve efficiency.+//!+//! There is not yet a physical counterpart to the [`Analyzer`]. The rules that+//! make an [`ExecutionPlan`] executable, by enforcing its distribution and+//! ordering requirements, currently run at the start of the+//! [`PhysicalOptimizer`]. Moving them into a separate physical analyzer phase+//! is tracked in <https://github.com/apache/datafusion/issues/26481>.//!//! ## Data Sources//!@@ -739,6 +746,9 @@//! [`Expr`]: datafusion_expr::Expr//! [`LogicalPlan`]: datafusion_expr::LogicalPlan//! [`AnalyzerRule`]: datafusion_optimizer::analyzer::AnalyzerRule+//! [`Analyzer`]: datafusion_optimizer::Analyzer+//! [`Optimizer`]: datafusion_optimizer::Optimizer+//! [`PhysicalOptimizer`]: datafusion_physical_optimizer::optimizer::PhysicalOptimizer//! [`OptimizerRule`]: optimizer::optimizer::OptimizerRule//! [`ExecutionPlan`]: physical_plan::ExecutionPlan//! [`PhysicalPlanner`]: physical_planner::PhysicalPlanner@@ -748,7 +758,6 @@//! [`RecordBatch`]: arrow::array::RecordBatch//! [`RecordBatchReader`]: arrow::record_batch::RecordBatchReader//! [`Array`]: arrow::array::Array-#![cfg_attr(doc, doc = include_str!("optimizer_rule_reference.md"))]extern crate core;#[cfg(feature = "sql")]@@ -774,9 +783,6 @@ pub use parquet;#[cfg(feature = "avro")]pub use datafusion_datasource_avro::arrow_avro;-#[cfg(test)]-mod optimizer_rule_reference;-// re-export DataFusion sub-crates at the top level. Use `pub use *`// so that the contents of the subcrates appears in rustdocs// for details, see https://github.com/apache/datafusion/issues/6648diff --git a/datafusion/core/src/optimizer_rule_reference.md b/datafusion/core/src/optimizer_rule_reference.mddeleted file mode 100644index 5f48a96009..0000000000--- a/datafusion/core/src/optimizer_rule_reference.md+++ /dev/null@@ -1,87 +0,0 @@-<!---SPDX-License-Identifier: Apache-2.0-SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership-SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements--->--## Built-in Optimizer Rules--DataFusion applies a default analyzer, logical optimizer, and physical-optimizer pipeline.--The rule names listed here match the names shown by `EXPLAIN VERBOSE`.--Rule order matters. The default pipeline may change between releases.--### Analyzer Rules--| order | rule | summary |-| ----- | --------------------------- | --------------------------------------------------------------------------------------- |-| 1 | `resolve_grouping_function` | Rewrites `GROUPING(...)` calls into expressions over DataFusion's internal grouping id. |-| 2 | `type_coercion` | Adds implicit casts so operators and functions receive valid input types. |--### Logical Optimizer Rules--| order | rule | summary |-| ----- | ----------------------------------------- | --------------------------------------------------------------------------------------------------------------------------- |-| 1 | `rewrite_set_comparison` | Rewrites `ANY` and `ALL` set-comparison subqueries into `EXISTS`-based boolean expressions with correct SQL NULL semantics. |-| 2 | `optimize_unions` | Flattens nested unions and removes unions with a single input. |-| 3 | `unions_to_filter` | Merges `UNION DISTINCT` branches that share the same source into a single filtered branch with a disjunctive predicate. |-| 4 | `simplify_expressions` | Constant-folds and simplifies expressions while preserving output names. |-| 5 | `replace_distinct_aggregate` | Rewrites `DISTINCT` and `DISTINCT ON` operators into aggregate-based plans that later rules can optimize further. |-| 6 | `eliminate_join` | Replaces keyless inner joins with a literal `false` filter by an empty relation. |-| 7 | `decorrelate_predicate_subquery` | Converts eligible `IN` and `EXISTS` predicate subqueries into semi or anti joins. |-| 8 | `scalar_subquery_to_join` | Rewrites eligible scalar subqueries into joins and adds schema-preserving projections. |-| 9 | `decorrelate_lateral_join` | Rewrites eligible lateral joins into regular joins. |-| 10 | `extract_equijoin_predicate` | Splits join filters into equijoin keys and residual predicates. |-| 11 | `eliminate_duplicated_expr` | Removes duplicate expressions from projections, aggregates, and similar operators. |-| 12 | `eliminate_filter` | Drops always-true filters and replaces always-false or NULL filters with empty relations. |-| 13 | `eliminate_cross_join` | Uses filter predicates to replace cross joins with inner joins when join keys can be found. |-| 14 | `eliminate_limit` | Removes no-op limits and simplifies trivial limit shapes. |-| 15 | `propagate_empty_relation` | Pushes empty-relation knowledge upward so operators fed by no rows collapse early. |-| 16 | `filter_null_join_keys` | Adds `IS NOT NULL` filters to nullable equijoin keys that can never match. |-| 17 | `eliminate_outer_join` | Rewrites outer joins to inner joins when later filters reject the NULL-extended rows. |-| 18 | `push_down_limit` | Moves literal limits closer to scans and unions and merges adjacent limits. |-| 19 | `push_down_filter` | Moves filters as early as possible through filter-commutative operators. |-| 20 | `eliminate_aggregate_distinct` | Drops the `DISTINCT` modifier from aggregates whose result cannot change, such as `min`, `max` and `bit_or`. |-| 21 | `single_distinct_aggregation_to_group_by` | Rewrites single-column `DISTINCT` aggregations into two-stage `GROUP BY` plans. |-| 22 | `eliminate_group_by_constant` | Removes constant or functionally redundant expressions from `GROUP BY`. |-| 23 | `common_sub_expression_eliminate` | Computes repeated subexpressions once and reuses the result. |-| 24 | `extract_leaf_expressions` | Pulls cheap leaf expressions closer to data sources so later pruning and filter rules can act earlier. |-| 25 | `push_down_leaf_projections` | Pushes the helper projections created by leaf extraction toward leaf inputs. |-| 26 | `optimize_projections` | Prunes unused columns and removes unnecessary logical projections. |-| 27 | `replace_filter_top1` | Rewrites `Filter(row_number() = 1)` over a partitioned window into a `first_value` aggregate. |--### Physical Optimizer Rules--The same rule name may appear more than once when the default pipeline runs it-in multiple phases.--After the last rule, including any rule added by the user, the physical planner-checks that the final plan meets its ordering, distribution, and infinite-input-safety requirements (`SanityCheckPlan`). If the ordering or distribution-requirements do not hold, the error names the rule that broke them.--| order | rule | phase | summary |-| ----- | ------------------------------ | ----------------------- | -------------------------------------------------------------------------------------------------------- |-| 1 | `OutputRequirements` | add phase | Adds helper nodes so output requirements survive later physical rewrites. |-| 2 | `aggregate_statistics` | - | Uses exact source statistics to answer some aggregates without scanning data. |-| 3 | `EliminateConstantJoin` | - | Replaces inner joins against a single row of literals with a filter that keeps the other input's order. |-| 4 | `join_selection` | - | Chooses join implementation, build side, and partition mode from statistics and stream properties. |-| 5 | `LimitedDistinctAggregation` | - | Pushes limit hints into grouped distinct-style aggregations when only a small result is needed. |-| 6 | `FilterPushdown` | pre-optimization phase | Pushes supported physical filters down toward data sources before distribution and sorting are enforced. |-| 7 | `WindowTopN` | - | Replaces eligible row-number window and filter patterns with per-partition TopK execution. |-| 8 | `EnsureRequirements` | - | Enforces both distribution and sorting requirements in a single idempotent rule. |-| 9 | `CombinePartialFinalAggregate` | - | Collapses adjacent partial and final aggregates when the distributed shape makes them redundant. |-| 10 | `OptimizeAggregateOrder` | - | Updates aggregate expressions to use the best ordering once sort requirements are known. |-| 11 | `ProjectionPushdown` | early pass | Pushes projections toward inputs before later physical rewrites add more limit and TopK structure. |-| 12 | `OutputRequirements` | remove phase | Removes the temporary output-requirement helper nodes after requirement-sensitive planning is done. |-| 13 | `LimitAggregation` | - | Passes a limit hint into eligible aggregations so they can keep fewer accumulator buckets. |-| 14 | `LimitPushPastWindows` | - | Pushes fetch limits through bounded window operators when doing so keeps the result correct. |-| 15 | `HashJoinBuffering` | - | Adds buffering on the probe side of hash joins so probing can start before build completion. |-| 16 | `LimitPushdown` | - | Moves physical limits into child operators or fetch-enabled variants to cut data early. |-| 17 | `TopKRepartition` | - | Pushes TopK below hash repartition when the partition key is a prefix of the sort key. |-| 18 | `ProjectionPushdown` | late pass | Runs projection pushdown again after limit and TopK rewrites expose new pruning opportunities. |-| 19 | `PushdownSort` | - | Pushes sort requirements into data sources that can already return sorted output. |-| 20 | `EnsureCooperative` | - | Wraps non-cooperative plan parts so long-running tasks yield fairly. |-| 21 | `FilterPushdown(Post)` | post-optimization phase | Pushes dynamic filters at the end of optimization, after plan references stop moving. |diff --git a/datafusion/core/src/optimizer_rule_reference.rs b/datafusion/core/src/optimizer_rule_reference.rsdeleted file mode 100644index 34f8d87593..0000000000--- a/datafusion/core/src/optimizer_rule_reference.rs+++ /dev/null@@ -1,72 +0,0 @@-// SPDX-License-Identifier: Apache-2.0-// SPDX-FileCopyrightText: See the NOTICE file distributed with this work for additional information regarding copyright ownership-// SPDX-FileContributor: Licensed to the Apache Software Foundation (ASF) under one or more contributor license agreements--use datafusion_optimizer::analyzer::Analyzer;-use datafusion_optimizer::optimizer::Optimizer;-use datafusion_physical_optimizer::optimizer::PhysicalOptimizer;--const OPTIMIZER_RULE_REFERENCE: &str = include_str!("optimizer_rule_reference.md");--fn documented_rules(section_heading: &str) -> Vec<String> {- let mut in_section = false;- let mut names = vec![];-- for line in OPTIMIZER_RULE_REFERENCE.lines() {- if line == section_heading {- in_section = true;- continue;- }-- if in_section && line.starts_with("### ") {- break;- }-- if !in_section || !line.starts_with('|') || line.contains("---") {- continue;- }-- let columns: Vec<_> = line.split('|').map(str::trim).collect();-- if columns.len() < 4 || columns[1] == "order" {- continue;- }-- names.push(columns[2].trim_matches('`').to_string());- }-- names-}--#[test]-fn analyzer_rules_match_documented_order() {- let rules: Vec<_> = Analyzer::new()- .rules- .iter()- .map(|rule| rule.name().to_string())- .collect();-- assert_eq!(documented_rules("### Analyzer Rules"), rules);-}--#[test]-fn logical_rules_match_documented_order() {- let rules: Vec<_> = Optimizer::new()- .rules- .iter()- .map(|rule| rule.name().to_string())- .collect();-- assert_eq!(documented_rules("### Logical Optimizer Rules"), rules);-}--#[test]-fn physical_rules_match_documented_order() {- let rules: Vec<_> = PhysicalOptimizer::new()- .rules- .iter()- .map(|rule| rule.name().to_string())- .collect();-- assert_eq!(documented_rules("### Physical Optimizer Rules"), rules);-}diff --git a/datafusion/optimizer/src/analyzer/mod.rs b/datafusion/optimizer/src/analyzer/mod.rsindex c96df1b448..191f240fb5 100644--- a/datafusion/optimizer/src/analyzer/mod.rs+++ b/datafusion/optimizer/src/analyzer/mod.rs@@ -48,13 +48,26 @@ pub trait AnalyzerRule: Debug {fn name(&self) -> &str;}-/// Rule-based Analyzer.+/// Rule-based Analyzer for [`LogicalPlan`]s.////// Applies [`FunctionRewrite`]s and [`AnalyzerRule`]s to transform a/// [`LogicalPlan`] in preparation for execution.////// For example, the `Analyzer` applies type coercion to ensure the types of/// operands match the types required by functions.+///+/// # Built-in Analyzer Rules+///+/// The Analyzer first applies any [`FunctionRewrite`]s added with+/// [`Self::add_function_rewrite`].+///+/// The rules are applied in the following order. Note: the list may change+/// between releases.+///+/// | rule (name in EXPLAIN plan) | struct name | summary |+/// | --------------------------- | --------------------------- | ------------------------------------------------------------------------- |+/// | `resolve_grouping_function` | [`ResolveGroupingFunction`] | Rewrites `GROUPING(...)` into expressions over the internal grouping id. |+/// | `type_coercion` | [`TypeCoercion`] | Adds implicit casts so operators and functions receive valid input types. |#[derive(Clone, Debug)]pub struct Analyzer {/// Expr --> Function writes to apply prior to analysis passes@@ -70,7 +83,7 @@ impl Default for Analyzer {}impl Analyzer {- /// Create a new analyzer using the recommended list of rules+ /// Create a new analyzer using the [built-in rules](Self#built-in-analyzer-rules)pub fn new() -> Self {let rules: Vec<Arc<dyn AnalyzerRule + Send + Sync>> = vec![Arc::new(ResolveGroupingFunction::new()),@@ -153,3 +166,27 @@ impl Analyzer {Ok(new_plan)}}++#[cfg(test)]+mod tests {+ use super::Analyzer;+ use datafusion_common::test_util::documented_rule_names;++ /// Verifies the rule table in the [`Analyzer`] docs matches the rules+ /// (and their order) returned by [`Analyzer::new`]+ #[test]+ fn analyzer_rules_match_documented_order() {+ let rules: Vec<_> = Analyzer::new()+ .rules+ .iter()+ .map(|rule| rule.name().to_string())+ .collect();++ let documented = documented_rule_names(+ include_str!("mod.rs"),+ "/// # Built-in Analyzer Rules",+ );++ assert_eq!(documented, rules);+ }+}diff --git a/datafusion/optimizer/src/eliminate_cross_join.rs b/datafusion/optimizer/src/eliminate_cross_join.rsindex 68497d2cff..9cb624b80c 100644--- a/datafusion/optimizer/src/eliminate_cross_join.rs+++ b/datafusion/optimizer/src/eliminate_cross_join.rs@@ -16,6 +16,8 @@ use datafusion_expr::logical_plan::{use datafusion_expr::utils::{can_hash, find_valid_equijoin_key_pair};use datafusion_expr::{ExprSchemable, Operator, and, build_join_schema};+/// Optimizer rule that uses filter predicates to replace cross joins with+/// inner joins when join keys can be found.#[derive(Default, Debug)]pub struct EliminateCrossJoin;diff --git a/datafusion/optimizer/src/optimizer.rs b/datafusion/optimizer/src/optimizer.rsindex 81d79541c9..fdf5e5bbb1 100644--- a/datafusion/optimizer/src/optimizer.rs+++ b/datafusion/optimizer/src/optimizer.rs@@ -233,10 +233,45 @@ impl OptimizerConfig for OptimizerContext {}}-/// A rule-based logical optimizer.+/// A rule-based optimizer for [`LogicalPlan`]s.////// Applies rules in order for up to `datafusion.optimizer.max_passes` passes,/// stopping early if the plan repeats.+///+/// # Built-in Optimizer Rules+///+/// The rules are applied in the following order. Note: the list may change+/// between releases.+///+/// | rule (name in EXPLAIN plan) | struct name | summary |+/// | ----------------------------------------- | -------------------------------- | -------------------------------------------------------------------------------------------------------------------------------- |+/// | `rewrite_set_comparison` | [`RewriteSetComparison`] | Rewrites `ANY` and `ALL` subqueries into `EXISTS` expressions with SQL NULL semantics. |+/// | `optimize_unions` | [`OptimizeUnions`] | Flattens nested unions and removes single-input unions. |+/// | `unions_to_filter` | [`UnionsToFilter`] | Merges `UNION DISTINCT` branches over the same source into one filtered branch. |+/// | `simplify_expressions` | [`SimplifyExpressions`] | Constant-folds and simplifies expressions. |+/// | `replace_distinct_aggregate` | [`ReplaceDistinctWithAggregate`] | Rewrites `DISTINCT` and `DISTINCT ON` into aggregates. |+/// | `eliminate_join` | [`EliminateJoin`] | Rewrites inner joins to semi joins, drops unused outer join sides, and replaces always-false inner joins with an empty relation. |+/// | `decorrelate_predicate_subquery` | [`DecorrelatePredicateSubquery`] | Rewrites `IN` and `EXISTS` subqueries into semi or anti joins. |+/// | `scalar_subquery_to_join` | [`ScalarSubqueryToJoin`] | Rewrites scalar subqueries into joins. |+/// | `decorrelate_lateral_join` | [`DecorrelateLateralJoin`] | Rewrites lateral joins into regular joins. |+/// | `extract_equijoin_predicate` | [`ExtractEquijoinPredicate`] | Splits join filters into equijoin keys and residual predicates. |+/// | `eliminate_duplicated_expr` | [`EliminateDuplicatedExpr`] | Removes duplicate expressions from projections, aggregates, and similar operators. |+/// | `eliminate_filter` | [`EliminateFilter`] | Drops always-true filters and replaces always-false or NULL filters with an empty relation. |+/// | `eliminate_cross_join` | [`EliminateCrossJoin`] | Replaces cross joins with inner joins when a filter supplies join keys. |+/// | `eliminate_limit` | [`EliminateLimit`] | Replaces `LIMIT 0` with an empty relation and removes no-op limits and offsets. |+/// | `propagate_empty_relation` | [`PropagateEmptyRelation`] | Collapses operators whose inputs produce no rows. |+/// | `filter_null_join_keys` | [`FilterNullJoinKeys`] | Adds `IS NOT NULL` filters on nullable equijoin keys. |+/// | `eliminate_outer_join` | [`EliminateOuterJoin`] | Rewrites outer joins to inner joins when a later filter rejects the NULL-extended rows. |+/// | `push_down_limit` | [`PushDownLimit`] | Pushes limits toward scans and merges adjacent limits. |+/// | `push_down_filter` | [`PushDownFilter`] | Pushes filters as close to the sources as possible. |+/// | `eliminate_aggregate_distinct` | [`EliminateAggregateDistinct`] | Drops `DISTINCT` from aggregates it cannot affect, such as `min` and `max`. |+/// | `single_distinct_aggregation_to_group_by` | [`SingleDistinctToGroupBy`] | Rewrites single-column `DISTINCT` aggregates into two-stage `GROUP BY` plans. |+/// | `eliminate_group_by_constant` | [`EliminateGroupByConstant`] | Removes constant expressions from `GROUP BY`. |+/// | `common_sub_expression_eliminate` | [`CommonSubexprEliminate`] | Computes repeated subexpressions once. |+/// | `extract_leaf_expressions` | [`ExtractLeafExpressions`] | Moves cheap leaf expressions closer to the sources. |+/// | `push_down_leaf_projections` | [`PushDownLeafProjections`] | Pushes the projections created by leaf extraction toward the sources. |+/// | `optimize_projections` | [`OptimizeProjections`] | Prunes unused columns and redundant projections. |+/// | `replace_filter_top1` | [`ReplaceFilterTop1`] | Rewrites `Filter(row_number() = 1)` over a partitioned window into a `first_value` aggregate. |#[derive(Clone, Debug)]pub struct Optimizer {/// All optimizer rules to apply@@ -262,7 +297,7 @@ impl Default for Optimizer {}impl Optimizer {- /// Create a new optimizer using the recommended list of rules+ /// Create a new optimizer using the [built-in rules](Self#built-in-optimizer-rules)pub fn new() -> Self {// NOTEs:// - The order of rules in this list is important, as it determines the@@ -800,9 +835,28 @@ mod tests {use crate::optimizer::Optimizer;use crate::test::{test_table_scan, test_table_scan_with_name};use crate::{OptimizerConfig, OptimizerContext, OptimizerRule};+ use datafusion_common::test_util::documented_rule_names;use super::ApplyOrder;+ /// Verifies the rule table in the [`Optimizer`] docs matches the rules+ /// (and their order) returned by [`Optimizer::new`]+ #[test]+ fn optimizer_rules_match_documented_order() {+ let rules: Vec<_> = Optimizer::new()+ .rules+ .iter()+ .map(|rule| rule.name().to_string())+ .collect();++ let documented = documented_rule_names(+ include_str!("optimizer.rs"),+ "/// # Built-in Optimizer Rules",+ );++ assert_eq!(documented, rules);+ }+#[test]fn skip_failing_rule() {let opt = Optimizer::with_rules(vec![Arc::new(BadRule {})]);diff --git a/datafusion/optimizer/src/unions_to_filter.rs b/datafusion/optimizer/src/unions_to_filter.rsindex 5275c158d3..41bd4e1fa5 100644--- a/datafusion/optimizer/src/unions_to_filter.rs+++ b/datafusion/optimizer/src/unions_to_filter.rs@@ -20,6 +20,8 @@ use datafusion_expr::{use log::debug;use std::sync::Arc;+/// Optimizer rule that merges `UNION DISTINCT` branches that share the same+/// source into a single filtered branch with a disjunctive predicate.#[derive(Default, Debug)]pub struct UnionsToFilter;diff --git a/datafusion/physical-optimizer/src/optimizer.rs b/datafusion/physical-optimizer/src/optimizer.rsindex 7c54b3f02a..589beadd58 100644--- a/datafusion/physical-optimizer/src/optimizer.rs+++ b/datafusion/physical-optimizer/src/optimizer.rs@@ -163,6 +163,44 @@ impl PhysicalOptimizerContext for ConfigOnlyContext<'_> {/// maintain. Instead, add a custom rule that rewrites your plan into the/// canonical shape recognized by the built-in rules. If extending a built-in/// rule still seems necessary, please open a DataFusion issue to discuss it.+///+/// # Built-in Physical Optimizer Rules+///+/// The rules are applied in the following order. Note: the list may change+/// between releases. Some rules run twice, in different phases. See the+/// [contract](#physical-optimizer-contract) above before reordering or+/// extending the list.+///+/// After the last rule, the physical planner checks the plan and returns an+/// error if it is not valid.+///+/// Note: The rules up to and including [`EnsureRequirements`] make the plan+/// executable rather than optimize it. This may be improved in the future.+/// See <https://github.com/apache/datafusion/issues/26481>.+///+/// | rule (name in EXPLAIN plan) | struct name | summary |+/// | ------------------------------ | -------------------------------- | --------------------------------------------------------------------------------------------- |+/// | `OutputRequirements` | [`OutputRequirements`] | Adds helper nodes so output requirements survive later rewrites (add mode). |+/// | `aggregate_statistics` | [`AggregateStatistics`] | Answers some aggregates from exact statistics without scanning data. |+/// | `EliminateConstantJoin` | [`EliminateConstantJoin`] | Replaces inner joins against a single row of literals with a filter. |+/// | `join_selection` | [`JoinSelection`] | Chooses the join implementation, build side, and partition mode. |+/// | `LimitedDistinctAggregation` | [`LimitedDistinctAggregation`] | Pushes limit hints into grouped distinct-style aggregations. |+/// | `FilterPushdown` | [`FilterPushdown`] | Pushes static filters toward data sources (pre-optimization phase). |+/// | `WindowTopN` | [`WindowTopN`] | Replaces row-number window and filter patterns with per-partition TopK. |+/// | `EnsureRequirements` | [`EnsureRequirements`] | Inserts the repartitioning and sorting needed to meet distribution and ordering requirements. |+/// | `CombinePartialFinalAggregate` | [`CombinePartialFinalAggregate`] | Merges adjacent partial and final aggregates. |+/// | `OptimizeAggregateOrder` | [`OptimizeAggregateOrder`] | Rewrites aggregate expressions to use the input ordering. |+/// | `ProjectionPushdown` | [`ProjectionPushdown`] | Pushes projections toward inputs (early pass). |+/// | `OutputRequirements` | [`OutputRequirements`] | Removes the helper nodes added by the first pass (remove mode). |+/// | `LimitAggregation` | [`TopKAggregation`] | Passes a limit hint into aggregations so they keep fewer groups. |+/// | `LimitPushPastWindows` | [`LimitPushPastWindows`] | Pushes limits through bounded window operators. |+/// | `HashJoinBuffering` | [`HashJoinBuffering`] | Buffers the probe side of hash joins so it is polled before the build side finishes. |+/// | `LimitPushdown` | [`LimitPushdown`] | Pushes limits into child operators or fetch-enabled variants. |+/// | `TopKRepartition` | [`TopKRepartition`] | Pushes TopK below hash repartition when the partition key is a prefix of the sort key. |+/// | `ProjectionPushdown` | [`ProjectionPushdown`] | Pushes projections again after limit and TopK rewrites (late pass). |+/// | `PushdownSort` | [`PushdownSort`] | Pushes sort requirements into data sources that can return sorted output. |+/// | `EnsureCooperative` | [`EnsureCooperative`] | Wraps non-cooperative operators so long-running tasks yield. |+/// | `FilterPushdown(Post)` | [`FilterPushdown`] | Pushes dynamic filters once plan references stop moving (post-optimization phase). |#[derive(Clone, Debug)]pub struct PhysicalOptimizer {/// All rules to apply@@ -176,7 +214,7 @@ impl Default for PhysicalOptimizer {}impl PhysicalOptimizer {- /// Create a new optimizer using the recommended list of rules+ /// Create a new optimizer using the [built-in rules](Self#built-in-physical-optimizer-rules)pub fn new() -> Self {// NOTEs:// - The order of rules in this list is important, as it determines the@@ -298,3 +336,27 @@ impl PhysicalOptimizer {Self { rules }}}++#[cfg(test)]+mod tests {+ use super::PhysicalOptimizer;+ use datafusion_common::test_util::documented_rule_names;++ /// Verifies the rule table in the [`PhysicalOptimizer`] docs matches the+ /// rules (and their order) returned by [`PhysicalOptimizer::new`]+ #[test]+ fn physical_rules_match_documented_order() {+ let rules: Vec<_> = PhysicalOptimizer::new()+ .rules+ .iter()+ .map(|rule| rule.name().to_string())+ .collect();++ let documented = documented_rule_names(+ include_str!("optimizer.rs"),+ "/// # Built-in Physical Optimizer Rules",+ );++ assert_eq!(documented, rules);+ }+}diff --git a/docs/source/library-user-guide/custom-table-providers.md b/docs/source/library-user-guide/custom-table-providers.mdindex 3bea2c7b1e..0c6b5d1aa6 100644--- a/docs/source/library-user-guide/custom-table-providers.md+++ b/docs/source/library-user-guide/custom-table-providers.md@@ -44,6 +44,8 @@ actually produced during execution.[viewtable]: https://docs.rs/datafusion/latest/datafusion/datasource/view/struct.ViewTable.html[planproperties]: https://docs.rs/datafusion/latest/datafusion/physical_plan/struct.PlanProperties.html[streamingtableexec]: https://docs.rs/datafusion/latest/datafusion/physical_plan/streaming/struct.StreamingTableExec.html+[`optimizer`]: https://docs.rs/datafusion/latest/datafusion/optimizer/struct.Optimizer.html#built-in-optimizer-rules+[`physicaloptimizer`]: https://docs.rs/datafusion/latest/datafusion/physical_optimizer/optimizer/struct.PhysicalOptimizer.html#built-in-physical-optimizer-rules[datasourceexec]: https://docs.rs/datafusion/latest/datafusion/datasource/source/struct.DataSourceExec.html## Background: Logical and Physical Planning@@ -80,6 +82,9 @@ optimizations include:- **Limit pushdown** -- pushes `LIMIT` earlier in the plan so operatorsproduce less data.+See the [`Optimizer`] documentation for the full list of built-in logical+optimizer rules.+### Physical PlanningThe **physical planner** converts the optimized logical plan into an@@ -96,6 +101,9 @@ to scan" are made. The physical optimizer then refines this tree further with re- **Aggregate optimization** -- combines partial and final aggregation stages,and can use exact statistics to skip scanning entirely.+See the [`PhysicalOptimizer`] documentation for the full list of built-in+physical optimizer rules.+### Why This Matters for Table ProvidersYour `TableProvider` sits at the boundary between logical and physical planning.diff --git a/docs/source/library-user-guide/query-optimizer.md b/docs/source/library-user-guide/query-optimizer.mdindex 4cedc05fd7..a9415fd89c 100644--- a/docs/source/library-user-guide/query-optimizer.md+++ b/docs/source/library-user-guide/query-optimizer.md@@ -16,7 +16,7 @@ contains an extensive set of [`OptimizerRule`]s and [`PhysicalOptimizerRule`]s tthey execute more quickly while still computing the same result.For a reference list of the built-in analyzer, logical optimizer, and physical optimizer rules,-see [Optimizer Rule Reference].+see the documentation of [`Analyzer`], [`Optimizer`], and [`PhysicalOptimizer`].For a deeper background on optimizer architecture and rule types and predicates, see[Optimizing SQL (and DataFrames) in DataFusion, Part 1], [Part 2],@@ -29,7 +29,9 @@ For a deeper background on optimizer architecture and rule types and predicates,[part 2]: https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-two[using ordering for better plans in apache datafusion]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis[dynamic filters: passing information between operators during execution for 25x faster queries]: https://datafusion.apache.org/blog/2025/09/10/dynamic-filters-[optimizer rule reference]: https://docs.rs/datafusion/latest/datafusion/index.html#built-in-optimizer-rules+[`analyzer`]: https://docs.rs/datafusion/latest/datafusion/optimizer/struct.Analyzer.html#built-in-analyzer-rules+[`optimizer`]: https://docs.rs/datafusion/latest/datafusion/optimizer/struct.Optimizer.html#built-in-optimizer-rules+[`physicaloptimizer`]: https://docs.rs/datafusion/latest/datafusion/physical_optimizer/optimizer/struct.PhysicalOptimizer.html#built-in-physical-optimizer-rules[`logicalplan`]: https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.LogicalPlan.html## Running the Optimizer@@ -358,6 +360,8 @@ There should also be a test in `integration-tests.rs` that tests the rule as parThe `EXPLAIN VERBOSE` command can be used to show the effect of each optimization rule on a query.In the following example, the `type_coercion` and `simplify_expressions` passes have simplified the plan so that it returns the constant `"3.2"` rather than doing a computation at execution time.+The rule names shown in the output are listed, in the order the rules run, in the+documentation of [`Analyzer`], [`Optimizer`], and [`PhysicalOptimizer`].```text> explain verbose select cast(1 + 2.2 as string) as foo;diff --git a/docs/source/user-guide/explain-usage.md b/docs/source/user-guide/explain-usage.mdindex f87a373da3..5672026934 100644--- a/docs/source/user-guide/explain-usage.md+++ b/docs/source/user-guide/explain-usage.md@@ -152,6 +152,14 @@ that is omitted from the default explain, as well as all intermediate physicalplans DataFusion generates before returning. This mode can be very helpful fordebugging to see why and when DataFusion added and removed operators from a plan.+Each intermediate plan is labeled with the name of the optimizer rule that+produced it. The rules, and the order in which they run, are listed in the+documentation of [`Analyzer`], [`Optimizer`], and [`PhysicalOptimizer`].++[`analyzer`]: https://docs.rs/datafusion/latest/datafusion/optimizer/struct.Analyzer.html#built-in-analyzer-rules+[`optimizer`]: https://docs.rs/datafusion/latest/datafusion/optimizer/struct.Optimizer.html#built-in-optimizer-rules+[`physicaloptimizer`]: https://docs.rs/datafusion/latest/datafusion/physical_optimizer/optimizer/struct.PhysicalOptimizer.html#built-in-physical-optimizer-rules+## Execution Counters: `EXPLAIN ANALYZE`During execution, DataFusion operators collect detailed metrics. You can access