Apache DataFusion SQL Query Engine
docs: Explain the concepts behind `EquivalenceProperties` (#27250)
Andrew Lamb6dc8d41dad42
diff --git a/datafusion/physical-expr-common/src/sort_expr.rs b/datafusion/physical-expr-common/src/sort_expr.rsindex 8af850e759..7f61536f31 100644--- a/datafusion/physical-expr-common/src/sort_expr.rs+++ b/datafusion/physical-expr-common/src/sort_expr.rs@@ -344,6 +344,11 @@ pub fn options_compatible(/// * `ORDER BY A DESC NULLS FIRST` matches/// * `ORDER BY A ASC NULLS FIRST` matches (`ASC` and `NULL` options ignored)/// * `ORDER BY B DESC NULLS FIRST` does not match (different expr)+///+/// See [Required orderings] for where requirements come from and how they+/// are checked against the orderings an input provides.+///+/// [Required orderings]: https://docs.rs/datafusion-physical-expr/latest/datafusion_physical_expr/equivalence/struct.EquivalenceProperties.html#required-orderings#[derive(Clone, Debug)]pub struct PhysicalSortRequirement {/// Physical expression representing the column to sort@@ -697,6 +702,11 @@ impl From<LexOrdering> for Vec<PhysicalSortExpr> {/// expressions after construction like a `LexOrdering` does, because it provides/// no mutation methods. If such methods become necessary, we will need to/// enforce uniqueness like the latter object.+///+/// See [Required orderings] for where requirements come from and how they+/// are checked against the orderings an input provides.+///+/// [Required orderings]: https://docs.rs/datafusion-physical-expr/latest/datafusion_physical_expr/equivalence/struct.EquivalenceProperties.html#required-orderings#[derive(Debug, Clone, PartialEq)]pub struct LexRequirement {reqs: Vec<PhysicalSortRequirement>,@@ -807,6 +817,11 @@ impl From<LexRequirement> for LexOrdering {/// 1. It is non-degenerate, meaning it contains at least one ordering. The/// absence of an input ordering requirement is represented by a `None` value/// in `ExecutionPlan` APIs, which return an `Option<OrderingRequirements>`.+///+/// See [Required orderings] for where requirements come from and how they+/// are checked against the orderings an input provides.+///+/// [Required orderings]: https://docs.rs/datafusion-physical-expr/latest/datafusion_physical_expr/equivalence/struct.EquivalenceProperties.html#required-orderings#[derive(Debug, Clone, PartialEq)]pub enum OrderingRequirements {/// The operator is not able to work without one of these requirements.diff --git a/datafusion/physical-expr/src/equivalence/properties/mod.rs b/datafusion/physical-expr/src/equivalence/properties/mod.rsindex 1206315bbb..b45ee313ca 100644--- a/datafusion/physical-expr/src/equivalence/properties/mod.rs+++ b/datafusion/physical-expr/src/equivalence/properties/mod.rs@@ -38,10 +38,12 @@ use datafusion_expr::sort_properties::{ExprProperties, SortProperties};use indexmap::IndexSet;use itertools::{Either, Itertools};-/// `EquivalenceProperties` stores information about the output of a plan node-/// that can be used to optimize the plan. Currently, it keeps track of:+/// `EquivalenceProperties` stores information about how values are distributed+/// across the output of a plan node that can be used to optimize the plan.+///+/// Currently, it tracks:/// - Sort expressions (orderings),-/// - Complete expression tuples whose values are contiguous,+/// - When all rows of equal values are contiguous for certain tuples (groupings),/// - Equivalent expressions; i.e. expressions known to have the same value./// - Constants expressions; i.e. expressions known to contain a single constant/// value.@@ -50,7 +52,24 @@ use itertools::{Either, Itertools};////// [Using Ordering for Better Plans]: https://datafusion.apache.org/blog/2025/03/11/ordering-analysis////-/// # Example equivalent sort expressions+/// This document explains the concepts and how the pieces fit together.+///+/// # Overview+///+/// | Concept | Example | Stored in |+/// |------------------------|-------------------------------------------|---------------------------------------------------------------------|+/// | Equivalent expressions | `a` and `b` after the filter `a = b` | An [`EquivalenceGroup`] of [`EquivalenceClass`]es |+/// | Constants | `a` after the filter `a = 5` | The `constant` of an [`EquivalenceClass`], added as a [`ConstExpr`] |+/// | Orderings | `[a ASC, b DESC]` after a sort | An [`OrderingEquivalenceClass`] |+/// | Groupings | `[a, b]` declared by a custom data source | A [`GroupingEquivalenceClass`] |+/// | Constraints | A primary key or unique constraint | [`Constraints`] |+///+/// [`EquivalenceClass`]: crate::equivalence::EquivalenceClass+/// [Required orderings]: #required-orderings+/// [`ExecutionPlan::required_input_ordering`]: https://docs.rs/datafusion-physical-plan/latest/datafusion_physical_plan/trait.ExecutionPlan.html#method.required_input_ordering+/// [Checking whether a requirement is satisfied]: #checking-whether-a-requirement-is-satisfied+///+/// # Equivalent sort expressions////// Consider table below:///@@ -67,11 +86,11 @@ use itertools::{Either, Itertools};////// In this case, both `a ASC` and `b DESC` can describe the table ordering./// `EquivalenceProperties` tracks these different valid sort expressions and-/// treat `a ASC` and `b DESC` on an equal footing. For example, if the query+/// treats `a ASC` and `b DESC` on an equal footing. For example, if the query/// specifies the output sorted by EITHER `a ASC` or `b DESC`, the sort can be/// avoided.///-/// # Example equivalent expressions+/// # Equivalent expressions////// Similarly, consider the table below:///@@ -86,11 +105,242 @@ use itertools::{Either, Itertools};/// └---┴---┘/// ```///-/// In this case, columns `a` and `b` always have the same value. With this-/// information, Datafusion can optimize various operations. For example, if-/// the partition requirement is `Hash(a)` and output partitioning is-/// `Hash(b)`, then DataFusion avoids repartitioning the data as the existing-/// partitioning satisfies the requirement.+/// In this case, columns `a` and `b` always have the same value. DataFusion+/// uses this information to optimize various operations. For example, if the+/// partition requirement is `Hash(a)` and output partitioning is `Hash(b)`,+/// then DataFusion avoids repartitioning the data as the existing partitioning+/// satisfies the requirement.+///+/// # Constants+///+/// A constant is an expression with a single value in the output of a plan+/// node. A constant is *uniform* if it has the same value in every output+/// partition, and *heterogeneous* if it has one value within each output+/// partition but may differ between output partitions (see+/// [`AcrossPartitions`]).+///+/// ## Example: Uniform Constant from a Filter+///+/// Consider an input sorted by `b ASC` and the filter `a = 5`, which produces+/// the following table:+///+/// ```text+/// ┌───┬───┐+/// │ a │ b │+/// ├───┼───┤+/// │ 5 │ 1 │+/// │ 5 │ 2 │+/// │ 5 │ 4 │+/// └───┴───┘+/// ```+///+/// The filter adds `a` as a uniform [`ConstExpr`]. If the query requests+/// sorting on `[a ASC, b ASC]` DataFusion knows it can skip sorting.+///+/// ## Example: Heterogeneous Constant from `UNION ALL`+///+/// Consider `SELECT 1 AS c, x FROM t1 UNION ALL SELECT 2 AS c, x FROM t2`,+/// where each input is sorted by `x ASC` and has one output partition. The+/// union has two output partitions, one from each input:+///+/// ```text+/// Partition 0 Partition 1+/// ┌───┬───┐ ┌───┬───┐+/// │ c │ x │ │ c │ x │+/// ├───┼───┤ ├───┼───┤+/// │ 1 │ 1 │ │ 2 │ 2 │+/// │ 1 │ 3 │ │ 2 │ 5 │+/// └───┴───┘ └───┴───┘+/// ```+///+/// Observations:+/// - Within each output partition `c` has one value, but the values differ+/// between output partitions, so `c` is an+/// [`AcrossPartitions::Heterogeneous`] constant.+/// - Each output partition is sorted by `[x ASC]`, and therefore also by+/// `[c ASC, x ASC]`.+/// - However, when the output partitions are merged by `[x ASC]` alone, the+/// result is NOT sorted by `c`, unlike with an [`AcrossPartitions::Uniform`]+/// constant, which would have the same value in every output partition and+/// thus be sorted by `[c ASC, x ASC]` as well.+///+/// Given those observations, `c` must stay in the ordering `[c ASC, x ASC]`+/// rather than being removed as a uniform constant would be.+///+/// # Window functions+///+/// NOTE: it is important to distinguish between *window partition*, the rows+/// with equal `PARTITION BY` keys, and *output partition*, how the data is+/// divided up across execution units during execution.+///+/// ## Example: Window Functions with `PARTITION BY`+///+/// Consider an input sorted by `k ASC` and the window expression+/// `SUM(x) OVER (PARTITION BY k) AS w`, which produces the following rows:+///+/// ```text+/// ┌───┬───┬────┐+/// │ k │ x │ w │+/// ├───┼───┼────┤+/// │ 1 │ 3 │ 5 │+/// │ 1 │ 2 │ 5 │+/// │ 2 │ 4 │ 10 │+/// │ 2 │ 6 │ 10 │+/// │ 3 │ 1 │ 1 │+/// └───┴───┴────┘+/// ```+///+/// Observations:+/// - `w` is not a constant because it differs between window partitions.+/// - The output is sorted by `k` (the window partition key), because+/// DataFusion's window implementation keeps the input order.+/// - Within each run of equal `k`, `w` has one value.+///+/// Given those observations, the output is sorted by `[k, w]` in *every*+/// direction of `w`: ascending or descending, nulls first or last:+///+/// - `[k ASC, w ASC NULLS FIRST]`+/// - `[k ASC, w ASC NULLS LAST]`+/// - `[k ASC, w DESC NULLS FIRST]`+/// - `[k ASC, w DESC NULLS LAST]`+///+/// ## Example: Window Functions without `PARTITION BY`+///+/// Consider the same input and the window expression `SUM(x) OVER () AS w`,+/// which produces the following rows:+///+/// ```text+/// ┌───┬───┬────┐+/// │ k │ x │ w │+/// ├───┼───┼────┤+/// │ 1 │ 3 │ 16 │+/// │ 1 │ 2 │ 16 │+/// │ 2 │ 4 │ 16 │+/// │ 2 │ 6 │ 16 │+/// │ 3 │ 1 │ 16 │+/// └───┴───┴────┘+/// ```+///+/// Observations:+/// - The frame of every row is the whole table, so `w` has one value.+/// - The output is still sorted by `k`, as the input was.+///+/// Given those observations, `w` is added as a constant (a [`ConstExpr`])+/// rather than as orderings.+///+/// ## Example: Set-Monotonic Window Functions+///+/// Consider an input sorted by `[ts ASC]` and the window function+/// `MAX(x) OVER (ORDER BY ts) AS m`, which produces the following rows:+///+/// ```text+/// ┌────┬───┬───┐+/// │ ts │ x │ m │+/// ├────┼───┼───┤+/// │ 1 │ 3 │ 3 │+/// │ 2 │ 1 │ 3 │+/// │ 3 │ 5 │ 5 │+/// │ 4 │ 2 │ 5 │+/// └────┴───┴───┘+/// ```+///+/// Observations:+/// - When written with explicit defaults, this query is equivalent to+/// `SELECT ts, x, MAX(x) OVER (ORDER BY ts RANGE BETWEEN UNBOUNDED PRECEDING AND CURRENT ROW) AS m`,+/// so each row's frame contains every earlier row's frame.+/// - `MAX` is set-monotonically increasing: adding rows to its input can only+/// raise its value or leave it unchanged (see [`SetMonotonicity`]).+///+/// Given those observations, `m` never decreases as `ts` increases, so the+/// output sorting is described by both `[ts ASC]` and `[m ASC]`.+///+/// If the query had a `PARTITION BY k`, with the input sorted by `k` first,+/// the added ordering would be `[k, m ASC]`.+///+/// A decreasing aggregate such as `MIN` results in `DESC`.+///+/// A frame that shrinks as rows advance, such as `CURRENT ROW` to+/// `UNBOUNDED FOLLOWING`, reverses the direction: `MAX` then results in+/// `DESC`.+///+/// [`SetMonotonicity`]: datafusion_expr::SetMonotonicity+///+/// # Required orderings+///+/// Some operators only work on sorted input. A sort merge join needs+/// inputs sorted on the join keys, and a streaming window needs its input+/// sorted by its `PARTITION BY` and then its `ORDER BY` expressions.+///+/// Each operator describes its requirements via+/// [`ExecutionPlan::required_input_ordering`].+///+/// **Orderings** describe how the data is actually distributed.+/// **Requirements** describe what an operator *needs*. The physical optimizer+/// (`EnforceSorting`) compares orderings to requirements with+/// [`Self::ordering_satisfy_requirement`] and inserts or removes Sort+/// operations so that they match (see [Checking whether a requirement is+/// satisfied]).+///+/// # Normal form+///+/// Orderings and requirements (see [Required orderings]) are compared in+/// *normal form*: every expression is replaced by the first member of its+/// equivalence class, uniform constants and expressions that repeat an earlier+/// one are removed.+///+/// For example, if `a` and `b` are equal and `d` is a uniform constant, the+/// requirement `[b ASC, d DESC, c ASC, a ASC]` normalizes to `[a ASC, c ASC]`.+///+/// # Checking whether a requirement is satisfied+///+/// The requirement `[a, b]` holds if the rows are sorted by `a`, and the rows+/// that tie on `a` are sorted by `b`.+///+/// [`Self::ordering_satisfy_requirement`] checks whether a requirement is+/// satisfied by an ordering, and is quite complex.+///+/// # Discovering orderings+///+/// `EquivalenceProperties` discovers and accounts for additional orderings that+/// may not be obvious from various additional information such as constants and+/// equality.+///+/// ## Example: Ordering Implied by a `CAST`+///+/// Consider a table `t` sorted by `[c ASC, a ASC, b ASC]` and a query `SELECT+/// a, b, c FROM t WHERE c = CAST(a AS BIGINT) ORDER BY a, b` that produces the+/// following rows:+///+/// ```text+/// ┌───┬───┬───┐+/// │ a │ b │ c │+/// ├───┼───┼───┤+/// │ 1 │ 2 │ 1 │+/// │ 1 │ 5 │ 1 │+/// │ 2 │ 1 │ 2 │+/// │ 3 │ 4 │ 3 │+/// └───┴───┴───┘+/// ```+///+/// Observations:+/// - The filter keeps the input order, so the rows are still sorted by+/// `[c ASC, a ASC, b ASC]`.+/// - Casting an `INT` to a `BIGINT` preserves order and never maps two+/// different values to the same one, so rows with different `a` are in the+/// order of `c`, which is the order of `a`.+/// - Rows with equal `c` are sorted by `[a, b]`.+///+/// Given those observations, the rows are also sorted by `[a ASC, b ASC]`,+/// even though `a` is not the leading sort key. `EquivalenceProperties`+/// derives this ordering when the equality is added, so the `ORDER BY a, b`+/// needs no sort.+///+/// This holds in general for a function `f` whose output follows the+/// lexicographical order of its arguments, which `f` declares by setting+/// [`ExprProperties::preserves_lex_ordering`].+///+/// [`Self::add_equal_conditions`] and [`Self::add_constants`] look for such+/// orderings whenever they learn something new.////// # Code Example/// ```@@ -107,10 +357,10 @@ use itertools::{Either, Itertools};/// # let col_a = col("a", &schema).unwrap();/// # let col_b = col("b", &schema).unwrap();/// # let col_c = col("c", &schema).unwrap();-/// // This object represents data that is sorted by a ASC, c DESC-/// // with a single constant value of b/// let mut eq_properties = EquivalenceProperties::new(schema);+/// // Single constant value of b/// eq_properties.add_constants(vec![ConstExpr::from(col_b)]);+/// // ORDER BY a ASC, c DESC/// eq_properties.add_ordering([/// PhysicalSortExpr::new_default(col_a).asc(),/// PhysicalSortExpr::new_default(col_c).desc(),@@ -364,8 +614,9 @@ impl EquivalenceProperties {self.geq_cache.clear();}- /// Removes constant expressions that may change across partitions.- /// This method should be used when merging data from different partitions.+ /// Removes constant expressions that may change across output partitions.+ /// This method should be used when merging data from different output+ /// partitions.pub fn clear_per_partition_constants(&mut self) {if self.eq_group.clear_per_partition_constants() {// Renormalize orderings if the equivalence group changes:@@ -754,6 +1005,10 @@ impl EquivalenceProperties {/// tie on `a`./// 3. The orderings now start at `[b, c]` and `[d]`./// 4. The function returns `true` as `[d]` satisfies `d`.+ ///+ /// See [Required orderings] for what a requirement is.+ ///+ /// [Required orderings]: EquivalenceProperties#required-orderingspub fn ordering_satisfy_requirement(&self,given: impl IntoIterator<Item = PhysicalSortRequirement>,@@ -1425,7 +1680,7 @@ impl EquivalenceProperties {/// Returns a `Some` value if the expression is constant according to/// equivalence group, and `None` otherwise. The `Some` variant contains/// an `AcrossPartitions` value indicating whether the expression is- /// constant across partitions, and its actual value (if available).+ /// constant across output partitions, and its actual value (if available).pub fn is_expr_constant(&self,expr: &Arc<dyn PhysicalExpr>,diff --git a/datafusion/physical-plan/src/execution_plan.rs b/datafusion/physical-plan/src/execution_plan.rsindex b236cc2f7c..fce0a8391b 100644--- a/datafusion/physical-plan/src/execution_plan.rs+++ b/datafusion/physical-plan/src/execution_plan.rs@@ -205,6 +205,11 @@ pub trait ExecutionPlan: Any + Debug + DisplayAs + Send + Sync {/// NOTE that checking `!is_empty()` does **not** check for a/// required input ordering. Instead, the correct check is that at/// least one entry must be `Some`+ ///+ /// See [Required orderings] for how the planner checks these requirements+ /// against the orderings a child provides.+ ///+ /// [Required orderings]: EquivalenceProperties#required-orderingsfn required_input_ordering(&self) -> Vec<Option<OrderingRequirements>> {vec![None; self.children().len()]}diff --git a/datafusion/physical-plan/src/windows/mod.rs b/datafusion/physical-plan/src/windows/mod.rsindex 1e5eea0b01..404f225387 100644--- a/datafusion/physical-plan/src/windows/mod.rs+++ b/datafusion/physical-plan/src/windows/mod.rs@@ -364,6 +364,11 @@ pub(crate) fn get_partition_by_sort_exprs(}}+/// Computes the [`EquivalenceProperties`] of a window operator's output.+///+/// See [Window functions] for what this computes along with examples.+///+/// [Window functions]: EquivalenceProperties#window-functionspub(crate) fn window_equivalence_properties(schema: &SchemaRef,input: &Arc<dyn ExecutionPlan>,diff --git a/docs/source/library-user-guide/custom-table-providers.md b/docs/source/library-user-guide/custom-table-providers.mdindex 0c6b5d1aa6..6e6ce5ab2f 100644--- a/docs/source/library-user-guide/custom-table-providers.md+++ b/docs/source/library-user-guide/custom-table-providers.md@@ -43,6 +43,7 @@ actually produced during execution.[listingtable]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html[viewtable]: https://docs.rs/datafusion/latest/datafusion/datasource/view/struct.ViewTable.html[planproperties]: https://docs.rs/datafusion/latest/datafusion/physical_plan/struct.PlanProperties.html+[equivalenceproperties]: https://docs.rs/datafusion/latest/datafusion/physical_expr/struct.EquivalenceProperties.html[streamingtableexec]: https://docs.rs/datafusion/latest/datafusion/physical_plan/streaming/struct.StreamingTableExec.html[`optimizer`]: https://docs.rs/datafusion/latest/datafusion/optimizer/struct.Optimizer.html#built-in-optimizer-rules[`physicaloptimizer`]: https://docs.rs/datafusion/latest/datafusion/physical_optimizer/optimizer/struct.PhysicalOptimizer.html#built-in-physical-optimizer-rules@@ -278,7 +279,8 @@ by file or by shard), expose that here.**Output ordering** declares whether your data is naturally sorted. Thisenables the optimizer to avoid inserting a `SortExec` when a query requires-ordered data. Getting this right can be a significant performance win.+ordered data. Getting this right can be a significant performance win. See+[EquivalenceProperties] for more details.### Partitioning Strategiesdiff --git a/docs/source/library-user-guide/query-optimizer.md b/docs/source/library-user-guide/query-optimizer.mdindex a9415fd89c..b939333c5f 100644--- a/docs/source/library-user-guide/query-optimizer.md+++ b/docs/source/library-user-guide/query-optimizer.md@@ -23,6 +23,10 @@ For a deeper background on optimizer architecture and rule types and predicates,[Using Ordering for Better Plans in Apache DataFusion], and[Dynamic Filters: Passing Information Between Operators During Execution for 25x Faster Queries].+The physical optimizer's reasoning about sort orders, equivalent expressions,+and constants is implemented by [`EquivalenceProperties`], whose documentation+explains the details.+[`optimizerrule`]: https://docs.rs/datafusion/latest/datafusion/optimizer/trait.OptimizerRule.html[`physicaloptimizerrule`]: https://docs.rs/datafusion/latest/datafusion/physical_optimizer/trait.PhysicalOptimizerRule.html[optimizing sql (and dataframes) in datafusion, part 1]: https://datafusion.apache.org/blog/2025/06/15/optimizing-sql-dataframes-part-one@@ -32,6 +36,7 @@ For a deeper background on optimizer architecture and rule types and predicates,[`analyzer`]: https://docs.rs/datafusion/latest/datafusion/optimizer/struct.Analyzer.html#built-in-analyzer-rules[`optimizer`]: https://docs.rs/datafusion/latest/datafusion/optimizer/struct.Optimizer.html#built-in-optimizer-rules[`physicaloptimizer`]: https://docs.rs/datafusion/latest/datafusion/physical_optimizer/optimizer/struct.PhysicalOptimizer.html#built-in-physical-optimizer-rules+[`equivalenceproperties`]: https://docs.rs/datafusion/latest/datafusion/physical_expr/struct.EquivalenceProperties.html[`logicalplan`]: https://docs.rs/datafusion/latest/datafusion/logical_expr/enum.LogicalPlan.html## Running the Optimizer