Apache DataFusion SQL Query Engine
refactor: deprecate the bundled statistics providers (#25969)
Alessandro Solimandobb24345ad5fd
diff --git a/benchmarks/src/statistics.rs b/benchmarks/src/statistics.rsindex 6c14264356..f574add42b 100644--- a/benchmarks/src/statistics.rs+++ b/benchmarks/src/statistics.rs@@ -249,7 +249,7 @@ enum QError {}fn capture_statistics(plan: &dyn ExecutionPlan) -> Result<Vec<CapturedStatistics>> {- let statistics_context = StatisticsRegistry::default_with_builtin_providers();+ let statistics_context = StatisticsRegistry::new();let mut result = vec![];capture_statistics_inner(plan, &statistics_context, "0", &mut result)?;Ok(result)diff --git a/datafusion-examples/README.md b/datafusion-examples/README.mdindex 6c7aad31a9..42e3410025 100644--- a/datafusion-examples/README.md+++ b/datafusion-examples/README.md@@ -220,9 +220,9 @@ cargo run --example dataframe -- dataframe#### Category: Single Process-| Subcommand | File Path | Description |-| ------------ | ------------------------------------------------------------------- | ----------------------------------------------------------------------- |-| join_reorder | [`statistics/join_reorder.rs`](examples/statistics/join_reorder.rs) | Supply and refine column statistics via a provider to flip a join order |+| Subcommand | File Path | Description |+| ------------ | ------------------------------------------------------------------- | -------------------------------------------------------------------- |+| join_reorder | [`statistics/join_reorder.rs`](examples/statistics/join_reorder.rs) | Supply catalog column statistics via a provider to flip a join order |## UDF Examplesdiff --git a/datafusion-examples/examples/statistics/join_reorder.rs b/datafusion-examples/examples/statistics/join_reorder.rsindex 0eccc93630..42ab2cfbcc 100644--- a/datafusion-examples/examples/statistics/join_reorder.rs+++ b/datafusion-examples/examples/statistics/join_reorder.rs@@ -15,21 +15,18 @@// specific language governing permissions and limitations// under the License.-//! Plug refined cardinality estimation into the optimizer via the `StatisticsRegistry`.+//! Supply catalog statistics to the optimizer via the `StatisticsRegistry`.//!-//! DataFusion's built-in statistics are intentionally simple defaults; the registry-//! is the seam for plugging in more refined estimation (formulas or heuristics-//! suited to your system or data) without changing the core.+//! The registry is the extension point for statistics the core cannot derive,+//! such as column statistics a catalog knows.//!-//! `(SELECT user_id FROM events WHERE amount < 50 GROUP BY user_id) JOIN dims`: a-//! provider supplies the column stats a catalog knows (`amount` range, `user_id`-//! distinct count); the built-in `FilterStatisticsProvider` then refines the-//! post-filter distinct count with the survival formula-//! `NDV * (1 - (1 - selectivity)^(rows / NDV))` (Yao/Cardenas) to ~32, below `dims`-//! (48), flipping the join build side. Core's simpler `min(NDV, rows)` cap would-//! give 50 (> 48) and keep the other order; the refinement is the point. The-//! ground-truth query prints the true surviving distinct count (below 48),-//! confirming the flip.+//! `(SELECT user_id FROM events WHERE amount < 50 GROUP BY user_id) JOIN dims`:+//! the in-memory `events` table has no column statistics, so the grouped side is+//! estimated at 1000 rows and `dims` (60 rows) becomes the build side. A provider+//! supplies the `amount` range and the `user_id` distinct count a catalog knows;+//! with them the operators estimate the grouped side at 50 rows, below `dims`,+//! which flips the join build side. The ground-truth query prints the true+//! distinct count (below 60), confirming the flip.use std::sync::Arc;@@ -111,11 +108,9 @@ fn build_ctx(with_registry: bool) -> Result<SessionContext> {.with_config(config).with_default_features();if with_registry {- let mut registry = StatisticsRegistry::default_with_builtin_providers();- registry.register(Arc::new(ClosureStatisticsProvider::with_matches(- catalog_matches,- catalog_stats,- )));+ let registry = StatisticsRegistry::with_providers(vec![Arc::new(+ ClosureStatisticsProvider::with_matches(catalog_matches, catalog_stats),+ )]);builder = builder.with_statistics_registry(registry);}let ctx = SessionContext::new_with_state(builder.build());@@ -135,8 +130,8 @@ fn build_ctx(with_registry: bool) -> Result<SessionContext> {ctx.register_table("dims",mem_table(&[- ("user_id", int_col(&(0..48).collect::<Vec<_>>())),- ("label", int_col(&(0..48).collect::<Vec<_>>())),+ ("user_id", int_col(&(0..60).collect::<Vec<_>>())),+ ("label", int_col(&(0..60).collect::<Vec<_>>())),])?,)?;Ok(ctx)@@ -167,13 +162,13 @@ pub async fn join_reorder() -> Result<()> {"A hash join builds its in-memory hash table from one input and probes with\n\the other, so the smaller input should be the build side. Default estimation\n\sizes the grouped `events` at 1000 rows and builds from `dims`; the\n\- registry's refined ~32 estimate is below `dims` (48 rows), so it flips the\n\- build side to `events`. The ground-truth count above (also below 48)\n\+ catalog statistics bring the estimate to 50, below `dims` (60 rows), so the\n\+ build side flips to `events`. The ground-truth count above (also below 60)\n\confirms `events` really is the smaller, cheaper side.\n");println!("-- Without the registry (default estimation) --");println!("{}\n", explain(&build_ctx(false)?).await?);- println!("-- With the registry (built-in refinement) --");+ println!("-- With the registry (catalog statistics) --");println!("{}", explain(&build_ctx(true)?).await?);Ok(())}diff --git a/datafusion-examples/examples/statistics/main.rs b/datafusion-examples/examples/statistics/main.rsindex 313e18645d..25e74fcf82 100644--- a/datafusion-examples/examples/statistics/main.rs+++ b/datafusion-examples/examples/statistics/main.rs@@ -30,7 +30,7 @@//! - `all`: run all examples included in this module//!//! - `join_reorder`-//! (file: join_reorder.rs, desc: Supply and refine column statistics via a provider to flip a join order)+//! (file: join_reorder.rs, desc: Supply catalog column statistics via a provider to flip a join order)mod join_reorder;diff --git a/datafusion/ffi/src/physical_optimizer.rs b/datafusion/ffi/src/physical_optimizer.rsindex 87fe3314af..e74a1cb9ff 100644--- a/datafusion/ffi/src/physical_optimizer.rs+++ b/datafusion/ffi/src/physical_optimizer.rs@@ -586,7 +586,7 @@ mod tests {let ctx = ContextWithRegistry {config: ConfigOptions::new(),- registry: StatisticsRegistry::default_with_builtin_providers(),+ registry: StatisticsRegistry::new(),};let plan = create_test_plan();diff --git a/datafusion/physical-plan/src/operator_statistics/mod.rs b/datafusion/physical-plan/src/operator_statistics/mod.rsindex e09aa6b86a..fd5c203503 100644--- a/datafusion/physical-plan/src/operator_statistics/mod.rs+++ b/datafusion/physical-plan/src/operator_statistics/mod.rs@@ -37,17 +37,12 @@//! - [`StatisticsRegistry`]: Chains providers, lives in SessionState//! - [`ExtendedStatistics`]: Statistics with type-safe custom extensions//!-//! # Built-in Providers+//! # Bundled Providers//!-//! The following providers are included and can be registered in this order:-//!-//! 1. [`FilterStatisticsProvider`] - selectivity-based filter estimation-//! 2. [`ProjectionStatisticsProvider`] - column mapping through projections-//! 3. [`PassthroughStatisticsProvider`] - passthrough for cardinality-preserving operators-//! 4. [`AggregateStatisticsProvider`] - NDV-based GROUP BY cardinality estimation-//! 5. [`JoinStatisticsProvider`] - NDV-based join output estimation (hash, sort-merge, cross)-//! 6. [`LimitStatisticsProvider`] - caps output at the fetch limit (local and global)-//! 7. [`UnionStatisticsProvider`] - sums input row counts+//! The providers bundled in this module are deprecated: the default statistics+//! estimation is each operator's [`ExecutionPlan::statistics_from_inputs`], and+//! estimation improvements belong there. Register user-defined providers to+//! plug in other estimation.//!//! # Statistics walk//!@@ -392,7 +387,7 @@ impl StatisticsRegistry {Self { providers }}- /// Create a registry pre-loaded with the standard built-in providers.+ /// Create a registry with the deprecated bundled providers.////// Provider order (first match wins):/// 1. [`FilterStatisticsProvider`]@@ -402,6 +397,11 @@ impl StatisticsRegistry {/// 5. [`JoinStatisticsProvider`]/// 6. [`LimitStatisticsProvider`]/// 7. [`UnionStatisticsProvider`]+ #[deprecated(+ since = "56.0.0",+ note = "the bundled providers are deprecated; the statistics walk uses each operator's `statistics_from_inputs` when no provider matches"+ )]+ #[expect(deprecated)]pub fn default_with_builtin_providers() -> Self {Self::with_providers(vec![Arc::new(FilterStatisticsProvider),@@ -590,9 +590,14 @@ fn computed_with_row_count(/// estimation logic as `FilterExec::statistics_helper`, then additionally/// adjusts each column's `distinct_count` using [`ndv_after_selectivity`] based/// on the computed selectivity ratio.+#[deprecated(+ since = "56.0.0",+ note = "duplicates `FilterExec::statistics_from_inputs` and adds a distinct count adjustment that belongs in the operator, see https://github.com/apache/datafusion/issues/26052; without a matching provider the statistics walk uses the operator's own estimate"+)]#[derive(Debug, Default)]pub struct FilterStatisticsProvider;+#[expect(deprecated)]impl StatisticsProvider for FilterStatisticsProvider {fn matches(&self, plan: &dyn ExecutionPlan) -> bool {plan.downcast_ref::<FilterExec>().is_some()@@ -649,9 +654,14 @@ impl StatisticsProvider for FilterStatisticsProvider {/// Maps enhanced child column statistics to output columns based on the/// projection expressions, preserving NDV and other statistics through/// column references.+#[deprecated(+ since = "56.0.0",+ note = "duplicates the `statistics_from_inputs` of `ProjectionExec`; without a matching provider the statistics walk uses the operator's own estimate"+)]#[derive(Debug, Default)]pub struct ProjectionStatisticsProvider;+#[expect(deprecated)]impl StatisticsProvider for ProjectionStatisticsProvider {fn matches(&self, plan: &dyn ExecutionPlan) -> bool {plan.downcast_ref::<ProjectionExec>().is_some()@@ -691,9 +701,14 @@ impl StatisticsProvider for ProjectionStatisticsProvider {/// transform statistics, so we pass through the enhanced child stats directly./// This avoids the fallback calling `statistics_from_inputs` (overall) which would/// trigger a redundant internal recursion with raw (non-enhanced) stats.+#[deprecated(+ since = "56.0.0",+ note = "duplicates the `statistics_from_inputs` of the cardinality-preserving operators; without a matching provider the statistics walk uses the operator's own estimate"+)]#[derive(Debug, Default)]pub struct PassthroughStatisticsProvider;+#[expect(deprecated)]impl StatisticsProvider for PassthroughStatisticsProvider {fn matches(&self, plan: &dyn ExecutionPlan) -> bool {plan.children().len() == 1@@ -742,9 +757,14 @@ impl StatisticsProvider for PassthroughStatisticsProvider {/// - GROUP BY is empty (scalar aggregate)/// - Any GROUP BY expression is not a simple column reference/// - Any GROUP BY column lacks NDV information+#[deprecated(+ since = "56.0.0",+ note = "duplicates the `statistics_from_inputs` of `AggregateExec`; without a matching provider the statistics walk uses the operator's own estimate"+)]#[derive(Debug, Default)]pub struct AggregateStatisticsProvider;+#[expect(deprecated)]impl StatisticsProvider for AggregateStatisticsProvider {fn matches(&self, plan: &dyn ExecutionPlan) -> bool {plan.downcast_ref::<AggregateExec>().is_some()@@ -835,9 +855,14 @@ impl StatisticsProvider for AggregateStatisticsProvider {/// Delegates when:/// - The plan is not a supported join type/// - Either input lacks row count information+#[deprecated(+ since = "56.0.0",+ note = "replaces the estimates of `HashJoinExec`, `SortMergeJoinExec` and `CrossJoinExec` with separate estimation logic; without a matching provider the statistics walk uses the operator's own estimate"+)]#[derive(Debug, Default)]pub struct JoinStatisticsProvider;+#[expect(deprecated)]impl StatisticsProvider for JoinStatisticsProvider {fn matches(&self, plan: &dyn ExecutionPlan) -> bool {plan.downcast_ref::<HashJoinExec>().is_some()@@ -960,9 +985,14 @@ impl StatisticsProvider for JoinStatisticsProvider {////// Caps output row count at the limit value, accounting for any leading skip offset/// in `GlobalLimitExec`.+#[deprecated(+ since = "56.0.0",+ note = "duplicates the `statistics_from_inputs` of the limit operators; without a matching provider the statistics walk uses the operator's own estimate"+)]#[derive(Debug, Default)]pub struct LimitStatisticsProvider;+#[expect(deprecated)]impl StatisticsProvider for LimitStatisticsProvider {fn matches(&self, plan: &dyn ExecutionPlan) -> bool {plan.downcast_ref::<LocalLimitExec>().is_some()@@ -1011,9 +1041,14 @@ impl StatisticsProvider for LimitStatisticsProvider {/// Statistics provider for [`UnionExec`].////// Sums row counts across all inputs.+#[deprecated(+ since = "56.0.0",+ note = "duplicates the `statistics_from_inputs` of `UnionExec`; without a matching provider the statistics walk uses the operator's own estimate"+)]#[derive(Debug, Default)]pub struct UnionStatisticsProvider;+#[expect(deprecated)]impl StatisticsProvider for UnionStatisticsProvider {fn matches(&self, plan: &dyn ExecutionPlan) -> bool {plan.downcast_ref::<UnionExec>().is_some()@@ -1134,6 +1169,7 @@ impl StatisticsProvider for ClosureStatisticsProvider {}#[cfg(test)]+#[expect(deprecated)]mod tests {use super::*;use crate::filter::FilterExec;diff --git a/datafusion/sqllogictest/src/test_context.rs b/datafusion/sqllogictest/src/test_context.rsindex bc2859aaa9..0c8c2c5add 100644--- a/datafusion/sqllogictest/src/test_context.rs+++ b/datafusion/sqllogictest/src/test_context.rs@@ -37,6 +37,7 @@ use datafusion::catalog::{CatalogProvider, MemoryCatalogProvider, MemorySchemaProvider, SchemaProvider, Session,};use datafusion::common::config::Dialect;+use datafusion::common::stats::Precision;use datafusion::common::{DataFusionError, Result, not_impl_err};use datafusion::functions::math::abs;use datafusion::logical_expr::async_udf::{AsyncScalarUDF, AsyncScalarUDFImpl};@@ -63,7 +64,11 @@ use datafusion::common::cast::as_float64_array;use datafusion::execution::SessionStateBuilder;use datafusion::execution::memory_pool::UnboundedMemoryPool;use datafusion::execution::runtime_env::RuntimeEnvBuilder;-use datafusion::physical_plan::operator_statistics::StatisticsRegistry;+use datafusion::physical_plan::joins::HashJoinExec;+use datafusion::physical_plan::operator_statistics::{+ ClosureStatisticsProvider, StatisticsRegistry, StatisticsResult,+};+use datafusion::physical_plan::statistics::StatisticsArgs;use log::info;use sqlparser::ast;use tempfile::TempDir;@@ -144,9 +149,31 @@ impl TestContext {relative_path.file_name().and_then(|name| name.to_str()),Some("statistics_registry.slt")) {- state_builder = state_builder.with_statistics_registry(- StatisticsRegistry::default_with_builtin_providers(),+ // Replaces the join estimate with the Cartesian product+ let join_provider = ClosureStatisticsProvider::with_matches(+ |plan| plan.downcast_ref::<HashJoinExec>().is_some(),+ |plan, child_stats| {+ let (Some(&left_rows), Some(&right_rows)) = (+ child_stats[0].base().num_rows.get_value(),+ child_stats[1].base().num_rows.get_value(),+ ) else {+ return Ok(StatisticsResult::Delegate);+ };+ let child_base = child_stats+ .iter()+ .map(|c| Arc::clone(c.base_arc()))+ .collect::<Vec<_>>();+ let mut stats = Arc::unwrap_or_clone(+ plan.statistics_from_inputs(&child_base, &StatisticsArgs::new())?,+ );+ stats.num_rows =+ Precision::Inexact(left_rows.saturating_mul(right_rows));+ Ok(StatisticsResult::Computed(stats.into()))+ },);+ let registry =+ StatisticsRegistry::with_providers(vec![Arc::new(join_provider)]);+ state_builder = state_builder.with_statistics_registry(registry);}let state = state_builder.build();diff --git a/datafusion/sqllogictest/test_files/statistics_registry.slt b/datafusion/sqllogictest/test_files/statistics_registry.sltindex 22209e5eb1..0314c0d9e5 100644--- a/datafusion/sqllogictest/test_files/statistics_registry.slt+++ b/datafusion/sqllogictest/test_files/statistics_registry.slt@@ -18,16 +18,17 @@# StatisticsRegistry: demonstrates improved join ordering via more conservative# cardinality estimates on a skewed dataset.#+# The session registers a test-only provider in test_context.rs that replaces+# the join estimate with the Cartesian product of the input row counts.+## customers (10 rows, 3 distinct customer_ids, skewed 8:1:1)# orders (10 rows, same distribution)# dim_small (50 rows)#-# Built-in operator stats give 10*10 / NDV(3) = 33 < 50, which would keep the+# The operators' own estimates give 10*10 / NDV(3) = 33 < 50, which would keep the# inner join on the build side (wrong; actual output is 66). The registry's# conservative estimate 10*10 = 100 > 50 swaps dim_small to the build side.#-# Parquet files written by COPY TO carry min/max stats (NDV=3 via range) but no-# distinct_count, so the registry falls back to the cartesian product upper bound.# Threshold settings force Partitioned mode so statistics alone drive the swap.statement okdiff --git a/docs/source/library-user-guide/upgrading/56.0.0.md b/docs/source/library-user-guide/upgrading/56.0.0.mdindex 6d616a90a8..fceda8f611 100644--- a/docs/source/library-user-guide/upgrading/56.0.0.md+++ b/docs/source/library-user-guide/upgrading/56.0.0.md@@ -448,9 +448,7 @@ previous default (`use_statistics_registry = false`).- Drop any `use_statistics_registry` setting.- To use statistics providers, register them on the session with- `SessionStateBuilder::with_statistics_registry(...)`. To keep the built-in- NDV-aware providers the flag previously enabled, register- `StatisticsRegistry::default_with_builtin_providers()`.+ `SessionStateBuilder::with_statistics_registry(...)`.- To opt out, register nothing (the default).### Generated protobuf `JsonWriterOptions` gained a `compression_level` field@@ -559,6 +557,25 @@ part of `StatisticsRegistry::default_with_builtin_providers()`.- Remove `DefaultStatisticsProvider` from any custom provider chain; register noterminal provider instead (the walk falls back on its own).+### The bundled statistics providers are deprecated++`StatisticsRegistry::default_with_builtin_providers()` and all the statistics+providers bundled in `datafusion_physical_plan::operator_statistics`+(`FilterStatisticsProvider`, `ProjectionStatisticsProvider`,+`PassthroughStatisticsProvider`, `AggregateStatisticsProvider`,+`JoinStatisticsProvider`, `LimitStatisticsProvider` and+`UnionStatisticsProvider`) are deprecated. They duplicate or replace the+estimation each operator already does in `statistics_from_inputs`, which is the+default. `StatisticsRegistry` remains the extension point for user-defined+providers.++**Migration guide:**++- Stop registering the bundled providers; with no matching provider, the+ statistics walk uses the operator's own estimate.+- To keep a bundled estimation technique, implement it in a user-defined+ `StatisticsProvider`.+### `StatisticsRegistry::compute` and `compute_base` are deprecatedUse the walk instead: