Apache DataFusion SQL Query Engine
Check the final physical plan in the planner and name the rule that broke it (#26461)
Jay Zhan0d3aac0bde72
diff --git a/datafusion/core/src/optimizer_rule_reference.md b/datafusion/core/src/optimizer_rule_reference.mdindex 13e7f62e72..a1393835c4 100644--- a/datafusion/core/src/optimizer_rule_reference.md+++ b/datafusion/core/src/optimizer_rule_reference.md@@ -69,27 +69,31 @@ Rule order matters. The default pipeline may change between releases.The same rule name may appear more than once when the default pipeline runs itin multiple phases.-| order | rule | phase | summary |-| ----- | ------------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------ |-| 1 | `OutputRequirements` | add phase | Adds helper nodes so output requirements survive later physical rewrites. |-| 2 | `aggregate_statistics` | - | Uses exact source statistics to answer some aggregates without scanning data. |-| 3 | `EliminateConstantJoin` | - | Replaces inner joins against a single row of literals with a filter that keeps the other input's order. |-| 4 | `join_selection` | - | Chooses join implementation, build side, and partition mode from statistics and stream properties. |-| 5 | `LimitedDistinctAggregation` | - | Pushes limit hints into grouped distinct-style aggregations when only a small result is needed. |-| 6 | `FilterPushdown` | pre-optimization phase | Pushes supported physical filters down toward data sources before distribution and sorting are enforced. |-| 7 | `WindowTopN` | - | Replaces eligible row-number window and filter patterns with per-partition TopK execution. |-| 8 | `EnsureRequirements` | - | Enforces both distribution and sorting requirements in a single idempotent rule. |-| 9 | `CombinePartialFinalAggregate` | - | Collapses adjacent partial and final aggregates when the distributed shape makes them redundant. |-| 10 | `OptimizeAggregateOrder` | - | Updates aggregate expressions to use the best ordering once sort requirements are known. |-| 11 | `ProjectionPushdown` | early pass | Pushes projections toward inputs before later physical rewrites add more limit and TopK structure. |-| 12 | `OutputRequirements` | remove phase | Removes the temporary output-requirement helper nodes after requirement-sensitive planning is done. |-| 13 | `LimitAggregation` | - | Passes a limit hint into eligible aggregations so they can keep fewer accumulator buckets. |-| 14 | `LimitPushPastWindows` | - | Pushes fetch limits through bounded window operators when doing so keeps the result correct. |-| 15 | `HashJoinBuffering` | - | Adds buffering on the probe side of hash joins so probing can start before build completion. |-| 16 | `LimitPushdown` | - | Moves physical limits into child operators or fetch-enabled variants to cut data early. |-| 17 | `TopKRepartition` | - | Pushes TopK below hash repartition when the partition key is a prefix of the sort key. |-| 18 | `ProjectionPushdown` | late pass | Runs projection pushdown again after limit and TopK rewrites expose new pruning opportunities. |-| 19 | `PushdownSort` | - | Pushes sort requirements into data sources that can already return sorted output. |-| 20 | `EnsureCooperative` | - | Wraps non-cooperative plan parts so long-running tasks yield fairly. |-| 21 | `FilterPushdown(Post)` | post-optimization phase | Pushes dynamic filters at the end of optimization, after plan references stop moving. |-| 22 | `SanityCheckPlan` | - | Validates that the final physical plan meets ordering, distribution, and infinite-input safety requirements. |+After the last rule, including any rule added by the user, the physical planner+checks that the final plan meets its ordering, distribution, and infinite-input+safety requirements (`SanityCheckPlan`). If the ordering or distribution+requirements do not hold, the error names the rule that broke them.++| order | rule | phase | summary |+| ----- | ------------------------------ | ----------------------- | -------------------------------------------------------------------------------------------------------- |+| 1 | `OutputRequirements` | add phase | Adds helper nodes so output requirements survive later physical rewrites. |+| 2 | `aggregate_statistics` | - | Uses exact source statistics to answer some aggregates without scanning data. |+| 3 | `EliminateConstantJoin` | - | Replaces inner joins against a single row of literals with a filter that keeps the other input's order. |+| 4 | `join_selection` | - | Chooses join implementation, build side, and partition mode from statistics and stream properties. |+| 5 | `LimitedDistinctAggregation` | - | Pushes limit hints into grouped distinct-style aggregations when only a small result is needed. |+| 6 | `FilterPushdown` | pre-optimization phase | Pushes supported physical filters down toward data sources before distribution and sorting are enforced. |+| 7 | `WindowTopN` | - | Replaces eligible row-number window and filter patterns with per-partition TopK execution. |+| 8 | `EnsureRequirements` | - | Enforces both distribution and sorting requirements in a single idempotent rule. |+| 9 | `CombinePartialFinalAggregate` | - | Collapses adjacent partial and final aggregates when the distributed shape makes them redundant. |+| 10 | `OptimizeAggregateOrder` | - | Updates aggregate expressions to use the best ordering once sort requirements are known. |+| 11 | `ProjectionPushdown` | early pass | Pushes projections toward inputs before later physical rewrites add more limit and TopK structure. |+| 12 | `OutputRequirements` | remove phase | Removes the temporary output-requirement helper nodes after requirement-sensitive planning is done. |+| 13 | `LimitAggregation` | - | Passes a limit hint into eligible aggregations so they can keep fewer accumulator buckets. |+| 14 | `LimitPushPastWindows` | - | Pushes fetch limits through bounded window operators when doing so keeps the result correct. |+| 15 | `HashJoinBuffering` | - | Adds buffering on the probe side of hash joins so probing can start before build completion. |+| 16 | `LimitPushdown` | - | Moves physical limits into child operators or fetch-enabled variants to cut data early. |+| 17 | `TopKRepartition` | - | Pushes TopK below hash repartition when the partition key is a prefix of the sort key. |+| 18 | `ProjectionPushdown` | late pass | Runs projection pushdown again after limit and TopK rewrites expose new pruning opportunities. |+| 19 | `PushdownSort` | - | Pushes sort requirements into data sources that can already return sorted output. |+| 20 | `EnsureCooperative` | - | Wraps non-cooperative plan parts so long-running tasks yield fairly. |+| 21 | `FilterPushdown(Post)` | post-optimization phase | Pushes dynamic filters at the end of optimization, after plan references stop moving. |diff --git a/datafusion/core/src/physical_planner.rs b/datafusion/core/src/physical_planner.rsindex bcabd26922..198e5bb741 100644--- a/datafusion/core/src/physical_planner.rs+++ b/datafusion/core/src/physical_planner.rs@@ -94,6 +94,10 @@ use datafusion_physical_expr::expressions::Literal;use datafusion_physical_expr::{LexOrdering, PhysicalSortExpr, create_physical_sort_exprs,};+use datafusion_physical_optimizer::ensure_requirements::EnsureRequirements;+use datafusion_physical_optimizer::sanity_checker::{+ check_plan_tree_sanity, input_requirements_hold,+};use datafusion_physical_plan::empty::EmptyExec;use datafusion_physical_plan::execution_plan::InvariantLevel;use datafusion_physical_plan::joins::PiecewiseMergeJoinExec;@@ -3481,9 +3485,18 @@ impl DefaultPhysicalPlanner {observer(new_plan.as_ref(), optimizer.as_ref())}- // This runs once after all optimizer runs are complete,- // to verify that the plan is executable.- InvariantChecker(InvariantLevel::Executable).check(&new_plan)?;+ // This runs once after all optimizer runs are complete, including+ // rules appended by the user, to verify that the plan is executable.+ let optimizer_options = &session_state.config_options().optimizer;+ if let Err(e) = check_plan_tree_sanity(&new_plan, optimizer_options) {+ return Err(sanity_check_error(+ e,+ &new_plan,+ plan,+ optimizers,+ &optimizer_context,+ ));+ }debug!("Optimized physical plan:\n{}\n",@@ -3693,6 +3706,73 @@ fn tuple_err<T, R>(value: (Result<T>, Result<R>)) -> Result<(T, R)> {}}+/// Adds context to an error from the final [`check_plan_tree_sanity`].+///+/// `EnsureRequirements` makes the order and distribution requirements hold,+/// and every rule after it must keep them. When they do not hold for+/// `final_plan`, the context names the rule that broke this contract: either+/// `EnsureRequirements` itself, or the rule after which the requirements+/// stopped holding. To find it, `optimizers` run again from `input_plan`, so+/// planning that succeeds keeps no intermediate plans.+fn sanity_check_error(+ error: DataFusionError,+ final_plan: &Arc<dyn ExecutionPlan>,+ input_plan: Arc<dyn ExecutionPlan>,+ optimizers: &[Arc<dyn PhysicalOptimizerRule + Send + Sync>],+ optimizer_context: &dyn PhysicalOptimizerContext,+) -> DataFusionError {+ const SANITY_CHECK: &str = "SanityCheckPlan";+ let plain_error =+ |error| DataFusionError::Context(SANITY_CHECK.to_string(), Box::new(error));+ if input_requirements_hold(final_plan) {+ // The error is about unbounded inputs or another invariant.+ return plain_error(error);+ }++ let ensure_requirements = EnsureRequirements::new();+ let enforced = optimizers+ .iter()+ .rposition(|rule| rule.name() == ensure_requirements.name());+ // Plans before `EnsureRequirements` need not satisfy the requirements, so+ // only plans from its last run onwards are checked. Without it, every+ // plan is checked, starting with the input plan.+ let first_checked = enforced.unwrap_or(0);+ let mut held = enforced.is_none() && input_requirements_hold(&input_plan);+ let mut broken_by = None;+ let mut plan = input_plan;+ for (idx, rule) in optimizers.iter().enumerate() {+ plan = match rule.optimize_with_context(plan, optimizer_context) {+ Ok(plan) => plan,+ // The rules did not run as they did before, so no rule is named.+ Err(_) => return plain_error(error),+ };+ if idx >= first_checked {+ let holds = input_requirements_hold(&plan);+ if held && !holds {+ broken_by = Some(rule);+ }+ held = holds;+ }+ }+ let context = match (broken_by, enforced) {+ (Some(rule), _) => format!(+ "{SANITY_CHECK}: order and distribution requirements stopped holding \+ after physical optimizer rule '{}'",+ rule.name()+ ),+ (None, Some(_)) => format!(+ "{SANITY_CHECK}: the plan produced by physical optimizer rule '{}' \+ does not satisfy the order and distribution requirements",+ ensure_requirements.name()+ ),+ (None, None) => format!(+ "{SANITY_CHECK}: no physical optimizer rule produced a plan that \+ satisfies the order and distribution requirements"+ ),+ };+ DataFusionError::Context(context, Box::new(error))+}+struct OptimizationInvariantChecker<'a> {rule: &'a Arc<dyn PhysicalOptimizerRule + Send + Sync>,}diff --git a/datafusion/core/tests/memory_limit/mod.rs b/datafusion/core/tests/memory_limit/mod.rsindex a370ca51c5..8d67c27e96 100644--- a/datafusion/core/tests/memory_limit/mod.rs+++ b/datafusion/core/tests/memory_limit/mod.rs@@ -578,7 +578,7 @@ async fn sort_merge_join_no_spill() {]).with_memory_limit(1_000).with_config(config)- .with_scenario(Scenario::AccessLogStreaming)+ .with_scenario(Scenario::AccessLogStreamingByPodTime).run().await}@@ -597,7 +597,7 @@ async fn sort_merge_join_spill() {.with_memory_limit(1_000).with_config(config).with_disk_manager_builder(DiskManagerBuilder::default())- .with_scenario(Scenario::AccessLogStreaming)+ .with_scenario(Scenario::AccessLogStreamingByPodTime).with_expected_success().run().await@@ -1461,6 +1461,11 @@ enum Scenario {/// [`StreamingTable`]AccessLogStreaming,+ /// [`Self::AccessLogStreaming`], sorted by `pod, time`, with a+ /// [`StreamingTable`] that declares that order. A sort merge join on+ /// `pod, time` then needs no sort.+ AccessLogStreamingByPodTime,+/// N partitions of sorted, dictionary encoded strings.DictionaryStrings {partitions: usize,@@ -1522,6 +1527,23 @@ impl Scenario {.with_infinite_table(true);Arc::new(table)}+ Self::AccessLogStreamingByPodTime => {+ let batches = access_log_batches_by_pod_time();+ let table = StreamingTable::try_new(+ batches[0].schema(),+ vec![Arc::new(DummyStreamPartition {+ schema: batches[0].schema(),+ batches,+ })],+ )+ .unwrap()+ .with_infinite_table(true)+ .with_sort_order(vec![+ datafusion_expr::col("pod").sort(true, true),+ datafusion_expr::col("time").sort(true, true),+ ]);+ Arc::new(table)+ }Self::DictionaryStrings {partitions,single_row_batches,@@ -1570,7 +1592,7 @@ impl Scenario {// has a memory budget which we'll likely hit first)Some(vec![Arc::new(JoinSelection::new())])}- Self::AccessLogStreaming => {+ Self::AccessLogStreaming | Self::AccessLogStreamingByPodTime => {// Disable all physical optimizer rules except the// JoinSelection rule to avoid sorts or repartition,// as they also have memory budgets that may be hit@@ -1646,6 +1668,26 @@ fn access_log_batches() -> Vec<RecordBatch> {.collect()}+/// [`access_log_batches`] sorted by `pod, time`, in batches of 50 rows+fn access_log_batches_by_pod_time() -> Vec<RecordBatch> {+ let batches = access_log_batches();+ let batch = arrow::compute::concat_batches(&batches[0].schema(), &batches).unwrap();+ let sort_column = |name: &str| arrow::compute::SortColumn {+ values: Arc::clone(batch.column_by_name(name).unwrap()),+ options: Some(SortOptions::default()),+ };+ let indices = arrow::compute::lexsort_to_indices(+ &[sort_column("pod"), sort_column("time")],+ None,+ )+ .unwrap();+ let batch = arrow::compute::take_record_batch(&batch, &indices).unwrap();+ (0..batch.num_rows())+ .step_by(50)+ .map(|offset| batch.slice(offset, 50.min(batch.num_rows() - offset)))+ .collect()+}+/// If `one_row_batches` is true, then returns new record batches that/// are one row in sizefn maybe_split_batches(diff --git a/datafusion/core/tests/physical_optimizer/enforce_distribution.rs b/datafusion/core/tests/physical_optimizer/enforce_distribution.rsindex a6ef2938b4..e728a5512e 100644--- a/datafusion/core/tests/physical_optimizer/enforce_distribution.rs+++ b/datafusion/core/tests/physical_optimizer/enforce_distribution.rs@@ -55,7 +55,7 @@ use datafusion_physical_optimizer::enforce_distribution::*;use datafusion_physical_optimizer::ensure_requirements::EnsureRequirements;use datafusion_physical_optimizer::join_selection::JoinSelection;use datafusion_physical_optimizer::output_requirements::OutputRequirements;-use datafusion_physical_optimizer::sanity_checker::SanityCheckPlan;+use datafusion_physical_optimizer::sanity_checker::check_plan_tree_sanity;use datafusion_physical_plan::aggregates::{AggregateExec, AggregateMode, PhysicalGroupBy,};@@ -1053,7 +1053,7 @@ async fn aligned_hour_groups_avoid_shuffle_through_partial_aggregate() -> Result.with_query_execution_partitions(2).config;let optimized = EnsureRequirements::new().optimize(aggregate, &config)?;- SanityCheckPlan::new().optimize(Arc::clone(&optimized), &config)?;+ check_plan_tree_sanity(&optimized, &config.optimizer)?;assert_plan!(optimized,@@ -1089,7 +1089,7 @@ async fn crossing_day_groups_keep_shuffle_through_partial_aggregate() -> Result<.with_query_execution_partitions(2).config;let optimized = EnsureRequirements::new().optimize(aggregate, &config)?;- SanityCheckPlan::new().optimize(Arc::clone(&optimized), &config)?;+ check_plan_tree_sanity(&optimized, &config.optimizer)?;assert_plan!(optimized,@@ -1126,10 +1126,10 @@ fn range_transform_aggregate_optimization_is_idempotent() -> Result<()> {let input = timestamp_range_input(hour)?;let aggregate = partial_final_date_trunc_count(input, precision)?;let optimized = EnsureRequirements::new().optimize(aggregate, &config)?;- SanityCheckPlan::new().optimize(Arc::clone(&optimized), &config)?;+ check_plan_tree_sanity(&optimized, &config.optimizer)?;let reoptimized =EnsureRequirements::new().optimize(Arc::clone(&optimized), &config)?;- SanityCheckPlan::new().optimize(Arc::clone(&reoptimized), &config)?;+ check_plan_tree_sanity(&reoptimized, &config.optimizer)?;assert_plan!(optimized, reoptimized);}Ok(())@@ -1155,7 +1155,7 @@ async fn range_transform_locality_does_not_satisfy_join_distribution() -> Result.with_query_execution_partitions(2).config;let optimized = EnsureRequirements::new().optimize(join, &config)?;- SanityCheckPlan::new().optimize(Arc::clone(&optimized), &config)?;+ check_plan_tree_sanity(&optimized, &config.optimizer)?;assert_plan!(optimized,@r"@@ -1275,7 +1275,7 @@ fn range_inner_hash_join_rehashes_incompatible_range_partitioning() -> Result<()/// Optimizes `plan` under both `top_down_join_key_reordering` settings with/// round-robin and file-scan repartitioning disabled. Checks the expected plan,-/// validates it with [`SanityCheckPlan`], and verifies that a second+/// validates it with [`check_plan_tree_sanity`], and verifies that a second/// [`EnsureRequirements`] pass leaves it unchanged.fn check_range_join_key_alignment(plan: Arc<dyn ExecutionPlan>,@@ -1299,7 +1299,7 @@ fn check_range_join_key_alignment_with_partitions(let optimized = config.try_to_plan(Arc::clone(&plan), &[Run::Distribution])?;check(&optimized);- SanityCheckPlan::new().optimize(Arc::clone(&optimized), &config.config)?;+ check_plan_tree_sanity(&optimized, &config.config.optimizer)?;let second = config.try_to_plan(Arc::clone(&optimized), &[Run::Distribution])?;assert_eq!(@@ -1307,7 +1307,7 @@ fn check_range_join_key_alignment_with_partitions(displayable(second.as_ref()).indent(true).to_string(),"plan changed on the second pass; top_down={top_down}");- SanityCheckPlan::new().optimize(second, &config.config)?;+ check_plan_tree_sanity(&second, &config.config.optimizer)?;}Ok(())}@@ -4255,15 +4255,11 @@ fn interleave_broken_by_later_rewrite_is_repaired_on_next_pass() -> Result<()> {Partitioning::UnknownPartitioning(10)));// Without a repair, the plan is rejected rather than executed.- assert!(- SanityCheckPlan::new()- .optimize(Arc::clone(&rewritten), &config)- .is_err()- );+ assert!(check_plan_tree_sanity(&rewritten, &config.optimizer).is_err());// The next distribution pass restores a valid interleave.let pass2 = EnsureRequirements::new().optimize(rewritten, &config)?;- SanityCheckPlan::new().optimize(Arc::clone(&pass2), &config)?;+ check_plan_tree_sanity(&pass2, &config.optimizer)?;assert_plan!(pass2,@r"InterleaveExec@@ -4333,16 +4329,12 @@ fn issue_21826_join_selection_after_distribution_pass() -> Result<()> {Partitioning::UnknownPartitioning(10)));// Without a repair, the plan is rejected rather than executed.- assert!(- SanityCheckPlan::new()- .optimize(Arc::clone(&reordered), &config)- .is_err()- );+ assert!(check_plan_tree_sanity(&reordered, &config.optimizer).is_err());// The second distribution pass repairs it. The two joins no longer share// a partitioning, so the result is a plain union.let pass2 = EnsureRequirements::new().optimize(reordered, &config)?;- SanityCheckPlan::new().optimize(Arc::clone(&pass2), &config)?;+ check_plan_tree_sanity(&pass2, &config.optimizer)?;assert_plan!(pass2,@r"UnionExecdiff --git a/datafusion/core/tests/physical_optimizer/enforce_sorting.rs b/datafusion/core/tests/physical_optimizer/enforce_sorting.rsindex b6c8a6b53a..02f0c9156c 100644--- a/datafusion/core/tests/physical_optimizer/enforce_sorting.rs+++ b/datafusion/core/tests/physical_optimizer/enforce_sorting.rs@@ -44,7 +44,7 @@ use datafusion_physical_plan::{displayable, get_plan_string, ExecutionPlan, Execuse datafusion::datasource::physical_plan::CsvSource;use datafusion::datasource::listing::PartitionedFile;use datafusion_physical_optimizer::enforce_sorting::{PlanWithCorrespondingCoalescePartitions, PlanWithCorrespondingSort, parallelize_sorts, ensure_sorting};-use datafusion_physical_optimizer::sanity_checker::SanityCheckPlan;+use datafusion_physical_optimizer::sanity_checker::check_plan_tree_sanity;use datafusion_physical_optimizer::enforce_sorting::replace_with_order_preserving_variants::{replace_with_order_preserving_variants, OrderPreservationContext};use datafusion_physical_optimizer::enforce_sorting::sort_pushdown::{SortPushDown, assign_initial_requirements, pushdown_sorts};use datafusion_physical_optimizer::ensure_requirements::EnsureRequirements;@@ -297,7 +297,7 @@ async fn test_hash_join_interleaved_projection_preserves_parent_sort() -> Resultconfig.execution.target_partitions = 10;let optimized_plan =EnsureRequirements::new().optimize(Arc::clone(&physical_plan), &config)?;- let optimized_plan = SanityCheckPlan::new().optimize(optimized_plan, &config)?;+ check_plan_tree_sanity(&optimized_plan, &config.optimizer)?;assert_snapshot!(displayable(optimized_plan.as_ref()).indent(true), @r"SortPreservingMergeExec: [nullable_col@1 ASC]@@ -378,7 +378,7 @@ fn test_do_not_remove_sort_with_limit_after_projection() -> Result<()> {] {let optimized =EnsureRequirements::new().optimize(Arc::clone(&plan), &config)?;- let optimized = SanityCheckPlan::new().optimize(optimized, &config)?;+ check_plan_tree_sanity(&optimized, &config.optimizer)?;assert_eq!(get_plan_string(&optimized), get_plan_string(&plan));}}@@ -413,7 +413,7 @@ fn test_required_sort_after_projection() -> Result<()> {input,);let optimized = EnsureRequirements::new().optimize(window, &config)?;- let optimized = SanityCheckPlan::new().optimize(optimized, &config)?;+ check_plan_tree_sanity(&optimized, &config.optimizer)?;let twice =EnsureRequirements::new().optimize(Arc::clone(&optimized), &config)?;assert_eq!(get_plan_string(&twice), get_plan_string(&optimized));@@ -471,7 +471,7 @@ async fn test_do_not_remove_sort_below_fetch_after_projection() -> Result<()> {let mut config = ConfigOptions::new();config.execution.target_partitions = 1;let optimized = EnsureRequirements::new().optimize(window, &config)?;- let optimized = SanityCheckPlan::new().optimize(optimized, &config)?;+ check_plan_tree_sanity(&optimized, &config.optimizer)?;let twice = EnsureRequirements::new().optimize(Arc::clone(&optimized), &config)?;assert_eq!(get_plan_string(&twice), get_plan_string(&optimized));@@ -3289,8 +3289,7 @@ async fn test_parallelize_sorts_remaps_index_through_reordering_projection() ->// but kept the key `score@1` (valid only in the projection output), while its// child schema `[a, b, score]` has `score` at index 2 — so `SanityCheckPlan`// reports `does not satisfy order requirements: [...]. Child-0 order: []`.- SanityCheckPlan::new()- .optimize(Arc::clone(&optimized), &ConfigOptions::default())+ check_plan_tree_sanity(&optimized, &ConfigOptions::default().optimizer).unwrap_or_else(|e| {panic!("Sort enforcement produced a plan that fails SanityCheckPlan \diff --git a/datafusion/core/tests/physical_optimizer/ensure_requirements.rs b/datafusion/core/tests/physical_optimizer/ensure_requirements.rsindex 751645c687..85c969ca00 100644--- a/datafusion/core/tests/physical_optimizer/ensure_requirements.rs+++ b/datafusion/core/tests/physical_optimizer/ensure_requirements.rs@@ -41,7 +41,7 @@ use datafusion_physical_plan::{};use datafusion_physical_optimizer::output_requirements::OutputRequirementExec;-use datafusion_physical_optimizer::sanity_checker::SanityCheckPlan;+use datafusion_physical_optimizer::sanity_checker::check_plan_tree_sanity;use datafusion_common::{JoinType, NullEquality};use datafusion_physical_expr::Distribution;@@ -170,7 +170,7 @@ fn optimize_and_sanity_check(let config = test_config();let optimized = EnsureRequirements::new().optimize(plan, &config)?;// SanityCheckPlan must pass- SanityCheckPlan::new().optimize(Arc::clone(&optimized), &config)?;+ check_plan_tree_sanity(&optimized, &config.optimizer)?;Ok(optimized)}@@ -192,11 +192,9 @@ fn assert_idempotent(plan: Arc<dyn ExecutionPlan>) {);// Both must pass SanityCheckPlan- SanityCheckPlan::new()- .optimize(p1, &config)+ check_plan_tree_sanity(&p1, &config.optimizer).expect("SanityCheckPlan failed on first pass");- SanityCheckPlan::new()- .optimize(p2, &config)+ check_plan_tree_sanity(&p2, &config.optimizer).expect("SanityCheckPlan failed on second pass");}@@ -238,8 +236,7 @@ macro_rules! assert_ensure_requirements_plan {let p1 = EnsureRequirements::new().optimize($plan, &config).expect("EnsureRequirements::optimize failed (pass 1)");- SanityCheckPlan::new()- .optimize(Arc::clone(&p1), &config)+ check_plan_tree_sanity(&p1, &config.optimizer).expect("SanityCheckPlan failed (pass 1)");let p1_str = plan_string(&p1);insta::assert_snapshot!(p1_str, @ $snapshot);@@ -253,8 +250,7 @@ macro_rules! assert_ensure_requirements_plan {p1_str, p2_str,"EnsureRequirements is NOT idempotent!\nPass 1:\n{p1_str}\nPass 2:\n{p2_str}",);- SanityCheckPlan::new()- .optimize(p2, &config)+ check_plan_tree_sanity(&p2, &config.optimizer).expect("SanityCheckPlan failed (pass 2)");}};}@@ -776,11 +772,9 @@ fn test_issue_21973_parallel_sort_survives_multiple_passes() {// All passes must produce sanity-checkable plans.for (i, p) in [p1, p2, p3].into_iter().enumerate() {- SanityCheckPlan::new()- .optimize(p, &config)- .unwrap_or_else(|e| {- panic!("SanityCheckPlan failed on pass {}: {e:?}", i + 1)- });+ check_plan_tree_sanity(&p, &config.optimizer).unwrap_or_else(|e| {+ panic!("SanityCheckPlan failed on pass {}: {e:?}", i + 1)+ });}}@@ -1188,11 +1182,9 @@ fn test_idempotent_parallelize_sorts() {"parallelize_sorts NOT idempotent!\nFirst:\n{s1}\nSecond:\n{s2}");- SanityCheckPlan::new()- .optimize(p1, &config)+ check_plan_tree_sanity(&p1, &config.optimizer).expect("SanityCheckPlan failed on first pass");- SanityCheckPlan::new()- .optimize(p2, &config)+ check_plan_tree_sanity(&p2, &config.optimizer).expect("SanityCheckPlan failed on second pass");}@@ -1454,7 +1446,7 @@ fn parallelize_sorts_and_sanity_check() -> Result<Arc<dyn ExecutionPlan>> {let ctx = PlanWithCorrespondingCoalescePartitions::new_default(plan);let rewritten = ctx.transform_up(parallelize_sorts).data()?.plan;- SanityCheckPlan::new().optimize(Arc::clone(&rewritten), &test_config())?;+ check_plan_tree_sanity(&rewritten, &test_config().optimizer)?;Ok(rewritten)}diff --git a/datafusion/core/tests/physical_optimizer/sanity_checker.rs b/datafusion/core/tests/physical_optimizer/sanity_checker.rsindex 479b867174..3fac55836c 100644--- a/datafusion/core/tests/physical_optimizer/sanity_checker.rs+++ b/datafusion/core/tests/physical_optimizer/sanity_checker.rs@@ -15,16 +15,20 @@ use crate::physical_optimizer::test_utils::{use arrow::compute::SortOptions;use arrow::datatypes::{DataType, Field, Schema, SchemaRef};use datafusion::datasource::stream::{FileStreamProvider, StreamConfig, StreamTable};+use datafusion::execution::SessionStateBuilder;+use datafusion::execution::session_state::SessionState;use datafusion::prelude::{CsvReadOptions, SessionContext};use datafusion_common::config::ConfigOptions;+use datafusion_common::tree_node::{Transformed, TransformedResult, TreeNode};use datafusion_common::{JoinType, NullEquality, Result, ScalarValue};use datafusion_physical_expr::expressions::{Literal, col};use datafusion_physical_expr::{Partitioning, RangePartitioning, SplitPoint};use datafusion_physical_expr_common::sort_expr::LexOrdering;use datafusion_physical_optimizer::PhysicalOptimizerRule;-use datafusion_physical_optimizer::sanity_checker::SanityCheckPlan;+use datafusion_physical_optimizer::sanity_checker::check_plan_tree_sanity;use datafusion_physical_plan::joins::{StreamJoinPartitionMode, SymmetricHashJoinExec};use datafusion_physical_plan::repartition::RepartitionExec;+use datafusion_physical_plan::sorts::sort::SortExec;use datafusion_physical_plan::{ExecutionPlan, displayable};use async_trait::async_trait;@@ -441,10 +445,9 @@ fn create_test_schema2() -> SchemaRef {/// Check if sanity checker should accept or reject plans.fn assert_sanity_check(plan: &Arc<dyn ExecutionPlan>, is_sane: bool) {- let sanity_checker = SanityCheckPlan::new();let opts = ConfigOptions::default();assert_eq!(- sanity_checker.optimize(plan.clone(), &opts).is_ok(),+ check_plan_tree_sanity(plan, &opts.optimizer).is_ok(),is_sane);}@@ -999,3 +1002,88 @@ async fn test_union_with_sorts_and_constants() -> Result<()> {Ok(())}++/// Test rule that removes every [`SortExec`], so that the final check fails+/// and its error can be checked.+///+/// [`SortExec`]: datafusion_physical_plan::sorts::sort::SortExec+#[derive(Debug)]+struct RemoveSorts;++impl PhysicalOptimizerRule for RemoveSorts {+ fn optimize(+ &self,+ plan: Arc<dyn ExecutionPlan>,+ _config: &ConfigOptions,+ ) -> Result<Arc<dyn ExecutionPlan>> {+ plan.transform_up(|plan| {+ if plan.is::<SortExec>() {+ Ok(Transformed::yes(Arc::clone(plan.children()[0])))+ } else {+ Ok(Transformed::no(plan))+ }+ })+ .data()+ }++ fn name(&self) -> &str {+ "remove_sorts"+ }++ fn schema_check(&self) -> bool {+ true+ }+}++async fn plan_window_query(state: SessionState) -> Result<Arc<dyn ExecutionPlan>> {+ let ctx = SessionContext::new_with_state(state);+ ctx.sql("CREATE TABLE t(a INT) AS VALUES (3), (1), (2)")+ .await?+ .collect()+ .await?;+ ctx.sql("SELECT a, row_number() OVER (ORDER BY a) AS rn FROM t")+ .await?+ .create_physical_plan()+ .await+}++/// A rule appended after the default rules is checked, and the error names it.+#[tokio::test]+async fn test_final_check_names_appended_rule() -> Result<()> {+ let state = SessionStateBuilder::new()+ .with_default_features()+ .with_physical_optimizer_rule(Arc::new(RemoveSorts))+ .build();+ let error = plan_window_query(state).await.unwrap_err().to_string();+ assert!(+ error.starts_with(+ "SanityCheckPlan: order and distribution requirements stopped holding \+ after physical optimizer rule 'remove_sorts'"+ ),+ "{error}"+ );+ assert!(+ error.contains("does not satisfy order requirements"),+ "{error}"+ );+ Ok(())+}++/// A custom rule list is checked too. Without `EnsureRequirements`, the+/// window's sort is never added.+#[tokio::test]+async fn test_final_check_without_ensure_requirements() -> Result<()> {+ let state = SessionStateBuilder::new()+ .with_default_features()+ .with_physical_optimizer_rules(vec![Arc::new(RemoveSorts)])+ .build();+ let error = plan_window_query(state).await.unwrap_err().to_string();+ assert!(+ error.starts_with(+ "SanityCheckPlan: no physical optimizer rule produced a plan that \+ satisfies the order and distribution requirements"+ ),+ "{error}"+ );+ Ok(())+}diff --git a/datafusion/physical-optimizer/src/optimizer.rs b/datafusion/physical-optimizer/src/optimizer.rsindex 07f64c4a23..7c54b3f02a 100644--- a/datafusion/physical-optimizer/src/optimizer.rs+++ b/datafusion/physical-optimizer/src/optimizer.rs@@ -20,7 +20,6 @@ use crate::limit_pushdown::LimitPushdown;use crate::limited_distinct_aggregation::LimitedDistinctAggregation;use crate::output_requirements::OutputRequirements;use crate::projection_pushdown::ProjectionPushdown;-use crate::sanity_checker::SanityCheckPlan;use crate::topk_aggregation::TopKAggregation;use crate::topk_repartition::TopKRepartition;use crate::update_aggr_exprs::OptimizeAggregateOrder;@@ -104,13 +103,15 @@ impl PhysicalOptimizerContext for ConfigOnlyContext<'_> {/// - [`EnsureRequirements`] inserts the repartitioning and sorting needed to/// satisfy the distribution and ordering requirements./// - [`OutputRequirements`] records output requirements.-/// - [`SanityCheckPlan`] rejects plans whose requirements are not satisfied.-/// It should run last.////// Custom pipelines must keep these rules in a compatible relative order. We/// are working to remove these exceptions, so that all rules can be reordered/// freely.///+/// After the last rule, including rules added by the user, the physical+/// planner rejects the plan if its requirements are not satisfied (see+/// [`check_plan_tree_sanity`](crate::sanity_checker::check_plan_tree_sanity)).+////// ## Optimization Coverage Is Not Guaranteed////// The invariants above guarantee *correctness*, not that every optimization is@@ -280,15 +281,10 @@ impl PhysicalOptimizer {// Therefore, it should be run at the end of the optimization process since any changes to the plan may break the dynamic filter's references.// See `FilterPushdownPhase` for more details.Arc::new(FilterPushdown::new_post_optimization()),- // The SanityCheckPlan rule checks whether the order and- // distribution requirements of each node in the plan- // is satisfied. It will also reject non-runnable query- // plans that use pipeline-breaking operators on infinite- // input(s). The rule generates a diagnostic error- // message for invalid plans. It makes no changes to the- // given query plan; i.e. it only acts as a final- // gatekeeping rule.- Arc::new(SanityCheckPlan::new()),+ // There is no `SanityCheckPlan` rule at the end of this list. The+ // physical planner runs the same check on the final plan, after+ // every rule including the ones appended by the user. See+ // `crate::sanity_checker`.];Self::with_rules(rules)diff --git a/datafusion/physical-optimizer/src/sanity_checker.rs b/datafusion/physical-optimizer/src/sanity_checker.rsindex 7f120aa1f3..bdfb70b9bb 100644--- a/datafusion/physical-optimizer/src/sanity_checker.rs+++ b/datafusion/physical-optimizer/src/sanity_checker.rs@@ -7,6 +7,12 @@//! non-runnable query plans that use pipeline-breaking operators on//! infinite input(s). In addition, it will check if all order and//! distribution requirements of a plan are satisfied by its children.+//!+//! The physical planner runs the same check on the final plan of every query,+//! after all physical optimizer rules, including rules appended by the user.+//! The rule is therefore not part of the default rule list and is deprecated.+//! Code that drives optimizer rules directly can call+//! [`check_plan_tree_sanity`] on the final plan.use std::sync::Arc;@@ -35,9 +41,14 @@ use itertools::izip;/// are not satisfied by their children./// 2. Plans that use pipeline-breaking operators on infinite input(s),/// it is impossible to execute such queries (they will never generate output nor finish)+#[deprecated(+ since = "56.0.0",+ note = "The physical planner checks the final plan after all rules. To check a plan outside the planner, use `check_plan_tree_sanity`"+)]#[derive(Default, Debug)]pub struct SanityCheckPlan {}+#[expect(deprecated)]impl SanityCheckPlan {#[expect(missing_docs)]pub fn new() -> Self {@@ -45,6 +56,7 @@ impl SanityCheckPlan {}}+#[expect(deprecated)]impl PhysicalOptimizerRule for SanityCheckPlan {fn optimize(&self,@@ -64,6 +76,29 @@ impl PhysicalOptimizerRule for SanityCheckPlan {}}+/// Checks every node of `plan` with [`check_plan_sanity`], children first.+pub fn check_plan_tree_sanity(+ plan: &Arc<dyn ExecutionPlan>,+ optimizer_options: &OptimizerOptions,+) -> Result<()> {+ check_plan_sanity_recursive(plan, optimizer_options).map(|_| ())+}++/// Returns whether every node of `plan` has its order and distribution+/// requirements satisfied by its children, as checked by+/// [`check_input_requirements`].+pub fn input_requirements_hold(plan: &Arc<dyn ExecutionPlan>) -> bool {+ let mut hold = true;+ let _ = plan.apply(|node| {+ if check_input_requirements(node).is_err() {+ hold = false;+ return Ok(TreeNodeRecursion::Stop);+ }+ Ok(TreeNodeRecursion::Continue)+ });+ hold+}+/// Bottom-up (post-order) read-only traversal that checks plan sanity.#[cfg_attr(feature = "recursive_protection", recursive::recursive)]fn check_plan_sanity_recursive(@@ -133,6 +168,15 @@ pub fn check_plan_sanity(optimizer_options: &OptimizerOptions,) -> Result<()> {check_finiteness_requirements(plan.as_ref(), optimizer_options)?;+ check_input_requirements(plan)?;+ plan.check_invariants(InvariantLevel::Executable)?;++ Ok(())+}++/// Checks that the order and distribution requirements of `plan` are+/// satisfied by its children.+pub fn check_input_requirements(plan: &Arc<dyn ExecutionPlan>) -> Result<()> {let input_distributions = plan.input_distribution_requirements();for ((idx, child), sort_req, dist_req) in izip!(@@ -177,8 +221,6 @@ pub fn check_plan_sanity(}}- plan.check_invariants(InvariantLevel::Executable)?;-Ok(())}diff --git a/datafusion/sqllogictest/test_files/explain.slt b/datafusion/sqllogictest/test_files/explain.sltindex 9113388dc4..65547d63fc 100644--- a/datafusion/sqllogictest/test_files/explain.slt+++ b/datafusion/sqllogictest/test_files/explain.slt@@ -242,7 +242,6 @@ physical_plan after ProjectionPushdown SAME TEXT AS ABOVEphysical_plan after PushdownSort SAME TEXT AS ABOVEphysical_plan after EnsureCooperative SAME TEXT AS ABOVEphysical_plan after FilterPushdown(Post) SAME TEXT AS ABOVE-physical_plan after SanityCheckPlan SAME TEXT AS ABOVEphysical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/example.csv]]}, projection=[a, b, c], file_type=csv, has_header=truephysical_plan_with_stats DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/example.csv]]}, projection=[a, b, c], file_type=csv, has_header=true, statistics=[Rows=Absent, Bytes=Absent, [(Col[0]:),(Col[1]:),(Col[2]:)]]physical_plan_with_schema DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/example.csv]]}, projection=[a, b, c], file_type=csv, has_header=true, schema=[a:Int32;N, b:Int32;N, c:Int32;N]@@ -360,7 +359,6 @@ physical_plan after ProjectionPushdown SAME TEXT AS ABOVEphysical_plan after PushdownSort SAME TEXT AS ABOVEphysical_plan after EnsureCooperative SAME TEXT AS ABOVEphysical_plan after FilterPushdown(Post) SAME TEXT AS ABOVE-physical_plan after SanityCheckPlan SAME TEXT AS ABOVEphysical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/parquet-testing/data/alltypes_plain.parquet]]}, projection=[id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col], limit=10, file_type=parquet, statistics=[Rows=Exact(8), Bytes=Absent, [(Col[0]: ScanBytes=Exact(32)),(Col[1]: ScanBytes=Inexact(24)),(Col[2]: ScanBytes=Exact(32)),(Col[3]: ScanBytes=Exact(32)),(Col[4]: ScanBytes=Exact(32)),(Col[5]: ScanBytes=Exact(64)),(Col[6]: ScanBytes=Exact(32)),(Col[7]: ScanBytes=Exact(64)),(Col[8]: ScanBytes=Inexact(88)),(Col[9]: ScanBytes=Inexact(49)),(Col[10]: ScanBytes=Exact(64))]]physical_plan_with_schema DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/parquet-testing/data/alltypes_plain.parquet]]}, projection=[id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col], limit=10, file_type=parquet, schema=[id:Int32;N, bool_col:Boolean;N, tinyint_col:Int32;N, smallint_col:Int32;N, int_col:Int32;N, bigint_col:Int64;N, float_col:Float32;N, double_col:Float64;N, date_string_col:BinaryView;N, string_col:BinaryView;N, timestamp_col:Timestamp(ns);N]@@ -407,7 +405,6 @@ physical_plan after ProjectionPushdown SAME TEXT AS ABOVEphysical_plan after PushdownSort SAME TEXT AS ABOVEphysical_plan after EnsureCooperative SAME TEXT AS ABOVEphysical_plan after FilterPushdown(Post) SAME TEXT AS ABOVE-physical_plan after SanityCheckPlan SAME TEXT AS ABOVEphysical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/parquet-testing/data/alltypes_plain.parquet]]}, projection=[id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col], limit=10, file_type=parquetphysical_plan_with_stats DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/parquet-testing/data/alltypes_plain.parquet]]}, projection=[id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col], limit=10, file_type=parquet, statistics=[Rows=Exact(8), Bytes=Absent, [(Col[0]: ScanBytes=Exact(32)),(Col[1]: ScanBytes=Inexact(24)),(Col[2]: ScanBytes=Exact(32)),(Col[3]: ScanBytes=Exact(32)),(Col[4]: ScanBytes=Exact(32)),(Col[5]: ScanBytes=Exact(64)),(Col[6]: ScanBytes=Exact(32)),(Col[7]: ScanBytes=Exact(64)),(Col[8]: ScanBytes=Inexact(88)),(Col[9]: ScanBytes=Inexact(49)),(Col[10]: ScanBytes=Exact(64))]]physical_plan_with_schema DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/parquet-testing/data/alltypes_plain.parquet]]}, projection=[id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col], limit=10, file_type=parquet, schema=[id:Int32;N, bool_col:Boolean;N, tinyint_col:Int32;N, smallint_col:Int32;N, int_col:Int32;N, bigint_col:Int64;N, float_col:Float32;N, double_col:Float64;N, date_string_col:BinaryView;N, string_col:BinaryView;N, timestamp_col:Timestamp(ns);N]@@ -661,7 +658,6 @@ physical_plan after ProjectionPushdown SAME TEXT AS ABOVEphysical_plan after PushdownSort SAME TEXT AS ABOVEphysical_plan after EnsureCooperative SAME TEXT AS ABOVEphysical_plan after FilterPushdown(Post) SAME TEXT AS ABOVE-physical_plan after SanityCheckPlan SAME TEXT AS ABOVEphysical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/example.csv]]}, projection=[a, b, c], file_type=csv, has_header=truephysical_plan_with_stats DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/example.csv]]}, projection=[a, b, c], file_type=csv, has_header=true, statistics=[Rows=Absent, Bytes=Absent, [(Col[0]:),(Col[1]:),(Col[2]:)]]physical_plan_with_schema DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/example.csv]]}, projection=[a, b, c], file_type=csv, has_header=true, schema=[a:Int32;N, b:Int32;N, c:Int32;N]diff --git a/docs/source/library-user-guide/upgrading/56.0.0.md b/docs/source/library-user-guide/upgrading/56.0.0.mdindex 45899c2759..e5ee4c2b14 100644--- a/docs/source/library-user-guide/upgrading/56.0.0.md+++ b/docs/source/library-user-guide/upgrading/56.0.0.md@@ -1353,3 +1353,22 @@ without declaring one, use `FileScanConfigBuilder::with_preserve_order(true)`.Code that reads `FileScanConfig::preserve_order` to decide whether files may bereordered should call `FileScanConfig::preserves_file_order()` instead, whichalso accounts for the output ordering.++### `SanityCheckPlan` is deprecated and no longer in the default physical optimizer rules++The physical planner now runs the `SanityCheckPlan` check itself on the final+plan, after every physical optimizer rule. This includes rules added with+`SessionStateBuilder::with_physical_optimizer_rule` and custom rule lists. When+the ordering or distribution requirements do not hold, the error names the rule+that broke them. `EXPLAIN VERBOSE` no longer prints a+`physical_plan after SanityCheckPlan` line.++To upgrade, remove `SanityCheckPlan` from custom rule lists, and drop code that+looks it up in `PhysicalOptimizer::new()`. Code that runs optimizer rules+without the physical planner can call+`datafusion_physical_optimizer::sanity_checker::check_plan_tree_sanity` on the+final plan instead.++A custom rule list that left out `SanityCheckPlan` is now checked too. A plan+whose requirements do not hold, such as a `SortMergeJoinExec` over unsorted+input, now fails to plan instead of possibly returning wrong results.