Skip to content

Apache DataFusion SQL Query Engine

Check the final physical plan in the planner and name the rule that broke it (#26461)

Jay Zhan0d3aac0bde72
diff --git a/datafusion/core/src/optimizer_rule_reference.md b/datafusion/core/src/optimizer_rule_reference.md
index 13e7f62e72..a1393835c4 100644
--- a/datafusion/core/src/optimizer_rule_reference.md
+++ b/datafusion/core/src/optimizer_rule_reference.md
@@ -69,27 +69,31 @@ Rule order matters. The default pipeline may change between releases.
The same rule name may appear more than once when the default pipeline runs it
in multiple phases.
-| order | rule | phase | summary |
-| ----- | ------------------------------ | ----------------------- | ------------------------------------------------------------------------------------------------------------ |
-| 1 | `OutputRequirements` | add phase | Adds helper nodes so output requirements survive later physical rewrites. |
-| 2 | `aggregate_statistics` | - | Uses exact source statistics to answer some aggregates without scanning data. |
-| 3 | `EliminateConstantJoin` | - | Replaces inner joins against a single row of literals with a filter that keeps the other input's order. |
-| 4 | `join_selection` | - | Chooses join implementation, build side, and partition mode from statistics and stream properties. |
-| 5 | `LimitedDistinctAggregation` | - | Pushes limit hints into grouped distinct-style aggregations when only a small result is needed. |
-| 6 | `FilterPushdown` | pre-optimization phase | Pushes supported physical filters down toward data sources before distribution and sorting are enforced. |
-| 7 | `WindowTopN` | - | Replaces eligible row-number window and filter patterns with per-partition TopK execution. |
-| 8 | `EnsureRequirements` | - | Enforces both distribution and sorting requirements in a single idempotent rule. |
-| 9 | `CombinePartialFinalAggregate` | - | Collapses adjacent partial and final aggregates when the distributed shape makes them redundant. |
-| 10 | `OptimizeAggregateOrder` | - | Updates aggregate expressions to use the best ordering once sort requirements are known. |
-| 11 | `ProjectionPushdown` | early pass | Pushes projections toward inputs before later physical rewrites add more limit and TopK structure. |
-| 12 | `OutputRequirements` | remove phase | Removes the temporary output-requirement helper nodes after requirement-sensitive planning is done. |
-| 13 | `LimitAggregation` | - | Passes a limit hint into eligible aggregations so they can keep fewer accumulator buckets. |
-| 14 | `LimitPushPastWindows` | - | Pushes fetch limits through bounded window operators when doing so keeps the result correct. |
-| 15 | `HashJoinBuffering` | - | Adds buffering on the probe side of hash joins so probing can start before build completion. |
-| 16 | `LimitPushdown` | - | Moves physical limits into child operators or fetch-enabled variants to cut data early. |
-| 17 | `TopKRepartition` | - | Pushes TopK below hash repartition when the partition key is a prefix of the sort key. |
-| 18 | `ProjectionPushdown` | late pass | Runs projection pushdown again after limit and TopK rewrites expose new pruning opportunities. |
-| 19 | `PushdownSort` | - | Pushes sort requirements into data sources that can already return sorted output. |
-| 20 | `EnsureCooperative` | - | Wraps non-cooperative plan parts so long-running tasks yield fairly. |
-| 21 | `FilterPushdown(Post)` | post-optimization phase | Pushes dynamic filters at the end of optimization, after plan references stop moving. |
-| 22 | `SanityCheckPlan` | - | Validates that the final physical plan meets ordering, distribution, and infinite-input safety requirements. |
+After the last rule, including any rule added by the user, the physical planner
+checks that the final plan meets its ordering, distribution, and infinite-input
+safety requirements (`SanityCheckPlan`). If the ordering or distribution
+requirements do not hold, the error names the rule that broke them.
+
+| order | rule | phase | summary |
+| ----- | ------------------------------ | ----------------------- | -------------------------------------------------------------------------------------------------------- |
+| 1 | `OutputRequirements` | add phase | Adds helper nodes so output requirements survive later physical rewrites. |
+| 2 | `aggregate_statistics` | - | Uses exact source statistics to answer some aggregates without scanning data. |
+| 3 | `EliminateConstantJoin` | - | Replaces inner joins against a single row of literals with a filter that keeps the other input's order. |
+| 4 | `join_selection` | - | Chooses join implementation, build side, and partition mode from statistics and stream properties. |
+| 5 | `LimitedDistinctAggregation` | - | Pushes limit hints into grouped distinct-style aggregations when only a small result is needed. |
+| 6 | `FilterPushdown` | pre-optimization phase | Pushes supported physical filters down toward data sources before distribution and sorting are enforced. |
+| 7 | `WindowTopN` | - | Replaces eligible row-number window and filter patterns with per-partition TopK execution. |
+| 8 | `EnsureRequirements` | - | Enforces both distribution and sorting requirements in a single idempotent rule. |
+| 9 | `CombinePartialFinalAggregate` | - | Collapses adjacent partial and final aggregates when the distributed shape makes them redundant. |
+| 10 | `OptimizeAggregateOrder` | - | Updates aggregate expressions to use the best ordering once sort requirements are known. |
+| 11 | `ProjectionPushdown` | early pass | Pushes projections toward inputs before later physical rewrites add more limit and TopK structure. |
+| 12 | `OutputRequirements` | remove phase | Removes the temporary output-requirement helper nodes after requirement-sensitive planning is done. |
+| 13 | `LimitAggregation` | - | Passes a limit hint into eligible aggregations so they can keep fewer accumulator buckets. |
+| 14 | `LimitPushPastWindows` | - | Pushes fetch limits through bounded window operators when doing so keeps the result correct. |
+| 15 | `HashJoinBuffering` | - | Adds buffering on the probe side of hash joins so probing can start before build completion. |
+| 16 | `LimitPushdown` | - | Moves physical limits into child operators or fetch-enabled variants to cut data early. |
+| 17 | `TopKRepartition` | - | Pushes TopK below hash repartition when the partition key is a prefix of the sort key. |
+| 18 | `ProjectionPushdown` | late pass | Runs projection pushdown again after limit and TopK rewrites expose new pruning opportunities. |
+| 19 | `PushdownSort` | - | Pushes sort requirements into data sources that can already return sorted output. |
+| 20 | `EnsureCooperative` | - | Wraps non-cooperative plan parts so long-running tasks yield fairly. |
+| 21 | `FilterPushdown(Post)` | post-optimization phase | Pushes dynamic filters at the end of optimization, after plan references stop moving. |
diff --git a/datafusion/core/src/physical_planner.rs b/datafusion/core/src/physical_planner.rs
index bcabd26922..198e5bb741 100644
--- a/datafusion/core/src/physical_planner.rs
+++ b/datafusion/core/src/physical_planner.rs
@@ -94,6 +94,10 @@ use datafusion_physical_expr::expressions::Literal;
use datafusion_physical_expr::{
LexOrdering, PhysicalSortExpr, create_physical_sort_exprs,
};
+use datafusion_physical_optimizer::ensure_requirements::EnsureRequirements;
+use datafusion_physical_optimizer::sanity_checker::{
+ check_plan_tree_sanity, input_requirements_hold,
+};
use datafusion_physical_plan::empty::EmptyExec;
use datafusion_physical_plan::execution_plan::InvariantLevel;
use datafusion_physical_plan::joins::PiecewiseMergeJoinExec;
@@ -3481,9 +3485,18 @@ impl DefaultPhysicalPlanner {
observer(new_plan.as_ref(), optimizer.as_ref())
}
- // This runs once after all optimizer runs are complete,
- // to verify that the plan is executable.
- InvariantChecker(InvariantLevel::Executable).check(&new_plan)?;
+ // This runs once after all optimizer runs are complete, including
+ // rules appended by the user, to verify that the plan is executable.
+ let optimizer_options = &session_state.config_options().optimizer;
+ if let Err(e) = check_plan_tree_sanity(&new_plan, optimizer_options) {
+ return Err(sanity_check_error(
+ e,
+ &new_plan,
+ plan,
+ optimizers,
+ &optimizer_context,
+ ));
+ }
debug!(
"Optimized physical plan:\n{}\n",
@@ -3693,6 +3706,73 @@ fn tuple_err<T, R>(value: (Result<T>, Result<R>)) -> Result<(T, R)> {
}
}
+/// Adds context to an error from the final [`check_plan_tree_sanity`].
+///
+/// `EnsureRequirements` makes the order and distribution requirements hold,
+/// and every rule after it must keep them. When they do not hold for
+/// `final_plan`, the context names the rule that broke this contract: either
+/// `EnsureRequirements` itself, or the rule after which the requirements
+/// stopped holding. To find it, `optimizers` run again from `input_plan`, so
+/// planning that succeeds keeps no intermediate plans.
+fn sanity_check_error(
+ error: DataFusionError,
+ final_plan: &Arc<dyn ExecutionPlan>,
+ input_plan: Arc<dyn ExecutionPlan>,
+ optimizers: &[Arc<dyn PhysicalOptimizerRule + Send + Sync>],
+ optimizer_context: &dyn PhysicalOptimizerContext,
+) -> DataFusionError {
+ const SANITY_CHECK: &str = "SanityCheckPlan";
+ let plain_error =
+ |error| DataFusionError::Context(SANITY_CHECK.to_string(), Box::new(error));
+ if input_requirements_hold(final_plan) {
+ // The error is about unbounded inputs or another invariant.
+ return plain_error(error);
+ }
+
+ let ensure_requirements = EnsureRequirements::new();
+ let enforced = optimizers
+ .iter()
+ .rposition(|rule| rule.name() == ensure_requirements.name());
+ // Plans before `EnsureRequirements` need not satisfy the requirements, so
+ // only plans from its last run onwards are checked. Without it, every
+ // plan is checked, starting with the input plan.
+ let first_checked = enforced.unwrap_or(0);
+ let mut held = enforced.is_none() && input_requirements_hold(&input_plan);
+ let mut broken_by = None;
+ let mut plan = input_plan;
+ for (idx, rule) in optimizers.iter().enumerate() {
+ plan = match rule.optimize_with_context(plan, optimizer_context) {
+ Ok(plan) => plan,
+ // The rules did not run as they did before, so no rule is named.
+ Err(_) => return plain_error(error),
+ };
+ if idx >= first_checked {
+ let holds = input_requirements_hold(&plan);
+ if held && !holds {
+ broken_by = Some(rule);
+ }
+ held = holds;
+ }
+ }
+ let context = match (broken_by, enforced) {
+ (Some(rule), _) => format!(
+ "{SANITY_CHECK}: order and distribution requirements stopped holding \
+ after physical optimizer rule '{}'",
+ rule.name()
+ ),
+ (None, Some(_)) => format!(
+ "{SANITY_CHECK}: the plan produced by physical optimizer rule '{}' \
+ does not satisfy the order and distribution requirements",
+ ensure_requirements.name()
+ ),
+ (None, None) => format!(
+ "{SANITY_CHECK}: no physical optimizer rule produced a plan that \
+ satisfies the order and distribution requirements"
+ ),
+ };
+ DataFusionError::Context(context, Box::new(error))
+}
+
struct OptimizationInvariantChecker<'a> {
rule: &'a Arc<dyn PhysicalOptimizerRule + Send + Sync>,
}
diff --git a/datafusion/core/tests/memory_limit/mod.rs b/datafusion/core/tests/memory_limit/mod.rs
index a370ca51c5..8d67c27e96 100644
--- a/datafusion/core/tests/memory_limit/mod.rs
+++ b/datafusion/core/tests/memory_limit/mod.rs
@@ -578,7 +578,7 @@ async fn sort_merge_join_no_spill() {
])
.with_memory_limit(1_000)
.with_config(config)
- .with_scenario(Scenario::AccessLogStreaming)
+ .with_scenario(Scenario::AccessLogStreamingByPodTime)
.run()
.await
}
@@ -597,7 +597,7 @@ async fn sort_merge_join_spill() {
.with_memory_limit(1_000)
.with_config(config)
.with_disk_manager_builder(DiskManagerBuilder::default())
- .with_scenario(Scenario::AccessLogStreaming)
+ .with_scenario(Scenario::AccessLogStreamingByPodTime)
.with_expected_success()
.run()
.await
@@ -1461,6 +1461,11 @@ enum Scenario {
/// [`StreamingTable`]
AccessLogStreaming,
+ /// [`Self::AccessLogStreaming`], sorted by `pod, time`, with a
+ /// [`StreamingTable`] that declares that order. A sort merge join on
+ /// `pod, time` then needs no sort.
+ AccessLogStreamingByPodTime,
+
/// N partitions of sorted, dictionary encoded strings.
DictionaryStrings {
partitions: usize,
@@ -1522,6 +1527,23 @@ impl Scenario {
.with_infinite_table(true);
Arc::new(table)
}
+ Self::AccessLogStreamingByPodTime => {
+ let batches = access_log_batches_by_pod_time();
+ let table = StreamingTable::try_new(
+ batches[0].schema(),
+ vec![Arc::new(DummyStreamPartition {
+ schema: batches[0].schema(),
+ batches,
+ })],
+ )
+ .unwrap()
+ .with_infinite_table(true)
+ .with_sort_order(vec![
+ datafusion_expr::col("pod").sort(true, true),
+ datafusion_expr::col("time").sort(true, true),
+ ]);
+ Arc::new(table)
+ }
Self::DictionaryStrings {
partitions,
single_row_batches,
@@ -1570,7 +1592,7 @@ impl Scenario {
// has a memory budget which we'll likely hit first)
Some(vec![Arc::new(JoinSelection::new())])
}
- Self::AccessLogStreaming => {
+ Self::AccessLogStreaming | Self::AccessLogStreamingByPodTime => {
// Disable all physical optimizer rules except the
// JoinSelection rule to avoid sorts or repartition,
// as they also have memory budgets that may be hit
@@ -1646,6 +1668,26 @@ fn access_log_batches() -> Vec<RecordBatch> {
.collect()
}
+/// [`access_log_batches`] sorted by `pod, time`, in batches of 50 rows
+fn access_log_batches_by_pod_time() -> Vec<RecordBatch> {
+ let batches = access_log_batches();
+ let batch = arrow::compute::concat_batches(&batches[0].schema(), &batches).unwrap();
+ let sort_column = |name: &str| arrow::compute::SortColumn {
+ values: Arc::clone(batch.column_by_name(name).unwrap()),
+ options: Some(SortOptions::default()),
+ };
+ let indices = arrow::compute::lexsort_to_indices(
+ &[sort_column("pod"), sort_column("time")],
+ None,
+ )
+ .unwrap();
+ let batch = arrow::compute::take_record_batch(&batch, &indices).unwrap();
+ (0..batch.num_rows())
+ .step_by(50)
+ .map(|offset| batch.slice(offset, 50.min(batch.num_rows() - offset)))
+ .collect()
+}
+
/// If `one_row_batches` is true, then returns new record batches that
/// are one row in size
fn maybe_split_batches(
diff --git a/datafusion/core/tests/physical_optimizer/enforce_distribution.rs b/datafusion/core/tests/physical_optimizer/enforce_distribution.rs
index a6ef2938b4..e728a5512e 100644
--- a/datafusion/core/tests/physical_optimizer/enforce_distribution.rs
+++ b/datafusion/core/tests/physical_optimizer/enforce_distribution.rs
@@ -55,7 +55,7 @@ use datafusion_physical_optimizer::enforce_distribution::*;
use datafusion_physical_optimizer::ensure_requirements::EnsureRequirements;
use datafusion_physical_optimizer::join_selection::JoinSelection;
use datafusion_physical_optimizer::output_requirements::OutputRequirements;
-use datafusion_physical_optimizer::sanity_checker::SanityCheckPlan;
+use datafusion_physical_optimizer::sanity_checker::check_plan_tree_sanity;
use datafusion_physical_plan::aggregates::{
AggregateExec, AggregateMode, PhysicalGroupBy,
};
@@ -1053,7 +1053,7 @@ async fn aligned_hour_groups_avoid_shuffle_through_partial_aggregate() -> Result
.with_query_execution_partitions(2)
.config;
let optimized = EnsureRequirements::new().optimize(aggregate, &config)?;
- SanityCheckPlan::new().optimize(Arc::clone(&optimized), &config)?;
+ check_plan_tree_sanity(&optimized, &config.optimizer)?;
assert_plan!(
optimized,
@@ -1089,7 +1089,7 @@ async fn crossing_day_groups_keep_shuffle_through_partial_aggregate() -> Result<
.with_query_execution_partitions(2)
.config;
let optimized = EnsureRequirements::new().optimize(aggregate, &config)?;
- SanityCheckPlan::new().optimize(Arc::clone(&optimized), &config)?;
+ check_plan_tree_sanity(&optimized, &config.optimizer)?;
assert_plan!(
optimized,
@@ -1126,10 +1126,10 @@ fn range_transform_aggregate_optimization_is_idempotent() -> Result<()> {
let input = timestamp_range_input(hour)?;
let aggregate = partial_final_date_trunc_count(input, precision)?;
let optimized = EnsureRequirements::new().optimize(aggregate, &config)?;
- SanityCheckPlan::new().optimize(Arc::clone(&optimized), &config)?;
+ check_plan_tree_sanity(&optimized, &config.optimizer)?;
let reoptimized =
EnsureRequirements::new().optimize(Arc::clone(&optimized), &config)?;
- SanityCheckPlan::new().optimize(Arc::clone(&reoptimized), &config)?;
+ check_plan_tree_sanity(&reoptimized, &config.optimizer)?;
assert_plan!(optimized, reoptimized);
}
Ok(())
@@ -1155,7 +1155,7 @@ async fn range_transform_locality_does_not_satisfy_join_distribution() -> Result
.with_query_execution_partitions(2)
.config;
let optimized = EnsureRequirements::new().optimize(join, &config)?;
- SanityCheckPlan::new().optimize(Arc::clone(&optimized), &config)?;
+ check_plan_tree_sanity(&optimized, &config.optimizer)?;
assert_plan!(
optimized,
@r"
@@ -1275,7 +1275,7 @@ fn range_inner_hash_join_rehashes_incompatible_range_partitioning() -> Result<()
/// Optimizes `plan` under both `top_down_join_key_reordering` settings with
/// round-robin and file-scan repartitioning disabled. Checks the expected plan,
-/// validates it with [`SanityCheckPlan`], and verifies that a second
+/// validates it with [`check_plan_tree_sanity`], and verifies that a second
/// [`EnsureRequirements`] pass leaves it unchanged.
fn check_range_join_key_alignment(
plan: Arc<dyn ExecutionPlan>,
@@ -1299,7 +1299,7 @@ fn check_range_join_key_alignment_with_partitions(
let optimized = config.try_to_plan(Arc::clone(&plan), &[Run::Distribution])?;
check(&optimized);
- SanityCheckPlan::new().optimize(Arc::clone(&optimized), &config.config)?;
+ check_plan_tree_sanity(&optimized, &config.config.optimizer)?;
let second = config.try_to_plan(Arc::clone(&optimized), &[Run::Distribution])?;
assert_eq!(
@@ -1307,7 +1307,7 @@ fn check_range_join_key_alignment_with_partitions(
displayable(second.as_ref()).indent(true).to_string(),
"plan changed on the second pass; top_down={top_down}"
);
- SanityCheckPlan::new().optimize(second, &config.config)?;
+ check_plan_tree_sanity(&second, &config.config.optimizer)?;
}
Ok(())
}
@@ -4255,15 +4255,11 @@ fn interleave_broken_by_later_rewrite_is_repaired_on_next_pass() -> Result<()> {
Partitioning::UnknownPartitioning(10)
));
// Without a repair, the plan is rejected rather than executed.
- assert!(
- SanityCheckPlan::new()
- .optimize(Arc::clone(&rewritten), &config)
- .is_err()
- );
+ assert!(check_plan_tree_sanity(&rewritten, &config.optimizer).is_err());
// The next distribution pass restores a valid interleave.
let pass2 = EnsureRequirements::new().optimize(rewritten, &config)?;
- SanityCheckPlan::new().optimize(Arc::clone(&pass2), &config)?;
+ check_plan_tree_sanity(&pass2, &config.optimizer)?;
assert_plan!(pass2,
@r"
InterleaveExec
@@ -4333,16 +4329,12 @@ fn issue_21826_join_selection_after_distribution_pass() -> Result<()> {
Partitioning::UnknownPartitioning(10)
));
// Without a repair, the plan is rejected rather than executed.
- assert!(
- SanityCheckPlan::new()
- .optimize(Arc::clone(&reordered), &config)
- .is_err()
- );
+ assert!(check_plan_tree_sanity(&reordered, &config.optimizer).is_err());
// The second distribution pass repairs it. The two joins no longer share
// a partitioning, so the result is a plain union.
let pass2 = EnsureRequirements::new().optimize(reordered, &config)?;
- SanityCheckPlan::new().optimize(Arc::clone(&pass2), &config)?;
+ check_plan_tree_sanity(&pass2, &config.optimizer)?;
assert_plan!(pass2,
@r"
UnionExec
diff --git a/datafusion/core/tests/physical_optimizer/enforce_sorting.rs b/datafusion/core/tests/physical_optimizer/enforce_sorting.rs
index b6c8a6b53a..02f0c9156c 100644
--- a/datafusion/core/tests/physical_optimizer/enforce_sorting.rs
+++ b/datafusion/core/tests/physical_optimizer/enforce_sorting.rs
@@ -44,7 +44,7 @@ use datafusion_physical_plan::{displayable, get_plan_string, ExecutionPlan, Exec
use datafusion::datasource::physical_plan::CsvSource;
use datafusion::datasource::listing::PartitionedFile;
use datafusion_physical_optimizer::enforce_sorting::{PlanWithCorrespondingCoalescePartitions, PlanWithCorrespondingSort, parallelize_sorts, ensure_sorting};
-use datafusion_physical_optimizer::sanity_checker::SanityCheckPlan;
+use datafusion_physical_optimizer::sanity_checker::check_plan_tree_sanity;
use datafusion_physical_optimizer::enforce_sorting::replace_with_order_preserving_variants::{replace_with_order_preserving_variants, OrderPreservationContext};
use datafusion_physical_optimizer::enforce_sorting::sort_pushdown::{SortPushDown, assign_initial_requirements, pushdown_sorts};
use datafusion_physical_optimizer::ensure_requirements::EnsureRequirements;
@@ -297,7 +297,7 @@ async fn test_hash_join_interleaved_projection_preserves_parent_sort() -> Result
config.execution.target_partitions = 10;
let optimized_plan =
EnsureRequirements::new().optimize(Arc::clone(&physical_plan), &config)?;
- let optimized_plan = SanityCheckPlan::new().optimize(optimized_plan, &config)?;
+ check_plan_tree_sanity(&optimized_plan, &config.optimizer)?;
assert_snapshot!(displayable(optimized_plan.as_ref()).indent(true), @r"
SortPreservingMergeExec: [nullable_col@1 ASC]
@@ -378,7 +378,7 @@ fn test_do_not_remove_sort_with_limit_after_projection() -> Result<()> {
] {
let optimized =
EnsureRequirements::new().optimize(Arc::clone(&plan), &config)?;
- let optimized = SanityCheckPlan::new().optimize(optimized, &config)?;
+ check_plan_tree_sanity(&optimized, &config.optimizer)?;
assert_eq!(get_plan_string(&optimized), get_plan_string(&plan));
}
}
@@ -413,7 +413,7 @@ fn test_required_sort_after_projection() -> Result<()> {
input,
);
let optimized = EnsureRequirements::new().optimize(window, &config)?;
- let optimized = SanityCheckPlan::new().optimize(optimized, &config)?;
+ check_plan_tree_sanity(&optimized, &config.optimizer)?;
let twice =
EnsureRequirements::new().optimize(Arc::clone(&optimized), &config)?;
assert_eq!(get_plan_string(&twice), get_plan_string(&optimized));
@@ -471,7 +471,7 @@ async fn test_do_not_remove_sort_below_fetch_after_projection() -> Result<()> {
let mut config = ConfigOptions::new();
config.execution.target_partitions = 1;
let optimized = EnsureRequirements::new().optimize(window, &config)?;
- let optimized = SanityCheckPlan::new().optimize(optimized, &config)?;
+ check_plan_tree_sanity(&optimized, &config.optimizer)?;
let twice = EnsureRequirements::new().optimize(Arc::clone(&optimized), &config)?;
assert_eq!(get_plan_string(&twice), get_plan_string(&optimized));
@@ -3289,8 +3289,7 @@ async fn test_parallelize_sorts_remaps_index_through_reordering_projection() ->
// but kept the key `score@1` (valid only in the projection output), while its
// child schema `[a, b, score]` has `score` at index 2 — so `SanityCheckPlan`
// reports `does not satisfy order requirements: [...]. Child-0 order: []`.
- SanityCheckPlan::new()
- .optimize(Arc::clone(&optimized), &ConfigOptions::default())
+ check_plan_tree_sanity(&optimized, &ConfigOptions::default().optimizer)
.unwrap_or_else(|e| {
panic!(
"Sort enforcement produced a plan that fails SanityCheckPlan \
diff --git a/datafusion/core/tests/physical_optimizer/ensure_requirements.rs b/datafusion/core/tests/physical_optimizer/ensure_requirements.rs
index 751645c687..85c969ca00 100644
--- a/datafusion/core/tests/physical_optimizer/ensure_requirements.rs
+++ b/datafusion/core/tests/physical_optimizer/ensure_requirements.rs
@@ -41,7 +41,7 @@ use datafusion_physical_plan::{
};
use datafusion_physical_optimizer::output_requirements::OutputRequirementExec;
-use datafusion_physical_optimizer::sanity_checker::SanityCheckPlan;
+use datafusion_physical_optimizer::sanity_checker::check_plan_tree_sanity;
use datafusion_common::{JoinType, NullEquality};
use datafusion_physical_expr::Distribution;
@@ -170,7 +170,7 @@ fn optimize_and_sanity_check(
let config = test_config();
let optimized = EnsureRequirements::new().optimize(plan, &config)?;
// SanityCheckPlan must pass
- SanityCheckPlan::new().optimize(Arc::clone(&optimized), &config)?;
+ check_plan_tree_sanity(&optimized, &config.optimizer)?;
Ok(optimized)
}
@@ -192,11 +192,9 @@ fn assert_idempotent(plan: Arc<dyn ExecutionPlan>) {
);
// Both must pass SanityCheckPlan
- SanityCheckPlan::new()
- .optimize(p1, &config)
+ check_plan_tree_sanity(&p1, &config.optimizer)
.expect("SanityCheckPlan failed on first pass");
- SanityCheckPlan::new()
- .optimize(p2, &config)
+ check_plan_tree_sanity(&p2, &config.optimizer)
.expect("SanityCheckPlan failed on second pass");
}
@@ -238,8 +236,7 @@ macro_rules! assert_ensure_requirements_plan {
let p1 = EnsureRequirements::new()
.optimize($plan, &config)
.expect("EnsureRequirements::optimize failed (pass 1)");
- SanityCheckPlan::new()
- .optimize(Arc::clone(&p1), &config)
+ check_plan_tree_sanity(&p1, &config.optimizer)
.expect("SanityCheckPlan failed (pass 1)");
let p1_str = plan_string(&p1);
insta::assert_snapshot!(p1_str, @ $snapshot);
@@ -253,8 +250,7 @@ macro_rules! assert_ensure_requirements_plan {
p1_str, p2_str,
"EnsureRequirements is NOT idempotent!\nPass 1:\n{p1_str}\nPass 2:\n{p2_str}",
);
- SanityCheckPlan::new()
- .optimize(p2, &config)
+ check_plan_tree_sanity(&p2, &config.optimizer)
.expect("SanityCheckPlan failed (pass 2)");
}};
}
@@ -776,11 +772,9 @@ fn test_issue_21973_parallel_sort_survives_multiple_passes() {
// All passes must produce sanity-checkable plans.
for (i, p) in [p1, p2, p3].into_iter().enumerate() {
- SanityCheckPlan::new()
- .optimize(p, &config)
- .unwrap_or_else(|e| {
- panic!("SanityCheckPlan failed on pass {}: {e:?}", i + 1)
- });
+ check_plan_tree_sanity(&p, &config.optimizer).unwrap_or_else(|e| {
+ panic!("SanityCheckPlan failed on pass {}: {e:?}", i + 1)
+ });
}
}
@@ -1188,11 +1182,9 @@ fn test_idempotent_parallelize_sorts() {
"parallelize_sorts NOT idempotent!\nFirst:\n{s1}\nSecond:\n{s2}"
);
- SanityCheckPlan::new()
- .optimize(p1, &config)
+ check_plan_tree_sanity(&p1, &config.optimizer)
.expect("SanityCheckPlan failed on first pass");
- SanityCheckPlan::new()
- .optimize(p2, &config)
+ check_plan_tree_sanity(&p2, &config.optimizer)
.expect("SanityCheckPlan failed on second pass");
}
@@ -1454,7 +1446,7 @@ fn parallelize_sorts_and_sanity_check(
) -> Result<Arc<dyn ExecutionPlan>> {
let ctx = PlanWithCorrespondingCoalescePartitions::new_default(plan);
let rewritten = ctx.transform_up(parallelize_sorts).data()?.plan;
- SanityCheckPlan::new().optimize(Arc::clone(&rewritten), &test_config())?;
+ check_plan_tree_sanity(&rewritten, &test_config().optimizer)?;
Ok(rewritten)
}
diff --git a/datafusion/core/tests/physical_optimizer/sanity_checker.rs b/datafusion/core/tests/physical_optimizer/sanity_checker.rs
index 479b867174..3fac55836c 100644
--- a/datafusion/core/tests/physical_optimizer/sanity_checker.rs
+++ b/datafusion/core/tests/physical_optimizer/sanity_checker.rs
@@ -15,16 +15,20 @@ use crate::physical_optimizer::test_utils::{
use arrow::compute::SortOptions;
use arrow::datatypes::{DataType, Field, Schema, SchemaRef};
use datafusion::datasource::stream::{FileStreamProvider, StreamConfig, StreamTable};
+use datafusion::execution::SessionStateBuilder;
+use datafusion::execution::session_state::SessionState;
use datafusion::prelude::{CsvReadOptions, SessionContext};
use datafusion_common::config::ConfigOptions;
+use datafusion_common::tree_node::{Transformed, TransformedResult, TreeNode};
use datafusion_common::{JoinType, NullEquality, Result, ScalarValue};
use datafusion_physical_expr::expressions::{Literal, col};
use datafusion_physical_expr::{Partitioning, RangePartitioning, SplitPoint};
use datafusion_physical_expr_common::sort_expr::LexOrdering;
use datafusion_physical_optimizer::PhysicalOptimizerRule;
-use datafusion_physical_optimizer::sanity_checker::SanityCheckPlan;
+use datafusion_physical_optimizer::sanity_checker::check_plan_tree_sanity;
use datafusion_physical_plan::joins::{StreamJoinPartitionMode, SymmetricHashJoinExec};
use datafusion_physical_plan::repartition::RepartitionExec;
+use datafusion_physical_plan::sorts::sort::SortExec;
use datafusion_physical_plan::{ExecutionPlan, displayable};
use async_trait::async_trait;
@@ -441,10 +445,9 @@ fn create_test_schema2() -> SchemaRef {
/// Check if sanity checker should accept or reject plans.
fn assert_sanity_check(plan: &Arc<dyn ExecutionPlan>, is_sane: bool) {
- let sanity_checker = SanityCheckPlan::new();
let opts = ConfigOptions::default();
assert_eq!(
- sanity_checker.optimize(plan.clone(), &opts).is_ok(),
+ check_plan_tree_sanity(plan, &opts.optimizer).is_ok(),
is_sane
);
}
@@ -999,3 +1002,88 @@ async fn test_union_with_sorts_and_constants() -> Result<()> {
Ok(())
}
+
+/// Test rule that removes every [`SortExec`], so that the final check fails
+/// and its error can be checked.
+///
+/// [`SortExec`]: datafusion_physical_plan::sorts::sort::SortExec
+#[derive(Debug)]
+struct RemoveSorts;
+
+impl PhysicalOptimizerRule for RemoveSorts {
+ fn optimize(
+ &self,
+ plan: Arc<dyn ExecutionPlan>,
+ _config: &ConfigOptions,
+ ) -> Result<Arc<dyn ExecutionPlan>> {
+ plan.transform_up(|plan| {
+ if plan.is::<SortExec>() {
+ Ok(Transformed::yes(Arc::clone(plan.children()[0])))
+ } else {
+ Ok(Transformed::no(plan))
+ }
+ })
+ .data()
+ }
+
+ fn name(&self) -> &str {
+ "remove_sorts"
+ }
+
+ fn schema_check(&self) -> bool {
+ true
+ }
+}
+
+async fn plan_window_query(state: SessionState) -> Result<Arc<dyn ExecutionPlan>> {
+ let ctx = SessionContext::new_with_state(state);
+ ctx.sql("CREATE TABLE t(a INT) AS VALUES (3), (1), (2)")
+ .await?
+ .collect()
+ .await?;
+ ctx.sql("SELECT a, row_number() OVER (ORDER BY a) AS rn FROM t")
+ .await?
+ .create_physical_plan()
+ .await
+}
+
+/// A rule appended after the default rules is checked, and the error names it.
+#[tokio::test]
+async fn test_final_check_names_appended_rule() -> Result<()> {
+ let state = SessionStateBuilder::new()
+ .with_default_features()
+ .with_physical_optimizer_rule(Arc::new(RemoveSorts))
+ .build();
+ let error = plan_window_query(state).await.unwrap_err().to_string();
+ assert!(
+ error.starts_with(
+ "SanityCheckPlan: order and distribution requirements stopped holding \
+ after physical optimizer rule 'remove_sorts'"
+ ),
+ "{error}"
+ );
+ assert!(
+ error.contains("does not satisfy order requirements"),
+ "{error}"
+ );
+ Ok(())
+}
+
+/// A custom rule list is checked too. Without `EnsureRequirements`, the
+/// window's sort is never added.
+#[tokio::test]
+async fn test_final_check_without_ensure_requirements() -> Result<()> {
+ let state = SessionStateBuilder::new()
+ .with_default_features()
+ .with_physical_optimizer_rules(vec![Arc::new(RemoveSorts)])
+ .build();
+ let error = plan_window_query(state).await.unwrap_err().to_string();
+ assert!(
+ error.starts_with(
+ "SanityCheckPlan: no physical optimizer rule produced a plan that \
+ satisfies the order and distribution requirements"
+ ),
+ "{error}"
+ );
+ Ok(())
+}
diff --git a/datafusion/physical-optimizer/src/optimizer.rs b/datafusion/physical-optimizer/src/optimizer.rs
index 07f64c4a23..7c54b3f02a 100644
--- a/datafusion/physical-optimizer/src/optimizer.rs
+++ b/datafusion/physical-optimizer/src/optimizer.rs
@@ -20,7 +20,6 @@ use crate::limit_pushdown::LimitPushdown;
use crate::limited_distinct_aggregation::LimitedDistinctAggregation;
use crate::output_requirements::OutputRequirements;
use crate::projection_pushdown::ProjectionPushdown;
-use crate::sanity_checker::SanityCheckPlan;
use crate::topk_aggregation::TopKAggregation;
use crate::topk_repartition::TopKRepartition;
use crate::update_aggr_exprs::OptimizeAggregateOrder;
@@ -104,13 +103,15 @@ impl PhysicalOptimizerContext for ConfigOnlyContext<'_> {
/// - [`EnsureRequirements`] inserts the repartitioning and sorting needed to
/// satisfy the distribution and ordering requirements.
/// - [`OutputRequirements`] records output requirements.
-/// - [`SanityCheckPlan`] rejects plans whose requirements are not satisfied.
-/// It should run last.
///
/// Custom pipelines must keep these rules in a compatible relative order. We
/// are working to remove these exceptions, so that all rules can be reordered
/// freely.
///
+/// After the last rule, including rules added by the user, the physical
+/// planner rejects the plan if its requirements are not satisfied (see
+/// [`check_plan_tree_sanity`](crate::sanity_checker::check_plan_tree_sanity)).
+///
/// ## Optimization Coverage Is Not Guaranteed
///
/// The invariants above guarantee *correctness*, not that every optimization is
@@ -280,15 +281,10 @@ impl PhysicalOptimizer {
// Therefore, it should be run at the end of the optimization process since any changes to the plan may break the dynamic filter's references.
// See `FilterPushdownPhase` for more details.
Arc::new(FilterPushdown::new_post_optimization()),
- // The SanityCheckPlan rule checks whether the order and
- // distribution requirements of each node in the plan
- // is satisfied. It will also reject non-runnable query
- // plans that use pipeline-breaking operators on infinite
- // input(s). The rule generates a diagnostic error
- // message for invalid plans. It makes no changes to the
- // given query plan; i.e. it only acts as a final
- // gatekeeping rule.
- Arc::new(SanityCheckPlan::new()),
+ // There is no `SanityCheckPlan` rule at the end of this list. The
+ // physical planner runs the same check on the final plan, after
+ // every rule including the ones appended by the user. See
+ // `crate::sanity_checker`.
];
Self::with_rules(rules)
diff --git a/datafusion/physical-optimizer/src/sanity_checker.rs b/datafusion/physical-optimizer/src/sanity_checker.rs
index 7f120aa1f3..bdfb70b9bb 100644
--- a/datafusion/physical-optimizer/src/sanity_checker.rs
+++ b/datafusion/physical-optimizer/src/sanity_checker.rs
@@ -7,6 +7,12 @@
//! non-runnable query plans that use pipeline-breaking operators on
//! infinite input(s). In addition, it will check if all order and
//! distribution requirements of a plan are satisfied by its children.
+//!
+//! The physical planner runs the same check on the final plan of every query,
+//! after all physical optimizer rules, including rules appended by the user.
+//! The rule is therefore not part of the default rule list and is deprecated.
+//! Code that drives optimizer rules directly can call
+//! [`check_plan_tree_sanity`] on the final plan.
use std::sync::Arc;
@@ -35,9 +41,14 @@ use itertools::izip;
/// are not satisfied by their children.
/// 2. Plans that use pipeline-breaking operators on infinite input(s),
/// it is impossible to execute such queries (they will never generate output nor finish)
+#[deprecated(
+ since = "56.0.0",
+ note = "The physical planner checks the final plan after all rules. To check a plan outside the planner, use `check_plan_tree_sanity`"
+)]
#[derive(Default, Debug)]
pub struct SanityCheckPlan {}
+#[expect(deprecated)]
impl SanityCheckPlan {
#[expect(missing_docs)]
pub fn new() -> Self {
@@ -45,6 +56,7 @@ impl SanityCheckPlan {
}
}
+#[expect(deprecated)]
impl PhysicalOptimizerRule for SanityCheckPlan {
fn optimize(
&self,
@@ -64,6 +76,29 @@ impl PhysicalOptimizerRule for SanityCheckPlan {
}
}
+/// Checks every node of `plan` with [`check_plan_sanity`], children first.
+pub fn check_plan_tree_sanity(
+ plan: &Arc<dyn ExecutionPlan>,
+ optimizer_options: &OptimizerOptions,
+) -> Result<()> {
+ check_plan_sanity_recursive(plan, optimizer_options).map(|_| ())
+}
+
+/// Returns whether every node of `plan` has its order and distribution
+/// requirements satisfied by its children, as checked by
+/// [`check_input_requirements`].
+pub fn input_requirements_hold(plan: &Arc<dyn ExecutionPlan>) -> bool {
+ let mut hold = true;
+ let _ = plan.apply(|node| {
+ if check_input_requirements(node).is_err() {
+ hold = false;
+ return Ok(TreeNodeRecursion::Stop);
+ }
+ Ok(TreeNodeRecursion::Continue)
+ });
+ hold
+}
+
/// Bottom-up (post-order) read-only traversal that checks plan sanity.
#[cfg_attr(feature = "recursive_protection", recursive::recursive)]
fn check_plan_sanity_recursive(
@@ -133,6 +168,15 @@ pub fn check_plan_sanity(
optimizer_options: &OptimizerOptions,
) -> Result<()> {
check_finiteness_requirements(plan.as_ref(), optimizer_options)?;
+ check_input_requirements(plan)?;
+ plan.check_invariants(InvariantLevel::Executable)?;
+
+ Ok(())
+}
+
+/// Checks that the order and distribution requirements of `plan` are
+/// satisfied by its children.
+pub fn check_input_requirements(plan: &Arc<dyn ExecutionPlan>) -> Result<()> {
let input_distributions = plan.input_distribution_requirements();
for ((idx, child), sort_req, dist_req) in izip!(
@@ -177,8 +221,6 @@ pub fn check_plan_sanity(
}
}
- plan.check_invariants(InvariantLevel::Executable)?;
-
Ok(())
}
diff --git a/datafusion/sqllogictest/test_files/explain.slt b/datafusion/sqllogictest/test_files/explain.slt
index 9113388dc4..65547d63fc 100644
--- a/datafusion/sqllogictest/test_files/explain.slt
+++ b/datafusion/sqllogictest/test_files/explain.slt
@@ -242,7 +242,6 @@ physical_plan after ProjectionPushdown SAME TEXT AS ABOVE
physical_plan after PushdownSort SAME TEXT AS ABOVE
physical_plan after EnsureCooperative SAME TEXT AS ABOVE
physical_plan after FilterPushdown(Post) SAME TEXT AS ABOVE
-physical_plan after SanityCheckPlan SAME TEXT AS ABOVE
physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/example.csv]]}, projection=[a, b, c], file_type=csv, has_header=true
physical_plan_with_stats DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/example.csv]]}, projection=[a, b, c], file_type=csv, has_header=true, statistics=[Rows=Absent, Bytes=Absent, [(Col[0]:),(Col[1]:),(Col[2]:)]]
physical_plan_with_schema DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/example.csv]]}, projection=[a, b, c], file_type=csv, has_header=true, schema=[a:Int32;N, b:Int32;N, c:Int32;N]
@@ -360,7 +359,6 @@ physical_plan after ProjectionPushdown SAME TEXT AS ABOVE
physical_plan after PushdownSort SAME TEXT AS ABOVE
physical_plan after EnsureCooperative SAME TEXT AS ABOVE
physical_plan after FilterPushdown(Post) SAME TEXT AS ABOVE
-physical_plan after SanityCheckPlan SAME TEXT AS ABOVE
physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/parquet-testing/data/alltypes_plain.parquet]]}, projection=[id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col], limit=10, file_type=parquet, statistics=[Rows=Exact(8), Bytes=Absent, [(Col[0]: ScanBytes=Exact(32)),(Col[1]: ScanBytes=Inexact(24)),(Col[2]: ScanBytes=Exact(32)),(Col[3]: ScanBytes=Exact(32)),(Col[4]: ScanBytes=Exact(32)),(Col[5]: ScanBytes=Exact(64)),(Col[6]: ScanBytes=Exact(32)),(Col[7]: ScanBytes=Exact(64)),(Col[8]: ScanBytes=Inexact(88)),(Col[9]: ScanBytes=Inexact(49)),(Col[10]: ScanBytes=Exact(64))]]
physical_plan_with_schema DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/parquet-testing/data/alltypes_plain.parquet]]}, projection=[id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col], limit=10, file_type=parquet, schema=[id:Int32;N, bool_col:Boolean;N, tinyint_col:Int32;N, smallint_col:Int32;N, int_col:Int32;N, bigint_col:Int64;N, float_col:Float32;N, double_col:Float64;N, date_string_col:BinaryView;N, string_col:BinaryView;N, timestamp_col:Timestamp(ns);N]
@@ -407,7 +405,6 @@ physical_plan after ProjectionPushdown SAME TEXT AS ABOVE
physical_plan after PushdownSort SAME TEXT AS ABOVE
physical_plan after EnsureCooperative SAME TEXT AS ABOVE
physical_plan after FilterPushdown(Post) SAME TEXT AS ABOVE
-physical_plan after SanityCheckPlan SAME TEXT AS ABOVE
physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/parquet-testing/data/alltypes_plain.parquet]]}, projection=[id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col], limit=10, file_type=parquet
physical_plan_with_stats DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/parquet-testing/data/alltypes_plain.parquet]]}, projection=[id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col], limit=10, file_type=parquet, statistics=[Rows=Exact(8), Bytes=Absent, [(Col[0]: ScanBytes=Exact(32)),(Col[1]: ScanBytes=Inexact(24)),(Col[2]: ScanBytes=Exact(32)),(Col[3]: ScanBytes=Exact(32)),(Col[4]: ScanBytes=Exact(32)),(Col[5]: ScanBytes=Exact(64)),(Col[6]: ScanBytes=Exact(32)),(Col[7]: ScanBytes=Exact(64)),(Col[8]: ScanBytes=Inexact(88)),(Col[9]: ScanBytes=Inexact(49)),(Col[10]: ScanBytes=Exact(64))]]
physical_plan_with_schema DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/parquet-testing/data/alltypes_plain.parquet]]}, projection=[id, bool_col, tinyint_col, smallint_col, int_col, bigint_col, float_col, double_col, date_string_col, string_col, timestamp_col], limit=10, file_type=parquet, schema=[id:Int32;N, bool_col:Boolean;N, tinyint_col:Int32;N, smallint_col:Int32;N, int_col:Int32;N, bigint_col:Int64;N, float_col:Float32;N, double_col:Float64;N, date_string_col:BinaryView;N, string_col:BinaryView;N, timestamp_col:Timestamp(ns);N]
@@ -661,7 +658,6 @@ physical_plan after ProjectionPushdown SAME TEXT AS ABOVE
physical_plan after PushdownSort SAME TEXT AS ABOVE
physical_plan after EnsureCooperative SAME TEXT AS ABOVE
physical_plan after FilterPushdown(Post) SAME TEXT AS ABOVE
-physical_plan after SanityCheckPlan SAME TEXT AS ABOVE
physical_plan DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/example.csv]]}, projection=[a, b, c], file_type=csv, has_header=true
physical_plan_with_stats DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/example.csv]]}, projection=[a, b, c], file_type=csv, has_header=true, statistics=[Rows=Absent, Bytes=Absent, [(Col[0]:),(Col[1]:),(Col[2]:)]]
physical_plan_with_schema DataSourceExec: file_groups={1 group: [[WORKSPACE_ROOT/datafusion/core/tests/data/example.csv]]}, projection=[a, b, c], file_type=csv, has_header=true, schema=[a:Int32;N, b:Int32;N, c:Int32;N]
diff --git a/docs/source/library-user-guide/upgrading/56.0.0.md b/docs/source/library-user-guide/upgrading/56.0.0.md
index 45899c2759..e5ee4c2b14 100644
--- a/docs/source/library-user-guide/upgrading/56.0.0.md
+++ b/docs/source/library-user-guide/upgrading/56.0.0.md
@@ -1353,3 +1353,22 @@ without declaring one, use `FileScanConfigBuilder::with_preserve_order(true)`.
Code that reads `FileScanConfig::preserve_order` to decide whether files may be
reordered should call `FileScanConfig::preserves_file_order()` instead, which
also accounts for the output ordering.
+
+### `SanityCheckPlan` is deprecated and no longer in the default physical optimizer rules
+
+The physical planner now runs the `SanityCheckPlan` check itself on the final
+plan, after every physical optimizer rule. This includes rules added with
+`SessionStateBuilder::with_physical_optimizer_rule` and custom rule lists. When
+the ordering or distribution requirements do not hold, the error names the rule
+that broke them. `EXPLAIN VERBOSE` no longer prints a
+`physical_plan after SanityCheckPlan` line.
+
+To upgrade, remove `SanityCheckPlan` from custom rule lists, and drop code that
+looks it up in `PhysicalOptimizer::new()`. Code that runs optimizer rules
+without the physical planner can call
+`datafusion_physical_optimizer::sanity_checker::check_plan_tree_sanity` on the
+final plan instead.
+
+A custom rule list that left out `SanityCheckPlan` is now checked too. A plan
+whose requirements do not hold, such as a `SortMergeJoinExec` over unsorted
+input, now fails to plan instead of possibly returning wrong results.