Apache DataFusion SQL Query Engine
docs: clarify what a canceled partition means in the hash join dynamic filter (#27087)
Adrian Garcia Badaraccoac19b2d4ed09
diff --git a/datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs b/datafusion/physical-plan/src/joins/hash_join/shared_bounds.rsindex f083566acf..0a3ce6dcd3 100644--- a/datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs+++ b/datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs@@ -371,6 +371,17 @@ struct AccumulatorState {enum PartitionStatus {Pending,Reported(PartitionData),+ /// A parent operator dropped this partition's stream before the stream+ /// reported its build side. The build keys of this partition are unknown.+ ///+ /// The filter does not wait for this partition. Without this status, the+ /// other partitions wait forever for a report that never comes.+ ///+ /// The filter may keep or reject the probe rows routed to this partition.+ /// Both choices are correct, because the dropped stream never joins those+ /// rows and never produces output.+ ///+ /// `BuildReportHandle` explains when a parent operator drops a stream.CanceledUnknown,}@@ -505,6 +516,11 @@ impl SharedBuildAccumulator {self.wait_for_completion().await}+ /// Tells the filter that `partition_id` never reports its build side, so the+ /// other partitions stop waiting for it. See+ /// [`PartitionStatus::CanceledUnknown`].+ ///+ /// The call does nothing if the partition has already reported.pub(crate) fn report_canceled_partition(&self, partition_id: usize) {let finalize_input = {let mut guard = self.inner.lock();diff --git a/datafusion/physical-plan/src/joins/hash_join/stream.rs b/datafusion/physical-plan/src/joins/hash_join/stream.rsindex 38da5166e1..c03354ccff 100644--- a/datafusion/physical-plan/src/joins/hash_join/stream.rs+++ b/datafusion/physical-plan/src/joins/hash_join/stream.rs@@ -245,18 +245,43 @@ pub(super) struct EmitUnmatchedBuildRowsState {/// `Scheduled` means the reporting `OnceFut` has been constructed but is lazy:/// the coordinator has not necessarily observed the report. Only `Delivered`/// guarantees the coordinator saw it, so `Drop` must still cancel a `Scheduled`-/// partition — otherwise sibling partitions can wait forever for a report that+/// partition. Otherwise sibling partitions can wait forever for a report that/// never runs.#[derive(Debug, PartialEq, Eq)]enum BuildReportState {NotReported,Scheduled,Delivered,+ /// The stream was dropped before it started to probe. See+ /// [`BuildReportHandle`].Canceled,Finalized,}-/// Owns the stream-side lifecycle for one partition's build-data report.+/// Tracks whether this stream has reported its build side to the shared+/// dynamic filter. If the stream is dropped first, the handle cancels the+/// partition.+///+/// # Canceled partitions+///+/// In `Partitioned` mode, each partition of the join runs as its own stream.+/// A parent operator can drop one of these streams early, because it needs no+/// output from that partition. For example, a parent join whose own build+/// side is empty for a partition drops this join's stream for that partition+/// without polling it (<https://github.com/apache/datafusion/issues/21625>).+///+/// A partition is canceled when its stream is dropped before the stream+/// starts to read its probe side. Cancellation affects that one partition:+///+/// - The other partitions of this join keep running. The query is not+/// canceled.+/// - The probe input keeps running, because the other partitions read from+/// it. A `RepartitionExec` in that input discards the rows that it routes+/// to the canceled partition.+/// - The canceled stream has produced no output and never will.+///+/// The other partitions wait for the build side of every partition before+/// they probe. The handle tells them not to wait for the canceled partition.struct BuildReportHandle {partition: usize,mode: PartitionMode,