Skip to content

Apache DataFusion SQL Query Engine

docs: clarify what a canceled partition means in the hash join dynamic filter (#27087)

Adrian Garcia Badaraccoac19b2d4ed09
diff --git a/datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs b/datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs
index f083566acf..0a3ce6dcd3 100644
--- a/datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs
+++ b/datafusion/physical-plan/src/joins/hash_join/shared_bounds.rs
@@ -371,6 +371,17 @@ struct AccumulatorState {
enum PartitionStatus {
Pending,
Reported(PartitionData),
+ /// A parent operator dropped this partition's stream before the stream
+ /// reported its build side. The build keys of this partition are unknown.
+ ///
+ /// The filter does not wait for this partition. Without this status, the
+ /// other partitions wait forever for a report that never comes.
+ ///
+ /// The filter may keep or reject the probe rows routed to this partition.
+ /// Both choices are correct, because the dropped stream never joins those
+ /// rows and never produces output.
+ ///
+ /// `BuildReportHandle` explains when a parent operator drops a stream.
CanceledUnknown,
}
@@ -505,6 +516,11 @@ impl SharedBuildAccumulator {
self.wait_for_completion().await
}
+ /// Tells the filter that `partition_id` never reports its build side, so the
+ /// other partitions stop waiting for it. See
+ /// [`PartitionStatus::CanceledUnknown`].
+ ///
+ /// The call does nothing if the partition has already reported.
pub(crate) fn report_canceled_partition(&self, partition_id: usize) {
let finalize_input = {
let mut guard = self.inner.lock();
diff --git a/datafusion/physical-plan/src/joins/hash_join/stream.rs b/datafusion/physical-plan/src/joins/hash_join/stream.rs
index 38da5166e1..c03354ccff 100644
--- a/datafusion/physical-plan/src/joins/hash_join/stream.rs
+++ b/datafusion/physical-plan/src/joins/hash_join/stream.rs
@@ -245,18 +245,43 @@ pub(super) struct EmitUnmatchedBuildRowsState {
/// `Scheduled` means the reporting `OnceFut` has been constructed but is lazy:
/// the coordinator has not necessarily observed the report. Only `Delivered`
/// guarantees the coordinator saw it, so `Drop` must still cancel a `Scheduled`
-/// partition — otherwise sibling partitions can wait forever for a report that
+/// partition. Otherwise sibling partitions can wait forever for a report that
/// never runs.
#[derive(Debug, PartialEq, Eq)]
enum BuildReportState {
NotReported,
Scheduled,
Delivered,
+ /// The stream was dropped before it started to probe. See
+ /// [`BuildReportHandle`].
Canceled,
Finalized,
}
-/// Owns the stream-side lifecycle for one partition's build-data report.
+/// Tracks whether this stream has reported its build side to the shared
+/// dynamic filter. If the stream is dropped first, the handle cancels the
+/// partition.
+///
+/// # Canceled partitions
+///
+/// In `Partitioned` mode, each partition of the join runs as its own stream.
+/// A parent operator can drop one of these streams early, because it needs no
+/// output from that partition. For example, a parent join whose own build
+/// side is empty for a partition drops this join's stream for that partition
+/// without polling it (<https://github.com/apache/datafusion/issues/21625>).
+///
+/// A partition is canceled when its stream is dropped before the stream
+/// starts to read its probe side. Cancellation affects that one partition:
+///
+/// - The other partitions of this join keep running. The query is not
+/// canceled.
+/// - The probe input keeps running, because the other partitions read from
+/// it. A `RepartitionExec` in that input discards the rows that it routes
+/// to the canceled partition.
+/// - The canceled stream has produced no output and never will.
+///
+/// The other partitions wait for the build side of every partition before
+/// they probe. The handle tells them not to wait for the canceled partition.
struct BuildReportHandle {
partition: usize,
mode: PartitionMode,