Skip to content

Apache DataFusion SQL Query Engine

docs: Explain Grouped TopK aggregation in the `TopKAggregation` optimizer rule (#27211)

Andrew Lamb3d7262d87e3b
diff --git a/datafusion/physical-optimizer/src/topk_aggregation.rs b/datafusion/physical-optimizer/src/topk_aggregation.rs
index 5a0fa7abab..785f743f05 100644
--- a/datafusion/physical-optimizer/src/topk_aggregation.rs
+++ b/datafusion/physical-optimizer/src/topk_aggregation.rs
@@ -18,12 +18,78 @@ use datafusion_physical_plan::projection::ProjectionExec;
use datafusion_physical_plan::sorts::sort::SortExec;
use itertools::Itertools;
-/// An optimizer rule that passes a `limit` hint to aggregations if the whole result is not needed
+/// An optimizer rule that transforms Sort + Limit above a grouped Aggregate to use
+/// a more efficient Grouped TopK aggregation implementation when possible.
+///
+/// For example, a query like
+///
+/// ```sql
+/// SELECT trace_id, MAX(timestamp)
+/// FROM traces
+/// GROUP BY trace_id
+/// ORDER BY MAX(timestamp) DESC
+/// LIMIT 10
+/// ```
+///
+/// is planned as a `SortExec` with a `fetch` above an `AggregateExec` (left).
+/// After this rule, the `AggregateExec` uses a [`GroupedTopKAggregateStream`]
+/// that retains only the best `N` groups (right).
+///
+/// [`GroupedTopKAggregateStream`]: mod@datafusion_physical_plan::aggregates#4-grouped-topk-aggregation
+///
+/// ```text
+/// Before After
+///
+/// ┌────────────────────────────┐ ┌────────────────────────────┐
+/// │ SortExec: TopK(fetch=10) │ │ SortExec: TopK(fetch=10) │
+/// │ expr=[max(timestamp) DESC] │ │ expr=[max(timestamp) DESC] │
+/// └────────────────────────────┘ └────────────────────────────┘
+/// ▲ ▲
+/// │ every group │ at most 10 groups
+/// │ │
+/// ┌────────────────────────────┐ ┌────────────────────────────┐
+/// │ AggregateExec │ │ AggregateExec, lim=[10] │
+/// │ gby=[trace_id] │ │ gby=[trace_id] │
+/// │ aggr=[max(timestamp)] │ ────────▶ │ aggr=[max(timestamp)] │
+/// │ │ │ │
+/// │ hash table holding │ │ priority map holding only │
+/// │ every group │ │ the 10 best groups │
+/// └────────────────────────────┘ └────────────────────────────┘
+/// ▲ ▲
+/// │ │
+/// ┌────────────────────────────┐ ┌────────────────────────────┐
+/// │ input │ │ input │
+/// └────────────────────────────┘ └────────────────────────────┘
+/// ```
+///
+/// Without this rule, the aggregate computes the aggregates for **all** groups
+/// in the input, and the sort then discards all but `N` of them.
+///
+/// `GroupedTopKAggregateStream` keeps only the best `N` groups in a
+/// priority map rather than a hash table and the `SortExec` simply
+/// enforces the final ordering and row count.
+///
+/// See the "Grouped TopK aggregation" section of the
+/// [`aggregates`](mod@datafusion_physical_plan::aggregates) module documentation
+/// for more details.
+///
+/// This rule is enabled by default and is controlled by the
+/// [`datafusion.optimizer.enable_topk_aggregation`](datafusion_common::config::OptimizerOptions::enable_topk_aggregation)
+/// setting.
+///
+/// # Identifying the optimization in `EXPLAIN` output
+///
+/// When this rule applies, the `AggregateExec` lines in the physical plan show
+/// a `lim=[N]` entry, for example:
+///
+/// ```text
+/// AggregateExec: ... aggr=[max(...)], lim=[10] <-- LIM
+/// ```
#[derive(Debug)]
pub struct TopKAggregation {}
impl TopKAggregation {
- /// Create a new `LimitAggregation`
+ /// Create a new `TopKAggregation`
pub fn new() -> Self {
Self {}
}
@@ -131,4 +197,4 @@ impl PhysicalOptimizerRule for TopKAggregation {
}
}
-// see `aggregate.slt` for tests
+// see `aggregates_topk.slt` for tests
diff --git a/datafusion/physical-plan/src/aggregates/mod.rs b/datafusion/physical-plan/src/aggregates/mod.rs
index 369cf77ae5..39389ad77d 100644
--- a/datafusion/physical-plan/src/aggregates/mod.rs
+++ b/datafusion/physical-plan/src/aggregates/mod.rs
@@ -103,7 +103,7 @@
//! input
//! ```
//!
-//! [`GroupedTopKAggregateStream`] keeps a bounded priority map for a single group
+//! `GroupedTopKAggregateStream` keeps a bounded priority map for a single group
//! key. It supports group-by-only queries and compatible `MIN` or `MAX`
//! aggregates. An unordered group-by-only soft limit instead stays on the normal
//! hash aggregation path.