Apache DataFusion SQL Query Engine
docs: Explain Grouped TopK aggregation in the `TopKAggregation` optimizer rule (#27211)
Andrew Lamb3d7262d87e3b
diff --git a/datafusion/physical-optimizer/src/topk_aggregation.rs b/datafusion/physical-optimizer/src/topk_aggregation.rsindex 5a0fa7abab..785f743f05 100644--- a/datafusion/physical-optimizer/src/topk_aggregation.rs+++ b/datafusion/physical-optimizer/src/topk_aggregation.rs@@ -18,12 +18,78 @@ use datafusion_physical_plan::projection::ProjectionExec;use datafusion_physical_plan::sorts::sort::SortExec;use itertools::Itertools;-/// An optimizer rule that passes a `limit` hint to aggregations if the whole result is not needed+/// An optimizer rule that transforms Sort + Limit above a grouped Aggregate to use+/// a more efficient Grouped TopK aggregation implementation when possible.+///+/// For example, a query like+///+/// ```sql+/// SELECT trace_id, MAX(timestamp)+/// FROM traces+/// GROUP BY trace_id+/// ORDER BY MAX(timestamp) DESC+/// LIMIT 10+/// ```+///+/// is planned as a `SortExec` with a `fetch` above an `AggregateExec` (left).+/// After this rule, the `AggregateExec` uses a [`GroupedTopKAggregateStream`]+/// that retains only the best `N` groups (right).+///+/// [`GroupedTopKAggregateStream`]: mod@datafusion_physical_plan::aggregates#4-grouped-topk-aggregation+///+/// ```text+/// Before After+///+/// ┌────────────────────────────┐ ┌────────────────────────────┐+/// │ SortExec: TopK(fetch=10) │ │ SortExec: TopK(fetch=10) │+/// │ expr=[max(timestamp) DESC] │ │ expr=[max(timestamp) DESC] │+/// └────────────────────────────┘ └────────────────────────────┘+/// ▲ ▲+/// │ every group │ at most 10 groups+/// │ │+/// ┌────────────────────────────┐ ┌────────────────────────────┐+/// │ AggregateExec │ │ AggregateExec, lim=[10] │+/// │ gby=[trace_id] │ │ gby=[trace_id] │+/// │ aggr=[max(timestamp)] │ ────────▶ │ aggr=[max(timestamp)] │+/// │ │ │ │+/// │ hash table holding │ │ priority map holding only │+/// │ every group │ │ the 10 best groups │+/// └────────────────────────────┘ └────────────────────────────┘+/// ▲ ▲+/// │ │+/// ┌────────────────────────────┐ ┌────────────────────────────┐+/// │ input │ │ input │+/// └────────────────────────────┘ └────────────────────────────┘+/// ```+///+/// Without this rule, the aggregate computes the aggregates for **all** groups+/// in the input, and the sort then discards all but `N` of them.+///+/// `GroupedTopKAggregateStream` keeps only the best `N` groups in a+/// priority map rather than a hash table and the `SortExec` simply+/// enforces the final ordering and row count.+///+/// See the "Grouped TopK aggregation" section of the+/// [`aggregates`](mod@datafusion_physical_plan::aggregates) module documentation+/// for more details.+///+/// This rule is enabled by default and is controlled by the+/// [`datafusion.optimizer.enable_topk_aggregation`](datafusion_common::config::OptimizerOptions::enable_topk_aggregation)+/// setting.+///+/// # Identifying the optimization in `EXPLAIN` output+///+/// When this rule applies, the `AggregateExec` lines in the physical plan show+/// a `lim=[N]` entry, for example:+///+/// ```text+/// AggregateExec: ... aggr=[max(...)], lim=[10] <-- LIM+/// ```#[derive(Debug)]pub struct TopKAggregation {}impl TopKAggregation {- /// Create a new `LimitAggregation`+ /// Create a new `TopKAggregation`pub fn new() -> Self {Self {}}@@ -131,4 +197,4 @@ impl PhysicalOptimizerRule for TopKAggregation {}}-// see `aggregate.slt` for tests+// see `aggregates_topk.slt` for testsdiff --git a/datafusion/physical-plan/src/aggregates/mod.rs b/datafusion/physical-plan/src/aggregates/mod.rsindex 369cf77ae5..39389ad77d 100644--- a/datafusion/physical-plan/src/aggregates/mod.rs+++ b/datafusion/physical-plan/src/aggregates/mod.rs@@ -103,7 +103,7 @@//! input//! ```//!-//! [`GroupedTopKAggregateStream`] keeps a bounded priority map for a single group+//! `GroupedTopKAggregateStream` keeps a bounded priority map for a single group//! key. It supports group-by-only queries and compatible `MIN` or `MAX`//! aggregates. An unordered group-by-only soft limit instead stays on the normal//! hash aggregation path.