Skip to content

Apache DataFusion SQL Query Engine

docs: add HashJoinExec metrics to user guide (#24055)

Yuwei Zhao9a83691aadaa
diff --git a/datafusion/physical-plan/src/joins/utils.rs b/datafusion/physical-plan/src/joins/utils.rs
index 20467a7ec5..ecf056560e 100644
--- a/datafusion/physical-plan/src/joins/utils.rs
+++ b/datafusion/physical-plan/src/joins/utils.rs
@@ -1754,6 +1754,7 @@ fn append_probe_indices_in_order(
/// Metrics for build & probe joins
#[derive(Clone, Debug)]
pub(crate) struct BuildProbeJoinMetrics {
+ // Keep these metric descriptions in sync with docs/source/user-guide/metrics.md.
pub(crate) baseline: BaselineMetrics,
/// Total time for collecting build-side of join
pub(crate) build_time: metrics::Time,
@@ -1763,15 +1764,17 @@ pub(crate) struct BuildProbeJoinMetrics {
pub(crate) build_input_rows: metrics::Count,
/// Memory used by build-side in bytes
pub(crate) build_mem_used: metrics::Gauge,
- /// Total time for joining probe-side batches to the build-side batches
+ /// Total time for join processing after build-side collection
pub(crate) join_time: metrics::Time,
/// Number of batches consumed by probe-side of this operator
pub(crate) input_batches: metrics::Count,
/// Number of rows consumed by probe-side this operator
pub(crate) input_rows: metrics::Count,
- /// Fraction of probe rows that found more than one match
+ /// Fraction of probe rows with at least one build-side join-key match before
+ /// applying any join filter
pub(crate) probe_hit_rate: metrics::RatioMetrics,
- /// Average number of build matches per matched probe row
+ /// Average number of build-side join-key matches per matched probe row before
+ /// applying any join filter
pub(crate) avg_fanout: metrics::RatioMetrics,
}
diff --git a/docs/source/user-guide/metrics.md b/docs/source/user-guide/metrics.md
index 7e0363f4ce..306094c438 100644
--- a/docs/source/user-guide/metrics.md
+++ b/docs/source/user-guide/metrics.md
@@ -42,6 +42,25 @@ DataFusion operators expose runtime metrics so you can understand where time is
| ----------- | ----------------------------------------------------------------- |
| selectivity | Selectivity of the filter, calculated as output_rows / input_rows |
+### HashJoinExec
+
+`HashJoinExec` also exposes the common `BaselineMetrics`. Its
+`elapsed_compute` metric is the sum of the build-side collection time and the
+subsequent join processing time.
+
+| Metric | Description |
+| ----------------------- | --------------------------------------------------------------------------------------------------------- |
+| build_time | Total time spent collecting and building the build side of the join. |
+| build_input_batches | Number of input batches consumed from the build side. |
+| build_input_rows | Number of input rows consumed from the build side. |
+| build_mem_used | Peak tracked memory used by the build side, in bytes. |
+| join_time | Total time spent processing the join after build-side collection. |
+| input_batches | Number of input batches consumed from the probe side. |
+| input_rows | Number of input rows consumed from the probe side. |
+| probe_hit_rate | Fraction of probe-side rows with a build-side join-key match before applying any join filter. |
+| avg_fanout | Average number of build-side join-key matches per matched probe-side row before applying any join filter. |
+| array_map_created_count | Number of times `HashJoinExec` created an `ArrayMap` for perfect hash join lookup execution. |
+
## TODO
Add metrics for the remaining operators