Apache DataFusion SQL Query Engine
docs: Add point queries blog to Good Reads and Tuning Guide (#26751)
Andrew Lamb80dda10eef47
diff --git a/dev/update_config_docs.sh b/dev/update_config_docs.shindex 5bba39625f..f08ad673ee 100755--- a/dev/update_config_docs.sh+++ b/dev/update_config_docs.sh@@ -172,12 +172,14 @@ cat <<'EOF' >> "$TARGET_FILE"# Tuning Guide-## Short Queries+## Short Queries and Point LookupsBy default DataFusion will attempt to maximize parallelism and use all cores --For example, if you have 32 cores, each plan will split the data into 32-partitions. However, if your data is small, the overhead of splitting the data-to enable parallelization can dominate the actual computation.+partitions. However, if your data is small, or the query only reads a few rows+(for example, a point lookup such as `SELECT * FROM t WHERE id = 42`), the+overhead of planning, opening files, and splitting the data to enable+parallelization can dominate the actual computation.You can find out how many cores are being used via the [`EXPLAIN`] command and lookat the number of partitions in the plan.@@ -190,7 +192,7 @@ does not apply to user defined data sources and only works when DataFusion has aIf you know your data is small, you can set the `datafusion.execution.target_partitions`option to a smaller number to reduce the overhead of repartitioning. For very small datasets (e.g. less-than 1MB), we recommend setting `target_partitions` to 1 to avoid repartitioning altogether.+than 1MB) or point lookups, we recommend setting `target_partitions` to 1 to avoid repartitioning altogether.```sqlSET datafusion.execution.target_partitions = '1';@@ -198,6 +200,41 @@ SET datafusion.execution.target_partitions = '1';[`ListingTable`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html+For point lookups, the following techniques also help:++- Sort the data by the lookup key so the data that is fetched is physically nearby (clustered).+ This makes techniques such as Parquet min/max metadata pruning more effective.+- Write Parquet files with smaller row groups and data pages than the default+ so that less data needs to be read and decoded for each lookup.+- Enable `datafusion.execution.parquet.pushdown_filters` so filters are+ evaluated while decoding Parquet.+- Set the Parquet metadata cache size via+ `datafusion.runtime.metadata_cache_limit` so it is large enough to hold all the metadata for the+ files you query repeatedly.+- Reuse physical plans to avoid re-planning each query.++Here is an example of how to write Parquet files with smaller row groups and+data pages:++```sql+COPY (SELECT * FROM t) TO 't.parquet'+OPTIONS (+ MAX_ROW_GROUP_SIZE 131072,+ DATA_PAGE_ROW_COUNT_LIMIT 2048+);+```++Here is how to enable pushdown filters:++```sql+SET datafusion.execution.parquet.pushdown_filters = true;+```++See the [Best Practices for Single Row Lookups (Point Queries)] blog post for+more details.++[Best Practices for Single Row Lookups (Point Queries)]: https://datafusion.apache.org/blog/2026/10/24/point-queries/+## Memory-limited QueriesWhen executing a memory-consuming query under a tight memory limit, DataFusiondiff --git a/docs/source/user-guide/concepts-readings-events.md b/docs/source/user-guide/concepts-readings-events.mdindex 2024dc6474..5d7a066b3e 100644--- a/docs/source/user-guide/concepts-readings-events.md+++ b/docs/source/user-guide/concepts-readings-events.md@@ -37,6 +37,8 @@This is a list of DataFusion related blog posts, articles, and other resources. Please open a PR to add any new resources you create or find+- **2026-10-24** [Blog: Best Practices for Single Row Lookups (Point Queries)](https://datafusion.apache.org/blog/2026/10/24/point-queries/)+- **2026-10-01** [Blog: How we extended Apache DataFusion to execute one query across many machines](https://www.datadoghq.com/blog/engineering/distributed-datafusion/)- **2026-07-29** [Blog: Bringing DuckLake to DataFusion](https://ducklake.select/2026/07/29/bringing-ducklake-to-datafusion/)diff --git a/docs/source/user-guide/configs.md b/docs/source/user-guide/configs.mdindex 5029e0ac80..fe163e4add 100644--- a/docs/source/user-guide/configs.md+++ b/docs/source/user-guide/configs.md@@ -259,12 +259,14 @@ The following runtime configuration settings are available:# Tuning Guide-## Short Queries+## Short Queries and Point LookupsBy default DataFusion will attempt to maximize parallelism and use all cores --For example, if you have 32 cores, each plan will split the data into 32-partitions. However, if your data is small, the overhead of splitting the data-to enable parallelization can dominate the actual computation.+partitions. However, if your data is small, or the query only reads a few rows+(for example, a point lookup such as `SELECT * FROM t WHERE id = 42`), the+overhead of planning, opening files, and splitting the data to enable+parallelization can dominate the actual computation.You can find out how many cores are being used via the [`EXPLAIN`] command and lookat the number of partitions in the plan.@@ -277,7 +279,7 @@ does not apply to user defined data sources and only works when DataFusion has aIf you know your data is small, you can set the `datafusion.execution.target_partitions`option to a smaller number to reduce the overhead of repartitioning. For very small datasets (e.g. less-than 1MB), we recommend setting `target_partitions` to 1 to avoid repartitioning altogether.+than 1MB) or point lookups, we recommend setting `target_partitions` to 1 to avoid repartitioning altogether.```sqlSET datafusion.execution.target_partitions = '1';@@ -285,6 +287,41 @@ SET datafusion.execution.target_partitions = '1';[`listingtable`]: https://docs.rs/datafusion/latest/datafusion/datasource/listing/struct.ListingTable.html+For point lookups, the following techniques also help:++- Sort the data by the lookup key so the data that is fetched is physically nearby (clustered).+ This makes techniques such as Parquet min/max metadata pruning more effective.+- Write Parquet files with smaller row groups and data pages than the default+ so that less data needs to be read and decoded for each lookup.+- Enable `datafusion.execution.parquet.pushdown_filters` so filters are+ evaluated while decoding Parquet.+- Set the Parquet metadata cache size via+ `datafusion.runtime.metadata_cache_limit` so it is large enough to hold all the metadata for the+ files you query repeatedly.+- Reuse physical plans to avoid re-planning each query.++Here is an example of how to write Parquet files with smaller row groups and+data pages:++```sql+COPY (SELECT * FROM t) TO 't.parquet'+OPTIONS (+ MAX_ROW_GROUP_SIZE 131072,+ DATA_PAGE_ROW_COUNT_LIMIT 2048+);+```++Here is how to enable pushdown filters:++```sql+SET datafusion.execution.parquet.pushdown_filters = true;+```++See the [Best Practices for Single Row Lookups (Point Queries)] blog post for+more details.++[best practices for single row lookups (point queries)]: https://datafusion.apache.org/blog/2026/10/24/point-queries/+## Memory-limited QueriesWhen executing a memory-consuming query under a tight memory limit, DataFusion