Apache DataFusion SQL Query Engine
Apache DataFusion 54.1.0 Changelog
This release consists of 19 commits from 9 contributors. See credits at the end of this changelog for more information.
See the upgrade guide for information on how to upgrade from previous versions.
Documentation updates:
- [branch-54] Add datafusion.execution.enable_file_stream_work_stealing config #23296 (andygrove)
Other:
- [branch-54] fix: preserve null_aware on logical JoinNode proto round-trip (backport #22104) #22785 (mithuncy)
- [branch-54]: backport #22811 (bugfix: changed return type of spark's width_bucket to i64) #23087 (mbutrovich)
- [branch-54] backport #22857 (Skip loading Parquet page index when row-group statistics already prove it cannot prune) #23088 (mbutrovich)
- [branch-54] backport #23192
array_compacthandle edge case with NULLs #23196 (comphead) - [branch-54] fix: Avoid panicing when stats are not available for a file group split (backport #23277) #23340 (mkleen)
- [branch-54] fix:
approx_distinctover-counts for utf8view (backport #22815, adapted) #23576 (mbutrovich) - [branch-54] fix: isolate anonymous file statistics cache (backport #22950, adapted) #23573 (mbutrovich)
- [branch-54] fix:
= ANY (SELECT ...)/<> ALL (SELECT ...)schema error (backport #22915) #23575 (mbutrovich) - [branch-54] fix: NestedLoopJoinExec emits spurious unmatched-left rows with multiple probe partitions (backport #22791) #23577 (mbutrovich)
- [branch-54] fix: regex simplification of anchored patterns produces wrong results (backport #22727) #23578 (mbutrovich)
- [branch-54] fix: Correctly compute nullability in recursive CTE schemas (backport #22552) #23579 (mbutrovich)
- [branch-54] fix: handle
IS TRUEcorrectly inEliminateOuterJoin(backport #22444) #23580 (mbutrovich) - [branch-54] fix: preserve no-filter SMJ matches across pending outer batches (backport #23049) #23574 (mbutrovich)
- [branch-54] perf: avoid intermediate slice allocation in Spark slice function (backport #23481) #23582 (mbutrovich)
- [branch-54] fix: don't duplicate volatile expressions when pushing projection into file scan (backport #23395, adapted) #23585 (fordN)
- [branch-54] chore: fix cargo audit #23607 (alamb)
- [branch-54] Resolve lost wakeup in SpillPoolReader with multiple conc… #23654 (pepijnve)
- [branch-54] Handle nulls in type coercion of higher-order UDFs, map_extract, spark array_repeat (backport #23071) #23629 (gstvg)
Credits
Thank you to everyone who contributed to this release. Here is a breakdown of commits (PRs merged) per contributor.
11 Matt Butrovich
1 Andrew Lamb
1 Andy Grove
1 Ford
1 Michael Kleen
1 Mithun Chicklore Yogendra
1 Oleks V
1 Pepijn Van Eeckhoudt
1 gstvg
Thank you also to everyone who contributed in other ways such as filing issues, reviewing PRs, and providing feedback on this release.