Apache DataFusion SQL Query Engine
Add higher-order functions changes to upgrade guide (#22107)
gstvge80076bc4e50
diff --git a/docs/source/library-user-guide/upgrading/54.0.0.md b/docs/source/library-user-guide/upgrading/54.0.0.mdindex 9643d1e936..0c3bf20a91 100644--- a/docs/source/library-user-guide/upgrading/54.0.0.md+++ b/docs/source/library-user-guide/upgrading/54.0.0.md@@ -751,10 +751,10 @@ SET datafusion.runtime.file_statistics_cache_limit = '0'Use the file statistics cache provided by the `CacheManager` when initializing a new `ListingTable`:-````rust,ignore+```rust,ignoreListingTable::try_new(config)?.with_cache(ctx.runtime_env().cache_manager.get_file_statistic_cache())-[#21075]: https://github.com/apache/datafusion/pull/21075+```### `UnionsToFilter` optimizer rule is now disabled by default@@ -771,7 +771,7 @@ SELECT * FROM t WHERE a = 2-- After: one scanSELECT DISTINCT * FROM t WHERE a = 1 OR a = 2-````+```**Who is affected:**@@ -788,7 +788,158 @@ a combined `OR` (e.g., index-backed sources).SET datafusion.optimizer.enable_unions_to_filter = true;```-### The `RegisterFunction` enum has a new `HigherOrder` variant+See [PR #21075](https://github.com/apache/datafusion/pull/21075) for more details++### Higher-order functions and lambdas++The changes below are related to the added support for higher-order functions+and lambdas. For more info, see [#14205], [PR #18921], [PR #21679] and+[EPIC #21172].++[#14205]: https://github.com/apache/datafusion/issues/14205+[pr #18921]: https://github.com/apache/datafusion/pull/18921+[pr #21679]: https://github.com/apache/datafusion/pull/21679+[epic #21172]: https://github.com/apache/datafusion/issues/21172++#### `FunctionRegistry` exposes two additional methods++`FunctionRegistry` exposes two additional methods, `higher_order_function`+which returns the registered higher-order function with the given name, if+any, and `higher_order_function_names` which exposes the set of registered+user defined higher-order function names.++**Who is affected:**++- Users who implement the `FunctionRegistry` trait++**Migration guide:**++Add `higher_order_function` and `higher_order_function_names` to your implementation.++```diff+impl FunctionRegistry for FunctionRegistryImpl {+ fn udfs(&self) -> HashSet<String> {+ self.scalar_functions.keys().cloned().collect()+ }++++ fn higher_order_function(&self, name: &str) -> Result<Arc<dyn HigherOrderUDF>> {++ self.higher_order_functions++ .get(name)++ .cloned()++ .ok_or_else(|| plan_datafusion_err!("Higher-order function {name} not found"))++ }++++ fn higher_order_function_names(&self) -> HashSet<String> {++ self.higher_order_functions.keys().cloned().collect()++ }+}+```++#### `ContextProvider` exposes two additional methods++`ContextProvider` exposes two additional methods, `get_higher_order_meta`+which returns the registered higher-order function with the given name,+if any, and `higher_order_function_names` which exposes the registered+user defined higher-order function names.++**Who is affected:**++- Users who implement the `ContextProvider` trait++**Migration guide:**++Add `get_higher_order_meta` and `higher_order_function_names` to your implementation.++```diff+impl ContextProvider for ContextProviderImpl {+ fn udfs(&self) -> HashSet<String> {+ self.scalar_functions.keys().cloned().collect()+ }++++ fn get_higher_order_meta(&self, name: &str) -> Option<Arc<dyn HigherOrderUDF>> {++ self.higher_order_functions.get(name).cloned()++ }++++ fn higher_order_function_names(&self) -> Vec<String> {++ self.higher_order_functions.keys().cloned().collect()++ }+}+```++#### Add `higher_order_functions()` method to `Session`++The `higher_order_functions` method has been added to the `Session` trait,+which exposes the registered user defined higher-order functions.++**Who is affected:**++- Users who implement the `Session` trait++**Migration guide:**++Add `higher_order_functions` to your implementation.++```diff+impl Session for MySession {+ ...++ fn higher_order_functions(&self) -> &HashMap<String, Arc<dyn HigherOrderUDF>> {++ &self.higher_order_functions++ }+}+```++#### New argument on `TaskContext::new`++`TaskContext::new` expects a new argument, `higher_order_functions`, which is+a map of higher-order functions keyed by name.++**Who is affected:**++- Users who call `TaskContext::new`++**Migration guide:**++Provide the new argument to the function. An empty hash map is sufficient.++```diff++let higher_order_functions = HashMap::new();++TaskContext::new(+ task_id,+ session_id,+ session_config,+ scalar_functions,++ higher_order_functions,+ aggregate_functions,+ window_functions,+ runtime,+)+```++#### The `Expr` enum has three new variants++- `HigherOrderFunction`: Call a higher-order function with a set of arguments+- `Lambda`: A Lambda expression with a set of parameters names and a body+- `LambdaVariable`: A named reference to a lambda parameter++**Who is affected:**++- Users who match on an `Expr` without a default branch `_ => {}`++**Migration guide:**++Add the new branches to the match with the logic applicable to the context.++```diff+match expr {+ Expr::Column(column) => ...,+ ...,++ Expr::HigherOrderFunction(func) => {},++ Expr::Lambda(lambda) => {},++ Expr::LambdaVariable(lambda_var) => {},+```++#### The `RegisterFunction` enum has a new `HigherOrder` variant`RegisterFunction` now has a `HigherOrder(Arc<dyn HigherOrderUDF>)` variantso user-defined higher-order functions can be registered