Skip to content

Apache DataFusion SQL Query Engine

Add higher-order functions changes to upgrade guide (#22107)

gstvge80076bc4e50
diff --git a/docs/source/library-user-guide/upgrading/54.0.0.md b/docs/source/library-user-guide/upgrading/54.0.0.md
index 9643d1e936..0c3bf20a91 100644
--- a/docs/source/library-user-guide/upgrading/54.0.0.md
+++ b/docs/source/library-user-guide/upgrading/54.0.0.md
@@ -751,10 +751,10 @@ SET datafusion.runtime.file_statistics_cache_limit = '0'
Use the file statistics cache provided by the `CacheManager` when initializing a new `ListingTable`:
-````rust,ignore
+```rust,ignore
ListingTable::try_new(config)?
.with_cache(ctx.runtime_env().cache_manager.get_file_statistic_cache())
-[#21075]: https://github.com/apache/datafusion/pull/21075
+```
### `UnionsToFilter` optimizer rule is now disabled by default
@@ -771,7 +771,7 @@ SELECT * FROM t WHERE a = 2
-- After: one scan
SELECT DISTINCT * FROM t WHERE a = 1 OR a = 2
-````
+```
**Who is affected:**
@@ -788,7 +788,158 @@ a combined `OR` (e.g., index-backed sources).
SET datafusion.optimizer.enable_unions_to_filter = true;
```
-### The `RegisterFunction` enum has a new `HigherOrder` variant
+See [PR #21075](https://github.com/apache/datafusion/pull/21075) for more details
+
+### Higher-order functions and lambdas
+
+The changes below are related to the added support for higher-order functions
+and lambdas. For more info, see [#14205], [PR #18921], [PR #21679] and
+[EPIC #21172].
+
+[#14205]: https://github.com/apache/datafusion/issues/14205
+[pr #18921]: https://github.com/apache/datafusion/pull/18921
+[pr #21679]: https://github.com/apache/datafusion/pull/21679
+[epic #21172]: https://github.com/apache/datafusion/issues/21172
+
+#### `FunctionRegistry` exposes two additional methods
+
+`FunctionRegistry` exposes two additional methods, `higher_order_function`
+which returns the registered higher-order function with the given name, if
+any, and `higher_order_function_names` which exposes the set of registered
+user defined higher-order function names.
+
+**Who is affected:**
+
+- Users who implement the `FunctionRegistry` trait
+
+**Migration guide:**
+
+Add `higher_order_function` and `higher_order_function_names` to your implementation.
+
+```diff
+impl FunctionRegistry for FunctionRegistryImpl {
+ fn udfs(&self) -> HashSet<String> {
+ self.scalar_functions.keys().cloned().collect()
+ }
++
++ fn higher_order_function(&self, name: &str) -> Result<Arc<dyn HigherOrderUDF>> {
++ self.higher_order_functions
++ .get(name)
++ .cloned()
++ .ok_or_else(|| plan_datafusion_err!("Higher-order function {name} not found"))
++ }
++
++ fn higher_order_function_names(&self) -> HashSet<String> {
++ self.higher_order_functions.keys().cloned().collect()
++ }
+}
+```
+
+#### `ContextProvider` exposes two additional methods
+
+`ContextProvider` exposes two additional methods, `get_higher_order_meta`
+which returns the registered higher-order function with the given name,
+if any, and `higher_order_function_names` which exposes the registered
+user defined higher-order function names.
+
+**Who is affected:**
+
+- Users who implement the `ContextProvider` trait
+
+**Migration guide:**
+
+Add `get_higher_order_meta` and `higher_order_function_names` to your implementation.
+
+```diff
+impl ContextProvider for ContextProviderImpl {
+ fn udfs(&self) -> HashSet<String> {
+ self.scalar_functions.keys().cloned().collect()
+ }
++
++ fn get_higher_order_meta(&self, name: &str) -> Option<Arc<dyn HigherOrderUDF>> {
++ self.higher_order_functions.get(name).cloned()
++ }
++
++ fn higher_order_function_names(&self) -> Vec<String> {
++ self.higher_order_functions.keys().cloned().collect()
++ }
+}
+```
+
+#### Add `higher_order_functions()` method to `Session`
+
+The `higher_order_functions` method has been added to the `Session` trait,
+which exposes the registered user defined higher-order functions.
+
+**Who is affected:**
+
+- Users who implement the `Session` trait
+
+**Migration guide:**
+
+Add `higher_order_functions` to your implementation.
+
+```diff
+impl Session for MySession {
+ ...
++ fn higher_order_functions(&self) -> &HashMap<String, Arc<dyn HigherOrderUDF>> {
++ &self.higher_order_functions
++ }
+}
+```
+
+#### New argument on `TaskContext::new`
+
+`TaskContext::new` expects a new argument, `higher_order_functions`, which is
+a map of higher-order functions keyed by name.
+
+**Who is affected:**
+
+- Users who call `TaskContext::new`
+
+**Migration guide:**
+
+Provide the new argument to the function. An empty hash map is sufficient.
+
+```diff
++let higher_order_functions = HashMap::new();
+
+TaskContext::new(
+ task_id,
+ session_id,
+ session_config,
+ scalar_functions,
++ higher_order_functions,
+ aggregate_functions,
+ window_functions,
+ runtime,
+)
+```
+
+#### The `Expr` enum has three new variants
+
+- `HigherOrderFunction`: Call a higher-order function with a set of arguments
+- `Lambda`: A Lambda expression with a set of parameters names and a body
+- `LambdaVariable`: A named reference to a lambda parameter
+
+**Who is affected:**
+
+- Users who match on an `Expr` without a default branch `_ => {}`
+
+**Migration guide:**
+
+Add the new branches to the match with the logic applicable to the context.
+
+```diff
+match expr {
+ Expr::Column(column) => ...,
+ ...,
++ Expr::HigherOrderFunction(func) => {},
++ Expr::Lambda(lambda) => {},
++ Expr::LambdaVariable(lambda_var) => {},
+```
+
+#### The `RegisterFunction` enum has a new `HigherOrder` variant
`RegisterFunction` now has a `HigherOrder(Arc<dyn HigherOrderUDF>)` variant
so user-defined higher-order functions can be registered