Skip to content

Apache DataFusion SQL Query Engine

FFI: plumb with_updated_config for FFI_ScalarUDF (#22797)

Amogh Ramesh8e5c78964af3
diff --git a/.ai/skills/datafusion-ffi/SKILL.md b/.ai/skills/datafusion-ffi/SKILL.md
index c105da6536..ba02d22c09 100644
--- a/.ai/skills/datafusion-ffi/SKILL.md
+++ b/.ai/skills/datafusion-ffi/SKILL.md
@@ -1,6 +1,6 @@
---
name: datafusion-ffi
-description: Patterns and review checklist for the `datafusion-ffi` crate. Use whenever the user adds, edits, or reviews code under `datafusion/ffi/` — new `FFI_X` wrappers, `Foreign<X>` impls, codec changes, or expanding an existing wrapper to cover more of a trait's surface. Also use when reviewing PRs that touch this crate.
+description: Patterns and review checklist for the `datafusion-ffi` crate. Use whenever the user adds, edits, or reviews code under `datafusion/ffi/` — new `FFI_X` wrappers, `ForeignX` implementations, codec changes, or expanding an existing wrapper to cover more of a trait's surface. Also use when reviewing PRs that touch this crate.
---
# DataFusion FFI Skill
@@ -210,9 +210,11 @@ cargo test -p datafusion-ffi --features integration-tests
To add coverage for a new wrapper:
1. **Add a constructor** in `src/tests/<area>.rs` (or a new file there). Return a populated `FFI_X` from a known-good native type.
-2. **Wire it into `ForeignLibraryModule`** in `src/tests/mod.rs`: add a field of type `extern "C" fn(...) -> FFI_X` and populate it in `datafusion_ffi_get_module`. This struct is the cross-library contract — adding a field is itself an ABI change for the test module; integration tests will rebuild the cdylib automatically.
+2. **Wire it into `ForeignLibraryModule`** in `src/tests/mod.rs`: add a field of type `extern "C" fn(...) -> FFI_X` and populate it in `datafusion_ffi_get_module`. This struct is the cross-library contract. Adding a field is itself an ABI change for the test module; integration tests will rebuild the cdylib automatically.
3. **Add the test** in `tests/ffi_<area>.rs` under `#[cfg(feature = "integration-tests")] mod tests { … }`. Call `datafusion_ffi::tests::utils::get_module()` to load the cdylib, invoke your constructor through the returned `ForeignLibraryModule`, convert into `Arc<dyn X>`, and exercise every method.
+When adding a method to an existing wrapper, reuse an existing fixture and constructor when a small trait-method override can cover it. This applies both to omitted default methods and methods newly added to the trait. Add a dedicated test type or `ForeignLibraryModule` field only when the existing fixtures cannot cover the method.
+
What integration tests catch that unit tests cannot:
- **Real ABI layout bugs.** Two builds means the consumer's view of `FFI_X` is reconstructed from declaration, not aliased to the producer's memory. Mismatched alignment, padding, niche optimization, or accidentally non-`#[repr(C)]` types surface here.
@@ -235,7 +237,7 @@ What integration tests catch that unit tests cannot:
- Primitives (`u8`/`u64`/`bool`/`usize`, etc.) and `#[repr(u8)]` FFI enums (`FFI_TableType`, `Volatility`, `InsertOp`, `TableProviderFilterPushDown`).
- A `stabby::string::String` (`SString`) returned by value, with no other args or returns.
-Concrete skippable example: `fn name(&self) -> SString` reading a field already validated by another method. Concrete *non*-skippable examples: anything returning `SVec<T>`, `FFI_Option<T>`, `FFI_Result<T>`, `WrappedSchema`, `WrappedArray`, an `FfiFuture`, an `FFI_*` sub-struct, or any `*mut`/`*const` pointer — those exercise alignment / padding / niche-opt across the ABI boundary and need the two-build coverage. When unsure, write the integration test; the cost is one constructor + ~20 lines.
+Concrete skippable example: `fn name(&self) -> SString` reading a field already validated by another method. Concrete *non*-skippable examples: anything returning `SVec<T>`, `FFI_Option<T>`, `FFI_Result<T>`, `WrappedSchema`, `WrappedArray`, an `FfiFuture`, an `FFI_*` sub-struct, or any `*mut`/`*const` pointer. These exercise alignment / padding / niche-opt across the ABI boundary and need the two-build coverage. When unsure, write the integration test; the cost is one constructor + ~20 lines.
If you skip the integration test for a layout change, you have effectively shipped untested ABI.
@@ -313,6 +315,8 @@ If a method's body is non-trivial, the consumer-side default is non-trivial too.
- **Logical `Expr` / `LogicalPlan`**: serialize via `datafusion-proto` using the embedded `FFI_LogicalExtensionCodec`. Same for physical plans → `FFI_PhysicalExtensionCodec`.
- **Enums** (`Volatility`, `TableType`, `InsertOp`, `TableProviderFilterPushDown`): `#[repr(u8)]`, with `From<Native> for FFI_X` and `From<&FFI_X> for Native`. Always write a round-trip unit test that exercises every variant.
- **Errors**: every `FFI_X` method that can fail returns `FFI_Result<T>`. Use the `sresult!`, `sresult_return!`, `df_result!` macros from `src/util.rs` — do not roll your own.
+- **Infallible trait methods**: if an FFI call can fail but the native trait cannot return the error, log the transport error before returning the trait's fallback (`None`, `false`, or a default). Never discard it with `.ok()`, `.unwrap_or_default()`, or equivalent.
+- **Owned FFI returns**: consume an owned `FFI_X` with `From<FFI_X>` instead of converting through `&FFI_X` and cloning across the boundary. Keep the borrowed conversion's local marker fast path.
## Async, sessions, and task context
diff --git a/datafusion/ffi/src/tests/udf_udaf_udwf.rs b/datafusion/ffi/src/tests/udf_udaf_udwf.rs
index 04d6fb26c1..830c639c74 100644
--- a/datafusion/ffi/src/tests/udf_udaf_udwf.rs
+++ b/datafusion/ffi/src/tests/udf_udaf_udwf.rs
@@ -20,6 +20,7 @@ use std::sync::Arc;
use arrow_schema::DataType;
use datafusion_catalog::TableFunctionImpl;
use datafusion_common::ScalarValue;
+use datafusion_common::config::ConfigOptions;
use datafusion_expr::sort_properties::ExprProperties;
use datafusion_expr::{
AggregateUDF, ColumnarValue, ExpressionPlacement, ScalarFunctionArgs, ScalarUDF,
@@ -104,6 +105,13 @@ impl ScalarUDFImpl for TimeZoneUDF {
let tz = args.config_options.execution.time_zone.clone();
Ok(ColumnarValue::Scalar(ScalarValue::from(tz)))
}
+
+ fn with_updated_config(&self, config: &ConfigOptions) -> Option<ScalarUDF> {
+ config.execution.time_zone.as_ref()?;
+ Some(ScalarUDF::from(Self {
+ signature: self.signature.clone(),
+ }))
+ }
}
pub(crate) extern "C" fn create_timezone_func() -> FFI_ScalarUDF {
diff --git a/datafusion/ffi/src/udf/mod.rs b/datafusion/ffi/src/udf/mod.rs
index 8e96dd9013..fa08cbd042 100644
--- a/datafusion/ffi/src/udf/mod.rs
+++ b/datafusion/ffi/src/udf/mod.rs
@@ -45,7 +45,7 @@ use crate::expr::columnar_value::FFI_ColumnarValue;
use crate::expr::expr_properties::FFI_ExprProperties;
use crate::placement::FFI_ExpressionPlacement;
use crate::util::{
- FFI_Result, rvec_wrapped_to_vec_datatype, vec_datatype_to_rvec_wrapped,
+ FFI_Option, FFI_Result, rvec_wrapped_to_vec_datatype, vec_datatype_to_rvec_wrapped,
};
use crate::volatility::FFI_Volatility;
use crate::{df_result, sresult, sresult_return};
@@ -123,6 +123,13 @@ pub struct FFI_ScalarUDF {
udf: &Self,
inputs: SVec<FFI_ExprProperties>,
) -> FFI_Result<bool>,
+
+ /// FFI equivalent to [`ScalarUDFImpl::with_updated_config`].
+ pub with_updated_config:
+ unsafe extern "C" fn(
+ udf: &Self,
+ config: FFI_ConfigOptions,
+ ) -> FFI_Result<FFI_Option<FFI_ScalarUDF>>,
}
unsafe impl Send for FFI_ScalarUDF {}
@@ -199,6 +206,21 @@ unsafe extern "C" fn preserves_lex_ordering_fn_wrapper(
sresult!(result)
}
+unsafe extern "C" fn with_updated_config_fn_wrapper(
+ udf: &FFI_ScalarUDF,
+ config: FFI_ConfigOptions,
+) -> FFI_Result<FFI_Option<FFI_ScalarUDF>> {
+ let config = sresult_return!(ConfigOptions::try_from(config));
+
+ let updated: Option<FFI_ScalarUDF> = udf
+ .inner()
+ .inner()
+ .with_updated_config(&config)
+ .map(|updated| Arc::new(updated).into());
+
+ FFI_Result::Ok(updated.into())
+}
+
unsafe extern "C" fn invoke_with_args_fn_wrapper(
udf: &FFI_ScalarUDF,
args: SVec<WrappedArray>,
@@ -298,6 +320,7 @@ impl From<Arc<ScalarUDF>> for FFI_ScalarUDF {
private_data: Box::into_raw(private_data) as *mut c_void,
library_marker_id: crate::get_library_marker_id,
preserves_lex_ordering: preserves_lex_ordering_fn_wrapper,
+ with_updated_config: with_updated_config_fn_wrapper,
}
}
}
@@ -325,6 +348,21 @@ pub struct ForeignScalarUDF {
unsafe impl Send for ForeignScalarUDF {}
unsafe impl Sync for ForeignScalarUDF {}
+impl ForeignScalarUDF {
+ fn new(udf: FFI_ScalarUDF) -> Self {
+ let name = udf.name.to_string();
+ let signature = Signature::user_defined((&udf.volatility).into());
+ let aliases = udf.aliases.iter().map(|s| s.to_string()).collect();
+
+ Self {
+ name,
+ aliases,
+ udf,
+ signature,
+ }
+ }
+}
+
impl PartialEq for ForeignScalarUDF {
fn eq(&self, other: &Self) -> bool {
let Self {
@@ -356,22 +394,22 @@ impl Hash for ForeignScalarUDF {
}
}
+impl From<FFI_ScalarUDF> for Arc<dyn ScalarUDFImpl> {
+ fn from(udf: FFI_ScalarUDF) -> Self {
+ if (udf.library_marker_id)() == crate::get_library_marker_id() {
+ Arc::clone(udf.inner().inner())
+ } else {
+ Arc::new(ForeignScalarUDF::new(udf))
+ }
+ }
+}
+
impl From<&FFI_ScalarUDF> for Arc<dyn ScalarUDFImpl> {
fn from(udf: &FFI_ScalarUDF) -> Self {
if (udf.library_marker_id)() == crate::get_library_marker_id() {
Arc::clone(udf.inner().inner())
} else {
- let name = udf.name.to_string();
- let signature = Signature::user_defined((&udf.volatility).into());
-
- let aliases = udf.aliases.iter().map(|s| s.to_string()).collect();
-
- Arc::new(ForeignScalarUDF {
- name,
- udf: udf.clone(),
- aliases,
- signature,
- })
+ Arc::new(ForeignScalarUDF::new(udf.clone()))
}
}
}
@@ -494,6 +532,22 @@ impl ScalarUDFImpl for ForeignScalarUDF {
df_result!(result)
})
}
+
+ fn with_updated_config(&self, config: &ConfigOptions) -> Option<ScalarUDF> {
+ let config: FFI_ConfigOptions = config.into();
+
+ let result = unsafe { (self.udf.with_updated_config)(&self.udf, config) };
+
+ let updated = match df_result!(result) {
+ Ok(updated) => updated.into_option()?,
+ Err(error) => {
+ log::warn!("Unable to update scalar UDF configuration over FFI: {error}");
+ return None;
+ }
+ };
+
+ Some(ScalarUDF::new_from_shared_impl(updated.into()))
+ }
}
#[cfg(test)]
@@ -542,6 +596,12 @@ mod tests {
Ok(inputs.iter().all(|input| input.preserves_lex_ordering))
}
+
+ fn with_updated_config(&self, _config: &ConfigOptions) -> Option<ScalarUDF> {
+ Some(ScalarUDF::from(Self {
+ signature: self.signature.clone(),
+ }))
+ }
}
#[test]
@@ -555,6 +615,11 @@ mod tests {
let foreign_udf: Arc<dyn ScalarUDFImpl> = (&local_udf).into();
assert_eq!(original_udf.name(), foreign_udf.name());
+ assert!(
+ foreign_udf
+ .with_updated_config(&ConfigOptions::default())
+ .is_none()
+ );
Ok(())
}
@@ -629,6 +694,15 @@ mod tests {
);
assert!(foreign_udf.preserves_lex_ordering(&[]).is_err());
+ let updated = foreign_udf
+ .with_updated_config(&ConfigOptions::default())
+ .expect("provider should return an updated UDF");
+ assert_eq!(
+ updated
+ .placement(&[ExpressionPlacement::Column, ExpressionPlacement::Literal]),
+ ExpressionPlacement::MoveTowardsLeafNodes
+ );
+
Ok(())
}
}
diff --git a/datafusion/ffi/tests/ffi_udf.rs b/datafusion/ffi/tests/ffi_udf.rs
index d9e7263ccd..10e0bb5cc1 100644
--- a/datafusion/ffi/tests/ffi_udf.rs
+++ b/datafusion/ffi/tests/ffi_udf.rs
@@ -21,6 +21,7 @@
mod tests {
use arrow::array::{Array, AsArray, record_batch};
use arrow::datatypes::DataType;
+ use datafusion::common::config::ConfigOptions;
use datafusion::error::Result;
use datafusion::logical_expr::{ExpressionPlacement, ScalarUDF, ScalarUDFImpl};
use datafusion::prelude::{SessionContext, col};
@@ -157,4 +158,30 @@ mod tests {
Ok(())
}
+
+ /// Validates that a provider's `with_updated_config` override survives the
+ /// FFI boundary (the trait default returns `None`).
+ #[test]
+ fn test_with_updated_config_on_scalar_udf() -> Result<()> {
+ let module = get_module()?;
+
+ let ffi_udf = (module.create_timezone_udf)();
+ let foreign_udf: Arc<dyn ScalarUDFImpl> = (&ffi_udf).into();
+
+ assert!(
+ foreign_udf
+ .with_updated_config(&ConfigOptions::default())
+ .is_none()
+ );
+
+ let mut options = ConfigOptions::default();
+ options.execution.time_zone = Some("AEST".into());
+
+ let updated = foreign_udf
+ .with_updated_config(&options)
+ .expect("provider should return an updated UDF");
+ assert_eq!(updated.name(), "TimeZoneUDF");
+
+ Ok(())
+ }
}