Skip to content

Apache DataFusion SQL Query Engine

3.5 KB
Raw Blame

Upgrade Guides

DataFusion 57.0.0

Note: DataFusion 57.0.0 has not been released yet. The information provided in this section pertains to features and changes that have already been merged to the main branch and are awaiting release in this version.

Substrait consumers use explicit scope context

Add a final &SubstraitConsumerContext argument to custom relation and expression callbacks on SubstraitConsumer. Forward it when calling conversion helpers or consuming child relations and expressions.

Remove implementations of push_outer_schema, pop_outer_schema, get_outer_schema, push_lambda_parameters, pop_lambda_parameters, and lambda_variable, and remove embedded DefaultSubstraitLambdaConsumer values. Use the context's get_outer_schema and lambda_variable methods to resolve outer references and lambda parameters. To enter a scope, derive a child with context.with_outer_schema(...) or context.with_lambda_parameters(...) and pass it to nested conversions. No pop operation is needed; the parent stays unchanged.

from_substrait_plan and from_substrait_plan_with_consumer create the root context. For a new conversion started directly through a helper, pass &SubstraitConsumerContext::default(). Nested conversions must receive the current context.

Custom scoped values can be stored with context.with_extension(Arc::new(value)) and retrieved with context.get_extension::<MyScope>(). Pass the returned child context to nested conversions. Children inherit values; replacing a value leaves parents and siblings unchanged. The values themselves are shared through Arc.

Update logical-plan snapshots if needed: generated lambda parameter names are local to lexical scopes, so sibling lambdas may reuse names.

GroupClusteringMode::Full no longer implies sorted input

AggregateExec can now select GroupClusteringMode::Full when the input's equivalence properties declare the complete GROUP BY tuple as grouped (contiguous within each partition), even if the input is not sorted.

Full describes group contiguity only. Code that previously used group_clustering_mode() == GroupClusteringMode::Full to infer sorted input or output should use properties().output_ordering() on the input or aggregate plan, respectively.

datafusion-spark moved to its own repository

The datafusion-spark crate is now developed and released in apache/datafusion-spark, starting with 57.0.0. The crate name is unchanged.

datafusion-cli no longer has a --spark flag. The Spark SQL dialect is still available without the Spark function library:

SET datafusion.sql_parser.dialect = 'spark';

ParquetFileReaderFactory adds Any as a supertrait

To enable downcasting of dyn ParquetFileReaderFactory to concrete factory types (via is::<T>() / downcast_ref::<T>()), the ParquetFileReaderFactory trait now has Any as a supertrait:

- pub trait ParquetFileReaderFactory: Debug + Send + Sync + 'static
+ pub trait ParquetFileReaderFactory: Any + Debug + Send + Sync

The trait already required 'static, which Any implies, so existing implementations need no changes. ParquetSource::parquet_file_reader_factory returns the trait object to downcast.