Apache DataFusion SQL Query Engine
fix: support RANGE window frames over TIME keys (#24515)
Fredrik Fornwallf10331fee376
diff --git a/datafusion/optimizer/src/analyzer/type_coercion.rs b/datafusion/optimizer/src/analyzer/type_coercion.rsindex 2ca679bc51..0f82f0b0df 100644--- a/datafusion/optimizer/src/analyzer/type_coercion.rs+++ b/datafusion/optimizer/src/analyzer/type_coercion.rs@@ -1079,6 +1079,8 @@ fn extract_window_frame_target_type(col_type: &DataType) -> Result<DataType> {| DataType::LargeList(_)| DataType::FixedSizeList(_, _)| DataType::Boolean+ | DataType::Time32(_)+ | DataType::Time64(_)){Ok(col_type.clone())@@ -1109,9 +1111,8 @@ fn coerce_window_frame(let target_type = extract_window_frame_target_type(&col_type)?;// A finite offset bound (e.g. `5 PRECEDING`) is computed as// `current_value ± offset`, so it is only meaningful for target- // types that support arithmetic. Strings, binaries, booleans- // and lists are orderable -- which is all a free range frame- // needs -- but have no such arithmetic.+ // types that support arithmetic. Other orderable target types can+ // still use free range frames, whose bounds require comparison only.let supports_offset_arithmetic =target_type.is_numeric() || is_interval(&target_type);if !supports_offset_arithmetic && !window_frame.free_range() {diff --git a/datafusion/sqllogictest/test_files/window.slt b/datafusion/sqllogictest/test_files/window.sltindex 6238361af0..230fb02055 100644--- a/datafusion/sqllogictest/test_files/window.slt+++ b/datafusion/sqllogictest/test_files/window.slt@@ -6948,6 +6948,76 @@ ORDER BY x62 262 2+# RANGE window frame over a time ORDER BY key. Time is orderable, so free+# frames can locate peer groups by comparison even though finite offsets do not+# have suitable frame-bound arithmetic.+query DI+SELECT x, COUNT(*) OVER (ORDER BY x)+FROM (VALUES (arrow_cast('01:00:00', 'Time64(Microsecond)')),+ (arrow_cast('02:00:00', 'Time64(Microsecond)')),+ (arrow_cast('02:00:00', 'Time64(Microsecond)'))) t(x)+ORDER BY x+----+01:00:00 1+02:00:00 3+02:00:00 3++# A non-aggregate window function and descending order over nanosecond time.+query DI+SELECT x, RANK() OVER (ORDER BY x DESC)+FROM (VALUES (arrow_cast('01:00:00', 'Time64(Nanosecond)')),+ (arrow_cast('02:00:00', 'Time64(Nanosecond)')),+ (arrow_cast('02:00:00', 'Time64(Nanosecond)'))) t(x)+ORDER BY x+----+01:00:00 3+02:00:00 1+02:00:00 1++# Explicit free bounds use the same comparison-only support.+query DI+SELECT x, COUNT(*) OVER (ORDER BY x RANGE BETWEEN CURRENT ROW AND UNBOUNDED FOLLOWING)+FROM (VALUES (arrow_cast('01:00:00', 'Time32(Second)')),+ (arrow_cast('02:00:00', 'Time32(Second)')),+ (arrow_cast('02:00:00', 'Time32(Second)'))) t(x)+ORDER BY x+----+01:00:00 3+02:00:00 2+02:00:00 2++query DI+SELECT x, COUNT(*) OVER (ORDER BY x)+FROM (VALUES (arrow_cast('01:00:00.000', 'Time32(Millisecond)')),+ (arrow_cast('02:00:00.000', 'Time32(Millisecond)')),+ (arrow_cast('02:00:00.000', 'Time32(Millisecond)'))) t(x)+ORDER BY x+----+01:00:00 1+02:00:00 3+02:00:00 3++# A time ORDER BY key nested in a dictionary resolves through the same arm.+query ?I+SELECT x, COUNT(*) OVER (ORDER BY x)+FROM (VALUES (arrow_cast(arrow_cast('01:00:00', 'Time64(Microsecond)'), 'Dictionary(Int32, Time64(Microsecond))')),+ (arrow_cast(arrow_cast('02:00:00', 'Time64(Microsecond)'), 'Dictionary(Int32, Time64(Microsecond))')),+ (arrow_cast(arrow_cast('02:00:00', 'Time64(Microsecond)'), 'Dictionary(Int32, Time64(Microsecond))'))) t(x)+ORDER BY x+----+01:00:00 1+02:00:00 3+02:00:00 3++# Finite offsets over time are rejected for both the start and end bound.+query error DataFusion error: type_coercion\ncaused by\nError during planning: RANGE with offset PRECEDING/FOLLOWING is not supported for ORDER BY type Time64\(µs\)+SELECT COUNT(*) OVER (ORDER BY x RANGE BETWEEN INTERVAL '1' HOUR PRECEDING AND CURRENT ROW)+FROM (VALUES (arrow_cast('01:00:00', 'Time64(Microsecond)'))) t(x)++query error DataFusion error: type_coercion\ncaused by\nError during planning: RANGE with offset PRECEDING/FOLLOWING is not supported for ORDER BY type Time64\(µs\)+SELECT COUNT(*) OVER (ORDER BY x RANGE BETWEEN CURRENT ROW AND INTERVAL '1' HOUR FOLLOWING)+FROM (VALUES (arrow_cast('01:00:00', 'Time64(Microsecond)'))) t(x)+# Unsupported RANGE ORDER BY types still propagate the type-coercion error.query error DataFusion error: type_coercion\ncaused by\nInternal error: Cannot run range queries on datatype: Struct\("c0": Int64\)SELECT COUNT(*) OVER (ORDER BY x)diff --git a/dev/update_function_docs.sh b/dev/update_function_docs.shindex 2ead1253ac..005122c33e 100755--- a/dev/update_function_docs.sh+++ b/dev/update_function_docs.sh@@ -317,7 +317,7 @@ where **offset** is an non-negative integer.RANGE and GROUPS modes require an ORDER BY clause (with RANGE the ORDER BY must specify exactly one column).-In RANGE mode an **offset** is measured in ORDER BY values rather than in rows, so the bound is computed by adding it to or subtracting it from the current row's ORDER BY value. That restricts `offset PRECEDING` and `offset FOLLOWING` to ORDER BY types supporting such arithmetic, namely the numeric, date, and timestamp types. Other orderable types, such as strings and binaries, can still be used with `UNBOUNDED PRECEDING`, `CURRENT ROW` and `UNBOUNDED FOLLOWING`, which are located by comparing ORDER BY values.+In RANGE mode an **offset** is measured in ORDER BY values rather than in rows, so the bound is computed by adding it to or subtracting it from the current row's ORDER BY value. That restricts `offset PRECEDING` and `offset FOLLOWING` to ORDER BY types supporting such arithmetic, namely the numeric, date, and timestamp types. Other orderable types, such as strings, binaries, and times, can still be used with `UNBOUNDED PRECEDING`, `CURRENT ROW` and `UNBOUNDED FOLLOWING`, which are located by comparing ORDER BY values.## Filter clause for aggregate window functionsdiff --git a/docs/source/user-guide/sql/window_functions.md b/docs/source/user-guide/sql/window_functions.mdindex d13fb64f86..7aeb34a4a6 100644--- a/docs/source/user-guide/sql/window_functions.md+++ b/docs/source/user-guide/sql/window_functions.md@@ -145,7 +145,7 @@ where **offset** is an non-negative integer.RANGE and GROUPS modes require an ORDER BY clause (with RANGE the ORDER BY must specify exactly one column).-In RANGE mode an **offset** is measured in ORDER BY values rather than in rows, so the bound is computed by adding it to or subtracting it from the current row's ORDER BY value. That restricts `offset PRECEDING` and `offset FOLLOWING` to ORDER BY types supporting such arithmetic, namely the numeric, date, and timestamp types. Other orderable types, such as strings and binaries, can still be used with `UNBOUNDED PRECEDING`, `CURRENT ROW` and `UNBOUNDED FOLLOWING`, which are located by comparing ORDER BY values.+In RANGE mode an **offset** is measured in ORDER BY values rather than in rows, so the bound is computed by adding it to or subtracting it from the current row's ORDER BY value. That restricts `offset PRECEDING` and `offset FOLLOWING` to ORDER BY types supporting such arithmetic, namely the numeric, date, and timestamp types. Other orderable types, such as strings, binaries, and times, can still be used with `UNBOUNDED PRECEDING`, `CURRENT ROW` and `UNBOUNDED FOLLOWING`, which are located by comparing ORDER BY values.## Filter clause for aggregate window functions