Apache DataFusion SQL Query Engine
perf: use one null-aware mark join for hashable IN subqueries in projections (#25338)
Adrian Garcia Badaraccob9c8b3dce803
diff --git a/benchmarks/sql_benchmarks/README.md b/benchmarks/sql_benchmarks/README.mdindex 8047c03092..8b68311531 100644--- a/benchmarks/sql_benchmarks/README.md+++ b/benchmarks/sql_benchmarks/README.md@@ -48,7 +48,7 @@ in the community:| `predicate_eval` | Conjunctive (AND) filter-evaluation micro-benchmarks; each subgroup is a different predicate pattern, to test how an adaptive predicate-ordering system behaves across them ([#11262](https://github.com/apache/datafusion/issues/11262)). Subgroups (`--subgroup`): `costsel`, `cost`, `selectivity`, `cardinality`, `width`, `scale`, `neutral`, `correlation`, `drift`, `nulls`. The suite sets no engine config of its own, so by default it measures DataFusion's built-in left-deep `AND` short-circuit; point it at a system under test by exporting that system's own DataFusion setting (the harness builds its `SessionConfig` with `SessionConfig::from_env`). Every query is a `count(*)`, and the counts are checked in under `predicate_eval/results/`, so `--result-mode validate` also checks that a reordering under test still returns the same rows; the checked-in counts were persisted at the suite defaults (`PRED_ROWS=1000000`, `PRED_FILL=30`), so validation assumes those (the `scale` and `width` subgroups pin their own values per query and validate at any setting). || `parquet_row_filter_skip` | Micro-benchmark for the per-row-group fully-matched RowFilter skip on Parquet scans ([#23696](https://github.com/apache/datafusion/issues/23696)). Subgroups (`--subgroup`): `skip` (clustered key, most row groups fully matched by statistics so the per-row filter is skipped), `control` (scrambled key, no row group is ever fully matched). Size the data with `PRED_ROWS` and the row-group size with `RG_SIZE`. || `spill_views` | Sort and `GROUP BY` queries that spill `StringView` and `BinaryView` columns read from Parquet ([#23564](https://github.com/apache/datafusion/issues/23564)). Each query sets its own memory limit and `target_partitions = 4`, so it spills in any environment. Subgroups (`--subgroup`): `repeated` (sort with a low-cardinality payload that shares one dictionary buffer, where spill compaction can remove data), `distinct` (sort and `GROUP BY` on all-distinct strings, where compaction cannot remove data). Set the memory limit of each subgroup with `SPILL_VIEWS_LIMIT_REPEATED` (default `40M`) and `SPILL_VIEWS_LIMIT_DISTINCT` (default `96M`), for example to let both sides of a comparison finish when one of them runs out of memory at the default. The queries must still spill, so check a raised limit with `EXPLAIN ANALYZE` and keep it well below the working set: `q05` stops spilling at about `192M`. Give a whole number of K, M or G units, because the run asserts the limit that the memory pool reports back. |-| `projection_subquery` | `IN`, `NOT IN` and `EXISTS` subqueries that sit in the `SELECT` list instead of a filter ([#25341](https://github.com/apache/datafusion/issues/25341)). Every query projects the boolean subquery result and aggregates it, so the measured cost is the decorrelation plan and not the size of the output. `q07` correlates on `<` instead of `=`, so it keeps the nested-loop plan and is the control that must not change. Size the two inline tables with `PSQ_ROWS` (default `30000`). The counts are checked in under `projection_subquery/results/`, so `--result-mode validate` also proves that a change to the decorrelation still returns the same rows; the checked-in counts were persisted at the default `PSQ_ROWS`, so validation assumes that value. |+| `projection_subquery` | `IN`, `NOT IN` and `EXISTS` subqueries that sit in the `SELECT` list instead of a filter ([#25341](https://github.com/apache/datafusion/issues/25341)). Every query projects the boolean subquery result and aggregates it, so the measured cost is the decorrelation plan and not the size of the output. `q07` correlates on `<` instead of `=`, so the correlation stays a residual filter of the mark join. Size the two inline tables with `PSQ_ROWS` (default `30000`). The counts are checked in under `projection_subquery/results/`, so `--result-mode validate` also proves that a change to the decorrelation still returns the same rows; the checked-in counts were persisted at the default `PSQ_ROWS`, so validation assumes that value. |# Running Benchmarksdiff --git a/benchmarks/sql_benchmarks/projection_subquery/projection_subquery.suite b/benchmarks/sql_benchmarks/projection_subquery/projection_subquery.suiteindex f5f419b50a..e384036197 100644--- a/benchmarks/sql_benchmarks/projection_subquery/projection_subquery.suite+++ b/benchmarks/sql_benchmarks/projection_subquery/projection_subquery.suite@@ -1,4 +1,4 @@-description = "IN / NOT IN / EXISTS subqueries that appear in the SELECT list instead of a filter (apache/datafusion#25341). Each query projects the boolean result of a subquery and aggregates it, so the measured work is the decorrelation plan and not the output size. The decorrelation turns one projected IN into mark joins; when the join predicate is not hashable, or the correlation is not an equality, the plan keeps a nested-loop mark join and the cost grows with outer rows times inner rows. Query 07 correlates on `<` and stays on that path, so it is the control that must not change. The two tables are built inline by the load SQL and are sized with PSQ_ROWS."+description = "IN / NOT IN / EXISTS subqueries that appear in the SELECT list instead of a filter (apache/datafusion#25341). Each query projects the boolean result of a subquery and aggregates it, so the measured work is the decorrelation plan and not the output size. The decorrelation turns one projected IN into mark joins. Query 07 correlates on `<`, so the correlation stays a residual filter of the mark join. The two tables are built inline by the load SQL and are sized with PSQ_ROWS."query_pattern = "q{QUERY_ID_PADDED}.benchmark"diff --git a/benchmarks/sql_benchmarks/projection_subquery/queries/q07.sql b/benchmarks/sql_benchmarks/projection_subquery/queries/q07.sqlindex 54299f0cbd..4069da66a9 100644--- a/benchmarks/sql_benchmarks/projection_subquery/queries/q07.sql+++ b/benchmarks/sql_benchmarks/projection_subquery/queries/q07.sql@@ -1,6 +1,6 @@--- Shape: IN correlated on `<` instead of `=`. There is no equality to hash on,--- so this query keeps the nested-loop mark joins. It is the control: its cost--- must stay the same when the hashable cases get faster.+-- Shape: IN correlated on `<` instead of `=`. The `IN` equality is the hash key+-- and the `<` correlation stays a residual filter of the one null-aware mark+-- join, which applies it when it decides whether a NULL makes the mark UNKNOWN.SELECTcount(*) FILTER (WHERE m) AS true_count,count(*) FILTER (WHERE m IS NULL) AS null_countdiff --git a/datafusion/optimizer/src/decorrelate.rs b/datafusion/optimizer/src/decorrelate.rsindex 4f89532e50..50c34d439f 100644--- a/datafusion/optimizer/src/decorrelate.rs+++ b/datafusion/optimizer/src/decorrelate.rs@@ -30,7 +30,7 @@ use datafusion_common::{Column, DFSchemaRef, HashMap, Result, ScalarValue, assert_or_internal_err, plan_err,};use datafusion_expr::expr::{Alias, GroupingSet};-use datafusion_expr::logical_plan::Join;+use datafusion_expr::logical_plan::{Join, JoinType};use datafusion_expr::simplify::SimplifyContext;use datafusion_expr::utils::{collect_subquery_cols, conjunction, find_join_exprs, split_conjunction,@@ -76,6 +76,21 @@ pub struct PullUpCorrelatedExpr {/// whether we have converted a scalar aggregation into a group aggregation. When unnesting/// lateral joins, we need to produce a left outer join in such cases.pub pulled_up_scalar_agg: bool,+ /// Every correlated conjunct that a `Filter` of the subquery applies,+ /// before `remove_duplicated_filter` drops the ones that the `IN`+ /// predicate already covers. The outer references are kept, so that a+ /// subquery column and an outer column with the same qualified name stay+ /// apart.+ ///+ /// `join_filters` holds only the conjuncts that the join still needs.+ /// This list is what the subquery enforces on its own rows. The caller+ /// uses it to tell if a join key can be NULL inside the scope of an outer+ /// row: `x IN (SELECT y FROM .. WHERE y = x)` keeps every NULL `y` out of+ /// its result, although `join_filters` no longer says so.+ ///+ /// The list is cleared when the pull up passes a node that can put a NULL+ /// back into such a column: an outer join, a union or a grouping set.+ pub correlated_filters: Vec<Expr>,}impl Default for PullUpCorrelatedExpr {@@ -97,6 +112,7 @@ impl PullUpCorrelatedExpr {collected_count_expr_map: HashMap::new(),pull_up_having_expr: None,pulled_up_scalar_agg: false,+ correlated_filters: Vec::new(),}}@@ -186,6 +202,12 @@ impl TreeNodeRewriter for PullUpCorrelatedExpr {}fn f_up(&mut self, plan: LogicalPlan) -> Result<Transformed<LogicalPlan>> {+ // A node that null-extends or regroups rows can put a NULL back into+ // a column that a correlated filter below it rejected, so the filters+ // recorded so far no longer bound what the subquery returns.+ if !self.correlated_filters.is_empty() && may_reintroduce_nulls(&plan) {+ self.correlated_filters.clear();+ }let subquery_schema = plan.schema();match &plan {LogicalPlan::Filter(plan_filter) => {@@ -195,6 +217,11 @@ impl TreeNodeRewriter for PullUpCorrelatedExpr {.iter().filter(|e| e.contains_outer()).all(|&e| can_pullup_over_aggregation(e));+ for expr in &subquery_filter_exprs {+ if expr.contains_outer() && !self.correlated_filters.contains(expr) {+ self.correlated_filters.push((*expr).clone());+ }+ }let (mut join_filters, subquery_filters) =find_join_exprs(subquery_filter_exprs)?;if let Some(in_predicate) = &self.in_predicate_opt {@@ -548,6 +575,27 @@ impl PullUpCorrelatedExpr {}}+/// Whether `plan` can output a NULL in a column that a filter below it keeps+/// free of NULLs: an outer join fills the columns of its unmatched side with+/// NULL, a union adds the rows of its other inputs, and a grouping set fills+/// the columns it leaves out with NULL, such as the grand total row of `ROLLUP`.+fn may_reintroduce_nulls(plan: &LogicalPlan) -> bool {+ match plan {+ LogicalPlan::Join(join) => {+ matches!(+ join.join_type,+ JoinType::Left | JoinType::Right | JoinType::Full+ )+ }+ LogicalPlan::Union(_) => true,+ LogicalPlan::Aggregate(aggregate) => aggregate+ .group_expr+ .iter()+ .any(|expr| matches!(expr, Expr::GroupingSet(_))),+ _ => false,+ }+}+/// Whether every input of `join` that holds outer references is a side whose/// rows the join preserves, so a correlated filter below it can be pulled above/// the join without changing the result.diff --git a/datafusion/optimizer/src/decorrelate_predicate_subquery.rs b/datafusion/optimizer/src/decorrelate_predicate_subquery.rsindex cd5ca84f66..e2f4aa40b5 100644--- a/datafusion/optimizer/src/decorrelate_predicate_subquery.rs+++ b/datafusion/optimizer/src/decorrelate_predicate_subquery.rs@@ -21,6 +21,7 @@ use std::ops::Deref;use std::sync::Arc;use crate::decorrelate::PullUpCorrelatedExpr;+use crate::extract_equijoin_predicate::split_eq_and_noneq_join_predicate;use crate::optimizer::ApplyOrder;use crate::utils::replace_qualified_name;use crate::{OptimizerConfig, OptimizerRule};@@ -28,16 +29,17 @@ use crate::{OptimizerConfig, OptimizerRule};use datafusion_common::alias::AliasGenerator;use datafusion_common::tree_node::{Transformed, TransformedResult, TreeNode};use datafusion_common::{- Column, DFSchemaRef, ExprSchema, NullEquality, Result, ScalarValue,- assert_or_internal_err, plan_err,+ Column, DFSchema, NullEquality, Result, ScalarValue, assert_or_internal_err, plan_err,};use datafusion_expr::expr::{Exists, InSubquery};-use datafusion_expr::expr_rewriter::create_col_from_scalar_expr;+use datafusion_expr::expr_rewriter::{+ create_col_from_scalar_expr, strip_outer_reference,+};use datafusion_expr::logical_plan::{JoinType, Subquery};use datafusion_expr::utils::{conjunction, expr_to_columns, split_conjunction_owned};use datafusion_expr::{- BinaryExpr, Expr, Filter, LogicalPlan, LogicalPlanBuilder, Operator, exists,- in_subquery, lit, not, not_exists, not_in_subquery, when,+ BinaryExpr, Expr, ExprSchemable, Filter, LogicalPlan, LogicalPlanBuilder, Operator,+ exists, in_subquery, lit, not, not_exists, not_in_subquery, when,};use log::debug;@@ -128,8 +130,23 @@ impl OptimizerRule for DecorrelatePredicateSubquery {match build_join_top(&subquery, &cur_input, config.alias_generator())?{Some(plan) => cur_input = plan,- // If the subquery can not be converted to a Join, reconstruct the subquery expression and add it to the Filter- None => other_exprs.push(subquery.expr()),+ // The subquery cannot become a semi or anti join. A+ // `NOT IN` may still become mark joins that+ // materialize its three-valued result, see+ // `in_subquery_value_mark_join`. Any other subquery+ // expression goes back into the filter as it is.+ None => match subquery.expr() {+ expr @ Expr::InSubquery(InSubquery {+ negated: true, ..+ }) => {+ let (plan, expr) = rewrite_inner_subqueries(+ cur_input, expr, config, true, true,+ )?;+ cur_input = plan;+ other_exprs.push(expr);+ }+ expr => other_exprs.push(expr),+ },}}// The subquery expression is embedded within another expression@@ -236,7 +253,11 @@ fn rewrite_inner_subqueries(alias,needs_null_aware_mark,)? {- Some((plan, exists_expr)) => {+ Some(MarkJoin {+ plan,+ mark: exists_expr,+ ..+ }) => {cur_input = plan;Ok(Transformed::yes(exists_expr))}@@ -270,6 +291,7 @@ fn rewrite_inner_subqueries(alias,needs_null_aware_mark,)?+ .map(|join| (join.plan, join.mark))};match rewritten {Some((plan, exists_expr)) => {@@ -285,6 +307,21 @@ fn rewrite_inner_subqueries(Ok((cur_input, expr_without_subqueries.data))}+/// Rewrites an `IN` subquery that gives a value, for example in a SELECT list.+/// The value follows SQL three-valued logic: TRUE for a match, FALSE for a miss+/// and NULL (UNKNOWN) when the answer depends on a NULL.+///+/// There are two paths:+///+/// * One mark join. When the mark column is already exact under three-valued+/// logic (see [`MarkJoin::three_valued_exact`]), the mark column is the+/// answer and this single join is the full rewrite. This is the usual case.+/// * Three mark joins. In the other case the mark column only tells TRUE from+/// not-TRUE, so the UNKNOWN cases must be materialized: one more join tells+/// if the subquery gives a NULL, and one more tells if the subquery gives+/// any row. A `CASE` expression puts the three marks together. The two extra+/// joins have no join predicate, so use them only when the first path+/// cannot apply.fn in_subquery_value_mark_join(left: &LogicalPlan,subquery: &LogicalPlan,@@ -296,23 +333,41 @@ fn in_subquery_value_mark_join(.head_output_expr()?.map_or(plan_err!("single expression required."), Ok)?;let in_predicate = Expr::eq(expr.clone(), output_expr.clone());- let Some((matched_plan, matched)) =- mark_join(left, subquery, Some(&in_predicate), false, alias, true)?+ let Some(MarkJoin {+ plan: matched_plan,+ mark: matched,+ three_valued_exact,+ }) = mark_join(left, subquery, Some(&in_predicate), false, alias, true)?else {return Ok(None);};+ // The mark column is the full answer when it is exact. Negation does not+ // change that, because NOT UNKNOWN is UNKNOWN.+ if three_valued_exact {+ return Ok(Some((+ matched_plan,+ if negated { not(matched) } else { matched },+ )));+ }+// SQL IN needs three facts per outer row to distinguish FALSE from UNKNOWN.let null_subquery = LogicalPlanBuilder::from(subquery.clone()).filter(output_expr.is_null())?.build()?;- let Some((null_plan, subquery_has_null)) =- mark_join(&matched_plan, &null_subquery, None, false, alias, true)?+ let Some(MarkJoin {+ plan: null_plan,+ mark: subquery_has_null,+ ..+ }) = mark_join(&matched_plan, &null_subquery, None, false, alias, true)?else {return Ok(None);};- let Some((final_plan, subquery_non_empty)) =- mark_join(&null_plan, subquery, None, false, alias, true)?+ let Some(MarkJoin {+ plan: final_plan,+ mark: subquery_non_empty,+ ..+ }) = mark_join(&null_plan, subquery, None, false, alias, true)?else {return Ok(None);};@@ -428,14 +483,15 @@ fn build_join_top(};let subquery = query_info.query.subquery.as_ref();let subquery_alias = alias.next("__correlated_sq");- build_join(+ Ok(build_join(left,subquery,in_predicate_opt.as_ref(),join_type,- subquery_alias,+ &subquery_alias,true,- )+ )?+ .map(|join| join.plan))}/// This is used to handle the case when the subquery is embedded in a more complex boolean@@ -460,7 +516,7 @@ fn mark_join(negated: bool,alias_generator: &Arc<AliasGenerator>,needs_null_aware_mark: bool,-) -> Result<Option<(LogicalPlan, Expr)>> {+) -> Result<Option<MarkJoin>> {let alias = alias_generator.next("__correlated_sq");let exists_col = Expr::Column(Column::new(Some(alias.clone()), "mark"));@@ -471,43 +527,181 @@ fn mark_join(subquery,in_predicate_opt,JoinType::LeftMark,- alias,+ &alias,needs_null_aware_mark,)?- .map(|plan| (plan, exists_expr)))+ .map(|join| MarkJoin {+ plan: join.plan,+ mark: exists_expr,+ three_valued_exact: join.mark_is_three_valued_exact,+ }))}-/// Check if join keys in the join filter may contain NULL values-///-/// Returns true if any join key column is nullable on either side.-/// This is used to optimize null-aware anti joins: if all join keys are non-nullable,-/// we can use a regular anti join instead of the more expensive null-aware variant.-fn join_keys_may_be_null(+/// A [`JoinType::LeftMark`] join that replaces a subquery predicate.+struct MarkJoin {+ /// The outer plan with the subquery joined into it.+ plan: LogicalPlan,+ /// Reads the mark column of the join, negated if the caller asked for it.+ mark: Expr,+ /// True when the mark column already gives SQL three-valued `IN`+ /// semantics: TRUE for a match, FALSE for a miss and NULL for UNKNOWN.+ ///+ /// A plain mark is exact when neither side of the `IN` predicate can be+ /// NULL in scope. Otherwise the join must be null-aware, and the `IN`+ /// equality must be its first key, which is where the null-aware hash join+ /// reads the `IN` value. That join also applies a residual non-equality+ /// filter when it decides whether a NULL makes the mark UNKNOWN.+ ///+ /// In any other case the caller materializes the UNKNOWN rows with more+ /// joins, see [`in_subquery_value_mark_join`].+ three_valued_exact: bool,+}++/// Is the `IN` equality the first equality of `join_filter` that the hash+/// join can use as a key? A null-aware hash join reads its first key as the+/// `IN` value and the other keys as the correlation.+fn value_is_first_key(+ in_value: &InValue,join_filter: &Expr,- left_schema: &DFSchemaRef,- right_schema: &DFSchemaRef,+ left_schema: &DFSchema,+ right_schema: &DFSchema,) -> Result<bool> {- // Extract columns from the join filter- let mut columns = std::collections::HashSet::new();- expr_to_columns(join_filter, &mut columns)?;-- // Check if any column is nullable- for col in columns {- // Check in left schema- if let Ok(field) = left_schema.field_from_column(&col)- && field.as_ref().is_nullable()- {- return Ok(true);+ let (equijoin_keys, _) = split_eq_and_noneq_join_predicate(+ join_filter.clone(),+ left_schema,+ right_schema,+ )?;+ Ok(equijoin_keys.first().is_some_and(|(value, column)| {+ value == &in_value.value && column == &in_value.subquery_column+ }))+}++/// The two sides of an `IN` predicate, `value IN (SELECT output_expr ..)`.+struct InValue {+ /// The value from the outer plan, as the join filter refers to it. This is+ /// a projected column when `value_as_written` is a constant.+ value: Expr,+ /// The subquery output, as the join filter refers to it: a column of the+ /// aliased subquery.+ subquery_column: Expr,+ /// The value as the query writes it.+ value_as_written: Expr,+ /// The subquery output expression as the query writes it.+ output_expr: Expr,+}++impl InValue {+ /// Can either side be NULL for a row inside the scope of an outer row?+ /// See [`key_may_be_null_in_scope`]. The correlated filters name the+ /// expressions as the query writes them, so the test matches on those.+ ///+ /// Only then can `IN` be UNKNOWN, so only then does the join need+ /// null-aware semantics. A correlation key that is NULL just empties the+ /// scope, which makes `IN` FALSE.+ fn may_be_null_in_scope(+ &self,+ left_schema: &DFSchema,+ right_schema: &DFSchema,+ scope_filters: &[Expr],+ ) -> Result<bool> {+ Ok(key_may_be_null_in_scope(+ self.value.nullable(left_schema)?,+ &self.value_as_written,+ true,+ scope_filters,+ ) || key_may_be_null_in_scope(+ self.subquery_column.nullable(right_schema)?,+ &self.output_expr,+ false,+ scope_filters,+ ))+ }+}++/// Can this join key be NULL for a row inside the scope of an outer row?+///+/// `nullable` is what the schema says about the key. The key is a full+/// expression, not only a column. An expression can be NULL although none of+/// its columns is nullable: `NULLIF(id, 1)`, `TRY_CAST(s AS INT)`, a `CASE`+/// with no `ELSE` branch, or a scalar function that does not declare its+/// nullability. So the caller asks the key expression itself, against the+/// schema of its own side.+///+/// The subquery can still keep every NULL out of its result. A correlated+/// conjunct such as `y = x`, `y > x` or `y IS NOT NULL` is never TRUE for a+/// NULL `y`, so no row with a NULL `y` is in the scope of any outer row, and+/// an outer row with a NULL `x` has an empty scope. Neither can make `IN`+/// UNKNOWN, so such a key does not need null-aware semantics. The usual shape+/// is a correlation that repeats the `IN` predicate, `x IN (SELECT y FROM t+/// WHERE y = x)`: the pull up drops that conjunct from the join filter, but+/// it still bounds the subquery result+/// (<https://github.com/apache/datafusion/issues/25480>).+///+/// This is a sufficient test, not an exact one. A conjunct counts only if it+/// is a comparison or an `IS NOT NULL` on the key expression itself, casts+/// aside. Any other conjunct is assumed to let a NULL through, which keeps+/// the join null-aware.+///+/// `key_is_outer` tells on which side of the join the key is. The scope+/// filters keep their outer references, and a side of a conjunct matches an+/// outer key only if it names outer columns alone. A subquery column can have+/// the same qualified name as an outer column (`FROM t AS a` inside a query+/// over `a`), and it must not be read as the outer key.+fn key_may_be_null_in_scope(+ nullable: bool,+ key: &Expr,+ key_is_outer: bool,+ scope_filters: &[Expr],+) -> bool {+ if !nullable {+ return false;+ }+ let key = strip_casts(key);+ !scope_filters+ .iter()+ .any(|filter| filter_rejects_null(filter, key, key_is_outer))+}++/// Is `filter` never TRUE when `key` is NULL? See+/// [`key_may_be_null_in_scope`] for `key_is_outer`.+fn filter_rejects_null(filter: &Expr, key: &Expr, key_is_outer: bool) -> bool {+ let is_key = |side: &Expr| {+ let side = strip_casts(side);+ if key_is_outer {+ side.contains_outer()+ && side.column_refs().is_empty()+ && &strip_outer_reference(side.clone()) == key+ } else {+ !side.contains_outer() && side == key}- // Check in right schema- if let Ok(field) = right_schema.field_from_column(&col)- && field.as_ref().is_nullable()- {- return Ok(true);+ };+ match filter {+ Expr::BinaryExpr(BinaryExpr { left, op, right }) => {+ matches!(+ op,+ Operator::Eq+ | Operator::NotEq+ | Operator::Lt+ | Operator::LtEq+ | Operator::Gt+ | Operator::GtEq+ ) && (is_key(left) || is_key(right))}+ Expr::IsNotNull(expr) => is_key(expr),+ _ => false,}+}- Ok(false)+/// `CAST(NULL)` is NULL and `CAST(x)` is not NULL for a non-null `x`, so a+/// conjunct on `x` says the same about `CAST(x)`, and the other way round.+/// Type coercion adds such casts on one side only. `TRY_CAST` can make a NULL+/// from a value, so it is not unwrapped.+fn strip_casts(expr: &Expr) -> &Expr {+ let mut expr = expr;+ while let Expr::Cast(cast) = expr {+ expr = cast.expr.as_ref();+ }+ expr}/// Combines the `IN`/`NOT IN` equality with the correlation predicates, if any.@@ -522,14 +716,23 @@ fn in_predicate_first(in_predicate: Expr, correlation: Option<Expr>) -> Expr {}}+/// The outcome of [`build_join`].+struct BuiltJoin {+ /// The outer plan with the subquery joined into it.+ plan: LogicalPlan,+ /// See [`MarkJoin::three_valued_exact`]. This is always false unless the+ /// join is a [`JoinType::LeftMark`] join built for an `IN` predicate.+ mark_is_three_valued_exact: bool,+}+fn build_join(left: &LogicalPlan,subquery: &LogicalPlan,in_predicate_opt: Option<&Expr>,join_type: JoinType,- alias: String,+ alias: &str,needs_null_aware_mark: bool,-) -> Result<Option<LogicalPlan>> {+) -> Result<Option<BuiltJoin>> {let mut pull_up = PullUpCorrelatedExpr::new().with_in_predicate_opt(in_predicate_opt.cloned()).with_exists_sub_query(in_predicate_opt.is_none());@@ -551,54 +754,54 @@ fn build_join(// alias the join filterlet join_filter_opt = conjunction(pull_up.join_filters).map_or(Ok(None), |filter| {- replace_qualified_name(filter, &all_correlated_cols, &alias).map(Some)+ replace_qualified_name(filter, &all_correlated_cols, alias).map(Some)})?;- // The outer value expression of an `IN`/`NOT IN` predicate, recorded together- // with the subquery column it is compared against, a name for the column it- // can be projected as, and the correlation predicates that share the join- // filter with it.- let mut in_value_expr = None;-- let mut join_filter = match (join_filter_opt, in_predicate_opt.cloned()) {- (- correlation_opt,- Some(Expr::BinaryExpr(BinaryExpr {- left,- op: Operator::Eq,- right,- })),- ) => {- let value_name = format!("{alias}_value");- let right_col = create_col_from_scalar_expr(&right, alias)?;- let value = left.deref().clone();- let in_predicate = Expr::eq(value.clone(), Expr::Column(right_col.clone()));- let join_filter = in_predicate_first(in_predicate, correlation_opt.clone());- in_value_expr = Some((value, right_col, value_name, correlation_opt));-- join_filter+ // The two sides of the `IN` predicate: the value from the outer plan and+ // the subquery output it is compared with, renamed to the alias.+ let in_value = match in_predicate_opt {+ Some(Expr::BinaryExpr(BinaryExpr {+ left,+ op: Operator::Eq,+ right,+ })) => {+ let right_col = create_col_from_scalar_expr(right, alias.to_string())?;+ Some(InValue {+ value: left.deref().clone(),+ subquery_column: Expr::Column(right_col),+ value_as_written: left.deref().clone(),+ output_expr: right.deref().clone(),+ })}- (Some(join_filter), _) => join_filter,- (None, None) => lit(true),- _ => return Ok(None),+ Some(_) => return Ok(None),+ None => None,+ };++ // True if the `IN` value or the subquery column it is compared with can be+ // NULL inside the scope of an outer row. `EXISTS` has no value to compare.+ let value_may_be_null = match &in_value {+ Some(in_value) => in_value.may_be_null_in_scope(+ left.schema(),+ sub_query_alias.schema(),+ &pull_up.correlated_filters,+ )?,+ None => false,};// Whether this join needs null-aware (`NOT IN` three-valued) semantics.// Decided once: the constant projection below, the mark branch and the anti// branch all depend on the same answer. `NOT EXISTS` is two-valued and has- // no `in_predicate_opt`, so it never qualifies; non-nullable keys on both- // sides mean no NULL can reach the join in the first place.+ // no `IN` value, so it never qualifies.//- // The left projection below only adds a non-nullable literal column and the- // right projection only drops unreferenced columns, so neither changes the- // nullability this looks at.- let null_aware = in_predicate_opt.is_some()+ // The left projection below only adds a column for a constant value and+ // the right projection only drops unreferenced columns, so neither changes+ // the nullability this looks at.+ let null_aware = value_may_be_null&& match join_type {JoinType::LeftAnti => true,JoinType::LeftMark => needs_null_aware_mark,_ => false,- }- && join_keys_may_be_null(&join_filter, left.schema(), sub_query_alias.schema())?;+ };// `<constant> IN/NOT IN (<subquery>)`: the outer value expression holds no// column reference, so `<constant> = __correlated_sq.col` is not a valid@@ -614,34 +817,67 @@ fn build_join(// into a real equi-join key, which fixes all three. It only pays for itself// on a join that ends up null-aware.let mut projected_left = None;- if let Some((value, right_col, mut value_name, correlation_opt)) = in_value_expr- && value.column_refs().is_empty()- && null_aware- {- // The projected column is unqualified, so a left field that already has- // this name — however unlikely — would make the reference ambiguous.- let left_schema = left.schema();- while left_schema.fields().iter().any(|f| f.name() == &value_name) {- value_name.push('_');+ let in_value = match in_value {+ Some(in_value) if in_value.value.column_refs().is_empty() && null_aware => {+ // The projected column is unqualified, so a left field that already+ // has this name — however unlikely — would make the reference+ // ambiguous.+ let mut value_name = format!("{alias}_value");+ let left_schema = left.schema();+ while left_schema.fields().iter().any(|f| f.name() == &value_name) {+ value_name.push('_');+ }+ let value_col = Column::new_unqualified(value_name);+ let projections = left_schema+ .columns()+ .into_iter()+ .map(Expr::from)+ .chain(std::iter::once(in_value.value.alias(value_col.name())))+ .collect::<Vec<_>>();+ projected_left = Some(+ LogicalPlanBuilder::from(left.clone())+ .project(projections)?+ .build()?,+ );+ Some(InValue {+ value: Expr::Column(value_col),+ ..in_value+ })}- let value_col = Column::new_unqualified(value_name);- let projections = left_schema- .columns()- .into_iter()- .map(Expr::from)- .chain(std::iter::once(value.alias(value_col.name())))- .collect::<Vec<_>>();- projected_left = Some(- LogicalPlanBuilder::from(left.clone())- .project(projections)?- .build()?,- );- // Rebuild the `IN` equality against the projected column.- let in_predicate = Expr::eq(Expr::Column(value_col), Expr::Column(right_col));- join_filter = in_predicate_first(in_predicate, correlation_opt);- }+ other => other,+ };let left = projected_left.as_ref().unwrap_or(left);+ let join_filter = match (&in_value, join_filter_opt) {+ (Some(in_value), correlation_opt) => in_predicate_first(+ Expr::eq(in_value.value.clone(), in_value.subquery_column.clone()),+ correlation_opt,+ ),+ (None, Some(correlation)) => correlation,+ (None, None) => lit(true),+ };++ // Only a null-aware join reads the key order.+ let value_is_first_key = match &in_value {+ Some(in_value) if null_aware => value_is_first_key(+ in_value,+ &join_filter,+ left.schema(),+ sub_query_alias.schema(),+ )?,+ _ => false,+ };++ // A `NOT IN` in a filter builds a `LeftAnti` join. The null-aware hash+ // join reads its first key as the `NOT IN` value and the other keys as the+ // scope of the outer row (see `HashJoinExec::null_aware`). If the `IN`+ // equality did not become that first key, give up here. The caller then+ // materializes the UNKNOWN rows with more joins, see+ // `in_subquery_value_mark_join`.+ if join_type == JoinType::LeftAnti && null_aware && !value_is_first_key {+ return Ok(None);+ }+if matches!(join_type, JoinType::LeftMark | JoinType::RightMark) {let right_schema = sub_query_alias.schema();@@ -680,6 +916,9 @@ fn build_join(// nullable mark column. A non-equality correlation stays behind as a// join filter, which the hash join also applies when it decides// whether a NULL makes the mark UNKNOWN.+ let mark_is_three_valued_exact = in_value.is_some()+ && (!value_may_be_null || (null_aware && value_is_first_key));+let new_plan = LogicalPlanBuilder::from(left.clone()).join_detailed_with_options(right_projected,@@ -696,7 +935,10 @@ fn build_join(new_plan.display_indent());- return Ok(Some(new_plan));+ return Ok(Some(BuiltJoin {+ plan: new_plan,+ mark_is_three_valued_exact,+ }));}// join our sub query into the main plan@@ -721,7 +963,10 @@ fn build_join("predicate subquery optimized:\n{}",new_plan.display_indent());- Ok(Some(new_plan))+ Ok(Some(BuiltJoin {+ plan: new_plan,+ mark_is_three_valued_exact: false,+ }))}#[derive(Debug)]@@ -807,6 +1052,15 @@ mod tests {table_scan(Some(name), &schema, None)?.build()}+ /// `CASE WHEN test.c = 1 THEN NULL ELSE test.c END`: an expression that can+ /// be NULL although `test.c` is not nullable. `NULLIF(c, 1)` and+ /// `TRY_CAST(c AS INT)` have the same shape, but the optimizer crate cannot+ /// depend on the function crates.+ fn nullable_key_expr() -> Result<Expr> {+ when(col("test.c").eq(lit(1u32)), lit(ScalarValue::UInt32(None)))+ .otherwise(col("test.c"))+ }+fn has_null_aware_left_mark_join(plan: &LogicalPlan) -> bool {if let LogicalPlan::Join(join) = plan&& join.join_type == JoinType::LeftMark@@ -817,6 +1071,18 @@ mod tests {plan.inputs().into_iter().any(has_null_aware_left_mark_join)}+ fn has_non_null_aware_left_mark_join(plan: &LogicalPlan) -> bool {+ if let LogicalPlan::Join(join) = plan+ && join.join_type == JoinType::LeftMark+ {+ return !join.null_aware;+ }++ plan.inputs()+ .into_iter()+ .any(has_non_null_aware_left_mark_join)+ }+fn optimize_with_decorrelate(plan: LogicalPlan) -> Result<LogicalPlan> {let optimizer = crate::Optimizer::with_rules(vec![Arc::new(DecorrelatePredicateSubquery::new(),@@ -1530,20 +1796,127 @@ mod tests {assert_optimized_plan_equal!(plan,@r"- Projection: CASE WHEN __correlated_sq_1.mark THEN Boolean(true) WHEN __correlated_sq_2.mark OR test.c IS NULL AND __correlated_sq_3.mark THEN Boolean(NULL) ELSE Boolean(false) END AS is_present [is_present:Boolean;N]- LeftMark Join: Filter: Boolean(true) [a:UInt32, b:UInt32, c:UInt32, mark:Boolean;N, mark:Boolean;N, mark:Boolean;N]- LeftMark Join: Filter: Boolean(true) [a:UInt32, b:UInt32, c:UInt32, mark:Boolean;N, mark:Boolean;N]- LeftMark Join: Filter: test.c = __correlated_sq_1.c [a:UInt32, b:UInt32, c:UInt32, mark:Boolean;N]- TableScan: test [a:UInt32, b:UInt32, c:UInt32]- Projection: __correlated_sq_1.c [c:UInt32]- SubqueryAlias: __correlated_sq_1 [c:UInt32]- Projection: sq.c [c:UInt32]- TableScan: sq [a:UInt32, b:UInt32, c:UInt32]- SubqueryAlias: __correlated_sq_2 [c:UInt32]- Filter: sq.c IS NULL [c:UInt32]- Projection: sq.c [c:UInt32]- TableScan: sq [a:UInt32, b:UInt32, c:UInt32]- SubqueryAlias: __correlated_sq_3 [c:UInt32]+ Projection: __correlated_sq_1.mark AS is_present [is_present:Boolean;N]+ LeftMark Join: Filter: test.c = __correlated_sq_1.c [a:UInt32, b:UInt32, c:UInt32, mark:Boolean;N]+ TableScan: test [a:UInt32, b:UInt32, c:UInt32]+ Projection: __correlated_sq_1.c [c:UInt32]+ SubqueryAlias: __correlated_sq_1 [c:UInt32]+ Projection: sq.c [c:UInt32]+ TableScan: sq [a:UInt32, b:UInt32, c:UInt32]+ "+ )+ }++ /// A residual non-equality correlation stays on the one mark join. The+ /// null-aware hash join applies it when it decides whether a NULL makes+ /// the mark UNKNOWN, so the mark is still exact.+ #[test]+ fn in_subquery_in_projection_with_residual_filter() -> Result<()> {+ let subquery = Arc::new(+ LogicalPlanBuilder::from(nullable_scalar_mark_scan("inner_t")?)+ .filter(+ out_ref_col(DataType::Int32, "outer_t.grp").gt(col("inner_t.grp")),+ )?+ .project(vec![col("inner_t.id")])?+ .build()?,+ );++ let plan = LogicalPlanBuilder::from(nullable_scalar_mark_scan("outer_t")?)+ .project(vec![+ in_subquery(col("outer_t.id"), subquery).alias("is_present"),+ ])?+ .build()?;++ assert_optimized_plan_equal!(+ plan,+ @r"+ Projection: __correlated_sq_1.mark AS is_present [is_present:Boolean;N]+ LeftMark Join: Filter: outer_t.id = __correlated_sq_1.id AND outer_t.grp > __correlated_sq_1.grp null_aware [id:Int32;N, grp:Int32;N, mark:Boolean;N]+ TableScan: outer_t [id:Int32;N, grp:Int32;N]+ Projection: __correlated_sq_1.id, __correlated_sq_1.grp [id:Int32;N, grp:Int32;N]+ SubqueryAlias: __correlated_sq_1 [id:Int32;N, grp:Int32;N]+ Projection: inner_t.id, inner_t.grp [id:Int32;N, grp:Int32;N]+ TableScan: inner_t [id:Int32;N, grp:Int32;N]+ "+ )+ }++ /// `NOT IN` reads the same mark column, negated. The keys are nullable here,+ /// so the join is null-aware and the mark is NULL for the UNKNOWN rows.+ #[test]+ fn not_in_subquery_in_projection() -> Result<()> {+ let subquery = Arc::new(+ LogicalPlanBuilder::from(nullable_scalar_mark_scan("inner_t")?)+ .project(vec![col("inner_t.id")])?+ .build()?,+ );++ let plan = LogicalPlanBuilder::from(nullable_scalar_mark_scan("outer_t")?)+ .project(vec![+ not_in_subquery(col("outer_t.id"), subquery).alias("is_absent"),+ ])?+ .build()?;++ assert_optimized_plan_equal!(+ plan,+ @r"+ Projection: NOT __correlated_sq_1.mark AS is_absent [is_absent:Boolean;N]+ LeftMark Join: Filter: outer_t.id = __correlated_sq_1.id null_aware [id:Int32;N, grp:Int32;N, mark:Boolean;N]+ TableScan: outer_t [id:Int32;N, grp:Int32;N]+ Projection: __correlated_sq_1.id [id:Int32;N]+ SubqueryAlias: __correlated_sq_1 [id:Int32;N]+ Projection: inner_t.id [id:Int32;N]+ TableScan: inner_t [id:Int32;N, grp:Int32;N]+ "+ )+ }++ /// A key expression can be NULL although none of its columns is nullable.+ /// The mark join must then be null-aware, so the mark is NULL for the rows+ /// that give UNKNOWN.+ #[test]+ fn in_subquery_in_projection_with_nullable_key_expr() -> Result<()> {+ let plan = LogicalPlanBuilder::from(test_table_scan()?)+ .project(vec![+ in_subquery(nullable_key_expr()?, test_subquery_with_name("sq")?)+ .alias("is_present"),+ ])?+ .build()?;++ assert_optimized_plan_equal!(+ plan,+ @r"+ Projection: __correlated_sq_1.mark AS is_present [is_present:Boolean;N]+ LeftMark Join: Filter: CASE WHEN test.c = UInt32(1) THEN UInt32(NULL) ELSE test.c END = __correlated_sq_1.c null_aware [a:UInt32, b:UInt32, c:UInt32, mark:Boolean;N]+ TableScan: test [a:UInt32, b:UInt32, c:UInt32]+ Projection: __correlated_sq_1.c [c:UInt32]+ SubqueryAlias: __correlated_sq_1 [c:UInt32]+ Projection: sq.c [c:UInt32]+ TableScan: sq [a:UInt32, b:UInt32, c:UInt32]+ "+ )+ }++ /// The `NOT IN` filter path builds a `LeftAnti` join. It reads the key+ /// nullability the same way, so a nullable key expression over columns that+ /// are not nullable also makes that join null-aware.+ #[test]+ fn not_in_subquery_filter_with_nullable_key_expr() -> Result<()> {+ let plan = LogicalPlanBuilder::from(test_table_scan()?)+ .filter(not_in_subquery(+ nullable_key_expr()?,+ test_subquery_with_name("sq")?,+ ))?+ .project(vec![col("test.b")])?+ .build()?;++ assert_optimized_plan_equal!(+ plan,+ @r"+ Projection: test.b [b:UInt32]+ LeftAnti Join: Filter: CASE WHEN test.c = UInt32(1) THEN UInt32(NULL) ELSE test.c END = __correlated_sq_1.c null_aware [a:UInt32, b:UInt32, c:UInt32]+ TableScan: test [a:UInt32, b:UInt32, c:UInt32]+ SubqueryAlias: __correlated_sq_1 [c:UInt32]Projection: sq.c [c:UInt32]TableScan: sq [a:UInt32, b:UInt32, c:UInt32]"@@ -1704,6 +2077,84 @@ mod tests {Ok(())}+ /// A correlation that repeats the `IN` predicate keeps every NULL out of+ /// the subquery result, so the mark join must not be null-aware.+ #[test]+ fn mark_join_for_in_predicate_correlation_is_not_null_aware() -> Result<()> {+ let outer_scan = nullable_scalar_mark_scan("outer_t")?;+ let inner_scan = nullable_scalar_mark_scan("inner_t")?;++ let subquery = Arc::new(+ LogicalPlanBuilder::from(inner_scan)+ .filter(out_ref_col(DataType::Int32, "outer_t.id").eq(col("inner_t.id")))?+ .project(vec![col("inner_t.id")])?+ .build()?,+ );++ let plan = LogicalPlanBuilder::from(outer_scan)+ .filter(in_subquery(col("outer_t.id"), subquery).is_null())?+ .build()?;++ let optimized = optimize_with_decorrelate(plan)?;+ assert!(+ has_non_null_aware_left_mark_join(&optimized),+ "{}",+ optimized.display_indent_schema()+ );++ Ok(())+ }++ /// A grouping set above the correlation puts a NULL back into the key: the+ /// grand-total row of `ROLLUP` is a NULL for every outer row. The+ /// correlation then no longer bounds the subquery result, and the mark+ /// join stays null-aware.+ #[test]+ fn mark_join_for_in_predicate_correlation_below_rollup_is_null_aware() -> Result<()> {+ let subquery = correlated_grouping_set_subquery("sq", rollup(vec![col("sq.c")]))?;+ let plan = LogicalPlanBuilder::from(test_table_scan()?)+ .project(vec![in_subquery(col("test.c"), subquery).alias("m")])?+ .build()?;++ let optimized = optimize_with_decorrelate(plan)?;+ assert!(+ has_null_aware_left_mark_join(&optimized),+ "{}",+ optimized.display_indent_schema()+ );++ Ok(())+ }++ /// The same for the `LeftAnti` join that a `NOT IN` filter builds.+ #[test]+ fn anti_join_for_in_predicate_correlation_is_not_null_aware() -> Result<()> {+ let outer_scan = nullable_scalar_mark_scan("outer_t")?;+ let inner_scan = nullable_scalar_mark_scan("inner_t")?;++ let subquery = Arc::new(+ LogicalPlanBuilder::from(inner_scan)+ .filter(out_ref_col(DataType::Int32, "outer_t.id").eq(col("inner_t.id")))?+ .project(vec![col("inner_t.id")])?+ .build()?,+ );++ let plan = LogicalPlanBuilder::from(outer_scan)+ .filter(not_in_subquery(col("outer_t.id"), subquery))?+ .build()?;++ assert_optimized_plan_equal!(+ plan,+ @r"+ LeftAnti Join: Filter: outer_t.id = __correlated_sq_1.id [id:Int32;N, grp:Int32;N]+ TableScan: outer_t [id:Int32;N, grp:Int32;N]+ SubqueryAlias: __correlated_sq_1 [id:Int32;N]+ Projection: inner_t.id [id:Int32;N]+ TableScan: inner_t [id:Int32;N, grp:Int32;N]+ "+ )+ }+#[test]fn in_subquery_both_side_expr() -> Result<()> {let table_scan = test_table_scan()?;diff --git a/datafusion/sqllogictest/test_files/projection_pushdown.slt b/datafusion/sqllogictest/test_files/projection_pushdown.sltindex c92c95fdfb..f0b8b9eeae 100644--- a/datafusion/sqllogictest/test_files/projection_pushdown.slt+++ b/datafusion/sqllogictest/test_files/projection_pushdown.slt@@ -2281,10 +2281,11 @@ SET datafusion.execution.target_partitions = 4;## Each ingredient below is load-bearing:#-# * The `IN (<subquery>)` sits in the SELECT list, not in a WHERE clause, so-# `decorrelate_predicate_subquery` (which runs earlier) leaves it alone and it-# is still a subquery expression by the time extraction runs. A subquery in-# WHERE would be flattened into the main plan, where the old scan could see it.+# * The subquery must still be a subquery expression when extraction runs.+# `decorrelate_predicate_subquery` runs earlier and now flattens `IN` and+# `EXISTS` in a SELECT list too, so a plain uncorrelated `IN` is gone before+# extraction. This subquery is correlated and has a `LIMIT`, which that rule+# cannot pull up, so the rule leaves it alone.# * The alias inside the subquery is literally `__datafusion_extracted_1`. Rename# it to anything outside the reserved prefix and there is nothing to collide# with -- the query then passes with or without the fix and guards nothing.@@ -2295,11 +2296,12 @@ SET datafusion.execution.target_partitions = 4;# Without the fix, extraction reuses `__datafusion_extracted_1` and planning# aborts with: Optimizer rule 'push_down_leaf_projections' failed Schema error:# Schema contains duplicate unqualified field name __datafusion_extracted_1.-# With the fix, generated aliases remain distinct, as the plan below shows.+# With the fix, the generator starts at 2, so the alias made inside the subquery+# is `__datafusion_extracted_2` and the two names stay distinct.#-# Keep this as `EXPLAIN` under `logical_plan_only`: the logical plan exposes both-# the collision-free extracted aliases and the mark joins used to preserve the-# three-valued semantics of `IN` in a projection.+# Keep this as `EXPLAIN` under `logical_plan_only`: an `InSubquery` expression+# that survives decorrelation has no physical plan, so only the logical plan can+# show the collision-free extracted aliases.#####################statement ok@@ -2313,30 +2315,28 @@ SELECTSELECT idFROM (SELECT id, s['label'] AS __datafusion_extracted_1- FROM simple_struct- WHERE s['value'] > 120+ FROM simple_struct inner_t+ WHERE s['value'] > 120 AND inner_t.id = outer_t.id)WHERE __datafusion_extracted_1 <> 'delta'+ LIMIT 1) AS has_matching_label-FROM simple_struct;------logical_plan-01)Projection: simple_struct.id, __correlated_sq_1.mark IS NOT DISTINCT FROM Boolean(true) OR __correlated_sq_2.mark IS NOT DISTINCT FROM Boolean(true) AND __correlated_sq_1.mark IS DISTINCT FROM Boolean(true) AND Boolean(NULL) AS has_matching_label-02)--LeftMark Join:-03)----LeftMark Join:-04)------LeftMark Join: simple_struct.id = __correlated_sq_1.id-05)--------TableScan: simple_struct projection=[id]-06)--------SubqueryAlias: __correlated_sq_1-07)----------Projection: simple_struct.id-08)------------Filter: __datafusion_extracted_4 > Int64(120) AND __datafusion_extracted_1 != Utf8("delta")-09)--------------Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_4, simple_struct.id, get_field(simple_struct.s, Utf8("label")) AS __datafusion_extracted_1-10)----------------TableScan: simple_struct projection=[id, s], partial_filters=[get_field(simple_struct.s, Utf8("value")) > Int64(120)]-11)------EmptyRelation: rows=0-12)----SubqueryAlias: __correlated_sq_3-13)------Projection: simple_struct.id-14)--------Filter: __datafusion_extracted_6 > Int64(120) AND __datafusion_extracted_1 != Utf8("delta")-15)----------Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_6, simple_struct.id, get_field(simple_struct.s, Utf8("label")) AS __datafusion_extracted_1-16)------------TableScan: simple_struct projection=[id, s], partial_filters=[Boolean(true), get_field(simple_struct.s, Utf8("value")) > Int64(120)]+FROM simple_struct outer_t;+----+logical_plan+01)Projection: outer_t.id, outer_t.id IN (<subquery>) AS has_matching_label+02)--Subquery:+03)----Projection: inner_t.id+04)------Projection: inner_t.id, __datafusion_extracted_1+05)--------SubqueryAlias: inner_t+06)----------Projection: simple_struct.id, simple_struct.s, __datafusion_extracted_1+07)------------Limit: skip=0, fetch=1+08)--------------Filter: __datafusion_extracted_2 > Int64(120) AND __datafusion_extracted_1 != Utf8("delta")+09)----------------Filter: simple_struct.id = outer_ref(outer_t.id)+10)------------------Projection: get_field(simple_struct.s, Utf8("value")) AS __datafusion_extracted_2, get_field(simple_struct.s, Utf8("label")) AS __datafusion_extracted_1, simple_struct.id, simple_struct.s+11)--------------------TableScan: simple_struct, partial_filters=[simple_struct.id = outer_ref(outer_t.id), get_field(simple_struct.s, Utf8("value")) > Int64(120)]+12)--SubqueryAlias: outer_t+13)----TableScan: simple_struct projection=[id]statement okset datafusion.explain.logical_plan_only = false;diff --git a/datafusion/sqllogictest/test_files/subquery.slt b/datafusion/sqllogictest/test_files/subquery.sltindex d27c244e0d..7c5c0b76bc 100644--- a/datafusion/sqllogictest/test_files/subquery.slt+++ b/datafusion/sqllogictest/test_files/subquery.slt@@ -2791,9 +2791,18 @@ SELECT gs_outer.k, EXISTS (SELECT 1 FROM gs_inner WHERE gs_inner.k = gs_outer.kstatement error DataFusion error: This feature is not implemented: Physical plan does not support logical expression ExistsSELECT gs_outer.k, EXISTS (SELECT 1 FROM gs_inner WHERE gs_inner.k = gs_outer.k GROUP BY ROLLUP(gs_inner.j)) FROM gs_outer;-# The `IN` form of the same subquery.-statement error DataFusion error: This feature is not implemented: Physical plan does not support logical expression InSubquery-SELECT gs_outer.k, gs_outer.k IN (SELECT gs_inner.k FROM gs_inner WHERE gs_inner.k = gs_outer.k GROUP BY ROLLUP(gs_inner.k)) FROM gs_outer;+# The `IN` form of the same subquery decorrelates: the correlation repeats the+# `IN` predicate, so the pull up drops it and adds no column to the grouping+# set. The grand-total row is a NULL for every outer row, so a miss is UNKNOWN,+# and the null-aware mark join gives NULL for it.+query IB+SELECT gs_outer.k, gs_outer.k IN (SELECT gs_inner.k FROM gs_inner WHERE gs_inner.k = gs_outer.k GROUP BY ROLLUP(gs_inner.k)) FROM gs_outer ORDER BY gs_outer.k;+----+1 true+2 true+4 NULL+5 true+NULL NULL# `NOT EXISTS` too.statement error DataFusion error: This feature is not implemented: Physical plan does not support logical expression Existsdiff --git a/datafusion/sqllogictest/test_files/subquery_projection.slt b/datafusion/sqllogictest/test_files/subquery_projection.sltindex ab7c80d9de..f86de505fe 100644--- a/datafusion/sqllogictest/test_files/subquery_projection.slt+++ b/datafusion/sqllogictest/test_files/subquery_projection.slt@@ -97,3 +97,526 @@ FROM outer_values o;3 NULL4 true5 NULL++# Plan shapes and NULL semantics of a projected IN subquery.+#+# `n1.id` holds a NULL, `n2.id` holds a NULL, and `n3.id` holds none. The mark+# column of a LeftMark join carries the three-valued result on its own when the+# join filter is hashable only, so one join per subquery is enough.++statement ok+CREATE TABLE n1(id INT, z INT) AS VALUES (1, 10), (2, 20), (NULL, 30), (4, 40);++statement ok+CREATE TABLE n2(id INT, z INT) AS VALUES (1, 5), (NULL, 50);++statement ok+CREATE TABLE n3(id INT) AS VALUES (1), (2);++# One hash mark join per subquery. There is no materialization join, so no+# nested loop join over outer x inner rows.+query TT+EXPLAIN SELECT id, id IN (SELECT id FROM n3) AS m3, id IN (SELECT id FROM n2) AS m2 FROM n1;+----+logical_plan+01)Projection: n1.id, __correlated_sq_1.mark AS m3, __correlated_sq_2.mark AS m2+02)--LeftMark Join: n1.id = __correlated_sq_2.id null_aware+03)----LeftMark Join: n1.id = __correlated_sq_1.id null_aware+04)------TableScan: n1 projection=[id]+05)------SubqueryAlias: __correlated_sq_1+06)--------TableScan: n3 projection=[id]+07)----SubqueryAlias: __correlated_sq_2+08)------TableScan: n2 projection=[id]+physical_plan+01)ProjectionExec: expr=[id@0 as id, mark@1 as m3, mark@2 as m2]+02)--HashJoinExec: mode=CollectLeft, join_type=LeftMark, on=[(id@0, id@0)], null_aware+03)----HashJoinExec: mode=CollectLeft, join_type=LeftMark, on=[(id@0, id@0)], null_aware+04)------DataSourceExec: partitions=1, partition_sizes=[1]+05)------DataSourceExec: partitions=1, partition_sizes=[1]+06)----DataSourceExec: partitions=1, partition_sizes=[1]++# A non-equality correlation stays a residual join filter. The null-aware hash+# join applies it when it decides whether a NULL makes the mark UNKNOWN, so one+# join is still enough.+query TT+EXPLAIN SELECT id, id IN (SELECT n2.id FROM n2 WHERE n2.z < n1.z) AS m FROM n1;+----+logical_plan+01)Projection: n1.id, __correlated_sq_1.mark AS m+02)--LeftMark Join: n1.id = __correlated_sq_1.id Filter: __correlated_sq_1.z < n1.z null_aware+03)----TableScan: n1 projection=[id, z]+04)----SubqueryAlias: __correlated_sq_1+05)------TableScan: n2 projection=[id, z]+physical_plan+01)ProjectionExec: expr=[id@0 as id, mark@1 as m]+02)--HashJoinExec: mode=CollectLeft, join_type=LeftMark, on=[(id@0, id@0)], filter=z@1 < z@0, projection=[id@0, mark@2], null_aware+03)----DataSourceExec: partitions=1, partition_sizes=[1]+04)----DataSourceExec: partitions=1, partition_sizes=[1]++query IB rowsort+SELECT id, id IN (SELECT n2.id FROM n2 WHERE n2.z < n1.z) AS m FROM n1;+----+1 true+2 false+4 false+NULL NULL++query IBBB rowsort+SELECT+ id,+ id IN (SELECT id FROM n3) AS m3,+ EXISTS (SELECT 1 FROM n2 WHERE n2.id = n1.id) AS e2,+ id NOT IN (SELECT id FROM n3 WHERE n3.id > 1) AS nn3+FROM n1;+----+1 true true true+2 true false false+4 false false true+NULL NULL false NULL++query IB rowsort+SELECT id, id IN (SELECT id FROM n2) AS m FROM n1;+----+1 true+2 NULL+4 NULL+NULL NULL++query IB rowsort+SELECT id, id NOT IN (SELECT id FROM n2) AS m FROM n1;+----+1 false+2 NULL+4 NULL+NULL NULL++query IB rowsort+SELECT id, id IN (SELECT id FROM n3) AS m FROM n1;+----+1 true+2 true+4 false+NULL NULL++query IT rowsort+SELECT id, CASE WHEN NOT (id IN (SELECT id FROM n2)) THEN 'a' ELSE 'b' END AS c FROM n1;+----+1 b+2 b+4 b+NULL b++query IB rowsort+SELECT z, sum(id) IN (SELECT id FROM n3) AS m FROM n1 GROUP BY z;+----+10 true+20 true+30 NULL+40 false++query IB rowsort+SELECT id, COALESCE((id IN (SELECT id FROM n3))::boolean, false) AS matched FROM n1;+----+1 true+2 true+4 false+NULL false++query IT rowsort+SELECT id, CASE WHEN id NOT IN (SELECT n2.id FROM n2 WHERE n2.z < n1.z) THEN 'a' ELSE 'b' END AS c FROM n1;+----+1 b+2 a+4 a+NULL b++statement ok+DROP TABLE n1;++statement ok+DROP TABLE n2;++statement ok+DROP TABLE n3;++# Nullable key expressions over non-nullable columns.+#+# `nn.id` and `nn.s` are not nullable, but a key expression over them can still+# be NULL. `NULLIF(id, 1)` is NULL for `id = 1`, and `TRY_CAST(s AS INT)` is+# NULL when the text is not a number. The join must be null-aware for these+# keys, so the mark is NULL and `IN` gives UNKNOWN.++statement ok+CREATE TABLE nn(id INT NOT NULL, s VARCHAR NOT NULL) AS VALUES (1, '1'), (2, 'x'), (4, '4');++statement ok+CREATE TABLE r3(id INT NOT NULL) AS VALUES (1), (2);++statement ok+CREATE TABLE r3n(id INT NOT NULL) AS VALUES (1), (2), (5);++# The nullable key expression keeps the plan at one null-aware mark join.+query TT+EXPLAIN SELECT id, NULLIF(id, 1) IN (SELECT id FROM r3) AS m FROM nn;+----+logical_plan+01)Projection: nn.id, __correlated_sq_1.mark AS m+02)--LeftMark Join: nullif(CAST(nn.id AS Int64), Int64(1)) = __correlated_sq_1.r3.id null_aware+03)----TableScan: nn projection=[id]+04)----SubqueryAlias: __correlated_sq_1+05)------Projection: CAST(r3.id AS Int64)+06)--------TableScan: r3 projection=[id]+physical_plan+01)ProjectionExec: expr=[id@0 as id, mark@1 as m]+02)--HashJoinExec: mode=CollectLeft, join_type=LeftMark, on=[(nullif(nn.id,Int64(1))@1, r3.id@0)], projection=[id@0, mark@2], null_aware+03)----ProjectionExec: expr=[id@0 as id, nullif(CAST(id@0 AS Int64), 1) as nullif(nn.id,Int64(1))]+04)------DataSourceExec: partitions=1, partition_sizes=[1]+05)----ProjectionExec: expr=[CAST(id@0 AS Int64) as r3.id]+06)------DataSourceExec: partitions=1, partition_sizes=[1]++# `NULLIF(id, 1)` is NULL for `id = 1`, and `r3` has no NULL, so the answer is+# UNKNOWN for that row.+query IB rowsort+SELECT id, NULLIF(id, 1) IN (SELECT id FROM r3) AS m FROM nn;+----+1 NULL+2 true+4 false++# `TRY_CAST('x' AS INT)` is NULL, so the answer is UNKNOWN for that row.+query IB rowsort+SELECT id, TRY_CAST(s AS INT) IN (SELECT id FROM r3) AS m FROM nn;+----+1 true+2 NULL+4 false++# The same on the subquery side: the output of the subquery holds a NULL, so a+# row with no match is UNKNOWN.+query IB rowsort+SELECT id, id IN (SELECT NULLIF(id, 5) FROM r3n) AS m FROM nn;+----+1 true+2 true+4 NULL++# `NOT IN` reads the same mark column, negated.+query IB rowsort+SELECT id, NULLIF(id, 1) NOT IN (SELECT id FROM r3) AS m FROM nn;+----+1 NULL+2 false+4 true++# The `NOT IN` filter path builds a LeftAnti join and reads the key nullability+# the same way. UNKNOWN does not pass a filter, so `id = 1` drops out.+query I rowsort+SELECT id FROM nn WHERE NULLIF(id, 1) NOT IN (SELECT id FROM r3);+----+4++# An empty subquery gives `false` also for a NULL key, and the plan stays one+# null-aware mark join.+statement ok+CREATE TABLE r_empty(id INT NOT NULL) AS SELECT * FROM r3 WHERE false;++query TT+EXPLAIN SELECT id, NULLIF(id, 1) IN (SELECT id FROM r_empty) AS m FROM nn;+----+logical_plan+01)Projection: nn.id, __correlated_sq_1.mark AS m+02)--LeftMark Join: nullif(CAST(nn.id AS Int64), Int64(1)) = __correlated_sq_1.r_empty.id null_aware+03)----TableScan: nn projection=[id]+04)----SubqueryAlias: __correlated_sq_1+05)------Projection: CAST(r_empty.id AS Int64)+06)--------TableScan: r_empty projection=[id]+physical_plan+01)ProjectionExec: expr=[id@0 as id, mark@1 as m]+02)--HashJoinExec: mode=CollectLeft, join_type=LeftMark, on=[(nullif(nn.id,Int64(1))@1, r_empty.id@0)], projection=[id@0, mark@2], null_aware+03)----ProjectionExec: expr=[id@0 as id, nullif(CAST(id@0 AS Int64), 1) as nullif(nn.id,Int64(1))]+04)------DataSourceExec: partitions=1, partition_sizes=[1]+05)----ProjectionExec: expr=[CAST(id@0 AS Int64) as r_empty.id]+06)------DataSourceExec: partitions=1, partition_sizes=[0]++query IB rowsort+SELECT id, NULLIF(id, 1) IN (SELECT id FROM r_empty) AS m FROM nn;+----+1 false+2 false+4 false++query IB rowsort+SELECT id, NULLIF(id, 1) NOT IN (SELECT id FROM r_empty) AS m FROM nn;+----+1 true+2 true+4 true++statement ok+DROP TABLE r_empty;++# A correlated `NOT IN` filter builds a `LeftAnti` join with two keys: the+# value and the correlation. A function key over non-nullable columns that+# cannot be NULL must not make this join null-aware.+statement ok+CREATE TABLE t1(k INT NOT NULL, s VARCHAR NOT NULL) AS VALUES (1, 'a'), (2, 'b');++statement ok+CREATE TABLE t2(k INT NOT NULL, s VARCHAR NOT NULL) AS VALUES (1, 'B'), (2, 'B');++query IT rowsort+SELECT * FROM t1 WHERE upper(t1.s) NOT IN (SELECT t2.s FROM t2 WHERE t2.k = t1.k);+----+1 a++# `NULLIF(k, 1)` is NULL for `k = 1`, and that group of `t2` is not empty, so+# the correct result has no row for `k = 1`. The join has two keys, the value+# first and the correlation second, and is null-aware because the value+# expression can be NULL. `main` tests the columns of the key instead, which+# are not nullable, and keeps the `k = 1` row. That is+# https://github.com/apache/datafusion/issues/25347.+query I rowsort+SELECT k FROM t1 WHERE NULLIF(t1.k, 1) NOT IN (SELECT t2.k + 10 FROM t2 WHERE t2.k = t1.k);+----+2++query TT+EXPLAIN SELECT k FROM t1 WHERE NULLIF(t1.k, 1) NOT IN (SELECT t2.k + 10 FROM t2 WHERE t2.k = t1.k);+----+logical_plan+01)LeftAnti Join: nullif(CAST(t1.k AS Int64), Int64(1)) = __correlated_sq_1.t2.k + Int64(10), t1.k = __correlated_sq_1.k null_aware+02)--TableScan: t1 projection=[k]+03)--SubqueryAlias: __correlated_sq_1+04)----Projection: CAST(t2.k AS Int64) + Int64(10), t2.k+05)------TableScan: t2 projection=[k]+physical_plan+01)HashJoinExec: mode=CollectLeft, join_type=LeftAnti, on=[(nullif(t1.k,Int64(1))@1, t2.k + Int64(10)@0), (k@0, k@1)], projection=[k@0], null_aware+02)--ProjectionExec: expr=[k@0 as k, nullif(CAST(k@0 AS Int64), 1) as nullif(t1.k,Int64(1))]+03)----DataSourceExec: partitions=1, partition_sizes=[1]+04)--ProjectionExec: expr=[CAST(k@0 AS Int64) + 10 as t2.k + Int64(10), k@0 as k]+05)----DataSourceExec: partitions=1, partition_sizes=[1]++statement ok+DROP TABLE t1;++statement ok+DROP TABLE t2;++# A non-equality correlation leaves one key and a residual join filter. The+# null-aware `LeftAnti` hash join applies the residual filter when it looks+# for UNKNOWN rows. For `k = 1` the key is NULL and the correlated subquery+# result is empty, and `NULL NOT IN (<empty set>)` is TRUE. Both rows are+# correct.+statement ok+CREATE TABLE ra(k INT NOT NULL, z INT NOT NULL) AS VALUES (1, 10), (2, 20);++statement ok+CREATE TABLE rb(k INT NOT NULL, z INT NOT NULL) AS VALUES (5, 50);++query I rowsort+SELECT k FROM ra WHERE NULLIF(ra.k, 1) NOT IN (SELECT rb.k FROM rb WHERE rb.z < ra.z);+----+1+2++# The same shape where the correlated subquery result is not empty for the NULL+# key: `NULL NOT IN ({5})` is UNKNOWN, so `2` is the only row. `main` builds a+# plain anti join here and also keeps `1`.+query I rowsort+SELECT k FROM ra WHERE NULLIF(ra.k, 1) NOT IN (SELECT rb.k FROM rb WHERE rb.z > ra.z);+----+2++query TT+EXPLAIN SELECT k FROM ra WHERE NULLIF(ra.k, 1) NOT IN (SELECT rb.k FROM rb WHERE rb.z > ra.z);+----+logical_plan+01)Projection: ra.k+02)--LeftAnti Join: nullif(CAST(ra.k AS Int64), Int64(1)) = __correlated_sq_1.rb.k Filter: __correlated_sq_1.z > ra.z null_aware+03)----TableScan: ra projection=[k, z]+04)----SubqueryAlias: __correlated_sq_1+05)------Projection: CAST(rb.k AS Int64), rb.z+06)--------TableScan: rb projection=[k, z]+physical_plan+01)HashJoinExec: mode=CollectLeft, join_type=LeftAnti, on=[(nullif(ra.k,Int64(1))@2, rb.k@0)], filter=z@1 > z@0, projection=[k@0], null_aware+02)--ProjectionExec: expr=[k@0 as k, z@1 as z, nullif(CAST(k@0 AS Int64), 1) as nullif(ra.k,Int64(1))]+03)----DataSourceExec: partitions=1, partition_sizes=[1]+04)--ProjectionExec: expr=[CAST(k@0 AS Int64) as rb.k, z@1 as z]+05)----DataSourceExec: partitions=1, partition_sizes=[1]++statement ok+DROP TABLE ra;++statement ok+DROP TABLE rb;++statement ok+DROP TABLE nn;++statement ok+DROP TABLE r3;++statement ok+DROP TABLE r3n;++# A correlation that repeats the `IN` predicate.+#+# `x IN (SELECT y FROM .. WHERE y = x)` writes the `IN` equality a second time.+# The decorrelation drops the duplicate and adds the same equality back as the+# join filter, so the scope of the subquery is gone by the time the join is+# built. The result of this shape is never UNKNOWN: every row the subquery+# keeps for an outer row satisfies `y = x`, so `y` is not NULL there and the+# subquery result is either empty or `{x}`. The join must therefore not be+# null-aware. All results below agree with DuckDB 1.5.2.++statement ok+CREATE TABLE co(id INT, k INT) AS VALUES (1,1),(2,1),(NULL,1),(2,2),(NULL,3),(5,NULL),(9,9);++statement ok+CREATE TABLE ci(id INT, k INT) AS VALUES (1,1),(NULL,2),(7,1),(3,NULL);++# The mark join carries the whole result and is not null-aware.+query TT+EXPLAIN SELECT co.id, co.k IN (SELECT ci.k FROM ci WHERE ci.k = co.k) AS m FROM co;+----+logical_plan+01)Projection: co.id, __correlated_sq_1.mark AS m+02)--LeftMark Join: co.k = __correlated_sq_1.k+03)----TableScan: co projection=[id, k]+04)----SubqueryAlias: __correlated_sq_1+05)------TableScan: ci projection=[k]+physical_plan+01)ProjectionExec: expr=[id@0 as id, mark@1 as m]+02)--HashJoinExec: mode=CollectLeft, join_type=RightMark, on=[(k@0, k@1)], projection=[id@0, mark@2]+03)----DataSourceExec: partitions=1, partition_sizes=[1]+04)----DataSourceExec: partitions=1, partition_sizes=[1]++query IIB+SELECT co.id, co.k, co.k IN (SELECT ci.k FROM ci WHERE ci.k = co.k) AS m FROM co ORDER BY k, id;+----+1 1 true+2 1 true+NULL 1 true+2 2 true+NULL 3 false+9 9 false+5 NULL false++# The `IN` value carries a second correlation, which stays a join key.+query IIB+SELECT co.id, co.k, co.id IN (SELECT ci.id FROM ci WHERE ci.id = co.id AND ci.k = co.k) AS m FROM co ORDER BY k, id;+----+1 1 true+2 1 false+NULL 1 false+2 2 false+NULL 3 false+9 9 false+5 NULL false++# The same shape in a `WHERE` clause builds a `LeftAnti` join, which must not be+# null-aware either. `main` gives no row here, which is+# https://github.com/apache/datafusion/issues/25480.+query II+SELECT co.id, co.k FROM co WHERE co.k NOT IN (SELECT ci.k FROM ci WHERE ci.k = co.k) ORDER BY k, id;+----+NULL 3+9 9+5 NULL++# The same shape with an expression as the value. The subquery output is then a+# cast, and the join keys refer to it by its column name. The correlation still+# names the expression as the query writes it, and the join is not null-aware.+query IIB+SELECT co.id, co.k, (co.k + 0) IN (SELECT ci.k FROM ci WHERE ci.k = co.k + 0) AS m FROM co ORDER BY k, id;+----+1 1 true+2 1 true+NULL 1 true+2 2 true+NULL 3 false+9 9 false+5 NULL false++# A grouping set above the correlated filter puts a NULL back. The grand-total+# row of `ROLLUP` exists for every outer row, so a miss is UNKNOWN here, not+# FALSE. The filter repeats the `IN` predicate, so the pull up drops it and the+# aggregate needs no correlated column. The filter below the aggregate then+# does not bound the subquery result, and the join stays null-aware. Expected+# results verified with DuckDB.+query IIB+SELECT co.id, co.k, co.k IN (SELECT ci.k FROM ci WHERE ci.k = co.k GROUP BY ROLLUP(ci.k)) AS m FROM co ORDER BY k, id;+----+1 1 true+2 1 true+NULL 1 true+2 2 true+NULL 3 NULL+9 9 NULL+5 NULL NULL++query TT+EXPLAIN SELECT co.id, co.k, co.k IN (SELECT ci.k FROM ci WHERE ci.k = co.k GROUP BY ROLLUP(ci.k)) AS m FROM co;+----+logical_plan+01)Projection: co.id, co.k, __correlated_sq_1.mark AS m+02)--LeftMark Join: co.k = __correlated_sq_1.k null_aware+03)----TableScan: co projection=[id, k]+04)----SubqueryAlias: __correlated_sq_1+05)------Projection: ci.k+06)--------Aggregate: groupBy=[[ROLLUP (ci.k)]], aggr=[[]]+07)----------TableScan: ci projection=[k]+physical_plan+01)ProjectionExec: expr=[id@0 as id, k@1 as k, mark@2 as m]+02)--HashJoinExec: mode=CollectLeft, join_type=LeftMark, on=[(k@1, k@0)], null_aware+03)----DataSourceExec: partitions=1, partition_sizes=[1]+04)----ProjectionExec: expr=[k@0 as k]+05)------AggregateExec: mode=FinalPartitioned, gby=[k@0 as k, __grouping_id@1 as __grouping_id], aggr=[]+06)--------RepartitionExec: partitioning=Hash([k@0, __grouping_id@1], 4), input_partitions=1+07)----------AggregateExec: mode=Partial, gby=[(NULL as k), (k@0 as k)], aggr=[]+08)------------DataSourceExec: partitions=1, partition_sizes=[1]++# The `NOT IN` filter form of the same subquery. Every row is either FALSE or+# UNKNOWN, so no row is kept.+query II+SELECT co.id, co.k FROM co WHERE co.k NOT IN (SELECT ci.k FROM ci WHERE ci.k = co.k GROUP BY ROLLUP(ci.k)) ORDER BY k, id;+----++statement ok+DROP TABLE co;++statement ok+DROP TABLE ci;++# A subquery column with the same qualified name as the outer `IN` value.+#+# Inside the subquery, `a` is `sh_t`, so `a.x = b.k` bounds `sh_t.x`, not the+# outer `a.x`. It does not keep a NULL outer value out of the scope, and the+# join must stay null-aware. The subquery is `{7}` for every outer row, so the+# NULL row is UNKNOWN. Results verified with DuckDB 1.5.2.+statement ok+CREATE TABLE sh_o(x INT, k INT) AS VALUES (NULL, 1), (7, 1), (8, 1);++statement ok+CREATE TABLE sh_t(x INT, y INT NOT NULL) AS VALUES (1, 7);++query IB+SELECT a.x, a.x IN (SELECT a.y FROM sh_t AS a WHERE a.x = b.k) AS r+FROM sh_o AS a, (SELECT 1 AS k) AS b ORDER BY 1;+----+7 true+8 false+NULL NULL++# The `NOT IN` filter drops the UNKNOWN row.+query I+SELECT a.x FROM sh_o AS a, (SELECT 1 AS k) AS b+WHERE a.x NOT IN (SELECT a.y FROM sh_t AS a WHERE a.x = b.k);+----+8++statement ok+DROP TABLE sh_o;++statement ok+DROP TABLE sh_t;