Skip to content

Apache DataFusion SQL Query Engine

docs: add an agent skill for auditing datafusion-spark expressions (#23893)

Andy Grovee7e037dbeb6e
diff --git a/.ai/skills/audit-datafusion-spark-expression/SKILL.md b/.ai/skills/audit-datafusion-spark-expression/SKILL.md
new file mode 100644
index 0000000000..258b403236
--- /dev/null
+++ b/.ai/skills/audit-datafusion-spark-expression/SKILL.md
@@ -0,0 +1,996 @@
+---
+name: audit-datafusion-spark-expression
+description: Audit a datafusion-spark function implementation for correctness against Apache Spark 4.2.0. Studies the Spark source across versions, reviews the Rust implementation and its signature, verifies expected values against a real PySpark, fixes divergences, and captures the rest as issues and disabled tests.
+argument-hint: <function-name>
+---
+
+<!--
+Licensed to the Apache Software Foundation (ASF) under one
+or more contributor license agreements. See the NOTICE file
+distributed with this work for additional information
+regarding copyright ownership. The ASF licenses this file
+to you under the Apache License, Version 2.0 (the
+"License"); you may not use this file except in compliance
+with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing,
+software distributed under the License is distributed on an
+"AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+KIND, either express or implied. See the License for the
+specific language governing permissions and limitations
+under the License.
+-->
+
+Audit the `datafusion-spark` implementation of the `$ARGUMENTS` function for
+correctness against Apache Spark.
+
+## Baseline Version
+
+The baseline is **Spark 4.2.0**. Every expected value written into a `.slt`
+file asserts Spark 4.2.0 behavior.
+
+Spark 3.5.8, 4.0.4, and 4.1.3 are also read, but only to detect that behavior
+changed between versions. They never set an expectation. There is currently no
+mechanism for version-specific expectations in DataFusion's SLT files, which is
+tracked by https://github.com/apache/datafusion/issues/23887.
+
+Older versions are read rather than ignored because `datafusion-spark` is meant
+to be reusable by engines that target more than one Spark release. Comet
+supports Spark 3.4 through 4.0. A function audited against 4.2.0 alone can be
+correct there and silently wrong for such a consumer, and nothing in the crate
+records which of the two it is. Finding the divergence is useful even though
+the test suite cannot yet assert it.
+
+## Overview
+
+1. Study the Spark implementation across versions
+2. Harvest Spark's own test coverage
+3. Review the DataFusion implementation
+4. Cross-check Comet and Sail
+5. Review existing SLT coverage
+6. Gap analysis
+7. Establish ground truth with PySpark
+8. Apply findings
+
+Audit one function per invocation.
+
+## Paths in This Skill Are Hints
+
+Every file path below was correct when this skill was written and several will
+rot. `FunctionRegistry.scala`, the golden-file directories, and
+`datafusion/common/src/config.rs` all move on someone else's schedule.
+
+**A path that does not resolve is a stop condition, not a step to skip.** The
+failure mode this exists to prevent is a command that quietly matches nothing,
+an audit that reads the silence as "no ANSI branch" or "Spark does not test
+this", and a confident report built on an empty grep. A `grep` over a
+nonexistent directory and a `grep` that genuinely found nothing are the same
+output.
+
+So when a path in this skill does not exist, or a command returns nothing where
+something was expected:
+
+1. Say so out loud in the audit report. Name the path and the step.
+2. Search for where the thing moved to, by content rather than by path. The
+ grep patterns here are more durable than the directories they are pointed at.
+3. If it cannot be found, mark every conclusion that depended on that step as
+ unestablished. Do not let an unread source become an absence of findings.
+
+The same rule covers a `.slt` file, an implementation file, or a Spark class
+that is not where this skill says it is.
+
+## Step 1: Study the Spark Implementation
+
+Shallow-clone the four version tags. Skip any already present.
+
+```bash
+set -eu
+for tag in v3.5.8 v4.0.4 v4.1.3 v4.2.0; do
+ dir="/tmp/spark-${tag}"
+ [ -d "$dir" ] || git clone --depth 1 --branch "$tag" \
+ https://github.com/apache/spark.git "$dir"
+done
+```
+
+Find the expression class in each version. Spark registers SQL function names
+in `FunctionRegistry.scala`, so start there to map `$ARGUMENTS` to its Scala
+class name, then locate the class.
+
+```bash
+for tag in v3.5.8 v4.0.4 v4.1.3 v4.2.0; do
+ dir="/tmp/spark-${tag}"
+ echo "=== $tag ==="
+ grep -rn "\"$ARGUMENTS\"" \
+ "$dir/sql/catalyst/src/main/scala/org/apache/spark/sql/catalyst/analysis/FunctionRegistry.scala"
+done
+```
+
+With the class name in hand, read its source in each version:
+
+```bash
+for tag in v3.5.8 v4.0.4 v4.1.3 v4.2.0; do
+ dir="/tmp/spark-${tag}"
+ echo "=== $tag ==="
+ find "$dir/sql/catalyst/src/main/scala" -name "*.scala" \
+ | xargs grep -lE "case class <ClassName>[ (\[]" 2>/dev/null
+done
+```
+
+A Scala `case class` name is always followed by a space, a `(`, or a `[`, so
+that bracket expression is the portable way to avoid matching a longer class
+name that starts with the same prefix. Do not reach for `\b`, which is a GNU
+and ugrep extension and is not available in POSIX or BSD `grep`.
+
+If it is not in catalyst, widen the search to `$dir/sql`.
+
+For each version, record:
+
+- `eval` / `nullSafeEval` / `doGenCode` logic
+- `inputTypes` and `dataType`
+- `nullable`, and how nulls propagate
+- ANSI branches (`ansiEnabled`, `failOnError`)
+- guards, `require` assertions, thrown exceptions
+- any SQLConf the expression reads
+- for string functions in 4.0+, whether it routes through
+ `CollationSupport.X.exec(..., collationId)` or declares
+ `StringTypeWithCollation` inputs
+
+Produce a diff summary across 3.5.8 → 4.0.4 → 4.1.3 → 4.2.0. Note new or
+removed input types, changed edge-case behavior, new ANSI branches, new
+parameters, and collation changes.
+
+Two changes affect nearly every function and are easy to miss:
+
+- ANSI mode became the **default** in Spark 4.0. A function whose behavior
+ differs under ANSI has a different default behavior in 3.5 than in 4.x.
+- Collation support landed for many string functions in 4.0. Non-default
+ collations are a divergence axis that does not exist in 3.5.
+
+## Step 2: Harvest Spark's Own Test Coverage
+
+Spark's tests are a ready-made edge-case inventory. Two sources matter.
+
+Unit tests:
+
+```bash
+dir=/tmp/spark-v4.2.0
+find "$dir/sql" -name "*.scala" -path "*/test/*" \
+ | xargs grep -ln "$ARGUMENTS" 2>/dev/null
+```
+
+Golden files, which carry Spark's own verified query results and are the
+highest-value source:
+
+```bash
+for tag in v3.5.8 v4.0.4 v4.1.3 v4.2.0; do
+ dir="/tmp/spark-${tag}"
+ echo "=== $tag ==="
+ grep -rln "$ARGUMENTS" "$dir/sql/core/src/test/resources/sql-tests/results/" 2>/dev/null
+done
+```
+
+**The ANSI and non-ANSI golden directories swapped meaning in Spark 4.0.** Read
+the version banner before trusting a path:
+
+| Version | ANSI-mode results | Non-ANSI results |
+| ------- | ------------------------ | --------------------------- |
+| 3.5.8 | `results/ansi/X.sql.out` | `results/X.sql.out` |
+| 4.x | `results/X.sql.out` | `results/nonansi/X.sql.out` |
+
+ANSI became the default in 4.0, so the unqualified path flipped from meaning
+non-ANSI to meaning ANSI. An auditor who learns the layout on 3.5.8 and then
+reads `results/X.sql.out` on 4.2.0 will record exactly inverted expectations,
+and they will look plausible. Confirm the mode by checking whether the file
+contains error output or `NULL` for a known-invalid input.
+
+Read every match and extract:
+
+- input types covered
+- edge cases exercised: null, empty, overflow, negative, boundary, special
+ characters
+- ANSI-mode cases
+- error cases and their exact messages
+
+This list is the reference for the gap matrix in Step 6.
+
+## Step 3: Review the DataFusion Implementation
+
+```bash
+grep -rln "$ARGUMENTS" datafusion/spark/src/function/ --include="*.rs"
+```
+
+The implementation lives at
+`datafusion/spark/src/function/<category>/<function>.rs`. **Read every other
+file that grep returns as well.** In particular
+`datafusion/spark/src/function/<category>/mod.rs` carries the `export_functions!`
+registration and the user-facing doc string, and unimplemented-behavior `TODO`
+comments are written there too, not only in the function's own file. An audit
+that reads only `<function>.rs` will miss them and will leave a doc string that
+contradicts the code after a fix.
+
+**A `.slt` file under `test_files/spark/` does not imply an implementation in
+`datafusion/spark`.** `SessionStateBuilderSpark::with_spark_features` layers the
+Spark functions on top of `with_default_features`, so a name the spark crate
+does not define resolves to DataFusion's own function of that name. `md5`,
+`coalesce`, and `reverse` are served this way today, and `date_format` resolves
+through an alias on core's `to_char`. The query under audit still runs, and the
+code it runs is not in the directory the grep above searched. If that grep comes
+back empty, widen it before concluding the function is unimplemented:
+
+```bash
+grep -rln "\"$ARGUMENTS\"" \
+ datafusion/functions/src datafusion/functions-nested/src \
+ datafusion/functions-aggregate/src datafusion/functions-window/src \
+ datafusion/functions-table/src --include="*.rs"
+```
+
+A hit may be in a file named after something else, because core registers
+aliases. `date_format` lives in `to_char.rs`. Read the `aliases` vector, not
+just the file name.
+
+If that grep is also empty, the function is genuinely unimplemented and the
+`.slt` file is a porting stub with every query commented out. Check with
+`grep -cE '^(query|statement)' <file>.slt`, which returns zero for a stub.
+There is no implementation to audit in that case. Say so and stop, rather than
+reporting a clean audit of nothing.
+
+Otherwise audit what the grep found against Spark with the rest of this step's
+checklist. The findings land differently. A core function has non-Spark users,
+so Spark's semantics cannot be imposed on it. Where the two agree, record the
+coverage and leave the core function alone. Where they diverge, the fix is a
+Spark-specific implementation under `datafusion/spark/src/function/<category>/`
+that shadows the core one, never an edit to the core function.
+`datafusion/spark/src/function/math/abs.rs` shadowing
+`datafusion/functions/src/math/abs.rs` is the in-repo precedent. Say in the
+audit report which crate the audited code came from, because a reader who
+assumes `datafusion/spark` will look for a file that does not exist.
+
+Check each of the following. The first two have no Comet analog and are the
+most common source of silent divergence.
+
+**Signature and coercion.** Compare `Signature` (and `coerce_types`, if
+implemented) against Spark's `inputTypes`. Two failure directions, both real:
+
+- DataFusion rejects a type Spark accepts. The user gets a coercion error
+ where Spark returns a value.
+- DataFusion accepts a type Spark rejects. The user gets a value where Spark
+ raises an analysis error.
+
+Note that `Signature::exact` is narrow. A Spark function accepting any string
+type needs all of `Utf8`, `LargeUtf8`, and `Utf8View` handled, and a function
+accepting any integral type needs the full width range.
+
+**Return nullability.** Compare `return_field_from_args` (or `return_type`)
+against Spark's `nullable`. A function that reports non-nullable while Spark
+can return NULL produces wrong plans, not merely wrong values: downstream
+optimizer rules will fold away null checks that were load-bearing.
+
+**Null propagation.** Does every input-null combination return what Spark
+returns? Check the order in which nullness and validity are tested. Spark
+expressions that are null intolerant (a `NullIntolerant` mixin in 3.5, a
+`nullIntolerant` override in 4.x) short circuit to NULL *before* validating the
+other arguments, so a NULL in one argument suppresses an error the other
+argument would otherwise raise, even under ANSI mode. An implementation that
+validates first and checks nullness second raises where Spark returns NULL.
+
+**Arrow kernel semantics are not Java semantics.** A `datafusion-spark`
+function is usually a thin arrangement of Arrow kernels, and the kernels do not
+share Java's rules. Check each one the implementation leans on:
+
+- **Comparison kernels order floats by total order**, so `-0.0` compares as
+ *less than* `0.0` rather than equal to it. Any `eq`/`lt` used to detect a
+ special value, such as a zero divisor, silently misses `-0.0`. Java compares
+ them as equal. This is easy to miss because `-0.0` almost never appears in
+ hand-written test values, so write it in deliberately.
+- **Arithmetic kernels are checked, not wrapping.** Java wraps silently on
+ overflow, so a checked Arrow kernel raises where Spark returns a wrapped
+ value. Where Spark's own expression is written in `Int` arithmetic, a
+ wrapping kernel is the faithful choice, not the reckless one.
+- **Java promotes `byte` and `short` operands to `int`.** A Spark expression
+ whose Scala source reads `(r + n)` on `Byte` operands is evaluated at `Int`
+ width and cannot overflow, while the same source on `Int` operands can.
+ Reproducing it at `Int8` width wraps where Spark does not.
+- **Kernel coverage is type dependent.** Arrow's `rem` raises on a zero divisor
+ for integers and decimals but returns `NaN` for floats, so an ANSI check that
+ relies on the kernel to raise silently does nothing on the float path.
+
+Read the Scala source as arithmetic to be reproduced, not as pseudocode to be
+paraphrased. Then test each of these against a live Spark rather than reasoning
+about them.
+
+**`ColumnarValue` shape dispatch.** A two-argument function has four shape
+combinations, and each needs its own branch:
+`(Scalar, Scalar)`, `(Array, Scalar)`, `(Scalar, Array)`, `(Array, Array)`.
+A missing arm falls through to a catch-all `exec_err!` and turns a working
+Spark query into a hard error. `(Scalar, Array)` is the one most often
+forgotten, because the natural test query puts the column first. Write a query
+for every combination and run it. Do not infer coverage from reading the
+`match`.
+
+**How to run an ad-hoc query.** The only in-repo way to execute a
+`datafusion-spark` function is to add the query to
+`datafusion/sqllogictest/test_files/spark/<category>/$ARGUMENTS.slt` and run
+
+```bash
+cargo test --test sqllogictests -- spark/<category>/$ARGUMENTS
+```
+
+`datafusion-cli` does **not** register the `datafusion-spark` function library,
+so reaching for it fails in a way that looks like the function does not exist:
+
+```
+$ ./target/debug/datafusion-cli -c "SELECT next_day(arrow_cast(95026236,'Date32'),'Mon');"
+Error during planning: Invalid function 'next_day'. Did you mean 'today'?
+```
+
+Giving the CLI an opt-in for the Spark library is tracked by
+https://github.com/apache/datafusion/issues/24146. Until that lands, treat the
+error above as a registration gap and not as evidence about the function.
+
+For a throwaway probe that should not end up in the committed file, add a
+temporary `.slt` alongside it, run it by name, and delete it afterwards. To
+exercise a raw storage value that has no literal syntax, such as a `Date32`
+past the last date DataFusion can render, wrap it in `arrow_cast`.
+
+**Cross the shapes with the NULL cases, and with ANSI mode.** These are not
+three independent checklists to tick off separately, they are one product to
+enumerate. The bug this instruction exists to catch is a branch that gets NULL
+handling right in the row-wise path and wrong in the vectorized one, so the two
+shapes disagree with each other while each looks correct in isolation. For
+every shape, ask what happens when the *other* argument is NULL, and when it is
+NULL in only some rows. A shape that takes an array and a scalar has to make
+the scalar's validation depend on the array's null mask, because a per-row null
+intolerant function skips validation on the rows it short circuits. All-NULL,
+some-NULL, and no-NULL versions of the same array are three different tests.
+
+Note that a `(Array, Scalar)` branch returning `ColumnarValue::Scalar` is *not*
+by itself a bug. In DataFusion a returned `Scalar` means "this value for every
+row" and the caller broadcasts it to the batch length. Before reporting one,
+run an N-row query and count the rows that come back. Report it only if the
+row count is actually wrong, or if the scalar's value depends on per-row input.
+Otherwise it is at most a readability cleanup, and reporting it as a
+correctness bug costs the audit credibility.
+
+**Overflow and underflow.** Does it return `Err`, wrap, or panic? Spark's
+answer depends on ANSI mode. A panic is always a bug.
+
+**Type dispatch.** Every type the signature admits must have a branch. An
+`exec_err!` fallthrough for a type the signature accepts is a bug.
+
+**ANSI handling.** Compare against `datafusion.execution.enable_ansi_mode`
+(defined in `datafusion/common/src/config.rs`). The crate has scattered `TODO`
+comments about unimplemented ANSI paths. Treat every one found in this
+function as a finding, not as a known-acceptable state. Grep for them across
+every file Step 3's grep returned, including `mod.rs`:
+
+```bash
+grep -rn "TODO" datafusion/spark/src/function/<category>/$ARGUMENTS.rs \
+ datafusion/spark/src/function/<category>/mod.rs
+```
+
+Wiring ANSI up is usually mechanical rather than a semantics decision. The flag
+is already threaded through `ScalarFunctionArgs`, so the implementation reads
+it directly:
+
+```rust
+let ScalarFunctionArgs { args, config_options, .. } = args;
+let ansi_mode = config_options.execution.enable_ansi_mode;
+```
+
+`datafusion/spark/src/function/math/abs.rs` and
+`datafusion/spark/src/function/math/modulus.rs` are the in-repo precedents for
+both the plumbing and the test layout. DataFusion does not model Spark's error
+classes or SQLSTATE values, so say that in the audit either way.
+
+Whether to match Spark's message text depends on who constructs the error:
+
+- **The function constructs it** (an `exec_err!` you write). Match Spark's text
+ exactly, including trailing punctuation, and assert it in the `.slt`.
+- **An Arrow kernel constructs it** (`rem`, `add`, a cast). You do not control
+ the wording. Assert only that an error is raised, note the divergence in a
+ comment, and file an issue for the message text rather than contorting the
+ implementation to restate an Arrow error in Spark's words.
+
+Do not read the `abs.rs` precedent as licence to invent DataFusion-flavoured
+text for an error you *do* construct. `abs` reports `Int32 overflow on
+abs(-2147483648)` where Spark 4.2.0 reports `[ARITHMETIC_OVERFLOW] overflow.`,
+which is a known divergence tracked by
+https://github.com/apache/datafusion/issues/24145. Follow its ANSI plumbing and
+test layout, not its wording.
+
+`abs` also shows why the "who constructs it" question is worth asking rather
+than assuming. Its scalar path builds the message in the spark crate, but its
+array path borrows `make_try_abs_function!` from
+`datafusion/functions/src/math/abs.rs`, so that half of the text is owned by
+core DataFusion and shared with core's own `abs`. Wherever a spark function
+reuses a core macro or kernel, Spark-specific error text is not available
+without un-sharing the code first. Check where the message is actually built
+before promising to match Spark's.
+
+**Rust unit tests.** Read the `#[test]` blocks in the same file. Per the crate
+README, direct invocation through `test_scalar_function!()` bypasses input
+coercion, so passing unit tests never establish end-to-end correctness. Note
+which cases are covered by unit tests only.
+
+## Step 4: Cross-Check Comet and Sail
+
+The `datafusion-spark` README recommends checking both projects for existing
+implementations. Diff against them to surface edge cases they handle that this
+crate misses.
+
+Neither project needs to be checked out. Both are searched over HTTP, by
+content rather than by path, because both have moved their directory layouts
+more than once:
+
+```bash
+for repo in apache/datafusion-comet lakehq/sail; do
+ echo "=== $repo ==="
+ gh api -X GET search/code -f q="$ARGUMENTS repo:$repo" --jq '.items[].path'
+done
+
+# Then read a specific hit
+gh api "repos/<repo>/contents/<path from above>" --jq '.content' | base64 -d
+```
+
+If a clone happens to be on disk, grep it instead. It is faster and it is not
+rate limited. Do not silence the failure with `2>/dev/null`, though: a grep over
+a directory that is not there prints an error, and swallowing it turns "Comet
+not checked out" into "Comet has no implementation".
+
+```bash
+[ -d ../datafusion-comet ] \
+ && grep -rln "$ARGUMENTS" ../datafusion-comet/native/spark-expr/src/ --include="*.rs" \
+ || echo "no local Comet clone, use the API above"
+```
+
+`gh api search/code` requires authentication and is rate limited to a handful of
+requests per minute. If it fails, say so in the audit report and mark Step 4 as
+not performed rather than reporting no differences found.
+
+As of this writing the implementations live under
+`crates/sail-function/src/scalar/<category>/spark_<function>.rs`, and Sail also
+keeps behavior fixtures at
+`python/pysail/tests/spark/function/features/<function>.feature`. Treat both
+paths as hints, not as addresses.
+
+Sail matters more than a generic peer here: many `datafusion-spark` functions
+were ported from Sail, so Sail's current code is often the same code with
+subsequent fixes applied. A difference between the two is a strong lead.
+
+Treat both as peer implementations, not as authority. Each has its own audit
+backlog, and a shared assumption between them is not evidence of correctness.
+Spark's source and a running Spark are the only authorities.
+
+## Step 5: Review Existing SLT Coverage
+
+```bash
+ls datafusion/sqllogictest/test_files/spark/*/$ARGUMENTS.slt
+```
+
+Read the file and list the queries, the types exercised, which cases are
+commented out, and whether the file has an audit header comment from a
+previous audit.
+
+Also read `datafusion/sqllogictest/test_files/spark/README.md`. It is the
+binding local convention for this tree and it prescribes things this skill does
+not repeat: every function must be tested on both scalar and array inputs,
+every literal needs an explicit cast because DataFusion and Spark infer types
+differently, test cases contain `SELECT` statements only, and a function whose
+behavior differs under ANSI is wrapped in
+
+```sql
+statement ok
+set datafusion.execution.enable_ansi_mode = true;
+
+# queries
+
+statement ok
+set datafusion.execution.enable_ansi_mode = false;
+```
+
+Reset the flag to `false` at the end of the block. The setting leaks to
+whatever follows it in the file otherwise.
+
+Many files in this tree were machine-generated from Sail's gold data. Their
+expected values were not necessarily verified against Spark, so an existing
+passing expectation is not evidence of correctness. Re-verify in Step 7.
+
+## Step 6: Gap Analysis
+
+Build this matrix, filling each cell with yes, no, or n/a:
+
+| Dimension | Spark tests it | DataFusion SLT | Rust unit test | Gap? |
+| ---------------------------------------------- | -------------- | -------------- | -------------- | ---- |
+| All-scalar arguments | | | | |
+| All-array arguments | | | | |
+| Array first argument, scalar second | | | | |
+| Scalar first argument, array second | | | | |
+| NULL input, each argument independently | | | | |
+| Empty string / array / map | | | | |
+| Zero, negative zero, negatives | | | | |
+| Overflow, underflow | | | | |
+| Boundary values (INT_MIN, INT_MAX, i64 bounds) | | | | |
+| NaN, Infinity, -Infinity, subnormal | | | | |
+| Multibyte / special UTF-8 | | | | |
+| ANSI mode enabled | | | | |
+| ANSI mode disabled | | | | |
+| Every type in Spark's `inputTypes` | | | | |
+| Types Spark rejects | | | | |
+| Return nullability | | | | |
+| Cross-version divergence | | | | |
+
+Rows marked n/a need a one-line justification, for example "no float inputs"
+against the NaN row.
+
+Both ANSI rows matter independently. DataFusion defaults `enable_ansi_mode` to
+`false` while Spark 4.x defaults ANSI to `true`, so the mode users actually hit
+on Spark 4 is the one most likely to be untested here.
+
+## Step 7: Establish Ground Truth with PySpark
+
+No expected value goes into a `.slt` file without being observed from a real
+Spark. Reading `nullSafeEval` and predicting the output is exactly how wrong
+golden values get written in the first place.
+
+### Set up the environment
+
+Requires a JDK 17 or later on `PATH`. Use `uv` to create the venv rather than
+`python3 -m venv`: `pyspark==4.2.0` requires Python 3.10 or later, and the
+system `python3` on many machines is older. `uv` is already used by this
+repository and will fetch a suitable interpreter if none is installed.
+
+Unset `SPARK_HOME` and `PYSPARK_PYTHON` before doing anything else. If either
+is set, a local Spark install on `SPARK_HOME` silently overrides the
+pip-installed `pyspark` JARs, and `PYSPARK_PYTHON` can point PySpark's workers
+at a different Python than the venv you just built. The `SPARK_HOME` case is
+not hypothetical: it reproduces reliably on a machine with any other Spark
+install on `PATH`, and it fails late and unhelpfully, inside `getOrCreate()`,
+as `TypeError: 'JavaPackage' object is not callable`. If you hit that exact
+error, this is the cause: check `echo $SPARK_HOME` and unset it.
+
+**`unset` does not persist between tool calls.** Each shell invocation starts
+from the user's profile, which is where `SPARK_HOME` was exported in the first
+place. Unsetting it once in the setup block and then running the harness in a
+later call reintroduces the failure. Put `unset SPARK_HOME PYSPARK_PYTHON` in
+the same command as every JVM-starting invocation, not in a preceding one.
+
+```bash
+unset SPARK_HOME PYSPARK_PYTHON
+uv venv --python 3.12 /tmp/pyspark-4.2.0-venv
+uv pip install --python /tmp/pyspark-4.2.0-venv --quiet 'pyspark==4.2.0'
+/tmp/pyspark-4.2.0-venv/bin/python -c "import pyspark; print(pyspark.__version__)"
+```
+
+The last command must print `4.2.0`. If it prints anything else, stop and fix
+the environment: an audit against the wrong Spark version is worse than no
+audit, because it writes confident wrong expectations.
+
+That check is necessary but not sufficient. It only reads the installed
+package's metadata: it never starts a JVM, so it passes cleanly even on a
+machine primed to hit the `SPARK_HOME` shadowing failure above. The check
+that actually proves which Spark you are talking to is the
+`# Spark 4.2.0 tz=UTC` banner the harness prints from a live `SparkSession` in
+the next section. Do not trust the environment until you have seen that
+banner.
+
+### The harness
+
+Write this to the scratchpad as `spark_ground_truth.py`. It is deliberately not
+committed to the repo: `pyspark` is too heavy a dependency to add to the `uv`
+workspace in `dev/`, and `datafusion/spark/scripts/` is claimed by other
+in-flight work.
+
+Write it unconditionally, overwriting whatever is already there. A previous
+audit may have left an older copy in the same scratchpad, and silently reusing
+it means auditing with a harness whose fixes you do not have.
+
+```python
+#!/usr/bin/env python3
+"""Run SQL queries against a local PySpark and print SLT-ready results.
+
+Usage: spark_ground_truth.py QUERIES.sql
+
+Input is a file of SQL statements, each terminated by a semicolon at the end
+of a line. Each query is run under both spark.sql.ansi.enabled=true and
+=false, and the observed result is printed in a form that can be pasted into
+a .slt file.
+
+The session time zone is pinned to UTC so that results do not depend on the
+auditing machine's local zone. DataFusion's sqllogictest runner leaves
+execution.time_zone unset, which is effectively UTC, so anything time zone
+sensitive would otherwise produce golden values that are wrong everywhere
+except on the machine that generated them.
+
+Statement splitting is naive: it splits on a semicolon followed by a newline.
+That keeps a semicolon inside a string literal intact, as in
+SELECT ascii(';'), as long as the literal is not the last thing on the line.
+A query whose string literal ends a line with a semicolon still splits
+wrongly. Put such a query on a single line with trailing text after the
+literal, or run it on its own.
+"""
+
+import sys
+
+from pyspark.sql import SparkSession
+
+
+def format_value(value):
+ """Render one cell the way sqllogictest expects it."""
+ if value is None:
+ return "NULL"
+ if isinstance(value, bool):
+ return "true" if value else "false"
+ return str(value)
+
+
+def run_query(spark, sql):
+ try:
+ rows = spark.sql(sql).collect()
+ except Exception as exc: # noqa: BLE001 - any failure is a result here
+ first_line = str(exc).strip().split("\n")[0]
+ return f"ERROR: {type(exc).__name__}: {first_line}"
+ if not rows:
+ return "(no rows)"
+ return "\n".join("\t".join(format_value(v) for v in row) for row in rows)
+
+
+def main(path):
+ spark = (
+ SparkSession.builder.appName("slt-ground-truth")
+ .master("local[1]")
+ .getOrCreate()
+ )
+ spark.conf.set("spark.sql.session.timeZone", "UTC")
+ spark.sparkContext.setLogLevel("ERROR")
+ print(f"# Spark {spark.version} tz={spark.conf.get('spark.sql.session.timeZone')}")
+
+ raw = open(path).read()
+ if not raw.endswith("\n"):
+ raw += "\n"
+ queries = [q.strip() for q in raw.split(";\n") if q.strip()]
+
+ for sql in queries:
+ for ansi in (True, False):
+ spark.conf.set("spark.sql.ansi.enabled", str(ansi).lower())
+ print(f"### ansi={str(ansi).lower()}")
+ print(f"# {sql}")
+ print(run_query(spark, sql))
+ print()
+
+ spark.stop()
+
+
+if __name__ == "__main__":
+ if len(sys.argv) != 2:
+ sys.exit("usage: spark_ground_truth.py QUERIES.sql")
+ main(sys.argv[1])
+```
+
+### Using it
+
+Write every candidate query, including the ones already in the `.slt` file, to
+a scratch `.sql` file and run:
+
+```bash
+cd <scratchpad>
+unset SPARK_HOME PYSPARK_PYTHON
+/tmp/pyspark-4.2.0-venv/bin/python spark_ground_truth.py queries.sql \
+ > ground_truth.txt 2> spark_logs.txt
+```
+
+Keep the streams separate. Spark writes log4j output, including structured JSON
+error records, on a stream that will interleave with the results if you merge
+them with `2>&1`. The merged form is what an agent reaches for by reflex and it
+makes the output unreadable exactly on the error cases that matter most.
+
+Verify the banner line reads `# Spark 4.2.0 tz=UTC` before trusting any
+output. Both halves matter. The version half proves you are not talking to a
+Spark install that shadowed the pip-installed one, and the time zone half
+proves the session is not silently inheriting the auditing machine's local
+zone. DataFusion's sqllogictest runner leaves `execution.time_zone` unset,
+which behaves as UTC, so a session on any other zone generates golden values
+that only reproduce on the machine that generated them. This is invisible for
+functions that take a `DATE` and never touch a time zone, and it is a silent
+corruption for `to_utc_timestamp`, `date_format`, `from_unixtime`, and every
+other timestamp function. If the banner shows anything but `tz=UTC`, stop and
+fix the harness before recording a single expected value.
+
+**Wrap date, timestamp, and interval results in `CAST(... AS STRING)`.** The
+harness calls `collect()`, which converts every cell to a Python object, and
+Python's `datetime.date` cannot represent everything Spark's `DateType` can. A
+value outside Python's range surfaces as a line like
+
+```
+ERROR: ValueError: year 10000 is out of range
+```
+
+which reads exactly like a Spark error and is not one. Spark returned a value
+fine. Any `ValueError` or `OverflowError` in the output is a harness artifact,
+never a Spark result. Re-run the query wrapped in a cast to see what Spark
+actually produced:
+
+```sql
+SELECT CAST(next_day(DATE '9999-12-31', 'Mon') AS STRING);
+-- +10000-01-03
+```
+
+Re-verify expectations that already exist in the `.slt` file. Many files in
+this tree were machine-generated and their values were never checked against
+Spark. An existing expectation is a claim, not evidence.
+
+Write the queries in Spark SQL. The `.slt` file needs DataFusion syntax, which
+differs: DataFusion uses `'2015-01-14'::DATE` where Spark SQL uses
+`DATE '2015-01-14'`. Translate when transcribing, and keep the semantics
+identical. If a query cannot be expressed in both dialects, note it as a
+finding rather than silently testing something different.
+
+### Never use `--complete`
+
+The sqllogictest runner supports `--complete`, which fills in expected results
+with whatever DataFusion currently returns.
+
+**Never run it on files under `test_files/spark/`.** On an audit it would
+silently cement the bug being audited as the golden answer, which inverts the
+purpose of the whole exercise. Expected values come from PySpark and are
+written by hand.
+
+### If PySpark cannot be installed
+
+Continue the audit using values derived from the Spark source and the golden
+files from Step 2, and mark every such expectation as unverified in both the
+audit report and a comment in the `.slt` file:
+
+```sql
+# UNVERIFIED: derived from Spark source, not confirmed against a running Spark
+```
+
+Never present a derived value as a verified one. An audit that overstates its
+confidence is worse than one that admits a gap.
+
+## Step 8: Apply Findings
+
+Findings do not survive as prose. A recommendation in a PR description is
+buried the moment the PR merges. Every finding becomes code, a test, or a
+filed issue.
+
+### Fix what can be fixed
+
+Divergences from Spark 4.2.0 are fixed in the audit PR. Attempt the fix
+regardless of size. Fall back to filing an issue only when the fix needs a
+semantics decision a human should make, for example choosing which of two
+defensible behaviors DataFusion should adopt when Spark's own behavior is
+inconsistent across versions.
+
+Every fix is locked in by an SLT test that fails before the fix and passes
+after. A test that passes before the fix was testing something else.
+
+Verify both directions before committing. Stash only the implementation
+change, so the new `.slt` query runs against the unfixed code:
+
+```bash
+# Stash the implementation only, leaving the new .slt query in place
+git stash push -- datafusion/spark/src/function/<category>/<function>.rs
+cargo test --test sqllogictests -- spark/<category>/$ARGUMENTS # expect FAIL
+git stash pop
+cargo test --test sqllogictests -- spark/<category>/$ARGUMENTS # expect PASS
+```
+
+This verifies SLT tests only. Rust unit tests live inside the implementation
+file, so stashing that file stashes the tests with it and the run proves
+nothing about them. For a unit test, get the fail-before evidence by reverting
+the implementation body by hand, or by writing the test first and watching it
+fail before the fix exists.
+
+Read the failure list from the first run and confirm it names the queries you
+added. Do not judge the run by an exit code you captured through a pipe:
+`cargo test ... | head` reports the exit status of `head`, not of `cargo`, so a
+piped command that "succeeded" tells you nothing. Either run it unpiped or
+inspect the `External error: N errors in file ...` block directly.
+
+If the first run passes, the test is not exercising the fix. Either the
+query was stashed along with the implementation, or the assertion does not
+actually cover the divergent case. Investigate before continuing: a test
+that passes against the unfixed code locks in nothing.
+
+If the fix also changed a `mod.rs` doc string or a helper in another file, that
+file is not covered by the stash above. Add it to the `git stash push` argument
+list, or the "expect FAIL" run may fail for the wrong reason.
+
+Run this before committing the fix. If `git stash push` reports
+`No local changes to save`, the change is already committed and the check is
+invalid, because the "expect FAIL" run just tested the fixed code. Redo it
+against the pre-fix commit by checking that one file out of history and back,
+which touches only the working tree and never moves `HEAD` or a branch:
+
+```bash
+# <pre-fix-sha> is the commit before the one that introduced the fix
+git checkout <pre-fix-sha> -- datafusion/spark/src/function/<category>/<function>.rs
+cargo test --test sqllogictests -- spark/<category>/$ARGUMENTS # expect FAIL
+git checkout HEAD -- datafusion/spark/src/function/<category>/<function>.rs
+cargo test --test sqllogictests -- spark/<category>/$ARGUMENTS # expect PASS
+```
+
+Find `<pre-fix-sha>` with `git log --oneline -- <the implementation file>` and
+take the commit before the fix. If the fix also touched other files, list every
+one of them in both `git checkout` commands.
+
+Do not reach for `git reset`, `git revert`, or `git rebase` here. None of them
+are needed and all of them can leave the branch in a state the audit then has
+to repair.
+
+### Capture what cannot be fixed
+
+A deferred divergence becomes a commented-out SLT query carrying the correct
+Spark 4.2.0 result and the issue link:
+
+```sql
+# Spark 4.2.0 raises ILLEGAL_DAY_OF_WEEK under ANSI mode; DataFusion returns NULL
+# https://github.com/apache/datafusion/issues/23889 <- the real number, not a placeholder
+# query error
+# SELECT next_day('2015-01-14'::DATE, 'NOT_A_DAY'::string);
+```
+
+Uncommenting the block is the contract for verifying the eventual fix. Note
+that DataFusion `.slt` files use `#` for comments, not `--`.
+
+**File the issue first, then write the block.** The block cannot be written
+until its issue number exists, so do the filing described below before typing
+the comment. Never commit a placeholder in place of the number. A string like
+`<ISSUE-URL-PENDING>`, `NNNNN`, or `TBD` inside a URL ships a dead link, and
+because the surrounding block is commented out no test ever catches it.
+
+If the issue cannot be filed at all, for example because `gh` is not
+authenticated or the user asked that nothing be filed yet, omit the link line
+entirely. A block with no link is honest and self-contained. Then say so
+explicitly in the audit report: name the divergence, state that no issue was
+filed and why, and ask the user to file it. Do not leave the reader to
+discover the gap from a broken URL.
+
+When you do assert an error, remember that the runner normalizes whitespace in
+the `query error` regex, so literal leading, trailing, and repeated spaces in
+the expected message cannot be written out. An input like `' MO '` produces
+`Illegal input for day of week: MO .` with two spaces, and a regex containing
+those two spaces is collapsed to one before matching. Write `\s+` instead,
+which carries no literal whitespace and survives normalization:
+
+```sql
+query error Illegal input for day of week:\s+MO \.
+SELECT next_day('2015-01-14'::DATE, ' MO '::string);
+```
+
+That asserts the padding is present without pinning its exact width. Anchor
+the right-hand end of the match on a non-whitespace character from Spark's
+message, such as the trailing period in Spark 4.x error templates, so the
+assertion still fails if the padding runs into the rest of the message. If
+even that cannot be expressed, fall back to the stable prefix and say in a
+comment why the assertion stops where it does.
+
+Match Spark's message text exactly, including trailing punctuation. Spark 4.x
+error templates in `error-conditions.json` generally end in a period, and the
+pre-4.0 exception strings they replaced generally did not. Copying the older
+string leaves the implementation asserting a version the audit does not claim,
+and a `query error` prefix match will not catch it.
+
+Before filing, search for an existing issue:
+
+```bash
+gh issue list --repo apache/datafusion --search "$ARGUMENTS <symptom> in:title,body" --state all --limit 5
+```
+
+If a candidate comes back, open it with `gh issue view <N> --repo apache/datafusion`
+and confirm it actually describes this divergence and is still open. A closed
+issue cited as a known divergence is worse than no citation, because the reader
+follows the link and finds a fix that already shipped.
+
+If no issue exists, file one with the `bug` label plus `spark`. Title format:
+`[Bug] <function> <one-line symptom>`. The body includes the Spark version
+range affected, a minimal repro in both Spark SQL and DataFusion SQL, the
+divergent result, and the relevant file and line.
+
+### Capture cross-version divergences
+
+A behavior difference between Spark versions cannot be tested today. Capture it
+in two places.
+
+A comment at the query site naming the versions:
+
+```sql
+# Spark 3.5.8 returns NULL here; Spark 4.0.4 and later throw under ANSI mode.
+# Expectation below asserts 4.2.0. Version-specific expectations are tracked by
+# https://github.com/apache/datafusion/issues/23887
+```
+
+And its own filed issue, so the divergence is scheduled once the version
+mechanism lands.
+
+### Record the audit
+
+Add a header comment block to the `.slt` file, directly below the license
+header and the existing provenance comment:
+
+```sql
+# Audited against Spark 4.2.0 on YYYY-MM-DD.
+# Expected values verified with pyspark==4.2.0.
+# Known divergences are captured as commented-out queries below.
+```
+
+Use today's real date. If PySpark was unavailable, say so on the second line
+instead of claiming verification.
+
+### Attribution
+
+Every issue and PR the skill files ends with:
+
+```
+Surfaced by the audit-datafusion-spark-expression skill.
+```
+
+Do not mention Claude, AI, or automated generation anywhere in an issue or PR.
+
+### Before committing
+
+Per the repository `CLAUDE.md`:
+
+```bash
+cargo fmt --all
+cargo clippy --all-targets --all-features -- -D warnings
+cargo test -p datafusion-spark --lib $ARGUMENTS
+cargo test --test sqllogictests -- spark/<category>/$ARGUMENTS
+```
+
+The `--lib` run is not optional. `cargo test --test sqllogictests` does not
+compile or run the `#[test]` blocks inside `datafusion/spark/src/`, so a
+refactor that renames or changes the signature of a helper those tests call
+passes the SLT gate and fails CI.
+
+Do not use `cargo build -p datafusion-spark` as a quick compile check. The
+crate does not currently build in isolation and the resulting error is
+unrelated to your change. Use `cargo test -p datafusion-spark --lib` or the
+sqllogictest target instead.
+
+If the audit touched documentation:
+
+```bash
+./ci/scripts/doc_prettier_check.sh --write --allow-dirty
+```
+
+Follow `.github/pull_request_template.md` for the PR.
+
+## Output Format
+
+Present the audit as:
+
+1. **Function summary.** What `$ARGUMENTS` does, its input and output types,
+ and its null behavior.
+2. **Spark version differences.** Across 3.5.8, 4.0.4, 4.1.3, and 4.2.0. Say
+ explicitly if there are none.
+3. **DataFusion implementation notes.** Including signature, coercion, and
+ nullability findings.
+4. **Gap matrix.** From Step 6, plus implementation gaps.
+5. **Findings.** Split into fixed in this PR, deferred with a filed issue, and
+ low-risk coverage gaps.
+6. **Offer to add the low-risk coverage tests.**
+
+Low-risk coverage gaps are the only category that pauses for user input.
+Correctness findings are always either fixed or filed.
+
+The line between the two is the code you touched. Coverage for a path this PR
+changed goes into the `.slt` file without asking: it is part of the fix. The
+offer is for cases on paths the PR did not touch and that already look correct,
+where adding tests is optional scope the user may not want. Phrase the offer
+as:
+
+> Spark exercises the following cases that have no DataFusion test, on code
+> paths that appear correct. Would you like me to add them?
+>
+> - [list each case]
+
+## Tone and Style
+
+- Write in clear, concise prose.
+- Use backticks around function names, file paths, type names, and config keys.
+- Acknowledge what is already well covered before raising gaps.
+- Avoid em dashes and semicolons. Use separate sentences.
+- Do not overstate confidence. An unverified expectation is labeled as one.