Skip to content

Apache DataFusion SQL Query Engine

docs: Make the preference for sqllogictest over Rust unit tests more visible (#25756)

Andrew Lamb266b827f5e60
diff --git a/AGENTS.md b/AGENTS.md
index 8fdf314ed4..c35c66a3a1 100644
--- a/AGENTS.md
+++ b/AGENTS.md
@@ -4,6 +4,7 @@
- [Quick Start Setup](docs/source/contributor-guide/development_environment.md#quick-start)
- [Testing Quick Start](docs/source/contributor-guide/testing.md#testing-quick-start)
+- [Choosing What Kind of Test to Write](docs/source/contributor-guide/testing.md#choosing-what-kind-of-test-to-write)
- [Before Submitting a PR](docs/source/contributor-guide/index.md#before-submitting-a-pr)
- [Reviewing Pull Requests](docs/source/contributor-guide/pr_review.md)
- [Contributor Guide](docs/source/contributor-guide/index.md)
@@ -38,7 +39,10 @@ When creating a PR, you MUST follow the [PR template](.github/pull_request_templ
## Testing
-If documentation files changed then run
+When adding tests, you MUST follow
+[Choosing What Kind of Test to Write](docs/source/contributor-guide/testing.md#choosing-what-kind-of-test-to-write):
+
+If documentation files changed then run
```bash
./ci/scripts/doc_prettier_check.sh --write --allow-dirty
```
diff --git a/docs/source/contributor-guide/index.md b/docs/source/contributor-guide/index.md
index 6f1a0f1c19..d6a16680dd 100644
--- a/docs/source/contributor-guide/index.md
+++ b/docs/source/contributor-guide/index.md
@@ -122,7 +122,12 @@ before merge.
# ./dev/rust_lint.sh --write --allow-dirty
```
-You should also run any relevant commands from the [testing quick start](testing.md#testing-quick-start).
+Please ensure your PR follows the [testing guide](testing.md). In particular:
+
+- Prefer end-to-end Public API tests such as `sqllogictest` (`.slt`) and DataFrame API,
+ over Rust unit tests where possible. See
+ [Choosing What Kind of Test to Write](testing.md#choosing-what-kind-of-test-to-write).
+- Run any relevant commands from the [testing quick start](testing.md#testing-quick-start).
## Conventional Commits & Labeling PRs
diff --git a/docs/source/contributor-guide/pr_review.md b/docs/source/contributor-guide/pr_review.md
index 1154ae0e96..6166a741aa 100644
--- a/docs/source/contributor-guide/pr_review.md
+++ b/docs/source/contributor-guide/pr_review.md
@@ -133,7 +133,8 @@ Guidelines for evaluating tests:
1. Prefer `sqllogictest` (`.slt`) tests or DataFrame API tests where
possible, as they exercise **user-visible behavior** and are less coupled
- to internal implementation details than unit tests.
+ to internal implementation details than unit tests (see
+ [Choosing What Kind of Test to Write](testing.md#choosing-what-kind-of-test-to-write)).
2. Verify tests cover edge cases and common failure scenarios, not just the
common successful path. However, it is NOT necessary to test every possible
error path, especially if it is difficult to trigger or unlikely to occur in
diff --git a/docs/source/contributor-guide/testing.md b/docs/source/contributor-guide/testing.md
index 4c097ae498..3e39ef2eb1 100644
--- a/docs/source/contributor-guide/testing.md
+++ b/docs/source/contributor-guide/testing.md
@@ -74,6 +74,30 @@ cargo nextest run
[the book]: https://doc.rust-lang.org/book/
[cargo-nextest]: https://nexte.st/
+## Choosing What Kind of Test to Write
+
+When adding tests, prefer tests that exercise **user-visible behavior** over
+tests of internal implementation details. Doing so means your tests are more
+likely to keep working (and keep catching regressions) while the code is
+refactored. In rough order of preference:
+
+1. Use [sqllogictest](#sqllogictests-tests) (`.slt`) tests for any behavior that can
+ be expressed in SQL.
+
+2. End-to-end tests that use public APIs such as the DataFrame API, for behavior
+ that cannot be expressed in SQL. For example, DataFrame tests live in
+ [`datafusion/core/tests/dataframe`] and run as part of the `core_integration`
+ test binary:
+
+ ```shell
+ cargo test -p datafusion --test core_integration -- dataframe
+ ```
+
+3. Rust unit tests for logic that cannot practically be reached through
+ SQL or DataFrame public APIs.
+
+[`datafusion/core/tests/dataframe`]: https://github.com/apache/datafusion/tree/main/datafusion/core/tests/dataframe
+
## Unit tests
Tests for code in an individual module are defined in the same source file with a `test` module, following Rust convention.