Skip to content

Apache DataFusion SQL Query Engine

docs: Add SKILL.md for designing benchmark (#25285)

Yongting You56361282fcf1
diff --git a/.ai/skills/add-benchmarks/SKILL.md b/.ai/skills/add-benchmarks/SKILL.md
new file mode 100644
index 0000000000..d25ec6a057
--- /dev/null
+++ b/.ai/skills/add-benchmarks/SKILL.md
@@ -0,0 +1,84 @@
+---
+name: add-benchmarks
+description: Guidelines for designing or extending DataFusion benchmarks.
+---
+
+# Benchmark Design
+
+Follow these guidelines when designing or adding a new benchmark.
+
+## Design principles
+
+### Use the highest-level interface
+
+Use the highest-level interface possible, preferably SQL, while keeping the work
+around the operator being measured as cheap as possible. When benchmarking a
+function, exercise its evaluation path instead of benchmarking internal utility
+functions in isolation.
+
+This makes benchmarks easier to maintain and helps assess how much an optimization
+matters to end-to-end runtime. It also helps avoid spending time optimizing code
+that accounts for only a small fraction of the total runtime.
+
+Practical criteria for choosing SQL or Rust benchmarks: Try implementing the benchmark in SQL first. If a Rust microbenchmark still seems like a better fit, use Criterion.
+
+### Vary the key workload axes
+
+First identify the key axes to vary. For example, for a join benchmark:
+
+- Input size on each side.
+- Join-filter selectivity.
+- ...
+
+Then choose benchmark cases that exercise representative variations. Full
+combinatorial coverage is unnecessary; focus on typical workloads that reflect
+real use cases.
+
+When adding benchmark queries, simply tag each query with the decision made
+for every axis. For example, for a join benchmark with input sizes/filter selectivity
+to tune:
+
+```sql
+-- Q1: Small left input, large right input; 0.1% of pairs match.
+SELECT *
+FROM generate_series(1, 100) AS l
+JOIN generate_series(1, 100000) AS r
+ ON (l.value + r.value) % 1000 = 0;
+
+-- Q2: Medium inputs on both sides; no filter.
+SELECT *
+FROM generate_series(1, 1000) AS l
+CROSS JOIN generate_series(1, 1000) AS r;
+```
+
+## SQL benchmarks
+
+For implementation details, see the
+[SQL benchmark README](../../../benchmarks/sql_benchmarks/README.md).
+
+1. **Keep other operators cheap.**
+
+ When a SQL benchmark targets a specific operator, keep the work done by
+ other operators as lightweight as possible. For example, use a data source
+ such as `generate_series()` instead of a Parquet scan so scan overhead does
+ not dominate the measurement. See the [`nlj` benchmark](../../../benchmarks/sql_benchmarks/nlj/) for examples.
+
+2. **Integrate with the top-level benchmark script.**
+
+ Ensure the benchmark can be prepared and run through `bench.sh`:
+
+ ```bash
+ # Run from the benchmarks directory.
+
+ # Generate any required dataset.
+ ./bench.sh data new_bench
+
+ # Run the benchmark.
+ ./bench.sh run new_bench
+ ```
+
+3. **Keep query runtimes practical.**
+
+ Tune the workload so each query takes roughly a few seconds per execution.
+ This helps reduce the relative impact of timing noise while keeping the
+ suite practical to run.
\ No newline at end of file