Skip to content

Apache DataFusion SQL Query Engine

docs: explain Parquet content-defined chunking (#24155)

Goutam Adwantd5868e683435
diff --git a/docs/source/index.rst b/docs/source/index.rst
index b939be86a0..0e3f56a7e1 100644
--- a/docs/source/index.rst
+++ b/docs/source/index.rst
@@ -126,6 +126,7 @@ To get started, see
user-guide/sql/index
user-guide/configs
user-guide/explain-usage
+ user-guide/parquet-content-defined-chunking
user-guide/metrics
user-guide/faq
diff --git a/docs/source/user-guide/parquet-content-defined-chunking.md b/docs/source/user-guide/parquet-content-defined-chunking.md
new file mode 100644
index 0000000000..d456200e69
--- /dev/null
+++ b/docs/source/user-guide/parquet-content-defined-chunking.md
@@ -0,0 +1,163 @@
+<!---
+ Licensed to the Apache Software Foundation (ASF) under one
+ or more contributor license agreements. See the NOTICE file
+ distributed with this work for additional information
+ regarding copyright ownership. The ASF licenses this file
+ to you under the Apache License, Version 2.0 (the
+ "License"); you may not use this file except in compliance
+ with the License. You may obtain a copy of the License at
+
+ http://www.apache.org/licenses/LICENSE-2.0
+
+ Unless required by applicable law or agreed to in writing,
+ software distributed under the License is distributed on an
+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
+ KIND, either express or implied. See the License for the
+ specific language governing permissions and limitations
+ under the License.
+-->
+
+# Parquet Content-Defined Chunking
+
+Content-defined chunking (CDC) is an experimental Parquet writer feature that
+makes data page boundaries depend on column values rather than fixed row or byte
+counts. This makes unchanged regions more likely to produce identical pages when
+closely related versions of a dataset are written with the same settings.
+
+CDC is useful when the resulting files are stored or transferred through a
+content-addressable or block-deduplicating system. Such a system can reuse the
+identical pages instead of storing or transferring them again. For example, a
+small insertion near the beginning of a dataset can change one page while later
+page boundaries converge back to those of the previous version.
+
+CDC does not itself deduplicate data or provide a page store. On a conventional
+filesystem or object store, each Parquet file is still stored in full. The output
+is a normal Parquet file and requires no CDC-specific reader support.
+
+## When to enable CDC
+
+Consider CDC when all of the following apply:
+
+- You regularly write similar versions of the same dataset.
+- Your storage or transfer layer detects and reuses duplicate byte ranges.
+- Reducing storage or network transfer is more important than maximizing write
+ parallelism for an individual file.
+
+Leave CDC disabled for ordinary Parquet output unless you have measured a benefit
+in the system that stores or transfers the files. CDC is disabled by default.
+
+When CDC is enabled, DataFusion uses the sequential Arrow writer for each output
+file because the chunker's state must persist across row groups. This can reduce
+write throughput compared with DataFusion's parallel writer path. Writing
+different output files can still proceed concurrently.
+
+CDC operates independently for each output file. When `COPY` targets a
+directory, DataFusion distributes input RecordBatches in round-robin order across
+parallel output files; `datafusion.execution.minimum_parallel_output_files`
+defaults to four. If batching or file assignment changes between dataset
+versions, unchanged rows can move between files and reduce deduplication. For
+the best results, keep the input order and output file layout stable. Use a
+filename target for single-file output, or partition by stable keys when
+multiple files are required. See
+[Configuration Settings](configs.md#setting-configuration-options).
+
+## Enable CDC with SQL
+
+Set CDC for one [`COPY`](sql/dml.md#copy) operation with Parquet format options.
+The filename target in this example selects single-file output:
+
+```sql
+COPY (
+ SELECT
+ value AS id,
+ CONCAT('event-', CAST(value AS VARCHAR)) AS event
+ FROM generate_series(1, 100000)
+) TO 'cdc-output.parquet'
+STORED AS PARQUET
+OPTIONS (
+ 'format.content_defined_chunking.enabled' 'true'
+);
+```
+
+The default chunking parameters are a good starting point. The next example
+specifies those defaults explicitly for one write; it does not change their
+values:
+
+```sql
+COPY source_table TO 'cdc-output.parquet'
+STORED AS PARQUET
+OPTIONS (
+ 'format.content_defined_chunking.enabled' 'true',
+ 'format.content_defined_chunking.min_chunk_size' '262144',
+ 'format.content_defined_chunking.max_chunk_size' '1048576',
+ 'format.content_defined_chunking.norm_level' '0'
+);
+```
+
+Change these values only after measuring with representative data.
+
+You can instead enable CDC for subsequent Parquet writes in the session:
+
+```sql
+SET datafusion.execution.parquet.content_defined_chunking.enabled = true;
+```
+
+The corresponding environment variable is
+`DATAFUSION_EXECUTION_PARQUET_CONTENT_DEFINED_CHUNKING_ENABLED`. See
+[Configuration Settings](configs.md#setting-configuration-options) for all ways
+to set session options.
+
+## Enable CDC with the Rust API
+
+Pass [`TableParquetOptions`] to [`DataFrame::write_parquet`]:
+
+```rust
+use datafusion::config::{ParquetCdcOptions, TableParquetOptions};
+use datafusion::dataframe::DataFrameWriteOptions;
+use datafusion::error::Result;
+use datafusion::prelude::SessionContext;
+
+#[tokio::main]
+async fn main() -> Result<()> {
+ let ctx = SessionContext::new();
+ let df = ctx
+ .sql("SELECT value AS id FROM generate_series(1, 100000)")
+ .await?;
+
+ let mut parquet_options = TableParquetOptions::default();
+ parquet_options.global.content_defined_chunking = ParquetCdcOptions::enabled();
+
+ df.write_parquet(
+ "cdc-output.parquet",
+ DataFrameWriteOptions::new().with_single_file_output(true),
+ Some(parquet_options),
+ )
+ .await?;
+
+ Ok(())
+}
+```
+
+Set the fields of `ParquetCdcOptions` directly to use non-default chunk sizes or
+normalization.
+
+## Tuning
+
+| Option | Default | Effect |
+| ---------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| `min_chunk_size` | 256 KiB | Minimum logical size before the rolling hash can select a boundary. |
+| `max_chunk_size` | 1 MiB | Maximum logical size before the writer forces a boundary. It must be greater than `min_chunk_size`. |
+| `norm_level` | `0` | Controls how aggressively boundaries are selected. Higher values can improve deduplication but create more small pages; recommended range is `-3` through `3`. |
+
+Chunk sizes are measured from logical column data before encoding and
+compression. Definition and repetition levels for nested data also count toward
+the size.
+
+Use the same CDC, encoding, compression, and schema settings when comparing
+dataset versions. Changing writer settings can change the page bytes and reduce
+deduplication even when the logical data is unchanged. Measure the deduplication
+ratio, output size, network transfer, and write time with representative data
+before changing the defaults.
+
+[`dataframe::write_parquet`]: https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html#method.write_parquet
+[`tableparquetoptions`]: https://docs.rs/datafusion/latest/datafusion/common/config/struct.TableParquetOptions.html
diff --git a/docs/source/user-guide/sql/format_options.md b/docs/source/user-guide/sql/format_options.md
index ca79858dae..719fd5bd6b 100644
--- a/docs/source/user-guide/sql/format_options.md
+++ b/docs/source/user-guide/sql/format_options.md
@@ -164,6 +164,10 @@ The following options are available when reading or writing Parquet files. If an
| ALLOW_SINGLE_FILE_PARALLELISM | No | Enables parallel serialization of columns in a single file. | `'allow_single_file_parallelism'` | true |
| MAXIMUM_PARALLEL_ROW_GROUP_WRITERS | No | Maximum number of parallel row group writers. | `'maximum_parallel_row_group_writers'` | 1 |
| MAXIMUM_BUFFERED_RECORD_BATCHES_PER_STREAM | No | Maximum number of buffered record batches per stream. | `'maximum_buffered_record_batches_per_stream'` | 2 |
+| CONTENT_DEFINED_CHUNKING_ENABLED | No | Enables experimental content-defined chunking when writing Parquet files. Enabling it uses the sequential writer for each output file so chunker state persists across row groups. See [Parquet Content-Defined Chunking](../parquet-content-defined-chunking.md). | `'content_defined_chunking.enabled'` | false |
+| CONTENT_DEFINED_CHUNKING_MIN_CHUNK_SIZE | No | Minimum logical size in bytes before the rolling hash can select a chunk boundary. | `'content_defined_chunking.min_chunk_size'` | 262144 (256 KiB) |
+| CONTENT_DEFINED_CHUNKING_MAX_CHUNK_SIZE | No | Maximum logical size in bytes before the writer forces a chunk boundary. Must be greater than `content_defined_chunking.min_chunk_size`. | `'content_defined_chunking.max_chunk_size'` | 1048576 (1 MiB) |
+| CONTENT_DEFINED_CHUNKING_NORM_LEVEL | No | Controls how aggressively chunk boundaries are selected. Higher values can improve deduplication but increase fragmentation. The recommended range is `-3` through `3`. | `'content_defined_chunking.norm_level'` | 0 |
| KEY_VALUE_METADATA | No (Key is specific) | Adds custom key-value pairs to the file metadata. Use the format `'metadata::your_key_name' 'your_value'`. Multiple entries allowed. | `'metadata::key_name'` | None |
**Example:**