Apache DataFusion SQL Query Engine
docs: explain Parquet content-defined chunking (#24155)
Goutam Adwantd5868e683435
diff --git a/docs/source/index.rst b/docs/source/index.rstindex b939be86a0..0e3f56a7e1 100644--- a/docs/source/index.rst+++ b/docs/source/index.rst@@ -126,6 +126,7 @@ To get started, seeuser-guide/sql/indexuser-guide/configsuser-guide/explain-usage+ user-guide/parquet-content-defined-chunkinguser-guide/metricsuser-guide/faqdiff --git a/docs/source/user-guide/parquet-content-defined-chunking.md b/docs/source/user-guide/parquet-content-defined-chunking.mdnew file mode 100644index 0000000000..d456200e69--- /dev/null+++ b/docs/source/user-guide/parquet-content-defined-chunking.md@@ -0,0 +1,163 @@+<!---+ Licensed to the Apache Software Foundation (ASF) under one+ or more contributor license agreements. See the NOTICE file+ distributed with this work for additional information+ regarding copyright ownership. The ASF licenses this file+ to you under the Apache License, Version 2.0 (the+ "License"); you may not use this file except in compliance+ with the License. You may obtain a copy of the License at++ http://www.apache.org/licenses/LICENSE-2.0++ Unless required by applicable law or agreed to in writing,+ software distributed under the License is distributed on an+ "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY+ KIND, either express or implied. See the License for the+ specific language governing permissions and limitations+ under the License.+-->++# Parquet Content-Defined Chunking++Content-defined chunking (CDC) is an experimental Parquet writer feature that+makes data page boundaries depend on column values rather than fixed row or byte+counts. This makes unchanged regions more likely to produce identical pages when+closely related versions of a dataset are written with the same settings.++CDC is useful when the resulting files are stored or transferred through a+content-addressable or block-deduplicating system. Such a system can reuse the+identical pages instead of storing or transferring them again. For example, a+small insertion near the beginning of a dataset can change one page while later+page boundaries converge back to those of the previous version.++CDC does not itself deduplicate data or provide a page store. On a conventional+filesystem or object store, each Parquet file is still stored in full. The output+is a normal Parquet file and requires no CDC-specific reader support.++## When to enable CDC++Consider CDC when all of the following apply:++- You regularly write similar versions of the same dataset.+- Your storage or transfer layer detects and reuses duplicate byte ranges.+- Reducing storage or network transfer is more important than maximizing write+ parallelism for an individual file.++Leave CDC disabled for ordinary Parquet output unless you have measured a benefit+in the system that stores or transfers the files. CDC is disabled by default.++When CDC is enabled, DataFusion uses the sequential Arrow writer for each output+file because the chunker's state must persist across row groups. This can reduce+write throughput compared with DataFusion's parallel writer path. Writing+different output files can still proceed concurrently.++CDC operates independently for each output file. When `COPY` targets a+directory, DataFusion distributes input RecordBatches in round-robin order across+parallel output files; `datafusion.execution.minimum_parallel_output_files`+defaults to four. If batching or file assignment changes between dataset+versions, unchanged rows can move between files and reduce deduplication. For+the best results, keep the input order and output file layout stable. Use a+filename target for single-file output, or partition by stable keys when+multiple files are required. See+[Configuration Settings](configs.md#setting-configuration-options).++## Enable CDC with SQL++Set CDC for one [`COPY`](sql/dml.md#copy) operation with Parquet format options.+The filename target in this example selects single-file output:++```sql+COPY (+ SELECT+ value AS id,+ CONCAT('event-', CAST(value AS VARCHAR)) AS event+ FROM generate_series(1, 100000)+) TO 'cdc-output.parquet'+STORED AS PARQUET+OPTIONS (+ 'format.content_defined_chunking.enabled' 'true'+);+```++The default chunking parameters are a good starting point. The next example+specifies those defaults explicitly for one write; it does not change their+values:++```sql+COPY source_table TO 'cdc-output.parquet'+STORED AS PARQUET+OPTIONS (+ 'format.content_defined_chunking.enabled' 'true',+ 'format.content_defined_chunking.min_chunk_size' '262144',+ 'format.content_defined_chunking.max_chunk_size' '1048576',+ 'format.content_defined_chunking.norm_level' '0'+);+```++Change these values only after measuring with representative data.++You can instead enable CDC for subsequent Parquet writes in the session:++```sql+SET datafusion.execution.parquet.content_defined_chunking.enabled = true;+```++The corresponding environment variable is+`DATAFUSION_EXECUTION_PARQUET_CONTENT_DEFINED_CHUNKING_ENABLED`. See+[Configuration Settings](configs.md#setting-configuration-options) for all ways+to set session options.++## Enable CDC with the Rust API++Pass [`TableParquetOptions`] to [`DataFrame::write_parquet`]:++```rust+use datafusion::config::{ParquetCdcOptions, TableParquetOptions};+use datafusion::dataframe::DataFrameWriteOptions;+use datafusion::error::Result;+use datafusion::prelude::SessionContext;++#[tokio::main]+async fn main() -> Result<()> {+ let ctx = SessionContext::new();+ let df = ctx+ .sql("SELECT value AS id FROM generate_series(1, 100000)")+ .await?;++ let mut parquet_options = TableParquetOptions::default();+ parquet_options.global.content_defined_chunking = ParquetCdcOptions::enabled();++ df.write_parquet(+ "cdc-output.parquet",+ DataFrameWriteOptions::new().with_single_file_output(true),+ Some(parquet_options),+ )+ .await?;++ Ok(())+}+```++Set the fields of `ParquetCdcOptions` directly to use non-default chunk sizes or+normalization.++## Tuning++| Option | Default | Effect |+| ---------------- | ------- | -------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| `min_chunk_size` | 256 KiB | Minimum logical size before the rolling hash can select a boundary. |+| `max_chunk_size` | 1 MiB | Maximum logical size before the writer forces a boundary. It must be greater than `min_chunk_size`. |+| `norm_level` | `0` | Controls how aggressively boundaries are selected. Higher values can improve deduplication but create more small pages; recommended range is `-3` through `3`. |++Chunk sizes are measured from logical column data before encoding and+compression. Definition and repetition levels for nested data also count toward+the size.++Use the same CDC, encoding, compression, and schema settings when comparing+dataset versions. Changing writer settings can change the page bytes and reduce+deduplication even when the logical data is unchanged. Measure the deduplication+ratio, output size, network transfer, and write time with representative data+before changing the defaults.++[`dataframe::write_parquet`]: https://docs.rs/datafusion/latest/datafusion/dataframe/struct.DataFrame.html#method.write_parquet+[`tableparquetoptions`]: https://docs.rs/datafusion/latest/datafusion/common/config/struct.TableParquetOptions.htmldiff --git a/docs/source/user-guide/sql/format_options.md b/docs/source/user-guide/sql/format_options.mdindex ca79858dae..719fd5bd6b 100644--- a/docs/source/user-guide/sql/format_options.md+++ b/docs/source/user-guide/sql/format_options.md@@ -164,6 +164,10 @@ The following options are available when reading or writing Parquet files. If an| ALLOW_SINGLE_FILE_PARALLELISM | No | Enables parallel serialization of columns in a single file. | `'allow_single_file_parallelism'` | true || MAXIMUM_PARALLEL_ROW_GROUP_WRITERS | No | Maximum number of parallel row group writers. | `'maximum_parallel_row_group_writers'` | 1 || MAXIMUM_BUFFERED_RECORD_BATCHES_PER_STREAM | No | Maximum number of buffered record batches per stream. | `'maximum_buffered_record_batches_per_stream'` | 2 |+| CONTENT_DEFINED_CHUNKING_ENABLED | No | Enables experimental content-defined chunking when writing Parquet files. Enabling it uses the sequential writer for each output file so chunker state persists across row groups. See [Parquet Content-Defined Chunking](../parquet-content-defined-chunking.md). | `'content_defined_chunking.enabled'` | false |+| CONTENT_DEFINED_CHUNKING_MIN_CHUNK_SIZE | No | Minimum logical size in bytes before the rolling hash can select a chunk boundary. | `'content_defined_chunking.min_chunk_size'` | 262144 (256 KiB) |+| CONTENT_DEFINED_CHUNKING_MAX_CHUNK_SIZE | No | Maximum logical size in bytes before the writer forces a chunk boundary. Must be greater than `content_defined_chunking.min_chunk_size`. | `'content_defined_chunking.max_chunk_size'` | 1048576 (1 MiB) |+| CONTENT_DEFINED_CHUNKING_NORM_LEVEL | No | Controls how aggressively chunk boundaries are selected. Higher values can improve deduplication but increase fragmentation. The recommended range is `-3` through `3`. | `'content_defined_chunking.norm_level'` | 0 || KEY_VALUE_METADATA | No (Key is specific) | Adds custom key-value pairs to the file metadata. Use the format `'metadata::your_key_name' 'your_value'`. Multiple entries allowed. | `'metadata::key_name'` | None |**Example:**