Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

feat: use OpenS2V for diffusion calibration (#2368)

Wang, Chang924b39873bb3
diff --git a/auto_round/compressors/diffusion/README.md b/auto_round/compressors/diffusion/README.md
index ba6b53beb..255cab04e 100644
--- a/auto_round/compressors/diffusion/README.md
+++ b/auto_round/compressors/diffusion/README.md
@@ -10,7 +10,7 @@ Quantization for diffusion models is limited:
1. Only the transformer module of diffusion models is quantized.
2. Use `auto_round` (the default) for a packed, reloadable checkpoint; use `fake` only for research or debugging.
-3. Calibration dataset currently supports `coco2014` and user customized `.tsv` files.
+3. Calibration datasets currently support `opens2v`, `coco2014`, and user customized `.tsv` files.
### API Usage (CPU/GPU) Recommended
@@ -34,7 +34,8 @@ output_dir = "./tmp_autoround"
autoround.quantize_and_save(output_dir, format="auto_round", inplace=True)
```
-- `dataset`: the dataset for quantization training. Currently supports `coco2014` and user customized `.tsv` files.
+- `dataset`: the dataset for quantization training. Currently supports `opens2v`, `coco2014`, and user customized
+ `.tsv` files.
- `calib_num_inference_steps`: the number of inference steps requested when the scheduler builds its native
short calibration schedule. Higher-order schedulers may expand these into more internal timesteps.
- `num_inference_steps`: the number of denoising steps used for diffusion generation/evaluation; it does not
@@ -80,27 +81,28 @@ For diffusion models, currently we validate quantization on the following models
| stepfun-ai/NextStep-1.1 | COCO2014 | - |
| AIDC-AI/Ovis-Image-7B | COCO2014 | - |
| stabilityai/stable-audio-open-1.0 | audiocaps | - |
-| Wan-AI/Wan2.2-I2V-A14B-Diffusers | COCO2014 | - |
-| Wan-AI/Wan2.2-TI2V-5B-Diffusers | COCO2014 | - |
-| Wan-AI/Wan2.2-T2V-A14B-Diffusers | COCO2014 | - |
+| Wan-AI/Wan2.2-I2V-A14B-Diffusers | OpenS2V | - |
+| Wan-AI/Wan2.2-TI2V-5B-Diffusers | OpenS2V | - |
+| Wan-AI/Wan2.2-T2V-A14B-Diffusers | OpenS2V | - |
| nvidia/Cosmos3-Nano | COCO2014 | - |
| nvidia/Cosmos3-Super | COCO2014 | - |
<details>
<summary style="font-size:17px;">Calibration Dataset</summary>
-For diffusion models, we use [**coco2014**](https://github.com/mlcommons/inference/raw/refs/heads/master/text_to_image/coco2014/captions/captions_source.tsv) calibration dataset as the default.
+For diffusion models, we use the compact [**OpenS2V calibration set**](https://huggingface.co/datasets/changwangss/opens2v-calibration) by default. Pass `--dataset coco2014` to use COCO2014 explicitly.
To use a custom dataset, build a `.tsv` file with the following structure and pass it through `--dataset`:
```text
-id caption
-0 YOUR_PROMPT
-1 YOUR_PROMPT
-... ...
+id caption image
+0 YOUR_PROMPT OPTIONAL_IMAGE_PATH
+1 YOUR_PROMPT OPTIONAL_IMAGE_PATH
+... ... ...
```
- `id`: the id used to map generated images and prompts.
- `caption`: the text prompt used to generate the images.
+- `image`: an optional local image path required by image-to-video pipelines.
</details>
diff --git a/auto_round/compressors/diffusion/README_CN.md b/auto_round/compressors/diffusion/README_CN.md
new file mode 100644
index 000000000..bec0b8400
--- /dev/null
+++ b/auto_round/compressors/diffusion/README_CN.md
@@ -0,0 +1,15 @@
+# AutoRound 扩散模型量化(实验性)
+
+本功能为实验性功能,目前仅量化扩散模型的 transformer 模块。
+
+使用默认的 `auto_round` 格式导出打包后可重新加载的检查点;`fake` 格式仅用于研究或调试。
+
+```python
+autoround.quantize_and_save("./tmp_autoround", format="auto_round", inplace=True)
+```
+
+校准默认使用精简的 [OpenS2V 校准集](https://huggingface.co/datasets/changwangss/opens2v-calibration)。
+也可通过 `--dataset coco2014` 显式选择 COCO2014,或传入自定义 `.tsv` 文件。
+自定义数据需要包含 `id` 和 `caption` 列;图生视频还需要 `image` 列提供参考图像路径。
+
+完整 API、CLI 示例及支持模型列表见[英文说明](README.md)。
diff --git a/auto_round/compressors/diffusion/dataset.py b/auto_round/compressors/diffusion/dataset.py
index f30a4203f..0b73b177b 100644
--- a/auto_round/compressors/diffusion/dataset.py
+++ b/auto_round/compressors/diffusion/dataset.py
@@ -15,6 +15,7 @@
import json
import os
import zipfile
+from concurrent.futures import ThreadPoolExecutor
from io import StringIO
from pathlib import Path
from typing import Dict, Optional
@@ -36,6 +37,12 @@ COCO_URL = {
)
}
+OPENS2V_DATASET_REVISION = "09a03003234151c14c7e0989e5a0c125039c9e0d"
+OPENS2V_ASSET_BASE_URL = (
+ f"https://huggingface.co/datasets/changwangss/opens2v-calibration/resolve/{OPENS2V_DATASET_REVISION}/data"
+)
+OPENS2V_MANIFEST_URL = f"{OPENS2V_ASSET_BASE_URL}/opens2v_calibration.tsv"
+
COCO_ANNOTATIONS_URL = "https://s3.amazonaws.com/images.cocodataset.org/annotations/annotations_trainval2014.zip"
COCO_CAPTIONS_MEMBER = "annotations/captions_val2014.json"
COCO_ALLOWED_LICENSE_ID = 4 # Creative Commons Attribution 2.0
@@ -51,6 +58,16 @@ def _get_coco_cache_dir() -> Path:
return cache_root / "datasets" / "coco2014"
+def _get_opens2v_cache_dir() -> Path:
+ """Return the persistent cache directory for OpenS2V calibration data."""
+ from auto_round import envs as _envs
+
+ cache_root = (
+ Path(_envs.AUTO_ROUND_CACHE).expanduser() if _envs.AUTO_ROUND_CACHE else Path.home() / ".cache" / "auto_round"
+ )
+ return cache_root / "datasets" / "opens2v"
+
+
def _download_to_cache(url: str, destination: Path, timeout: int) -> None:
"""Download a file atomically unless a non-empty cached copy exists."""
if destination.is_file() and destination.stat().st_size > 0:
@@ -152,6 +169,38 @@ def _load_coco_dataframe(dataset: str, nsamples: int, image_required: bool) -> p
return selected
+def _load_opens2v_dataframe(nsamples: int, image_required: bool) -> pd.DataFrame:
+ """Load the compact OpenS2V manifest and cache reference images only for I2V."""
+ cache_dir = _get_opens2v_cache_dir()
+ manifest_path = cache_dir / "opens2v_calibration.tsv"
+ _download_to_cache(OPENS2V_MANIFEST_URL, manifest_path, timeout=30)
+ dataframe = pd.read_csv(manifest_path, sep="\t")
+
+ required_cols = {"id", "caption", "image"}
+ if not required_cols.issubset(dataframe.columns):
+ raise ValueError(f"OpenS2V calibration requires columns {sorted(required_cols)}.")
+
+ selected = dataframe.iloc[:nsamples].copy() if nsamples > 0 else dataframe.copy()
+ if not image_required:
+ return selected.drop(columns="image")
+
+ image_paths = []
+ logger.info(f"Caching {len(selected)} OpenS2V reference images for I2V calibration in {cache_dir / 'images'}")
+ downloads = []
+ for image_reference in selected["image"]:
+ image_reference = str(image_reference).lstrip("/")
+ image_path = cache_dir / "images" / Path(image_reference).name
+ image_url = f"{OPENS2V_ASSET_BASE_URL}/{image_reference}"
+ downloads.append((image_url, image_path))
+ image_paths.append(str(image_path))
+ with ThreadPoolExecutor(max_workers=min(8, len(downloads) or 1)) as executor:
+ futures = [executor.submit(_download_to_cache, url, path, 60) for url, path in downloads]
+ for future in futures:
+ future.result()
+ selected["image"] = image_paths
+ return selected
+
+
def register_dataset(name_list):
"""Class decorator to register a DATASET subclass to the registry.
@@ -267,7 +316,7 @@ class AudioCapsDataset(Dataset):
return self.caption_ids[i], self.captions[i]
-def get_diffusion_dataloader(dataset="coco2014", bs=1, seed=42, nsamples=128, image_required=False):
+def get_diffusion_dataloader(dataset="opens2v", bs=1, seed=42, nsamples=128, image_required=False):
"""Generate a DataLoader for calibration using specified parameters.
Args:
Dataset_name (str): The name or path of the dataset.
@@ -281,6 +330,11 @@ def get_diffusion_dataloader(dataset="coco2014", bs=1, seed=42, nsamples=128, im
dataframe = _load_coco_dataframe(dataset, nsamples, image_required)
dataset = DIFFUSION_DATASET["local"](dataset, nsamples, dataframe=dataframe)
+ if dataset == "opens2v":
+ logger.info("use dataset opens2v, loading calibration data...")
+ dataframe = _load_opens2v_dataframe(nsamples, image_required)
+ dataset = DIFFUSION_DATASET["local"]("opens2v", nsamples, dataframe=dataframe)
+
if dataset in ("audiocaps",):
dataset = download_audiocaps_csv()
@@ -292,7 +346,7 @@ def get_diffusion_dataloader(dataset="coco2014", bs=1, seed=42, nsamples=128, im
else:
dataset = DIFFUSION_DATASET["local"](dataset, nsamples)
else:
- raise ValueError("Only support coco2014/audiocaps dataset or loading local tsv/csv file now.")
+ raise ValueError("Only support opens2v/coco2014/audiocaps dataset or loading local tsv/csv file now.")
if (
image_required
diff --git a/auto_round/compressors/diffusion_mixin.py b/auto_round/compressors/diffusion_mixin.py
index 4f5ba2dd7..415bfcbc7 100644
--- a/auto_round/compressors/diffusion_mixin.py
+++ b/auto_round/compressors/diffusion_mixin.py
@@ -88,9 +88,9 @@ class DiffusionMixin:
self.generator_seed = generator_seed
self.pipeline_call_kwargs = dict(kwargs.pop("pipeline_call_kwargs", {}) or {})
- # Default dataset for diffusion models is "coco2014", not "NeelNanda/pile-10k"
+ # Use video-domain calibration prompts by default for diffusion models.
if kwargs.get("dataset") in (None, "NeelNanda/pile-10k"):
- kwargs["dataset"] = "coco2014"
+ kwargs["dataset"] = "opens2v"
iters = kwargs.get("iters", None)
_alg_cfg = args[0] if args else None
diff --git a/test/unit/common/models/test_diffusion_dataset.py b/test/unit/common/models/test_diffusion_dataset.py
index e22bb37bc..f42941eee 100644
--- a/test/unit/common/models/test_diffusion_dataset.py
+++ b/test/unit/common/models/test_diffusion_dataset.py
@@ -3,7 +3,12 @@ import zipfile
import pytest
-from auto_round.compressors.diffusion.dataset import DIFFUSION_DATASET, _load_coco_dataframe, get_diffusion_dataloader
+from auto_round.compressors.diffusion.dataset import (
+ DIFFUSION_DATASET,
+ _load_coco_dataframe,
+ _load_opens2v_dataframe,
+ get_diffusion_dataloader,
+)
def test_text2img_dataset_rejects_missing_required_columns(tmp_path):
@@ -88,3 +93,59 @@ def test_coco_i2v_filters_cc_by_before_selecting_samples(monkeypatch, tmp_path):
assert dataframe.iloc[0]["flickr_url"] == "https://flickr.test/102"
assert dataframe.iloc[0]["license_url"] == "http://creativecommons.org/licenses/by/2.0/"
assert dataframe.iloc[0]["image"].endswith("/images/102.jpg")
+
+
+def test_opens2v_t2v_loads_prompts_without_downloading_images(monkeypatch, tmp_path):
+ manifest = "id\tcaption\timage\n1\thello\timages/1.jpg\n2\tworld\timages/2.jpg\n"
+ downloaded_urls = []
+
+ def _fake_download(url, destination, timeout):
+ downloaded_urls.append(url)
+ destination.parent.mkdir(parents=True, exist_ok=True)
+ destination.write_text(manifest, encoding="utf-8")
+
+ monkeypatch.setattr("auto_round.compressors.diffusion.dataset._get_opens2v_cache_dir", lambda: tmp_path)
+ monkeypatch.setattr("auto_round.compressors.diffusion.dataset._download_to_cache", _fake_download)
+
+ dataframe = _load_opens2v_dataframe(nsamples=2, image_required=False)
+
+ assert dataframe.to_dict("records") == [{"id": 1, "caption": "hello"}, {"id": 2, "caption": "world"}]
+ assert len(downloaded_urls) == 1
+
+
+def test_opens2v_i2v_caches_only_selected_reference_images(monkeypatch, tmp_path):
+ manifest = "id\tcaption\timage\n1\thello\timages/1.jpg\n2\tworld\timages/2.jpg\n"
+ downloaded_urls = []
+
+ def _fake_download(url, destination, timeout):
+ downloaded_urls.append(url)
+ destination.parent.mkdir(parents=True, exist_ok=True)
+ if destination.suffix == ".tsv":
+ destination.write_text(manifest, encoding="utf-8")
+ else:
+ destination.write_bytes(b"image")
+
+ monkeypatch.setattr("auto_round.compressors.diffusion.dataset._get_opens2v_cache_dir", lambda: tmp_path)
+ monkeypatch.setattr("auto_round.compressors.diffusion.dataset._download_to_cache", _fake_download)
+
+ dataframe = _load_opens2v_dataframe(nsamples=1, image_required=True)
+
+ assert dataframe[["id", "caption"]].to_dict("records") == [{"id": 1, "caption": "hello"}]
+ assert dataframe.iloc[0]["image"] == str(tmp_path / "images" / "1.jpg")
+ assert downloaded_urls[-1].endswith("/data/images/1.jpg")
+ assert len(downloaded_urls) == 2
+
+
+def test_get_diffusion_dataloader_defaults_to_opens2v(monkeypatch):
+ calls = []
+
+ def _fake_load(nsamples, image_required):
+ calls.append((nsamples, image_required))
+ return __import__("pandas").DataFrame({"id": [1], "caption": ["hello"]})
+
+ monkeypatch.setattr("auto_round.compressors.diffusion.dataset._load_opens2v_dataframe", _fake_load)
+
+ dataloader, _ = get_diffusion_dataloader(nsamples=1)
+
+ assert calls == [(1, False)]
+ assert dataloader.dataset.captions == ["hello"]