PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
feat: use OpenS2V for diffusion calibration (#2368)
Wang, Chang924b39873bb3
diff --git a/auto_round/compressors/diffusion/README.md b/auto_round/compressors/diffusion/README.mdindex ba6b53beb..255cab04e 100644--- a/auto_round/compressors/diffusion/README.md+++ b/auto_round/compressors/diffusion/README.md@@ -10,7 +10,7 @@ Quantization for diffusion models is limited:1. Only the transformer module of diffusion models is quantized.2. Use `auto_round` (the default) for a packed, reloadable checkpoint; use `fake` only for research or debugging.-3. Calibration dataset currently supports `coco2014` and user customized `.tsv` files.+3. Calibration datasets currently support `opens2v`, `coco2014`, and user customized `.tsv` files.### API Usage (CPU/GPU) Recommended@@ -34,7 +34,8 @@ output_dir = "./tmp_autoround"autoround.quantize_and_save(output_dir, format="auto_round", inplace=True)```-- `dataset`: the dataset for quantization training. Currently supports `coco2014` and user customized `.tsv` files.+- `dataset`: the dataset for quantization training. Currently supports `opens2v`, `coco2014`, and user customized+ `.tsv` files.- `calib_num_inference_steps`: the number of inference steps requested when the scheduler builds its nativeshort calibration schedule. Higher-order schedulers may expand these into more internal timesteps.- `num_inference_steps`: the number of denoising steps used for diffusion generation/evaluation; it does not@@ -80,27 +81,28 @@ For diffusion models, currently we validate quantization on the following models| stepfun-ai/NextStep-1.1 | COCO2014 | - || AIDC-AI/Ovis-Image-7B | COCO2014 | - || stabilityai/stable-audio-open-1.0 | audiocaps | - |-| Wan-AI/Wan2.2-I2V-A14B-Diffusers | COCO2014 | - |-| Wan-AI/Wan2.2-TI2V-5B-Diffusers | COCO2014 | - |-| Wan-AI/Wan2.2-T2V-A14B-Diffusers | COCO2014 | - |+| Wan-AI/Wan2.2-I2V-A14B-Diffusers | OpenS2V | - |+| Wan-AI/Wan2.2-TI2V-5B-Diffusers | OpenS2V | - |+| Wan-AI/Wan2.2-T2V-A14B-Diffusers | OpenS2V | - || nvidia/Cosmos3-Nano | COCO2014 | - || nvidia/Cosmos3-Super | COCO2014 | - |<details><summary style="font-size:17px;">Calibration Dataset</summary>-For diffusion models, we use [**coco2014**](https://github.com/mlcommons/inference/raw/refs/heads/master/text_to_image/coco2014/captions/captions_source.tsv) calibration dataset as the default.+For diffusion models, we use the compact [**OpenS2V calibration set**](https://huggingface.co/datasets/changwangss/opens2v-calibration) by default. Pass `--dataset coco2014` to use COCO2014 explicitly.To use a custom dataset, build a `.tsv` file with the following structure and pass it through `--dataset`:```text-id caption-0 YOUR_PROMPT-1 YOUR_PROMPT-... ...+id caption image+0 YOUR_PROMPT OPTIONAL_IMAGE_PATH+1 YOUR_PROMPT OPTIONAL_IMAGE_PATH+... ... ...```- `id`: the id used to map generated images and prompts.- `caption`: the text prompt used to generate the images.+- `image`: an optional local image path required by image-to-video pipelines.</details>diff --git a/auto_round/compressors/diffusion/README_CN.md b/auto_round/compressors/diffusion/README_CN.mdnew file mode 100644index 000000000..bec0b8400--- /dev/null+++ b/auto_round/compressors/diffusion/README_CN.md@@ -0,0 +1,15 @@+# AutoRound 扩散模型量化(实验性)++本功能为实验性功能,目前仅量化扩散模型的 transformer 模块。++使用默认的 `auto_round` 格式导出打包后可重新加载的检查点;`fake` 格式仅用于研究或调试。++```python+autoround.quantize_and_save("./tmp_autoround", format="auto_round", inplace=True)+```++校准默认使用精简的 [OpenS2V 校准集](https://huggingface.co/datasets/changwangss/opens2v-calibration)。+也可通过 `--dataset coco2014` 显式选择 COCO2014,或传入自定义 `.tsv` 文件。+自定义数据需要包含 `id` 和 `caption` 列;图生视频还需要 `image` 列提供参考图像路径。++完整 API、CLI 示例及支持模型列表见[英文说明](README.md)。diff --git a/auto_round/compressors/diffusion/dataset.py b/auto_round/compressors/diffusion/dataset.pyindex f30a4203f..0b73b177b 100644--- a/auto_round/compressors/diffusion/dataset.py+++ b/auto_round/compressors/diffusion/dataset.py@@ -15,6 +15,7 @@import jsonimport osimport zipfile+from concurrent.futures import ThreadPoolExecutorfrom io import StringIOfrom pathlib import Pathfrom typing import Dict, Optional@@ -36,6 +37,12 @@ COCO_URL = {)}+OPENS2V_DATASET_REVISION = "09a03003234151c14c7e0989e5a0c125039c9e0d"+OPENS2V_ASSET_BASE_URL = (+ f"https://huggingface.co/datasets/changwangss/opens2v-calibration/resolve/{OPENS2V_DATASET_REVISION}/data"+)+OPENS2V_MANIFEST_URL = f"{OPENS2V_ASSET_BASE_URL}/opens2v_calibration.tsv"+COCO_ANNOTATIONS_URL = "https://s3.amazonaws.com/images.cocodataset.org/annotations/annotations_trainval2014.zip"COCO_CAPTIONS_MEMBER = "annotations/captions_val2014.json"COCO_ALLOWED_LICENSE_ID = 4 # Creative Commons Attribution 2.0@@ -51,6 +58,16 @@ def _get_coco_cache_dir() -> Path:return cache_root / "datasets" / "coco2014"+def _get_opens2v_cache_dir() -> Path:+ """Return the persistent cache directory for OpenS2V calibration data."""+ from auto_round import envs as _envs++ cache_root = (+ Path(_envs.AUTO_ROUND_CACHE).expanduser() if _envs.AUTO_ROUND_CACHE else Path.home() / ".cache" / "auto_round"+ )+ return cache_root / "datasets" / "opens2v"++def _download_to_cache(url: str, destination: Path, timeout: int) -> None:"""Download a file atomically unless a non-empty cached copy exists."""if destination.is_file() and destination.stat().st_size > 0:@@ -152,6 +169,38 @@ def _load_coco_dataframe(dataset: str, nsamples: int, image_required: bool) -> preturn selected+def _load_opens2v_dataframe(nsamples: int, image_required: bool) -> pd.DataFrame:+ """Load the compact OpenS2V manifest and cache reference images only for I2V."""+ cache_dir = _get_opens2v_cache_dir()+ manifest_path = cache_dir / "opens2v_calibration.tsv"+ _download_to_cache(OPENS2V_MANIFEST_URL, manifest_path, timeout=30)+ dataframe = pd.read_csv(manifest_path, sep="\t")++ required_cols = {"id", "caption", "image"}+ if not required_cols.issubset(dataframe.columns):+ raise ValueError(f"OpenS2V calibration requires columns {sorted(required_cols)}.")++ selected = dataframe.iloc[:nsamples].copy() if nsamples > 0 else dataframe.copy()+ if not image_required:+ return selected.drop(columns="image")++ image_paths = []+ logger.info(f"Caching {len(selected)} OpenS2V reference images for I2V calibration in {cache_dir / 'images'}")+ downloads = []+ for image_reference in selected["image"]:+ image_reference = str(image_reference).lstrip("/")+ image_path = cache_dir / "images" / Path(image_reference).name+ image_url = f"{OPENS2V_ASSET_BASE_URL}/{image_reference}"+ downloads.append((image_url, image_path))+ image_paths.append(str(image_path))+ with ThreadPoolExecutor(max_workers=min(8, len(downloads) or 1)) as executor:+ futures = [executor.submit(_download_to_cache, url, path, 60) for url, path in downloads]+ for future in futures:+ future.result()+ selected["image"] = image_paths+ return selected++def register_dataset(name_list):"""Class decorator to register a DATASET subclass to the registry.@@ -267,7 +316,7 @@ class AudioCapsDataset(Dataset):return self.caption_ids[i], self.captions[i]-def get_diffusion_dataloader(dataset="coco2014", bs=1, seed=42, nsamples=128, image_required=False):+def get_diffusion_dataloader(dataset="opens2v", bs=1, seed=42, nsamples=128, image_required=False):"""Generate a DataLoader for calibration using specified parameters.Args:Dataset_name (str): The name or path of the dataset.@@ -281,6 +330,11 @@ def get_diffusion_dataloader(dataset="coco2014", bs=1, seed=42, nsamples=128, imdataframe = _load_coco_dataframe(dataset, nsamples, image_required)dataset = DIFFUSION_DATASET["local"](dataset, nsamples, dataframe=dataframe)+ if dataset == "opens2v":+ logger.info("use dataset opens2v, loading calibration data...")+ dataframe = _load_opens2v_dataframe(nsamples, image_required)+ dataset = DIFFUSION_DATASET["local"]("opens2v", nsamples, dataframe=dataframe)+if dataset in ("audiocaps",):dataset = download_audiocaps_csv()@@ -292,7 +346,7 @@ def get_diffusion_dataloader(dataset="coco2014", bs=1, seed=42, nsamples=128, imelse:dataset = DIFFUSION_DATASET["local"](dataset, nsamples)else:- raise ValueError("Only support coco2014/audiocaps dataset or loading local tsv/csv file now.")+ raise ValueError("Only support opens2v/coco2014/audiocaps dataset or loading local tsv/csv file now.")if (image_requireddiff --git a/auto_round/compressors/diffusion_mixin.py b/auto_round/compressors/diffusion_mixin.pyindex 4f5ba2dd7..415bfcbc7 100644--- a/auto_round/compressors/diffusion_mixin.py+++ b/auto_round/compressors/diffusion_mixin.py@@ -88,9 +88,9 @@ class DiffusionMixin:self.generator_seed = generator_seedself.pipeline_call_kwargs = dict(kwargs.pop("pipeline_call_kwargs", {}) or {})- # Default dataset for diffusion models is "coco2014", not "NeelNanda/pile-10k"+ # Use video-domain calibration prompts by default for diffusion models.if kwargs.get("dataset") in (None, "NeelNanda/pile-10k"):- kwargs["dataset"] = "coco2014"+ kwargs["dataset"] = "opens2v"iters = kwargs.get("iters", None)_alg_cfg = args[0] if args else Nonediff --git a/test/unit/common/models/test_diffusion_dataset.py b/test/unit/common/models/test_diffusion_dataset.pyindex e22bb37bc..f42941eee 100644--- a/test/unit/common/models/test_diffusion_dataset.py+++ b/test/unit/common/models/test_diffusion_dataset.py@@ -3,7 +3,12 @@ import zipfileimport pytest-from auto_round.compressors.diffusion.dataset import DIFFUSION_DATASET, _load_coco_dataframe, get_diffusion_dataloader+from auto_round.compressors.diffusion.dataset import (+ DIFFUSION_DATASET,+ _load_coco_dataframe,+ _load_opens2v_dataframe,+ get_diffusion_dataloader,+)def test_text2img_dataset_rejects_missing_required_columns(tmp_path):@@ -88,3 +93,59 @@ def test_coco_i2v_filters_cc_by_before_selecting_samples(monkeypatch, tmp_path):assert dataframe.iloc[0]["flickr_url"] == "https://flickr.test/102"assert dataframe.iloc[0]["license_url"] == "http://creativecommons.org/licenses/by/2.0/"assert dataframe.iloc[0]["image"].endswith("/images/102.jpg")+++def test_opens2v_t2v_loads_prompts_without_downloading_images(monkeypatch, tmp_path):+ manifest = "id\tcaption\timage\n1\thello\timages/1.jpg\n2\tworld\timages/2.jpg\n"+ downloaded_urls = []++ def _fake_download(url, destination, timeout):+ downloaded_urls.append(url)+ destination.parent.mkdir(parents=True, exist_ok=True)+ destination.write_text(manifest, encoding="utf-8")++ monkeypatch.setattr("auto_round.compressors.diffusion.dataset._get_opens2v_cache_dir", lambda: tmp_path)+ monkeypatch.setattr("auto_round.compressors.diffusion.dataset._download_to_cache", _fake_download)++ dataframe = _load_opens2v_dataframe(nsamples=2, image_required=False)++ assert dataframe.to_dict("records") == [{"id": 1, "caption": "hello"}, {"id": 2, "caption": "world"}]+ assert len(downloaded_urls) == 1+++def test_opens2v_i2v_caches_only_selected_reference_images(monkeypatch, tmp_path):+ manifest = "id\tcaption\timage\n1\thello\timages/1.jpg\n2\tworld\timages/2.jpg\n"+ downloaded_urls = []++ def _fake_download(url, destination, timeout):+ downloaded_urls.append(url)+ destination.parent.mkdir(parents=True, exist_ok=True)+ if destination.suffix == ".tsv":+ destination.write_text(manifest, encoding="utf-8")+ else:+ destination.write_bytes(b"image")++ monkeypatch.setattr("auto_round.compressors.diffusion.dataset._get_opens2v_cache_dir", lambda: tmp_path)+ monkeypatch.setattr("auto_round.compressors.diffusion.dataset._download_to_cache", _fake_download)++ dataframe = _load_opens2v_dataframe(nsamples=1, image_required=True)++ assert dataframe[["id", "caption"]].to_dict("records") == [{"id": 1, "caption": "hello"}]+ assert dataframe.iloc[0]["image"] == str(tmp_path / "images" / "1.jpg")+ assert downloaded_urls[-1].endswith("/data/images/1.jpg")+ assert len(downloaded_urls) == 2+++def test_get_diffusion_dataloader_defaults_to_opens2v(monkeypatch):+ calls = []++ def _fake_load(nsamples, image_required):+ calls.append((nsamples, image_required))+ return __import__("pandas").DataFrame({"id": [1], "caption": ["hello"]})++ monkeypatch.setattr("auto_round.compressors.diffusion.dataset._load_opens2v_dataframe", _fake_load)++ dataloader, _ = get_diffusion_dataloader(nsamples=1)++ assert calls == [(1, False)]+ assert dataloader.dataset.captions == ["hello"]