PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
update doc (#1856)
Wenhua Chengf06f9a1629fd
diff --git a/README.md b/README.mdindex 380704574..06f61a812 100644--- a/README.md+++ b/README.md@@ -36,9 +36,11 @@ See our papers [SignRoundV1](https://arxiv.org/pdf/2309.05516) and [SignRoundV2]## 🆕 What's New+* [2026/05] We provide **free** devices for calibration-free quantization via pure RTN mode; please visit [Intel Low Bit Open LLM Leaderboard](https://huggingface.co/spaces/Intel/low_bit_open_llm_leaderboard) for more details.+* [2026/05] **Model free** quantization is available, `auto-round-rtn` will now default to using the model-free approach: [Doc](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#model-free-mode).-* [2026/03] **Block-wise FP8** quantization is available via `auto-round-rtn --scheme FP8_BLOCK`.+* [2026/03] **Block-wise FP8** quantization is available and rtn mode is recommended. `auto-round-rtn --scheme FP8_BLOCK`.* [2026/03] **MTP layer quantization** has been supported in this [PR](https://github.com/intel/auto-round/pull/1526)@@ -90,7 +92,7 @@ Quantize 7B models in about 10 minutes on a single GPU. Details are shown in [quOut-of-the-box quantization for 10+ vision-language models [example models](https://huggingface.co/collections/OPEA/vlms-autoround-675bc712fdd6a55ebaf11bfa), [support matrix](https://github.com/intel/auto-round/tree/main/auto_round/compressors/mllm#vlm-support-matrix)✅ **Multiple Recipes**-Choose from `auto-round-best`, `auto-round`, and `auto-round-light` to suit your needs. Details are shown in [quantization recipes](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#recipe-recommendation)+Choose from `auto-round-best`, `auto-round`, `auto-round-light`, `auto-round-opt-rtn` (optimized RTN) and `auto-round-rtn` (pure RTN, fastest baseline) to suit your needs. Details are shown in [quantization recipes](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#recipe-recommendation)✅ Advanced UtilitiesIncludes [multiple gpus quantization](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#devicemulti-gpu-setting-in-quantization), [multiple calibration datasets](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#default-dataset) and support for [10+ runtime backends](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#specify-inference-backend).@@ -174,6 +176,21 @@ auto-round-light \```+ ```bash+# Optimized RTN (iters=0, opt_rtn enabled); fast baseline+auto-round-opt-rtn \+ --model Qwen/Qwen3-0.6B \+ --scheme "W4A16"+ ```++ ```bash+# Pure RTN (iters=0, no AutoRound optimization); fastest, lowest memory+# auto-routes to model-free mode for supported INT WOQ schemes+auto-round-rtn \+ --model Qwen/Qwen3-0.6B \+ --scheme "W4A16"+ ```+<!-- ```bashauto-round-fast \# Fast and low memory, 2-3X speedup, slight accuracy drop at W4G128diff --git a/README_CN.md b/README_CN.mdindex 645d8ca6d..85760179e 100644--- a/README_CN.md+++ b/README_CN.md@@ -35,7 +35,9 @@ AutoRound 是专为大语言模型(LLMs)和视觉-语言模型(VLMs)设## 🆕 最新进展-* [2026/05] 现已支持**免模型量化**模式, `auto-round-rtn`现在会默认启用免模型方式: [文档](https://github.com/intel/auto-round/blob/main/docs/step_by_step_CN.md#%E5%85%8D%E6%A8%A1%E5%9E%8B%E9%87%8F%E5%8C%96%E6%A8%A1%E5%BC%8F).+* [2026/05] 我们提供**免费**的硬件资源,用于通过纯 RTN 方法进行模型量化。更多详情请访问 [Intel Low Bit Open LLM Leaderboard](https://huggingface.co/spaces/Intel/low_bit_open_llm_leaderboard)。++* [2026/05] 现已支持**免模型架构量化**模式, `auto-round-rtn` 现在会默认启用免模型方式: [文档](https://github.com/intel/auto-round/blob/main/docs/step_by_step_CN.md#%E5%85%8D%E6%A8%A1%E5%9E%8B%E6%9E%B6%E6%9E%84%E9%87%8F%E5%8C%96%E6%A8%A1%E5%BC%8F).* [2026/03] 支持**Block-wise FP8**,只需要将scheme改成 `--scheme FP8_BLOCK`.为了更快的量化,我们推荐使用RTN模式,即`auto-round-rtn`@@ -81,7 +83,7 @@ AutoRound 是专为大语言模型(LLMs)和视觉-语言模型(VLMs)设✅ **支持十余种 VLM 模型** 已支持十余种视觉语言模型,让用户有“开盖即食”般的量化体验。详见:[示例模型](https://huggingface.co/collections/OPEA/vlms-autoround-675bc712fdd6a55ebaf11bfa),[支持矩阵](https://github.com/intel/auto-round/tree/main/auto_round/compressors/mllm#vlm-support-matrix)-✅ **多种量化方案可选** 提供 `auto-round-best`、`auto-round`、`auto-round-light` 等多种预设方案,能够满足多样化需求。详见:[量化方案](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#recipe-recommendation)+✅ **多种量化方案可选** 提供 `auto-round-best`、`auto-round`、`auto-round-light`、`auto-round-opt-rtn`(优化版 RTN)以及 `auto-round-rtn`(原始 RTN,最快基线)等多种预设方案,能够满足多样化需求。详见:[量化方案](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#recipe-recommendation)✅ **实用额外特性** 支持[多 GPU 量化](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#devicemulti-gpu-setting-in-quantization)和[多标定数据集](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#default-dataset),并兼容[十余种推理后端](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#specify-inference-backend)。@@ -162,6 +164,21 @@ auto-round-light \--scheme "W4A16"```+ ```bash+# 无需标定的优化版 RTN(iters=0,启用 opt_rtn),快速基线方案+auto-round-opt-rtn \+ --model Qwen/Qwen3-0.6B \+ --scheme "W4A16"+ ```++ ```bash+# 原始 RTN(iters=0,不做任何 AutoRound 优化),速度最快、显存占用最低+# 对受支持的 INT WOQ scheme 会自动路由到 model-free 模式+auto-round-rtn \+ --model Qwen/Qwen3-0.6B \+ --scheme "W4A16"+ ```+<!-- ```bashauto-round-fast \# Fast and low memory, 2-3X speedup, slight accuracy drop at W4G128diff --git a/docs/step_by_step.md b/docs/step_by_step.mdindex 562b88d0e..4a2a8052d 100644--- a/docs/step_by_step.md+++ b/docs/step_by_step.md@@ -27,14 +27,14 @@ This document presents step-by-step instructions for auto-round llm quantization- [API Usage](#api-usage-1)- [Hyperparameters in AutoScheme](#hyperparameters-in-autoscheme)+ [OPT RTN mode](#opt-rtn-mode)- + [AWQ Algorithm](#awq-algorithm)+ + [AWQ Algorithm-Experimental](#awq-algorithm)+ [Model-Free Mode](#model-free-mode)+ [GGUF format](#gguf-format)+ [Quantization Costs](#quantization-costs)+ [Device/Multi-GPU setting in Quantization](#devicemulti-gpu-setting-in-quantization)- [Enable multiple gpus calibration in lm_head quantization](#enable-multiple-gpus-calibration-in-lm_head-quantization)+ [Adjust Hyperparameters](#adjust-hyperparameters)- + [Hadamard Transform](#hadamard-transform)+ + [Hadamard Transform-Research Feature](#hadamard-transform)* [4 Inference](#4-inference)+ [CPU](#cpu)+ [Intel GPU](#intel-gpu)@@ -156,7 +156,7 @@ adopted within the community, **only 4-bits quantization is supported**. Please**LLM-Compressor Format**: **NVFP4, MXFP4(kernel in WIP), MXFP8 are supported**. Please set `--format llm_compressor`-**MLX Format**: This format targets Apple Silicon (M1/M2/M3/...) and is loaded directly by [`mlx-lm`](https://github.com/ml-explore/mlx-lm) (text-only LLM) or [`mlx-vlm`](https://github.com/Blaizzy/mlx-vlm) (vision/audio + language).+**MLX Format**[Experimental Feature]: This format targets Apple Silicon (M1/M2/M3/...) and is loaded directly by [`mlx-lm`](https://github.com/ml-explore/mlx-lm) (text-only LLM) or [`mlx-vlm`](https://github.com/Blaizzy/mlx-vlm) (vision/audio + language).- Supports **2, 3, 4, 5, 6, 8 bits** (5/6 bits are MLX-exclusive — GPTQ/AWQ have no standard packing for them).- Native **mixed-bit / mixed-group_size** via `layer_config` or AutoScheme (`--target_bits 3.5 --options "..."`);- Use `--format mlx` for a native MLX checkpoint; use `--format auto_round:mlx` if you want HuggingFace `transformers` + AutoRound to load it (post-init repacks each layer into MLX `QuantLinear` on Darwin).@@ -165,16 +165,16 @@ adopted within the community, **only 4-bits quantization is supported**. Please> Gray indicates the absence of a kernel or the presence of only an inefficient/reference kernel. BF16 is mainly for AutoScheme-| Format | Supported Schemes |-|:---|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-| **auto_round** | W4A16, W2A16, W3A16, W8A16, W2A16G64, W2A16G32, `MXFP4`, `MXFP8`, `MXFP4_RCEIL`, `MXFP8_RCEIL`, `NVFP4`, `FPW8A16`, `FP8_STATIC`, `FP8_BLOCK`, `BF16`, `MXINT4` |-| **auto_awq** | W4A16, BF16 |-| **auto_gptq** | W4A16, W2A16, W3A16, W8A16,W2A16G64, W2A16G32, BF16 |-| **llm_compressor** | NVFP4, `MXFP4`, `MXFP8`, `FPW8A16`, `FP8_STATIC`, FP8_BLOCK |+| Format | Supported Schemes |+|:-----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------|+| **auto_round** | W4A16, W2A16, W3A16, W8A16, W2A16G64, W2A16G32, `MXFP4`, `MXFP8`, `MXFP4_RCEIL`, `MXFP8_RCEIL`, `NVFP4`, `FPW8A16`, `FP8_STATIC`, `FP8_BLOCK`, `BF16`, `MXINT4` |+| **auto_awq** | W4A16, BF16 |+| **auto_gptq** | W4A16, W2A16, W3A16, W8A16,W2A16G64, W2A16G32, BF16 |+| **llm_compressor** | NVFP4, `MXFP4`, `MXFP8`, `FPW8A16`, `FP8_STATIC`, FP8_BLOCK || **mlx** / **auto_round:mlx** | W2A16, W3A16, W4A16, W5A16, W6A16, W8A16, BF16, mixed-bit / mixed-group_size (Apple Silicon only) |-| **gguf** | GGUF:Q4_K_M, GGUF:Q2_K_S, GGUF:Q3_K_S, GGUF:Q3_K_M, GGUF:Q3_K_L, GGUF:Q4_K_S, GGUF:Q5_K_S, GGUF:Q5_K_M, GGUF:Q6_K, GGUF:Q4_0, GGUF:Q4_1, GGUF:Q5_0, GGUF:Q5_1,GGUF:Q8_0 |-| **fp8** | FP8_BLOCK |-| **fake** | `all schemes (only for research)` |+| **gguf** | GGUF:Q4_K_M, GGUF:Q2_K_S, GGUF:Q3_K_S, GGUF:Q3_K_M, GGUF:Q3_K_L, GGUF:Q4_K_S, GGUF:Q5_K_S, GGUF:Q5_K_M, GGUF:Q6_K, GGUF:Q4_0, GGUF:Q4_1, GGUF:Q5_0, GGUF:Q5_1,GGUF:Q8_0 |+| **fp8** | FP8_BLOCK |+| **fake** | `all schemes (only for research)` |### Hardware Compatibility@@ -211,6 +211,22 @@ Before starting quantization, you may want to configure AutoRound's environmentauto-round-light --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_gptq,auto_awq,auto_round"```+- **AutoRoundOptRTN recipe (optimized RTN, without gradient computation):**++ This setting runs the optimized RTN (Round-To-Nearest) path (`iters=0` with `disable_opt_rtn=False`). It is calibration-free and several times faster than the default AutoRound recipe, while still applying AutoRound's RTN-side optimizations (e.g. improved scale/zero-point search and llamacpp-style refinements for GGUF). Recommended as a fast baseline when calibration data or tuning time is limited. See the [OPT RTN Mode](#opt-rtn-mode) section for details.++ ```bash+ auto-round-opt-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"+ ```++- **AutoRoundRTN recipe (pure RTN, calibration-free, no optimization):**++ This setting runs pure RTN (`iters=0` with `disable_opt_rtn=True`), without any AutoRound optimization. It is the fastest path and uses the least memory, but typically yields lower accuracy than `auto-round-opt-rtn`. When combined with a supported INT WOQ scheme, it is automatically routed through [Model-Free Mode](#model-free-mode) for minimal memory usage. Use this as a quick sanity-check or when you want a calibration-free baseline equivalent to traditional RTN.++ ```bash+ auto-round-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"+ ```+### API usage#### AutoRound API UsageThis setting offers a better trade-off between accuracy and tuning cost, and is recommended in all scenarios.@@ -292,11 +308,13 @@ configuration to suit your specific requirements and available resources.<details><summary>Recipe Configuration Details</summary>-| Recipe | batch_size | iters | seqlen | nsamples | lr |-|---------|------------|-------|--------|----------|-------|-| default | 8 | 200 | 2048 | 128 | None |-| best | 8 | 1000 | 2048 | 512 | None |-| light | 8 | 50 | 2048 | 128 | 5e-3 |+| Recipe | batch_size | iters | seqlen | nsamples | lr | disable_opt_rtn |+|---------|------------|-------|--------|----------|-------|-----------------|+| default | 8 | 200 | 2048 | 128 | None | False |+| best | 8 | 1000 | 2048 | 512 | None | False |+| light | 8 | 50 | 2048 | 128 | 5e-3 | False |+| opt_rtn | 8 | 0 | 2048 | 128 | None | False |+| rtn | 8 | 0 | 2048 | 0 | None | True |</details>@@ -325,6 +343,8 @@ W2G64 Average Accuracy of 13 tasks and Time Cost Results(Testing was conducted o### AWQ Algorithm+**Experimental feature: our current implementation does not apply weight clipping yet, so accuracy may drop compared to the original AWQ algorithm.**+AWQ (Activation-Aware Weight Quantization) is available as an alternative quantization algorithm. AWQ protects salient weight channels by analyzing activation patterns and applying channel-wise scaling before standard RTN quantization.The canonical AWQ deployment path is **W4A16** served by vLLM's AWQ/Marlin CUDA kernels. **W8A8** with AWQ smoothing can also be served via vLLM's compressed_tensors backend (cutlass INT8 GEMM).@@ -467,6 +487,24 @@ Embedding layer is not supported in AutoScheme, it will use the best scheme in oAutoRound also supports Optimized RTN (Round-To-Nearest) mode for fast, calibration-free baseline quantization. Setting `iters=0` tp enable it and we recommend using `group_size=32` for better results. Check [accuracy comparison](./opt_rtn.md) between RTN and OPT RTN modeFor the GGUF format, we have optimized the RTN algorithm inspired by llamacpp. To use the original (pure) RTN algorithm instead, enable the `--disable_opt_rtn` option.++#### CLI Usage++Two dedicated CLI entry points are provided as shortcuts:++- `auto-round-opt-rtn` — equivalent to `auto-round --iters 0 --enable_opt_rtn` (optimized RTN, recommended).+- `auto-round-rtn` — equivalent to `auto-round --iters 0 --disable_opt_rtn` (pure RTN, no optimization; auto-routes to [Model-Free Mode](#model-free-mode) for supported INT WOQ schemes).++```bash+# Optimized RTN (recommended fast baseline)+auto-round-opt-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"++# Pure RTN (fastest, lowest memory; baseline quality)+auto-round-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"+```++#### API Usage+```pythonfrom auto_round import AutoRound@@ -764,6 +802,8 @@ autoround.save_quantized(format="auto_awq", output_dir="tmp_autoround")### Hadamard Transform+**Research feature with no effective kernels currently available and typically low community adoption.**+AutoRound supports Hadamard transform as an optional weight/activation transformation technique, which can improve quantization accuracy by rotating the weight/activation matrix. This is particularly useful for certain quantization scenarios.#### Overviewdiff --git a/docs/step_by_step_CN.md b/docs/step_by_step_CN.mdindex c5a07fb79..cf4347df0 100644--- a/docs/step_by_step_CN.md+++ b/docs/step_by_step_CN.md@@ -27,15 +27,15 @@- [API 用法](#api-用法)- [AutoScheme 中的超参数](#autoscheme-超参数说明)+ [OPT RTN 模式](#opt-rtn-模式)- + [AWQ 算法](#awq-算法)- + [免模型量化模式](#免模型量化模式)+ + [AWQ 算法-实验性功能](#awq-算法)+ + [免模型架构量化模式](#免模型架构量化模式)+ [GGUF 格式](#gguf-格式量化)+ [量化成本](#量化成本)+ [设备及多卡量化设置](#设备及多卡量化设置)- [lm_head 量化中开启多 GPU 标定](#lm_head-量化中开启多-gpu-标定)- [手动配置设备映射](#手动配置设备映射)+ [超参数调整](#超参数调整)- + [Hadamard变换](#hadamard变换)+ + [Hadamard变换-研究功能](#hadamard变换)* [4 推理部署](#4-推理部署)+ [CPU](#cpu)+ [英特尔 GPU](#英特尔-gpu)@@ -146,7 +146,7 @@ AutoRound 支持多种量化配置:**LLM-Compressor 格式**:**支持 NVFP4、MXFP4(kernel 开发中)、MXFP8** 等。需设置 `--format llm_compressor`。-**MLX 格式**:面向 Apple Silicon (M1/M2/M3/...),可直接被 [`mlx-lm`](https://github.com/ml-explore/mlx-lm)(纯文本 LLM)或 [`mlx-vlm`](https://github.com/Blaizzy/mlx-vlm)(多模态 VLM)加载推理。+**MLX 格式(实验性功能)**:面向 Apple Silicon (M1/M2/M3/...),可直接被 [`mlx-lm`](https://github.com/ml-explore/mlx-lm)(纯文本 LLM)或 [`mlx-vlm`](https://github.com/Blaizzy/mlx-vlm)(多模态 VLM)加载推理。- 支持 **2、3、4、5、6、8 bits**(其中 5/6 bits 是 MLX 独有,GPTQ/AWQ 没有标准打包格式)。- 原生支持 **混合 bit / 混合 group_size**:通过 `layer_config` 或 AutoScheme(如 `--target_bits 3.5 --options "..."`),按层覆盖会写入 `config.json["quantization"]`,- `--format mlx` 导出原生 MLX checkpoint;`--format auto_round:mlx` 则让 HuggingFace `transformers` + AutoRound 加载它(在 Darwin 上 post-init 会把每层重新打包成 MLX 的 `QuantLinear`)。@@ -156,16 +156,16 @@ AutoRound 支持多种量化配置:> 灰色背景的 schemes 表示它没有专门优化的内核,或只有效率极低的参考内核。-| 格式 | 支持的量化方案 |-|:-------------- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |-| **auto_round** | W4A16、W2A16、W3A16、W8A16、W2A16G64、W2A16G32、`MXFP4`、`MXFP8`、`MXFP4_RCEIL`、`MXFP8_RCEIL`、`NVFP4`、`FPW8A16`、`FP8_STATIC`、`FP8_BLOCK`、`BF16`, `MXINT4` |-| **auto_awq** | W4A16、BF16 |-| **auto_gptq** | W4A16、W2A16、W3A16、W8A16、W2A16G64、W2A16G32、BF16 |-| **llm_compressor** | NVFP4、`MXFP4`、`MXFP8`、`FPW8A16`、`FP8_STATIC`、FP8_BLOCK |-| **mlx** / **auto_round:mlx** | W2A16、W3A16、W4A16、W5A16、W6A16、W8A16、BF16、混合 bit / 混合 group_size(仅 Apple Silicon) |-| **gguf** | GGUF:Q4_K_M、GGUF:Q2_K_S、GGUF:Q3_K_S、GGUF:Q3_K_M、GGUF:Q3_K_L、GGUF:Q4_K_S、GGUF:Q5_K_S、GGUF:Q5_K_M、GGUF:Q6_K、GGUF:Q4_0、GGUF:Q4_1、GGUF:Q5_0、GGUF:Q5_1、GGUF:Q8_0 |-| **fp8** | FP8_BLOCK |-| **fake** | `所有方案(仅用于研究场景)` |+| 格式 | 支持的量化方案 |+|:--------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |+| **auto_round** | W4A16、W2A16、W3A16、W8A16、W2A16G64、W2A16G32、`MXFP4`、`MXFP8`、`MXFP4_RCEIL`、`MXFP8_RCEIL`、`NVFP4`、`FPW8A16`、`FP8_STATIC`、`FP8_BLOCK`、`BF16`, `MXINT4` |+| **auto_awq** | W4A16、BF16 |+| **auto_gptq** | W4A16、W2A16、W3A16、W8A16、W2A16G64、W2A16G32、BF16 |+| **llm_compressor** | NVFP4、`MXFP4`、`MXFP8`、`FPW8A16`、`FP8_STATIC`、FP8_BLOCK |+| **mlx** / **auto_round:mlx** (实验性功能) | W2A16、W3A16、W4A16、W5A16、W6A16、W8A16、BF16、混合 bit / 混合 group_size(仅 Apple Silicon) |+| **gguf** | GGUF:Q4_K_M、GGUF:Q2_K_S、GGUF:Q3_K_S、GGUF:Q3_K_M、GGUF:Q3_K_L、GGUF:Q4_K_S、GGUF:Q5_K_S、GGUF:Q5_K_M、GGUF:Q6_K、GGUF:Q4_0、GGUF:Q4_1、GGUF:Q5_0、GGUF:Q5_1、GGUF:Q8_0 |+| **fp8** | FP8_BLOCK |+| **fake** | `所有方案(仅用于研究场景)` |### 硬件兼容性@@ -202,6 +202,22 @@ AutoRound 支持多种量化配置:auto-round-light --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_gptq,auto_awq,auto_round"```+- **AutoRoundOptRTN 优化 RTN 超参(无需梯度下降)**:++ 该方案启用优化版 RTN(Round-To-Nearest,就近舍入)模式(`iters=0` 且 `disable_opt_rtn=False`)。**无需标定数据**,相比基础方案快数倍,同时仍会执行 AutoRound 在 RTN 侧的优化(如更优的 scale/zero-point 搜索、GGUF 中借鉴 llamacpp 的优化等)。**推荐在标定数据不足或调优时间有限的场景下作为快速基线使用**。详情参见 [OPT-RTN 模式](#opt-rtn-模式)章节。++ ```bash+ auto-round-opt-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"+ ```++- **AutoRoundRTN 原始 RTN 超参(不做任何优化,不需要任何数据)**:++ 该方案使用原始 RTN(`iters=0` 且 `disable_opt_rtn=True`),不启用任何 AutoRound 优化。**速度最快、显存占用最低**,但精度通常低于 `auto-round-opt-rtn`。当搭配受支持的 INT WOQ scheme 时,会自动路由至[免模型架构量化模式](#免模型架构量化模式),进一步降低内存占用。适合用作快速验证或等价于传统 RTN 的无标定基线。++ ```bash+ auto-round-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"+ ```+### API 使用方法#### AutoRound API 基础用法该方案兼顾精度和训练耗时,**绝大多数场景下使用,2bit 等量化损失很大的场景尽量不要使用**。@@ -285,11 +301,13 @@ ar.quantize_and_save(output_dir, format="auto_round")<details><summary>各配置方案详细参数</summary>-| 配置方案 | 批次大小 | 迭代次数 | 序列长度 | 标定样本数 | 学习率 |-|---------|----------|----------|----------|-------|--------|-| 基础版 | 8 | 200 | 2048 | 128 | 自动适配 |-| 高精度版 | 8 | 1000 | 2048 | 512 | 自动适配 |-| 高速版 | 8 | 50 | 2048 | 128 | 5e-3 |+| 配置方案 | 批次大小 | 迭代次数 | 序列长度 | 标定样本数 | 学习率 | disable_opt_rtn |+|---------|----------|----------|----------|-------|--------|-----------------|+| 基础版(default) | 8 | 200 | 2048 | 128 | 自动适配 | False |+| 高精度版(best) | 8 | 1000 | 2048 | 512 | 自动适配 | False |+| 高速版(light) | 8 | 50 | 2048 | 128 | 5e-3 | False |+| 优化 RTN(opt_rtn) | 8 | 0 | 2048 | 128 | 自动适配 | False |+| 原始 RTN(rtn) | 8 | 0 | 2048 | 0 | 自动适配 | True |</details>@@ -320,6 +338,8 @@ W2G64 在 13 个任务上的平均精度与耗时### AWQ 算法+实验性功能:原始实现中未使用 weight clipping(权重裁剪)逻辑,因此相比原版 AWQ 算法,可能会存在一定精度下降+AWQ(Activation-Aware Weight Quantization,激活感知权重量化)是一种可选的量化算法。AWQ 通过分析激活模式来保护关键权重通道,在标准量化前对权重施加通道级缩放,从而降低量化误差。AWQ 的标准部署路径是 **W4A16**,通过 vLLM 的 AWQ/Marlin CUDA 内核提供服务。**W8A8** 搭配 AWQ 平滑化也可通过 vLLM 的 compressed_tensors 后端(cutlass INT8 GEMM)提供服务。@@ -464,6 +484,24 @@ AutoScheme 目前还**不支持对嵌入层(Embedding layer)进行自动量AutoRound 还提供优化版 RTN(Round-To-Nearest,就近舍入)模式,无需标定数据即可实现快速基线量化。**启用方式为 `iters=0`**。同时为获得更好的效果,推荐搭配 `group_size=32` 。RTN 与 OPT RTN 模式的精度对比详见[《精度对比报告》](./opt_rtn.md)。对于 GGUF 格式,我们参考 llamacpp 的思路,优化了 RTN 算法。若需使用原始(非优化)RTN 算法,开启 `--disable_opt_rtn` 即可。++#### 命令行使用++我们提供了两个专用的 CLI 入口作为快捷方式:++- `auto-round-opt-rtn` — 等价于 `auto-round --iters 0 --enable_opt_rtn`(优化版 RTN,推荐使用)。+- `auto-round-rtn` — 等价于 `auto-round --iters 0 --disable_opt_rtn`(原始 RTN,不做任何优化;对于受支持的 INT WOQ scheme 会自动路由到[免模型架构量化模式](#免模型架构量化模式))。++```bash+# 优化版 RTN(推荐的快速基线方案)+auto-round-opt-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"++# 原始 RTN(速度最快、显存占用最低;仅作基线参考)+auto-round-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"+```++#### API 使用+```pythonfrom auto_round import AutoRound@@ -478,9 +516,9 @@ output_dir = "./tmp_autoround"ar.quantize_and_save(output_dir, format="auto_round")```-### 免模型量化模式+### 免模型架构量化模式-免模型量化模式(Model-Free Mode)可以**无需将完整模型加载到内存中**即可执行 RTN WOQ 量化。它直接下载 safetensors 文件,逐分片地对每个 Linear 权重张量进行量化并保存打包结果。当您需要快速、无标定数据的量化且资源有限时,该模式非常实用。+免模型架构量化模式(Model-Free Mode)可以**无需将完整模型加载到内存中**即可执行 RTN WOQ 量化。它直接下载 safetensors 文件,逐分片地对每个 Linear 权重张量进行量化并保存打包结果。当您需要快速、无标定数据的量化且资源有限时,该模式非常实用。> **默认自动启用。** 自 v0.13 起,当您同时传入 `--iters 0 --disable_opt_rtn` 与一个受支持的 INT WOQ scheme 时,CLI 会自动走免模型路径。该路径与原始 `--iters 0 --disable_opt_rtn` 流程**位级(bit-exact)等价**,但内存占用大幅降低。如需关闭自动路由、强制使用原始流程,可加 `--disable_model_free`。@@ -746,8 +784,8 @@ AutoRound 提供两种类型的 Hadamard 变换:1. **确定性 Hadamard 变换**(`hadamard`):使用 Sylvester 构造法生成确定性的 Hadamard 矩阵,尺寸必须为 2 的幂次。2. **随机 Hadamard 变换**(`random_hadamard`):使用 N. J. A. Sloane 的 Hadamard 矩阵库中已知的矩阵。支持非 2 的幂次的尺寸,并支持确定性随机种子。-#### 使用 Hadamard 变换进行量化-+#### 使用 Hadamard 旋转进行量化+研究性功能:当前暂无高效可用的 kernel 支持,且社区兼容性通常较低。```pythonfrom auto_round import AutoRound