Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

update doc (#1856)

Wenhua Chengf06f9a1629fd
diff --git a/README.md b/README.md
index 380704574..06f61a812 100644
--- a/README.md
+++ b/README.md
@@ -36,9 +36,11 @@ See our papers [SignRoundV1](https://arxiv.org/pdf/2309.05516) and [SignRoundV2]
## 🆕 What's New
+* [2026/05] We provide **free** devices for calibration-free quantization via pure RTN mode; please visit [Intel Low Bit Open LLM Leaderboard](https://huggingface.co/spaces/Intel/low_bit_open_llm_leaderboard) for more details.
+
* [2026/05] **Model free** quantization is available, `auto-round-rtn` will now default to using the model-free approach: [Doc](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#model-free-mode).
-* [2026/03] **Block-wise FP8** quantization is available via `auto-round-rtn --scheme FP8_BLOCK`.
+* [2026/03] **Block-wise FP8** quantization is available and rtn mode is recommended. `auto-round-rtn --scheme FP8_BLOCK`.
* [2026/03] **MTP layer quantization** has been supported in this [PR](https://github.com/intel/auto-round/pull/1526)
@@ -90,7 +92,7 @@ Quantize 7B models in about 10 minutes on a single GPU. Details are shown in [qu
Out-of-the-box quantization for 10+ vision-language models [example models](https://huggingface.co/collections/OPEA/vlms-autoround-675bc712fdd6a55ebaf11bfa), [support matrix](https://github.com/intel/auto-round/tree/main/auto_round/compressors/mllm#vlm-support-matrix)
✅ **Multiple Recipes**
-Choose from `auto-round-best`, `auto-round`, and `auto-round-light` to suit your needs. Details are shown in [quantization recipes](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#recipe-recommendation)
+Choose from `auto-round-best`, `auto-round`, `auto-round-light`, `auto-round-opt-rtn` (optimized RTN) and `auto-round-rtn` (pure RTN, fastest baseline) to suit your needs. Details are shown in [quantization recipes](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#recipe-recommendation)
✅ Advanced Utilities
Includes [multiple gpus quantization](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#devicemulti-gpu-setting-in-quantization), [multiple calibration datasets](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#default-dataset) and support for [10+ runtime backends](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#specify-inference-backend).
@@ -174,6 +176,21 @@ auto-round-light \
```
+ ```bash
+# Optimized RTN (iters=0, opt_rtn enabled); fast baseline
+auto-round-opt-rtn \
+ --model Qwen/Qwen3-0.6B \
+ --scheme "W4A16"
+ ```
+
+ ```bash
+# Pure RTN (iters=0, no AutoRound optimization); fastest, lowest memory
+# auto-routes to model-free mode for supported INT WOQ schemes
+auto-round-rtn \
+ --model Qwen/Qwen3-0.6B \
+ --scheme "W4A16"
+ ```
+
<!-- ```bash
auto-round-fast \
# Fast and low memory, 2-3X speedup, slight accuracy drop at W4G128
diff --git a/README_CN.md b/README_CN.md
index 645d8ca6d..85760179e 100644
--- a/README_CN.md
+++ b/README_CN.md
@@ -35,7 +35,9 @@ AutoRound 是专为大语言模型(LLMs)和视觉-语言模型(VLMs)设
## 🆕 最新进展
-* [2026/05] 现已支持**免模型量化**模式, `auto-round-rtn`现在会默认启用免模型方式: [文档](https://github.com/intel/auto-round/blob/main/docs/step_by_step_CN.md#%E5%85%8D%E6%A8%A1%E5%9E%8B%E9%87%8F%E5%8C%96%E6%A8%A1%E5%BC%8F).
+* [2026/05] 我们提供**免费**的硬件资源,用于通过纯 RTN 方法进行模型量化。更多详情请访问 [Intel Low Bit Open LLM Leaderboard](https://huggingface.co/spaces/Intel/low_bit_open_llm_leaderboard)。
+
+* [2026/05] 现已支持**免模型架构量化**模式, `auto-round-rtn` 现在会默认启用免模型方式: [文档](https://github.com/intel/auto-round/blob/main/docs/step_by_step_CN.md#%E5%85%8D%E6%A8%A1%E5%9E%8B%E6%9E%B6%E6%9E%84%E9%87%8F%E5%8C%96%E6%A8%A1%E5%BC%8F).
* [2026/03] 支持**Block-wise FP8**,只需要将scheme改成 `--scheme FP8_BLOCK`.为了更快的量化,我们推荐使用RTN模式,即`auto-round-rtn`
@@ -81,7 +83,7 @@ AutoRound 是专为大语言模型(LLMs)和视觉-语言模型(VLMs)设
✅ **支持十余种 VLM 模型** 已支持十余种视觉语言模型,让用户有“开盖即食”般的量化体验。详见:[示例模型](https://huggingface.co/collections/OPEA/vlms-autoround-675bc712fdd6a55ebaf11bfa),[支持矩阵](https://github.com/intel/auto-round/tree/main/auto_round/compressors/mllm#vlm-support-matrix)
-✅ **多种量化方案可选** 提供 `auto-round-best`​、`auto-round`​、`auto-round-light`​ 等多种预设方案,能够满足多样化需求。详见:[量化方案](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#recipe-recommendation)
+✅ **多种量化方案可选** 提供 `auto-round-best`​、`auto-round`​、`auto-round-light`​、`auto-round-opt-rtn`(优化版 RTN)以及 `auto-round-rtn`(原始 RTN,最快基线)等多种预设方案,能够满足多样化需求。详见:[量化方案](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#recipe-recommendation)
✅ **实用额外特性** 支持[多 GPU 量化](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#devicemulti-gpu-setting-in-quantization)和[多标定数据集](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#default-dataset),并兼容[十余种推理后端](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#specify-inference-backend)。
@@ -162,6 +164,21 @@ auto-round-light \
--scheme "W4A16"
```
+ ```bash
+# 无需标定的优化版 RTN(iters=0,启用 opt_rtn),快速基线方案
+auto-round-opt-rtn \
+ --model Qwen/Qwen3-0.6B \
+ --scheme "W4A16"
+ ```
+
+ ```bash
+# 原始 RTN(iters=0,不做任何 AutoRound 优化),速度最快、显存占用最低
+# 对受支持的 INT WOQ scheme 会自动路由到 model-free 模式
+auto-round-rtn \
+ --model Qwen/Qwen3-0.6B \
+ --scheme "W4A16"
+ ```
+
<!-- ```bash
auto-round-fast \
# Fast and low memory, 2-3X speedup, slight accuracy drop at W4G128
diff --git a/docs/step_by_step.md b/docs/step_by_step.md
index 562b88d0e..4a2a8052d 100644
--- a/docs/step_by_step.md
+++ b/docs/step_by_step.md
@@ -27,14 +27,14 @@ This document presents step-by-step instructions for auto-round llm quantization
- [API Usage](#api-usage-1)
- [Hyperparameters in AutoScheme](#hyperparameters-in-autoscheme)
+ [OPT RTN mode](#opt-rtn-mode)
- + [AWQ Algorithm](#awq-algorithm)
+ + [AWQ Algorithm-Experimental](#awq-algorithm)
+ [Model-Free Mode](#model-free-mode)
+ [GGUF format](#gguf-format)
+ [Quantization Costs](#quantization-costs)
+ [Device/Multi-GPU setting in Quantization](#devicemulti-gpu-setting-in-quantization)
- [Enable multiple gpus calibration in lm_head quantization](#enable-multiple-gpus-calibration-in-lm_head-quantization)
+ [Adjust Hyperparameters](#adjust-hyperparameters)
- + [Hadamard Transform](#hadamard-transform)
+ + [Hadamard Transform-Research Feature](#hadamard-transform)
* [4 Inference](#4-inference)
+ [CPU](#cpu)
+ [Intel GPU](#intel-gpu)
@@ -156,7 +156,7 @@ adopted within the community, **only 4-bits quantization is supported**. Please
**LLM-Compressor Format**: **NVFP4, MXFP4(kernel in WIP), MXFP8 are supported**. Please set `--format llm_compressor`
-**MLX Format**: This format targets Apple Silicon (M1/M2/M3/...) and is loaded directly by [`mlx-lm`](https://github.com/ml-explore/mlx-lm) (text-only LLM) or [`mlx-vlm`](https://github.com/Blaizzy/mlx-vlm) (vision/audio + language).
+**MLX Format**[Experimental Feature]: This format targets Apple Silicon (M1/M2/M3/...) and is loaded directly by [`mlx-lm`](https://github.com/ml-explore/mlx-lm) (text-only LLM) or [`mlx-vlm`](https://github.com/Blaizzy/mlx-vlm) (vision/audio + language).
- Supports **2, 3, 4, 5, 6, 8 bits** (5/6 bits are MLX-exclusive — GPTQ/AWQ have no standard packing for them).
- Native **mixed-bit / mixed-group_size** via `layer_config` or AutoScheme (`--target_bits 3.5 --options "..."`);
- Use `--format mlx` for a native MLX checkpoint; use `--format auto_round:mlx` if you want HuggingFace `transformers` + AutoRound to load it (post-init repacks each layer into MLX `QuantLinear` on Darwin).
@@ -165,16 +165,16 @@ adopted within the community, **only 4-bits quantization is supported**. Please
> Gray indicates the absence of a kernel or the presence of only an inefficient/reference kernel. BF16 is mainly for AutoScheme
-| Format | Supported Schemes |
-|:---|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| **auto_round** | W4A16, W2A16, W3A16, W8A16, W2A16G64, W2A16G32, `MXFP4`, `MXFP8`, `MXFP4_RCEIL`, `MXFP8_RCEIL`, `NVFP4`, `FPW8A16`, `FP8_STATIC`, `FP8_BLOCK`, `BF16`, `MXINT4` |
-| **auto_awq** | W4A16, BF16 |
-| **auto_gptq** | W4A16, W2A16, W3A16, W8A16,W2A16G64, W2A16G32, BF16 |
-| **llm_compressor** | NVFP4, `MXFP4`, `MXFP8`, `FPW8A16`, `FP8_STATIC`, FP8_BLOCK |
+| Format | Supported Schemes |
+|:-----------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| **auto_round** | W4A16, W2A16, W3A16, W8A16, W2A16G64, W2A16G32, `MXFP4`, `MXFP8`, `MXFP4_RCEIL`, `MXFP8_RCEIL`, `NVFP4`, `FPW8A16`, `FP8_STATIC`, `FP8_BLOCK`, `BF16`, `MXINT4` |
+| **auto_awq** | W4A16, BF16 |
+| **auto_gptq** | W4A16, W2A16, W3A16, W8A16,W2A16G64, W2A16G32, BF16 |
+| **llm_compressor** | NVFP4, `MXFP4`, `MXFP8`, `FPW8A16`, `FP8_STATIC`, FP8_BLOCK |
| **mlx** / **auto_round:mlx** | W2A16, W3A16, W4A16, W5A16, W6A16, W8A16, BF16, mixed-bit / mixed-group_size (Apple Silicon only) |
-| **gguf** | GGUF:Q4_K_M, GGUF:Q2_K_S, GGUF:Q3_K_S, GGUF:Q3_K_M, GGUF:Q3_K_L, GGUF:Q4_K_S, GGUF:Q5_K_S, GGUF:Q5_K_M, GGUF:Q6_K, GGUF:Q4_0, GGUF:Q4_1, GGUF:Q5_0, GGUF:Q5_1,GGUF:Q8_0 |
-| **fp8** | FP8_BLOCK |
-| **fake** | `all schemes (only for research)` |
+| **gguf** | GGUF:Q4_K_M, GGUF:Q2_K_S, GGUF:Q3_K_S, GGUF:Q3_K_M, GGUF:Q3_K_L, GGUF:Q4_K_S, GGUF:Q5_K_S, GGUF:Q5_K_M, GGUF:Q6_K, GGUF:Q4_0, GGUF:Q4_1, GGUF:Q5_0, GGUF:Q5_1,GGUF:Q8_0 |
+| **fp8** | FP8_BLOCK |
+| **fake** | `all schemes (only for research)` |
### Hardware Compatibility
@@ -211,6 +211,22 @@ Before starting quantization, you may want to configure AutoRound's environment
auto-round-light --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_gptq,auto_awq,auto_round"
```
+- **AutoRoundOptRTN recipe (optimized RTN, without gradient computation):**
+
+ This setting runs the optimized RTN (Round-To-Nearest) path (`iters=0` with `disable_opt_rtn=False`). It is calibration-free and several times faster than the default AutoRound recipe, while still applying AutoRound's RTN-side optimizations (e.g. improved scale/zero-point search and llamacpp-style refinements for GGUF). Recommended as a fast baseline when calibration data or tuning time is limited. See the [OPT RTN Mode](#opt-rtn-mode) section for details.
+
+ ```bash
+ auto-round-opt-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"
+ ```
+
+- **AutoRoundRTN recipe (pure RTN, calibration-free, no optimization):**
+
+ This setting runs pure RTN (`iters=0` with `disable_opt_rtn=True`), without any AutoRound optimization. It is the fastest path and uses the least memory, but typically yields lower accuracy than `auto-round-opt-rtn`. When combined with a supported INT WOQ scheme, it is automatically routed through [Model-Free Mode](#model-free-mode) for minimal memory usage. Use this as a quick sanity-check or when you want a calibration-free baseline equivalent to traditional RTN.
+
+ ```bash
+ auto-round-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"
+ ```
+
### API usage
#### AutoRound API Usage
This setting offers a better trade-off between accuracy and tuning cost, and is recommended in all scenarios.
@@ -292,11 +308,13 @@ configuration to suit your specific requirements and available resources.
<details>
<summary>Recipe Configuration Details</summary>
-| Recipe | batch_size | iters | seqlen | nsamples | lr |
-|---------|------------|-------|--------|----------|-------|
-| default | 8 | 200 | 2048 | 128 | None |
-| best | 8 | 1000 | 2048 | 512 | None |
-| light | 8 | 50 | 2048 | 128 | 5e-3 |
+| Recipe | batch_size | iters | seqlen | nsamples | lr | disable_opt_rtn |
+|---------|------------|-------|--------|----------|-------|-----------------|
+| default | 8 | 200 | 2048 | 128 | None | False |
+| best | 8 | 1000 | 2048 | 512 | None | False |
+| light | 8 | 50 | 2048 | 128 | 5e-3 | False |
+| opt_rtn | 8 | 0 | 2048 | 128 | None | False |
+| rtn | 8 | 0 | 2048 | 0 | None | True |
</details>
@@ -325,6 +343,8 @@ W2G64 Average Accuracy of 13 tasks and Time Cost Results(Testing was conducted o
### AWQ Algorithm
+**Experimental feature: our current implementation does not apply weight clipping yet, so accuracy may drop compared to the original AWQ algorithm.**
+
AWQ (Activation-Aware Weight Quantization) is available as an alternative quantization algorithm. AWQ protects salient weight channels by analyzing activation patterns and applying channel-wise scaling before standard RTN quantization.
The canonical AWQ deployment path is **W4A16** served by vLLM's AWQ/Marlin CUDA kernels. **W8A8** with AWQ smoothing can also be served via vLLM's compressed_tensors backend (cutlass INT8 GEMM).
@@ -467,6 +487,24 @@ Embedding layer is not supported in AutoScheme, it will use the best scheme in o
AutoRound also supports Optimized RTN (Round-To-Nearest) mode for fast, calibration-free baseline quantization. Setting `iters=0` tp enable it and we recommend using `group_size=32` for better results. Check [accuracy comparison](./opt_rtn.md) between RTN and OPT RTN mode
For the GGUF format, we have optimized the RTN algorithm inspired by llamacpp. To use the original (pure) RTN algorithm instead, enable the `--disable_opt_rtn` option.
+
+#### CLI Usage
+
+Two dedicated CLI entry points are provided as shortcuts:
+
+- `auto-round-opt-rtn` — equivalent to `auto-round --iters 0 --enable_opt_rtn` (optimized RTN, recommended).
+- `auto-round-rtn` — equivalent to `auto-round --iters 0 --disable_opt_rtn` (pure RTN, no optimization; auto-routes to [Model-Free Mode](#model-free-mode) for supported INT WOQ schemes).
+
+```bash
+# Optimized RTN (recommended fast baseline)
+auto-round-opt-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"
+
+# Pure RTN (fastest, lowest memory; baseline quality)
+auto-round-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"
+```
+
+#### API Usage
+
```python
from auto_round import AutoRound
@@ -764,6 +802,8 @@ autoround.save_quantized(format="auto_awq", output_dir="tmp_autoround")
### Hadamard Transform
+**Research feature with no effective kernels currently available and typically low community adoption.**
+
AutoRound supports Hadamard transform as an optional weight/activation transformation technique, which can improve quantization accuracy by rotating the weight/activation matrix. This is particularly useful for certain quantization scenarios.
#### Overview
diff --git a/docs/step_by_step_CN.md b/docs/step_by_step_CN.md
index c5a07fb79..cf4347df0 100644
--- a/docs/step_by_step_CN.md
+++ b/docs/step_by_step_CN.md
@@ -27,15 +27,15 @@
- [API 用法](#api-用法)
- [AutoScheme 中的超参数](#autoscheme-超参数说明)
+ [OPT RTN 模式](#opt-rtn-模式)
- + [AWQ 算法](#awq-算法)
- + [免模型量化模式](#免模型量化模式)
+ + [AWQ 算法-实验性功能](#awq-算法)
+ + [免模型架构量化模式](#免模型架构量化模式)
+ [GGUF 格式](#gguf-格式量化)
+ [量化成本](#量化成本)
+ [设备及多卡量化设置](#设备及多卡量化设置)
- [lm_head 量化中开启多 GPU 标定](#lm_head-量化中开启多-gpu-标定)
- [手动配置设备映射](#手动配置设备映射)
+ [超参数调整](#超参数调整)
- + [Hadamard变换](#hadamard变换)
+ + [Hadamard变换-研究功能](#hadamard变换)
* [4 推理部署](#4-推理部署)
+ [CPU](#cpu)
+ [英特尔 GPU](#英特尔-gpu)
@@ -146,7 +146,7 @@ AutoRound 支持多种量化配置:
**LLM-Compressor 格式**:**支持 NVFP4、MXFP4(kernel 开发中)、MXFP8** 等。需设置 `--format llm_compressor`。
-**MLX 格式**:面向 Apple Silicon (M1/M2/M3/...),可直接被 [`mlx-lm`](https://github.com/ml-explore/mlx-lm)(纯文本 LLM)或 [`mlx-vlm`](https://github.com/Blaizzy/mlx-vlm)(多模态 VLM)加载推理。
+**MLX 格式(实验性功能)**:面向 Apple Silicon (M1/M2/M3/...),可直接被 [`mlx-lm`](https://github.com/ml-explore/mlx-lm)(纯文本 LLM)或 [`mlx-vlm`](https://github.com/Blaizzy/mlx-vlm)(多模态 VLM)加载推理。
- 支持 **2、3、4、5、6、8 bits**(其中 5/6 bits 是 MLX 独有,GPTQ/AWQ 没有标准打包格式)。
- 原生支持 **混合 bit / 混合 group_size**:通过 `layer_config` 或 AutoScheme(如 `--target_bits 3.5 --options "..."`),按层覆盖会写入 `config.json["quantization"]`,
- `--format mlx` 导出原生 MLX checkpoint;`--format auto_round:mlx` 则让 HuggingFace `transformers` + AutoRound 加载它(在 Darwin 上 post-init 会把每层重新打包成 MLX 的 `QuantLinear`)。
@@ -156,16 +156,16 @@ AutoRound 支持多种量化配置:
> 灰色背景的 schemes 表示它没有专门优化的内核,或只有效率极低的参考内核。
-| 格式 | 支持的量化方案 |
-|:-------------- |:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
-| **auto_round** | W4A16、W2A16、W3A16、W8A16、W2A16G64、W2A16G32、`MXFP4`、`MXFP8`、`MXFP4_RCEIL`、`MXFP8_RCEIL`、`NVFP4`、`FPW8A16`、`FP8_STATIC`、`FP8_BLOCK`、`BF16`, `MXINT4` |
-| **auto_awq** | W4A16、BF16 |
-| **auto_gptq** | W4A16、W2A16、W3A16、W8A16、W2A16G64、W2A16G32、BF16 |
-| **llm_compressor** | NVFP4、`MXFP4`、`MXFP8`、`FPW8A16`、`FP8_STATIC`、FP8_BLOCK |
-| **mlx** / **auto_round:mlx** | W2A16、W3A16、W4A16、W5A16、W6A16、W8A16、BF16、混合 bit / 混合 group_size(仅 Apple Silicon) |
-| **gguf** | GGUF:Q4_K_M、GGUF:Q2_K_S、GGUF:Q3_K_S、GGUF:Q3_K_M、GGUF:Q3_K_L、GGUF:Q4_K_S、GGUF:Q5_K_S、GGUF:Q5_K_M、GGUF:Q6_K、GGUF:Q4_0、GGUF:Q4_1、GGUF:Q5_0、GGUF:Q5_1、GGUF:Q8_0 |
-| **fp8** | FP8_BLOCK |
-| **fake** | `所有方案(仅用于研究场景)` |
+| 格式 | 支持的量化方案 |
+|:--------------------------------|:------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| **auto_round** | W4A16、W2A16、W3A16、W8A16、W2A16G64、W2A16G32、`MXFP4`、`MXFP8`、`MXFP4_RCEIL`、`MXFP8_RCEIL`、`NVFP4`、`FPW8A16`、`FP8_STATIC`、`FP8_BLOCK`、`BF16`, `MXINT4` |
+| **auto_awq** | W4A16、BF16 |
+| **auto_gptq** | W4A16、W2A16、W3A16、W8A16、W2A16G64、W2A16G32、BF16 |
+| **llm_compressor** | NVFP4、`MXFP4`、`MXFP8`、`FPW8A16`、`FP8_STATIC`、FP8_BLOCK |
+| **mlx** / **auto_round:mlx** (实验性功能) | W2A16、W3A16、W4A16、W5A16、W6A16、W8A16、BF16、混合 bit / 混合 group_size(仅 Apple Silicon) |
+| **gguf** | GGUF:Q4_K_M、GGUF:Q2_K_S、GGUF:Q3_K_S、GGUF:Q3_K_M、GGUF:Q3_K_L、GGUF:Q4_K_S、GGUF:Q5_K_S、GGUF:Q5_K_M、GGUF:Q6_K、GGUF:Q4_0、GGUF:Q4_1、GGUF:Q5_0、GGUF:Q5_1、GGUF:Q8_0 |
+| **fp8** | FP8_BLOCK |
+| **fake** | `所有方案(仅用于研究场景)` |
### 硬件兼容性
@@ -202,6 +202,22 @@ AutoRound 支持多种量化配置:
auto-round-light --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_gptq,auto_awq,auto_round"
```
+- **AutoRoundOptRTN 优化 RTN 超参(无需梯度下降)**:
+
+ 该方案启用优化版 RTN(Round-To-Nearest,就近舍入)模式(`iters=0` 且 `disable_opt_rtn=False`)。**无需标定数据**,相比基础方案快数倍,同时仍会执行 AutoRound 在 RTN 侧的优化(如更优的 scale/zero-point 搜索、GGUF 中借鉴 llamacpp 的优化等)。**推荐在标定数据不足或调优时间有限的场景下作为快速基线使用**。详情参见 [OPT-RTN 模式](#opt-rtn-模式)章节。
+
+ ```bash
+ auto-round-opt-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"
+ ```
+
+- **AutoRoundRTN 原始 RTN 超参(不做任何优化,不需要任何数据)**:
+
+ 该方案使用原始 RTN(`iters=0` 且 `disable_opt_rtn=True`),不启用任何 AutoRound 优化。**速度最快、显存占用最低**,但精度通常低于 `auto-round-opt-rtn`。当搭配受支持的 INT WOQ scheme 时,会自动路由至[免模型架构量化模式](#免模型架构量化模式),进一步降低内存占用。适合用作快速验证或等价于传统 RTN 的无标定基线。
+
+ ```bash
+ auto-round-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"
+ ```
+
### API 使用方法
#### AutoRound API 基础用法
该方案兼顾精度和训练耗时,**绝大多数场景下使用,2bit 等量化损失很大的场景尽量不要使用**。
@@ -285,11 +301,13 @@ ar.quantize_and_save(output_dir, format="auto_round")
<details>
<summary>各配置方案详细参数</summary>
-| 配置方案 | 批次大小 | 迭代次数 | 序列长度 | 标定样本数 | 学习率 |
-|---------|----------|----------|----------|-------|--------|
-| 基础版 | 8 | 200 | 2048 | 128 | 自动适配 |
-| 高精度版 | 8 | 1000 | 2048 | 512 | 自动适配 |
-| 高速版 | 8 | 50 | 2048 | 128 | 5e-3 |
+| 配置方案 | 批次大小 | 迭代次数 | 序列长度 | 标定样本数 | 学习率 | disable_opt_rtn |
+|---------|----------|----------|----------|-------|--------|-----------------|
+| 基础版(default) | 8 | 200 | 2048 | 128 | 自动适配 | False |
+| 高精度版(best) | 8 | 1000 | 2048 | 512 | 自动适配 | False |
+| 高速版(light) | 8 | 50 | 2048 | 128 | 5e-3 | False |
+| 优化 RTN(opt_rtn) | 8 | 0 | 2048 | 128 | 自动适配 | False |
+| 原始 RTN(rtn) | 8 | 0 | 2048 | 0 | 自动适配 | True |
</details>
@@ -320,6 +338,8 @@ W2G64 在 13 个任务上的平均精度与耗时
### AWQ 算法
+实验性功能:原始实现中未使用 weight clipping(权重裁剪)逻辑,因此相比原版 AWQ 算法,可能会存在一定精度下降
+
AWQ(Activation-Aware Weight Quantization,激活感知权重量化)是一种可选的量化算法。AWQ 通过分析激活模式来保护关键权重通道,在标准量化前对权重施加通道级缩放,从而降低量化误差。
AWQ 的标准部署路径是 **W4A16**,通过 vLLM 的 AWQ/Marlin CUDA 内核提供服务。**W8A8** 搭配 AWQ 平滑化也可通过 vLLM 的 compressed_tensors 后端(cutlass INT8 GEMM)提供服务。
@@ -464,6 +484,24 @@ AutoScheme 目前还**不支持对嵌入层(Embedding layer)进行自动量
AutoRound 还提供优化版 RTN(Round-To-Nearest,就近舍入)模式,无需标定数据即可实现快速基线量化。**启用方式为 `iters=0`**。同时为获得更好的效果,推荐搭配 `group_size=32` 。RTN 与 OPT RTN 模式的精度对比详见[《精度对比报告》](./opt_rtn.md)。
对于 GGUF 格式,我们参考 llamacpp 的思路,优化了 RTN 算法。若需使用原始(非优化)RTN 算法,开启 `--disable_opt_rtn` 即可。
+
+#### 命令行使用
+
+我们提供了两个专用的 CLI 入口作为快捷方式:
+
+- `auto-round-opt-rtn` — 等价于 `auto-round --iters 0 --enable_opt_rtn`(优化版 RTN,推荐使用)。
+- `auto-round-rtn` — 等价于 `auto-round --iters 0 --disable_opt_rtn`(原始 RTN,不做任何优化;对于受支持的 INT WOQ scheme 会自动路由到[免模型架构量化模式](#免模型架构量化模式))。
+
+```bash
+# 优化版 RTN(推荐的快速基线方案)
+auto-round-opt-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"
+
+# 原始 RTN(速度最快、显存占用最低;仅作基线参考)
+auto-round-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"
+```
+
+#### API 使用
+
```python
from auto_round import AutoRound
@@ -478,9 +516,9 @@ output_dir = "./tmp_autoround"
ar.quantize_and_save(output_dir, format="auto_round")
```
-### 免模型量化模式
+### 免模型架构量化模式
-免模型量化模式(Model-Free Mode)可以**无需将完整模型加载到内存中**即可执行 RTN WOQ 量化。它直接下载 safetensors 文件,逐分片地对每个 Linear 权重张量进行量化并保存打包结果。当您需要快速、无标定数据的量化且资源有限时,该模式非常实用。
+免模型架构量化模式(Model-Free Mode)可以**无需将完整模型加载到内存中**即可执行 RTN WOQ 量化。它直接下载 safetensors 文件,逐分片地对每个 Linear 权重张量进行量化并保存打包结果。当您需要快速、无标定数据的量化且资源有限时,该模式非常实用。
> **默认自动启用。** 自 v0.13 起,当您同时传入 `--iters 0 --disable_opt_rtn` 与一个受支持的 INT WOQ scheme 时,CLI 会自动走免模型路径。该路径与原始 `--iters 0 --disable_opt_rtn` 流程**位级(bit-exact)等价**,但内存占用大幅降低。如需关闭自动路由、强制使用原始流程,可加 `--disable_model_free`。
@@ -746,8 +784,8 @@ AutoRound 提供两种类型的 Hadamard 变换:
1. **确定性 Hadamard 变换**(`hadamard`):使用 Sylvester 构造法生成确定性的 Hadamard 矩阵,尺寸必须为 2 的幂次。
2. **随机 Hadamard 变换**(`random_hadamard`):使用 N. J. A. Sloane 的 Hadamard 矩阵库中已知的矩阵。支持非 2 的幂次的尺寸,并支持确定性随机种子。
-#### 使用 Hadamard 变换进行量化
-
+#### 使用 Hadamard 旋转进行量化
+研究性功能:当前暂无高效可用的 kernel 支持,且社区兼容性通常较低。
```python
from auto_round import AutoRound