Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

improve the robustness of scheme (#803)

Wenhua Cheng9534d2aca23a
diff --git a/README.md b/README.md
index 5f39b4d39..15bf1112d 100644
--- a/README.md
+++ b/README.md
@@ -38,11 +38,7 @@ and [fbaldassarri](https://huggingface.co/fbaldassarri). For usage instructions,
all bits other than 3 bits. Example
models: [Intel/Qwen3-235B-A22B-q2ks-mixed-AutoRound](https://huggingface.co/Intel/Qwen3-235B-A22B-q2ks-mixed-AutoRound)
and [Intel/DeepSeek-R1-0528-q2ks-mixed-AutoRound](https://huggingface.co/Intel/DeepSeek-R1-0528-q2ks-mixed-AutoRound). **A more advanced algorithm** tailored for specific configurations may be available in
- v0.6.2.
-
-[2025/05] AutoRound provides some recipes for **DeepSeek-R1-0528**, please refer
- to [OPEA/DeepSeek-R1-0528-int2-mixed-AutoRound](https://huggingface.co/OPEA/DeepSeek-R1-0528-int2-mixed-AutoRound) and [OPEA/DeepSeek-R1-0528-int4-AutoRound](https://huggingface.co/OPEA/DeepSeek-R1-0528-int4-AutoRound) for
- more details.
+ v0.7.1.
[2025/05] AutoRound has been integrated into **vLLM**. You can now run models in the AutoRound format directly with
vLLM versions later than v0.85.post1.
@@ -121,16 +117,16 @@ Please change to `auto-round-mllm` for visual-language models (VLMs) quantizatio
auto-round \
--model Qwen/Qwen3-0.6B \
--scheme "W4A16" \
- --format "auto_gptq,auto_awq,auto_round" \
+ --format "auto_round" \
--output_dir ./tmp_autoround
```
-We offer another two configurations, `auto-round-best` and `auto-round-light`, designed for optimal accuracy and improved speed, respectively. Details are as follows.
+We offer another two recipes, `auto-round-best` and `auto-round-light`, designed for optimal accuracy and improved speed, respectively. Details are as follows.
<details>
<summary>Other Recipes</summary>
```bash
-## best accuracy, 3X slower, low_gpu_mem_usage could save ~20G but ~30% slower
+# Best accuracy, 3X slower, low_gpu_mem_usage could save ~20G but ~30% slower
auto-round-best \
--model Qwen/Qwen3-0.6B \
--scheme "W4A16" \
@@ -138,7 +134,7 @@ auto-round-best \
```
```bash
-## light accuracy, 2-3X speedup, slight accuracy drop at W4 and larger accuracy drop at W2
+# 2-3X speedup, slight accuracy drop at W4 and larger accuracy drop at W2
auto-round-light \
--model Qwen/Qwen3-0.6B \
--scheme "W4A16"
@@ -147,7 +143,7 @@ auto-round-light \
<!-- ```bash
auto-round-fast \
-## fast and low memory, 2-3X speedup, slight accuracy drop at W4G128
+# Fast and low memory, 2-3X speedup, slight accuracy drop at W4G128
--model Qwen/Qwen3-0.6B \
--bits 4 \
--group_size 128 \
@@ -176,10 +172,8 @@ ar = AutoRound(model_name_or_path, scheme="W4A16")
# Faster quantization (2–3× speedup) with slight accuracy drop at W4G128.
# ar = AutoRound(model_name_or_path, nsamples=128, iters=50, lr=5e-3)
-# Save quantized model
-output_dir = "./tmp_autoround"
# Supported formats: "auto_round" (default), "auto_gptq", "auto_awq", "llm_compressor", "gguf:q4_k_m", etc.
-ar.quantize_and_save(output_dir, format="auto_round")
+ar.quantize_and_save(output_dir="./tmp_autoround", format="auto_round")
```
<details>
diff --git a/auto_round/schemes.py b/auto_round/schemes.py
index 2649e7f01..e9211959d 100644
--- a/auto_round/schemes.py
+++ b/auto_round/schemes.py
@@ -101,6 +101,7 @@ MXFP4 = QuantizationScheme.from_dict(
"data_type": "mx_fp",
"act_bits": 4,
"act_data_type": "mx_fp_rceil",
+ "act_group_size": 32,
}
)
@@ -111,6 +112,7 @@ MXFP8 = QuantizationScheme.from_dict(
"data_type": "mx_fp",
"act_bits": 8,
"act_data_type": "mx_fp_rceil",
+ "act_group_size": 32,
}
)
@@ -121,6 +123,7 @@ NVFP4 = QuantizationScheme.from_dict(
"data_type": "nv_fp",
"act_bits": 4,
"act_data_type": "nv_fp4_with_static_gs",
+ "act_group_size": 16,
}
)