PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
improve the robustness of scheme (#803)
Wenhua Cheng9534d2aca23a
diff --git a/README.md b/README.mdindex 5f39b4d39..15bf1112d 100644--- a/README.md+++ b/README.md@@ -38,11 +38,7 @@ and [fbaldassarri](https://huggingface.co/fbaldassarri). For usage instructions,all bits other than 3 bits. Examplemodels: [Intel/Qwen3-235B-A22B-q2ks-mixed-AutoRound](https://huggingface.co/Intel/Qwen3-235B-A22B-q2ks-mixed-AutoRound)and [Intel/DeepSeek-R1-0528-q2ks-mixed-AutoRound](https://huggingface.co/Intel/DeepSeek-R1-0528-q2ks-mixed-AutoRound). **A more advanced algorithm** tailored for specific configurations may be available in- v0.6.2.--[2025/05] AutoRound provides some recipes for **DeepSeek-R1-0528**, please refer- to [OPEA/DeepSeek-R1-0528-int2-mixed-AutoRound](https://huggingface.co/OPEA/DeepSeek-R1-0528-int2-mixed-AutoRound) and [OPEA/DeepSeek-R1-0528-int4-AutoRound](https://huggingface.co/OPEA/DeepSeek-R1-0528-int4-AutoRound) for- more details.+ v0.7.1.[2025/05] AutoRound has been integrated into **vLLM**. You can now run models in the AutoRound format directly withvLLM versions later than v0.85.post1.@@ -121,16 +117,16 @@ Please change to `auto-round-mllm` for visual-language models (VLMs) quantizatioauto-round \--model Qwen/Qwen3-0.6B \--scheme "W4A16" \- --format "auto_gptq,auto_awq,auto_round" \+ --format "auto_round" \--output_dir ./tmp_autoround```-We offer another two configurations, `auto-round-best` and `auto-round-light`, designed for optimal accuracy and improved speed, respectively. Details are as follows.+We offer another two recipes, `auto-round-best` and `auto-round-light`, designed for optimal accuracy and improved speed, respectively. Details are as follows.<details><summary>Other Recipes</summary>```bash-## best accuracy, 3X slower, low_gpu_mem_usage could save ~20G but ~30% slower+# Best accuracy, 3X slower, low_gpu_mem_usage could save ~20G but ~30% slowerauto-round-best \--model Qwen/Qwen3-0.6B \--scheme "W4A16" \@@ -138,7 +134,7 @@ auto-round-best \``````bash-## light accuracy, 2-3X speedup, slight accuracy drop at W4 and larger accuracy drop at W2+# 2-3X speedup, slight accuracy drop at W4 and larger accuracy drop at W2auto-round-light \--model Qwen/Qwen3-0.6B \--scheme "W4A16"@@ -147,7 +143,7 @@ auto-round-light \<!-- ```bashauto-round-fast \-## fast and low memory, 2-3X speedup, slight accuracy drop at W4G128+# Fast and low memory, 2-3X speedup, slight accuracy drop at W4G128--model Qwen/Qwen3-0.6B \--bits 4 \--group_size 128 \@@ -176,10 +172,8 @@ ar = AutoRound(model_name_or_path, scheme="W4A16")# Faster quantization (2–3× speedup) with slight accuracy drop at W4G128.# ar = AutoRound(model_name_or_path, nsamples=128, iters=50, lr=5e-3)-# Save quantized model-output_dir = "./tmp_autoround"# Supported formats: "auto_round" (default), "auto_gptq", "auto_awq", "llm_compressor", "gguf:q4_k_m", etc.-ar.quantize_and_save(output_dir, format="auto_round")+ar.quantize_and_save(output_dir="./tmp_autoround", format="auto_round")```<details>diff --git a/auto_round/schemes.py b/auto_round/schemes.pyindex 2649e7f01..e9211959d 100644--- a/auto_round/schemes.py+++ b/auto_round/schemes.py@@ -101,6 +101,7 @@ MXFP4 = QuantizationScheme.from_dict("data_type": "mx_fp","act_bits": 4,"act_data_type": "mx_fp_rceil",+ "act_group_size": 32,})@@ -111,6 +112,7 @@ MXFP8 = QuantizationScheme.from_dict("data_type": "mx_fp","act_bits": 8,"act_data_type": "mx_fp_rceil",+ "act_group_size": 32,})@@ -121,6 +123,7 @@ NVFP4 = QuantizationScheme.from_dict("data_type": "nv_fp","act_bits": 4,"act_data_type": "nv_fp4_with_static_gs",+ "act_group_size": 16,})