PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
fix critic bug of gguf tuning (#710)
Wenhua Chengd945afaad58b
diff --git a/README.md b/README.mdindex 467338a32..a2c5397d9 100644--- a/README.md+++ b/README.md@@ -25,6 +25,30 @@ and [fbaldassarri](https://huggingface.co/fbaldassarri).<img src="docs/imgs/autoround_overview.png" alt="AutoRound Overview" width="80%"></p>++## 🆕 What's New++[2025/07] AutoRound now offers experimental support for **GGUF** format, and recommends using optimized RTN mode (--iters 0) for+ all bits other than 3 bits. **A more advanced algorithm** tailored for specific configurations may be available in+ v0.6.1. Example+ models: [Intel/Qwen3-235B-A22B-q2ks-mixed-ar](https://huggingface.co/Intel/Qwen3-235B-A22B-q2ks-ar)+ and [Intel/DeepSeek-R1-0528-q2ks-mixed-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-q2ks-mixed-ar).++[2025/05] AutoRound provides some recipes for **DeepSeek-R1-0528**, please refer+ to [Intel/DeepSeek-R1-0528-int2-mixed-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-int2-mixed-ar), [Intel/DeepSeek-R1-0528-int4-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-int4-ar)+ and [Intel/DeepSeek-R1-0528-int4-asym-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-int4-asym-ar) for+ more details.++[2025/05] AutoRound has been integrated into **vLLM**. You can now run models in the AutoRound format directly with+ vLLM versions later than v0.85.post1.++[2025/04] AutoRound has been integrated into **Transformers**. You can run models in the AutoRound format directly+ with Transformers versions later than 4.51.3.++[2025/03] The INT2-mixed **DeepSeek-R1** model (~200GB) retains 97.9% accuracy. Check+ out [OPEA/DeepSeek-R1-int2-mixed-sym-inc](https://huggingface.co/OPEA/DeepSeek-R1-int2-mixed-sym-inc).++## ✨ Key Features@@ -32,7 +56,7 @@ and [fbaldassarri](https://huggingface.co/fbaldassarri).Delivers strong performance even at 2–3 bits [example models](https://huggingface.co/collections/OPEA/2-3-bits-67a5f0bc6b49d73c01b4753b), with leading results at 4 bits [benchmark](https://huggingface.co/spaces/Intel/low_bit_open_llm_leaderboard).✅ **Ecosystem Integration**-Seamlessly works with Transformers, vLLM, TorchAO, sglang(on going,[pr](https://github.com/sgl-project/sglang/pull/6226)) and more.+Seamlessly works with **Transformers, vLLM,** TorchAO, sglang(on going,[pr](https://github.com/sgl-project/sglang/pull/6226)) and more.✅ **Multiple Formats Export**Support **AutoRound, AutoAWQ, AutoGPTQ, and GGUF** for maximum compatibility. Details are shown in [export formats](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#supported-export-formats)@@ -57,27 +81,6 @@ Includes [multiple gpus quantization](https://github.com/intel/auto-round/blob/m🟨 Beyond weight only quantization. We are actively expanding support for additional datatypes such as **MXFP**, NVFP, W8A8, and more.-## 🆕 What's New--[2025/07] AutoRound now offers experimental support for **GGUF** format, and recommends using optimized RTN mode (--iters 0) for- all bits other than 3 bits. **A more advanced algorithm** tailored for specific configurations may be available in- v0.6.1. Example- models: [Intel/Qwen3-235B-A22B-q2ks-mixed-ar](https://huggingface.co/Intel/Qwen3-235B-A22B-q2ks-ar)- and [Intel/DeepSeek-R1-0528-q2ks-mixed-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-q2ks-mixed-ar).--[2025/05] AutoRound provides some recipes for **DeepSeek-R1-0528**, please refer- to [IntelDeepSeek-R1-0528-int2-mixed-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-int2-mixed-ar), [Intel/DeepSeek-R1-0528-int4-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-int4-ar)- and [Intel/DeepSeek-R1-0528-int4-asym-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-int4-asym-ar) for- more details.--[2025/05] AutoRound has been integrated into **vLLM**. You can now run models in the AutoRound format directly with- vLLM versions later than v0.85.post1.--[2025/04] AutoRound has been integrated into **Transformers**. You can run models in the AutoRound format directly- with Transformers versions later than 4.51.3.--[2025/03] The INT2-mixed **DeepSeek-R1** model (~200GB) retains 97.9% accuracy. Check- out [OPEA/DeepSeek-R1-int2-mixed-sym-inc](https://huggingface.co/OPEA/DeepSeek-R1-int2-mixed-sym-inc).## Installationdiff --git a/auto_round/data_type/gguf.py b/auto_round/data_type/gguf.pyindex 7ae7cfa46..0d61c2e03 100644--- a/auto_round/data_type/gguf.py+++ b/auto_round/data_type/gguf.py@@ -169,7 +169,7 @@ def make_qp_quants(nmax, data, quant_weights):L = torch.round(iscale * data)diffs = data - scale * L- best_mse = torch.sum(quant_weights * diffs * diffs)+ best_mse = torch.sum(quant_weights * diffs * diffs, dim=-1)for _is in range(-4, 5):if _is == 0:@@ -179,7 +179,7 @@ def make_qp_quants(nmax, data, quant_weights):tmp_L = torch.round(iscale_is * data).clip(max=nmax)diffs = data - scale_is * tmp_L- mse = torch.sum(quant_weights * diffs * diffs)+ mse = torch.sum(quant_weights * diffs * diffs, dim=-1)replace_idx = mse < best_msebest_mse[replace_idx] = mse[replace_idx]diff --git a/auto_round/export/export_to_gguf/packing.py b/auto_round/export/export_to_gguf/packing.pyindex 79e635914..a867ffae9 100644--- a/auto_round/export/export_to_gguf/packing.py+++ b/auto_round/export/export_to_gguf/packing.py@@ -257,7 +257,7 @@ def make_qp_quants(nmax, data, quant_weights):L = torch.round(iscale * data)diffs = data - scale * L- best_mse = torch.sum(quant_weights * diffs * diffs)+ best_mse = torch.sum(quant_weights * diffs * diffs, dim=-1)for _is in range(-4, 5):if _is == 0:@@ -267,7 +267,7 @@ def make_qp_quants(nmax, data, quant_weights):tmp_L = torch.round(iscale_is * data).clip(max=nmax)diffs = data - scale_is * tmp_L- mse = torch.sum(quant_weights * diffs * diffs)+ mse = torch.sum(quant_weights * diffs * diffs, dim=-1)replace_idx = mse < best_msebest_mse[replace_idx] = mse[replace_idx]