Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

fix critic bug of gguf tuning (#710)

Wenhua Chengd945afaad58b
diff --git a/README.md b/README.md
index 467338a32..a2c5397d9 100644
--- a/README.md
+++ b/README.md
@@ -25,6 +25,30 @@ and [fbaldassarri](https://huggingface.co/fbaldassarri).
<img src="docs/imgs/autoround_overview.png" alt="AutoRound Overview" width="80%">
</p>
+
+## 🆕 What's New
+
+[2025/07] AutoRound now offers experimental support for **GGUF** format, and recommends using optimized RTN mode (--iters 0) for
+ all bits other than 3 bits. **A more advanced algorithm** tailored for specific configurations may be available in
+ v0.6.1. Example
+ models: [Intel/Qwen3-235B-A22B-q2ks-mixed-ar](https://huggingface.co/Intel/Qwen3-235B-A22B-q2ks-ar)
+ and [Intel/DeepSeek-R1-0528-q2ks-mixed-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-q2ks-mixed-ar).
+
+[2025/05] AutoRound provides some recipes for **DeepSeek-R1-0528**, please refer
+ to [Intel/DeepSeek-R1-0528-int2-mixed-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-int2-mixed-ar), [Intel/DeepSeek-R1-0528-int4-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-int4-ar)
+ and [Intel/DeepSeek-R1-0528-int4-asym-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-int4-asym-ar) for
+ more details.
+
+[2025/05] AutoRound has been integrated into **vLLM**. You can now run models in the AutoRound format directly with
+ vLLM versions later than v0.85.post1.
+
+[2025/04] AutoRound has been integrated into **Transformers**. You can run models in the AutoRound format directly
+ with Transformers versions later than 4.51.3.
+
+[2025/03] The INT2-mixed **DeepSeek-R1** model (~200GB) retains 97.9% accuracy. Check
+ out [OPEA/DeepSeek-R1-int2-mixed-sym-inc](https://huggingface.co/OPEA/DeepSeek-R1-int2-mixed-sym-inc).
+
+
## ✨ Key Features
@@ -32,7 +56,7 @@ and [fbaldassarri](https://huggingface.co/fbaldassarri).
Delivers strong performance even at 2–3 bits [example models](https://huggingface.co/collections/OPEA/2-3-bits-67a5f0bc6b49d73c01b4753b), with leading results at 4 bits [benchmark](https://huggingface.co/spaces/Intel/low_bit_open_llm_leaderboard).
✅ **Ecosystem Integration**
-Seamlessly works with Transformers, vLLM, TorchAO, sglang(on going,[pr](https://github.com/sgl-project/sglang/pull/6226)) and more.
+Seamlessly works with **Transformers, vLLM,** TorchAO, sglang(on going,[pr](https://github.com/sgl-project/sglang/pull/6226)) and more.
✅ **Multiple Formats Export**
Support **AutoRound, AutoAWQ, AutoGPTQ, and GGUF** for maximum compatibility. Details are shown in [export formats](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#supported-export-formats)
@@ -57,27 +81,6 @@ Includes [multiple gpus quantization](https://github.com/intel/auto-round/blob/m
🟨 Beyond weight only quantization. We are actively expanding support for additional datatypes such as **MXFP**, NVFP, W8A8, and more.
-## 🆕 What's New
-
-[2025/07] AutoRound now offers experimental support for **GGUF** format, and recommends using optimized RTN mode (--iters 0) for
- all bits other than 3 bits. **A more advanced algorithm** tailored for specific configurations may be available in
- v0.6.1. Example
- models: [Intel/Qwen3-235B-A22B-q2ks-mixed-ar](https://huggingface.co/Intel/Qwen3-235B-A22B-q2ks-ar)
- and [Intel/DeepSeek-R1-0528-q2ks-mixed-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-q2ks-mixed-ar).
-
-[2025/05] AutoRound provides some recipes for **DeepSeek-R1-0528**, please refer
- to [IntelDeepSeek-R1-0528-int2-mixed-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-int2-mixed-ar), [Intel/DeepSeek-R1-0528-int4-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-int4-ar)
- and [Intel/DeepSeek-R1-0528-int4-asym-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-int4-asym-ar) for
- more details.
-
-[2025/05] AutoRound has been integrated into **vLLM**. You can now run models in the AutoRound format directly with
- vLLM versions later than v0.85.post1.
-
-[2025/04] AutoRound has been integrated into **Transformers**. You can run models in the AutoRound format directly
- with Transformers versions later than 4.51.3.
-
-[2025/03] The INT2-mixed **DeepSeek-R1** model (~200GB) retains 97.9% accuracy. Check
- out [OPEA/DeepSeek-R1-int2-mixed-sym-inc](https://huggingface.co/OPEA/DeepSeek-R1-int2-mixed-sym-inc).
## Installation
diff --git a/auto_round/data_type/gguf.py b/auto_round/data_type/gguf.py
index 7ae7cfa46..0d61c2e03 100644
--- a/auto_round/data_type/gguf.py
+++ b/auto_round/data_type/gguf.py
@@ -169,7 +169,7 @@ def make_qp_quants(nmax, data, quant_weights):
L = torch.round(iscale * data)
diffs = data - scale * L
- best_mse = torch.sum(quant_weights * diffs * diffs)
+ best_mse = torch.sum(quant_weights * diffs * diffs, dim=-1)
for _is in range(-4, 5):
if _is == 0:
@@ -179,7 +179,7 @@ def make_qp_quants(nmax, data, quant_weights):
tmp_L = torch.round(iscale_is * data).clip(max=nmax)
diffs = data - scale_is * tmp_L
- mse = torch.sum(quant_weights * diffs * diffs)
+ mse = torch.sum(quant_weights * diffs * diffs, dim=-1)
replace_idx = mse < best_mse
best_mse[replace_idx] = mse[replace_idx]
diff --git a/auto_round/export/export_to_gguf/packing.py b/auto_round/export/export_to_gguf/packing.py
index 79e635914..a867ffae9 100644
--- a/auto_round/export/export_to_gguf/packing.py
+++ b/auto_round/export/export_to_gguf/packing.py
@@ -257,7 +257,7 @@ def make_qp_quants(nmax, data, quant_weights):
L = torch.round(iscale * data)
diffs = data - scale * L
- best_mse = torch.sum(quant_weights * diffs * diffs)
+ best_mse = torch.sum(quant_weights * diffs * diffs, dim=-1)
for _is in range(-4, 5):
if _is == 0:
@@ -267,7 +267,7 @@ def make_qp_quants(nmax, data, quant_weights):
tmp_L = torch.round(iscale_is * data).clip(max=nmax)
diffs = data - scale_is * tmp_L
- mse = torch.sum(quant_weights * diffs * diffs)
+ mse = torch.sum(quant_weights * diffs * diffs, dim=-1)
replace_idx = mse < best_mse
best_mse[replace_idx] = mse[replace_idx]