Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

update readme (#569)

Wenhua Cheng90fef62868c9
diff --git a/README.md b/README.md
index 53ab09119..ce16b4e1f 100644
--- a/README.md
+++ b/README.md
@@ -34,16 +34,17 @@ and [fbaldassarri](https://huggingface.co/fbaldassarri).
## What's New
-* [2025/04] AutoRound provides some recipes for Qwen3 series, please refer
+* [2025/05] AutoRound now supports all GGUF `q*_k_s` formats. Improved algorithms for certain configurations (e.g., q2_k_s) are planned for release in about two months, stay tuned!
+
+* [2025/05] AutoRound has been integrated into **vLLM**. You can now run models in the AutoRound format directly with
+ vLLM versions later than v0.85.post1.
+* [2025/04] AutoRound provides some recipes for **Qwen3** series, please refer
to [Qwen3-8B-sym-recipe](./docs/Qwen3-8B-sym-recipe.md) and [Qwen3-14B-sym-recipe](./docs/Qwen3-14B-sym-recipe.md) for
more details.
-* [2025/04] AutoRound has been integrated into Transformers. You can run models in the AutoRound format directly with
+* [2025/04] AutoRound has been integrated into **Transformers**. You can run models in the AutoRound format directly with
Transformers versions later than 4.51.3.
-* [2025/03] The INT2-mixed R1 model (~200GB) retains 97.9% accuracy. Check
- out [OPEA/DeepSeek-R1-int2-mixed-sym-inc](https://huggingface.co/OPEA/DeepSeek-R1-int2-mixed-sym-inc).
-* [2025/01] We provide experimental support for GGUF q4_0 and q4_1 formats.
-* [2024/11] We provide experimental support for VLM quantization, please check out
- the [README](./auto_round/mllm/README.md)
+ * [2025/03] The INT2-mixed **DeepSeek-R1** model (~200GB) retains 97.9% accuracy. Check
+ out [OPEA/DeepSeek-R1-int2-mixed-sym-inc](https://huggingface.co/OPEA/DeepSeek-R1-int2-mixed-sym-inc).
## Installation
@@ -268,7 +269,7 @@ autoround.save_quantized(output_dir, format='auto_round', inplace=True)
### Export Formats
**AutoRound Format**: This format is well-suited for CPU, HPU devices, 2 bits, as well as mixed-precision
-inference. **[2,3,4,8] bits are supported**. However, it has not yet gained widespread community adoption.
+inference. **[2,3,4,8] bits are supported**.
**AutoGPTQ Format**: This format is well-suited for symmetric quantization on CUDA devices and is widely adopted by the
community, **[2,3,4,8] bits are supported**. However, **the
@@ -278,9 +279,8 @@ models. Besides, recently 3 bits may have some accuracy issues in Transformers.
**AutoAWQ Format**: This format is well-suited for asymmetric 4-bit quantization on CUDA devices and is widely
adopted within the community, **only 4-bits quantization is supported**.
-**GGUF** Format: This format is well-suited for CPU devices and is widely adopted by the community, **only q4_0 and
-q4_1 (W4G32) is supported in our repo**.
-
+**GGUF** Format: This format is well-suited for CPU devices and is widely adopted by the community. Mixed-bit
+configs like `q4_k_m` are not supported yet.
### Quantization Costs
Testing was conducted on the Nvidia A100 80G using the nightly version of PyTorch 2.6.0.dev20241029+cu124. Please note
@@ -323,7 +323,6 @@ this may cause unexpected exceptions.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
-from auto_round import AutoRoundConfig ## must import for auto-round format
quantized_model_path = "./tmp_autoround"
model = AutoModelForCausalLM.from_pretrained(quantized_model_path,
@@ -356,7 +355,7 @@ to the following table for the details and specify the backend you want.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
-from auto_round import AutoRoundConfig
+from transformers import AutoRoundConfig
quantized_model_path = "./tmp_autoround"
quantization_config = AutoRoundConfig(backend="auto")
@@ -376,7 +375,7 @@ Please note that the quantization config will be changed if the model is seriali
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
-from auto_round import AutoRoundConfig ## must import for auto-round format
+from transformers import AutoRoundConfig
model_name = "ybelkada/opt-125m-gptq-4bit"
quantization_config = AutoRoundConfig()
diff --git a/docs/step_by_step.md b/docs/step_by_step.md
index ee68d00c8..bc5d36595 100644
--- a/docs/step_by_step.md
+++ b/docs/step_by_step.md
@@ -143,6 +143,8 @@ Auto-GPTQ and Auto-AWQ only support a limited set of mixed-bit configurations. I
Also, avoid setting mixed bits to 3 for asymmetric quantization at this time, as models exported with this setting may not be compatible with future versions of the AutoRound format.
+vLLM and SGLang fuse MoE and QKV layers, so it's recommended not to assign different bit widths to these layers.
+
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_round import AutoRound
@@ -243,7 +245,8 @@ autoround.quantize_and_save(output_dir, format='auto_round')
### GGUF format
-This format is well-suited for CPU devices and is widely adopted by the community, **only q4_0 and q4_1 (W4G32) is supported in our repo**.
+This format is well-suited for CPU devices and is widely adopted by the community. Mixed bits
+configs like q4_k_m have not been supported yet. Please note: In contrast to the official implementation, AutoRound does not quantize the embedding layer or the LM head layer by default.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
@@ -264,7 +267,7 @@ autoround = AutoRound(
)
output_dir = "./tmp_autoround"
-autoround.quantize_and_save(output_dir, format='gguf:q4_0') # gguf:q4_1
+autoround.quantize_and_save(output_dir, format='gguf:q4_0') # gguf:q4_1, gguf:q*_k_s
```
### Device/Multi-GPU setting in Quantization
@@ -398,7 +401,6 @@ Supports 2, 4, and 8 bits. We recommend using intel-extension-for-pytorch (IPEX)
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
-from auto_round import AutoRoundConfig
model_name = "OPEA/Qwen2.5-1.5B-Instruct-int4-sym-inc"
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="cpu", torch_dtype="auto")
@@ -415,7 +417,6 @@ Supports 4 bits only. We recommend using intel-extension-for-pytorch (IPEX) for
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
-from auto_round import AutoRoundConfig
model_name = "OPEA/Qwen2.5-1.5B-Instruct-int4-sym-inc"
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="xpu", torch_dtype="auto")
@@ -431,7 +432,6 @@ Supports 2, 3, 4, and 8 bits. We recommend using GPTQModel for 4 and 8 bits infe
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
-from auto_round import AutoRoundConfig
model_name = "OPEA/Qwen2.5-1.5B-Instruct-int4-sym-inc"
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="cuda", torch_dtype="auto")
@@ -450,11 +450,11 @@ in [Gaudi Guide](https://docs.habana.ai/en/latest/).
import habana_frameworks.torch.core as htcore
import habana_frameworks.torch.hpu as hthpu
from transformers import AutoModelForCausalLM,AutoTokenizer
-from auto_round import AutoRoundConfig
+import torch
model_name = "Intel/Qwen2-7B-int4-inc"
tokenizer = AutoTokenizer.from_pretrained(model_name)
-model = AutoModelForCausalLM.from_pretrained(model_name).to('hpu').to(bfloat16)
+model = AutoModelForCausalLM.from_pretrained(model_name).to('hpu').to(torch.bfloat16)
text = "There is a girl who likes adventure,"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))