PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
update readme (#569)
Wenhua Cheng90fef62868c9
diff --git a/README.md b/README.mdindex 53ab09119..ce16b4e1f 100644--- a/README.md+++ b/README.md@@ -34,16 +34,17 @@ and [fbaldassarri](https://huggingface.co/fbaldassarri).## What's New-* [2025/04] AutoRound provides some recipes for Qwen3 series, please refer+* [2025/05] AutoRound now supports all GGUF `q*_k_s` formats. Improved algorithms for certain configurations (e.g., q2_k_s) are planned for release in about two months, stay tuned!++* [2025/05] AutoRound has been integrated into **vLLM**. You can now run models in the AutoRound format directly with+ vLLM versions later than v0.85.post1.+* [2025/04] AutoRound provides some recipes for **Qwen3** series, please referto [Qwen3-8B-sym-recipe](./docs/Qwen3-8B-sym-recipe.md) and [Qwen3-14B-sym-recipe](./docs/Qwen3-14B-sym-recipe.md) formore details.-* [2025/04] AutoRound has been integrated into Transformers. You can run models in the AutoRound format directly with+* [2025/04] AutoRound has been integrated into **Transformers**. You can run models in the AutoRound format directly withTransformers versions later than 4.51.3.-* [2025/03] The INT2-mixed R1 model (~200GB) retains 97.9% accuracy. Check- out [OPEA/DeepSeek-R1-int2-mixed-sym-inc](https://huggingface.co/OPEA/DeepSeek-R1-int2-mixed-sym-inc).-* [2025/01] We provide experimental support for GGUF q4_0 and q4_1 formats.-* [2024/11] We provide experimental support for VLM quantization, please check out- the [README](./auto_round/mllm/README.md)+ * [2025/03] The INT2-mixed **DeepSeek-R1** model (~200GB) retains 97.9% accuracy. Check+ out [OPEA/DeepSeek-R1-int2-mixed-sym-inc](https://huggingface.co/OPEA/DeepSeek-R1-int2-mixed-sym-inc).## Installation@@ -268,7 +269,7 @@ autoround.save_quantized(output_dir, format='auto_round', inplace=True)### Export Formats**AutoRound Format**: This format is well-suited for CPU, HPU devices, 2 bits, as well as mixed-precision-inference. **[2,3,4,8] bits are supported**. However, it has not yet gained widespread community adoption.+inference. **[2,3,4,8] bits are supported**.**AutoGPTQ Format**: This format is well-suited for symmetric quantization on CUDA devices and is widely adopted by thecommunity, **[2,3,4,8] bits are supported**. However, **the@@ -278,9 +279,8 @@ models. Besides, recently 3 bits may have some accuracy issues in Transformers.**AutoAWQ Format**: This format is well-suited for asymmetric 4-bit quantization on CUDA devices and is widelyadopted within the community, **only 4-bits quantization is supported**.-**GGUF** Format: This format is well-suited for CPU devices and is widely adopted by the community, **only q4_0 and-q4_1 (W4G32) is supported in our repo**.-+**GGUF** Format: This format is well-suited for CPU devices and is widely adopted by the community. Mixed-bit+configs like `q4_k_m` are not supported yet.### Quantization CostsTesting was conducted on the Nvidia A100 80G using the nightly version of PyTorch 2.6.0.dev20241029+cu124. Please note@@ -323,7 +323,6 @@ this may cause unexpected exceptions.```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizer-from auto_round import AutoRoundConfig ## must import for auto-round formatquantized_model_path = "./tmp_autoround"model = AutoModelForCausalLM.from_pretrained(quantized_model_path,@@ -356,7 +355,7 @@ to the following table for the details and specify the backend you want.```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizer-from auto_round import AutoRoundConfig+from transformers import AutoRoundConfigquantized_model_path = "./tmp_autoround"quantization_config = AutoRoundConfig(backend="auto")@@ -376,7 +375,7 @@ Please note that the quantization config will be changed if the model is seriali```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizer-from auto_round import AutoRoundConfig ## must import for auto-round format+from transformers import AutoRoundConfigmodel_name = "ybelkada/opt-125m-gptq-4bit"quantization_config = AutoRoundConfig()diff --git a/docs/step_by_step.md b/docs/step_by_step.mdindex ee68d00c8..bc5d36595 100644--- a/docs/step_by_step.md+++ b/docs/step_by_step.md@@ -143,6 +143,8 @@ Auto-GPTQ and Auto-AWQ only support a limited set of mixed-bit configurations. IAlso, avoid setting mixed bits to 3 for asymmetric quantization at this time, as models exported with this setting may not be compatible with future versions of the AutoRound format.+vLLM and SGLang fuse MoE and QKV layers, so it's recommended not to assign different bit widths to these layers.+```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom auto_round import AutoRound@@ -243,7 +245,8 @@ autoround.quantize_and_save(output_dir, format='auto_round')### GGUF format-This format is well-suited for CPU devices and is widely adopted by the community, **only q4_0 and q4_1 (W4G32) is supported in our repo**.+This format is well-suited for CPU devices and is widely adopted by the community. Mixed bits+configs like q4_k_m have not been supported yet. Please note: In contrast to the official implementation, AutoRound does not quantize the embedding layer or the LM head layer by default.```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizer@@ -264,7 +267,7 @@ autoround = AutoRound()output_dir = "./tmp_autoround"-autoround.quantize_and_save(output_dir, format='gguf:q4_0') # gguf:q4_1+autoround.quantize_and_save(output_dir, format='gguf:q4_0') # gguf:q4_1, gguf:q*_k_s```### Device/Multi-GPU setting in Quantization@@ -398,7 +401,6 @@ Supports 2, 4, and 8 bits. We recommend using intel-extension-for-pytorch (IPEX)```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizer-from auto_round import AutoRoundConfigmodel_name = "OPEA/Qwen2.5-1.5B-Instruct-int4-sym-inc"model = AutoModelForCausalLM.from_pretrained(model_name, device_map="cpu", torch_dtype="auto")@@ -415,7 +417,6 @@ Supports 4 bits only. We recommend using intel-extension-for-pytorch (IPEX) for```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizer-from auto_round import AutoRoundConfigmodel_name = "OPEA/Qwen2.5-1.5B-Instruct-int4-sym-inc"model = AutoModelForCausalLM.from_pretrained(model_name, device_map="xpu", torch_dtype="auto")@@ -431,7 +432,6 @@ Supports 2, 3, 4, and 8 bits. We recommend using GPTQModel for 4 and 8 bits infe```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizer-from auto_round import AutoRoundConfigmodel_name = "OPEA/Qwen2.5-1.5B-Instruct-int4-sym-inc"model = AutoModelForCausalLM.from_pretrained(model_name, device_map="cuda", torch_dtype="auto")@@ -450,11 +450,11 @@ in [Gaudi Guide](https://docs.habana.ai/en/latest/).import habana_frameworks.torch.core as htcoreimport habana_frameworks.torch.hpu as hthpufrom transformers import AutoModelForCausalLM,AutoTokenizer-from auto_round import AutoRoundConfig+import torchmodel_name = "Intel/Qwen2-7B-int4-inc"tokenizer = AutoTokenizer.from_pretrained(model_name)-model = AutoModelForCausalLM.from_pretrained(model_name).to('hpu').to(bfloat16)+model = AutoModelForCausalLM.from_pretrained(model_name).to('hpu').to(torch.bfloat16)text = "There is a girl who likes adventure,"inputs = tokenizer(text, return_tensors="pt").to(model.device)print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))