PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
change to new api in readme (#507)
Wenhua Cheng1d63c7bee999
diff --git a/README.md b/README.mdindex fd9bd3d29..37e4872d9 100644--- a/README.md+++ b/README.md@@ -30,8 +30,6 @@ and [fbaldassarri](https://huggingface.co/fbaldassarri).<div align="left">--## What's New* [2024/03] The INT2-mixed R1 model (~200GB) retains 97.9% accuracy. Check@@ -39,8 +37,8 @@ and [fbaldassarri](https://huggingface.co/fbaldassarri).* [2024/01] We provide experimental support for GGUF q4_0 and q4_1 formats.* [2024/11] We provide experimental support for VLM quantization, please check outthe [README](./auto_round/mllm/README.md)-## Installation+## Installation### Install from pypi@@ -89,7 +87,8 @@ auto-round \--output_dir ./tmp_autoround```-We offer two configurations, `auto-round-best` and `auto-round-light`, designed for optimal accuracy and improved speed, respectively. Details are as follows.+We offer two configurations, `auto-round-best` and `auto-round-light`, designed for optimal accuracy and improved speed,+respectively. Details are as follows.<details><summary>Other Recipes</summary>@@ -121,29 +120,30 @@ auto-round-fast \</details>-In conclusion, we recommend using **auto-round for INT4 and auto-round-best for INT2**. However, you may adjust the configuration to suit your specific requirements and available resources.+In conclusion, we recommend using **auto-round for INT4 and auto-round-best for INT2**. However, you may adjust the+configuration to suit your specific requirements and available resources.-W4G128 Average Accuracy of 13 tasks and Time Cost Results(Testing was conducted on the Nvidia A100 80G using the version of PyTorch 2.6.0 with enable_torch_compile):---| Model | Qwen2.5-0.5B-Instruct | Falcon3-3B | Qwen2.5-7B-Instruct | Meta-Llama-3.1-8B-Instruct | Falcon3-10B | Qwen2.5-72B-Instruct |-|---------|-----------------------|----------------------|---------------------|----------------------|----------------------|-----------------------|-| 16bits | 0.4192 | 0.5203 | 0.6470 | 0.6212 | 0.6151 | 0.7229 |-| Best | **0.4137**(7m) | **0.5142**(23m) | 0.6426(58m) | **0.6116**(81m) | **0.6092**(81m) | 0.7242(575m) |-| Default | 0.4129(2m) | 0.5133(6m) | 0.6441(13m) | 0.6106(13m) | 0.6080(18m) | **0.7252**(118m) |-| Light | 0.4052(2m) | 0.5108(3m) | **0.6453**(5m) | 0.6104(6m) | 0.6063(6m) | 0.7243(37m) |+W4G128 Average Accuracy of 13 tasks and Time Cost Results(Testing was conducted on the Nvidia A100 80G using the version+of PyTorch 2.6.0 with enable_torch_compile):+| Model | Qwen2.5-0.5B-Instruct | Falcon3-3B | Qwen2.5-7B-Instruct | Meta-Llama-3.1-8B-Instruct | Falcon3-10B | Qwen2.5-72B-Instruct |+|---------|-----------------------|-----------------|---------------------|----------------------------|-----------------|----------------------|+| 16bits | 0.4192 | 0.5203 | 0.6470 | 0.6212 | 0.6151 | 0.7229 |+| Best | **0.4137**(7m) | **0.5142**(23m) | 0.6426(58m) | **0.6116**(81m) | **0.6092**(81m) | 0.7242(575m) |+| Default | 0.4129(2m) | 0.5133(6m) | 0.6441(13m) | 0.6106(13m) | 0.6080(18m) | **0.7252**(118m) |+| Light | 0.4052(2m) | 0.5108(3m) | **0.6453**(5m) | 0.6104(6m) | 0.6063(6m) | 0.7243(37m) |<details><summary>W2G64 results</summary>W2G64 Average Accuracy of 13 tasks and Time Cost Results(Testing was conducted on the Nvidia A100 80G using the version of PyTorch 2.6.0 with enable_torch_compile). We recommend using higher precision for the head, tail, and non-expert modules to alleviate the significant accuracy drop.-| Model | Qwen2.5-0.5B-Instruct | Falcon3-3B | Qwen2.5-7B-Instruct | Falcon3-10B | Qwen2.5-72B-Instruct |- |---------|-----------------------|----------------------|---------------------|----------------------|-----------------------|- | 16bits | 0.4192 | 0.5203 | 0.6470 | 0.6151 | 0.7229 |- | Best | **0.2989**(6m) | **0.4267**(24m) | **0.5343**(56m)| **0.5207**(79m) | **0.6715**(564m) |- | Default | 0.2878(2m) | 0.4219(6m) | 0.5209(13m) | 0.5133(18m) | 0.6713(122m) |- | Light | 0.2760(2m) | 0.4063(3m) | 0.4764(5m) | 0.4810(7m) | 0.6581(38m) |+| Model | Qwen2.5-0.5B-Instruct | Falcon3-3B | Qwen2.5-7B-Instruct | Falcon3-10B | Qwen2.5-72B-Instruct |+ |---------|-----------------------|-----------------|---------------------|-----------------|----------------------|+| 16bits | 0.4192 | 0.5203 | 0.6470 | 0.6151 | 0.7229 |+| Best | **0.2989**(6m) | **0.4267**(24m) | **0.5343**(56m) | **0.5207**(79m) | **0.6715**(564m) |+| Default | 0.2878(2m) | 0.4219(6m) | 0.5209(13m) | 0.5133(18m) | 0.6713(122m) |+| Light | 0.2760(2m) | 0.4063(3m) | 0.4764(5m) | 0.4810(7m) | 0.6581(38m) |+</details>### API Usage (Gaudi/CPU/XPU/GPU)@@ -164,12 +164,11 @@ autoround = AutoRound(model, tokenizer, bits=bits, group_size=group_size, sym=sy# autoround = AutoRound(model, tokenizer, nsamples=512, iters=1000, low_gpu_mem_usage=True, bits=bits, group_size=group_size, sym=sym)## 2-3X speedup, slight accuracy drop at W4G128-# autoround = AutoRound(model, tokenizer, nsamples=128, iters=50, lr=5e-3, seqlen=512, batch_size=4, bits=bits, group_size=group_size, sym=sym )+# autoround = AutoRound(model, tokenizer, nsamples=128, iters=50, lr=5e-3, bits=bits, group_size=group_size, sym=sym )-autoround.quantize()output_dir = "./tmp_autoround"-## format= 'auto_round'(default in version>0.3.0), 'auto_gptq', 'auto_awq'-autoround.save_quantized(output_dir, format='auto_round', inplace=True)+## format= 'auto_round'(default), 'auto_gptq', 'auto_awq'+autoround.quantize_and_save(output_dir, format='auto_round')```<details>@@ -224,10 +223,8 @@ autoround.save_quantized(output_dir, format='auto_round', inplace=True)</details>-### API Usage for VLMs-<details><summary>Click to expand</summary>@@ -259,11 +256,11 @@ autoround.quantize()output_dir = "./tmp_autoround"autoround.save_quantized(output_dir, format='auto_round', inplace=True)```-</details>-+</details>### Export Formats+**AutoRound Format**: This format is well-suited for CPU, HPU devices, 2 bits, as well as mixed-precisioninference. **[2,3,4,8] bits are supported**. However, it has not yet gained widespread community adoption.@@ -273,13 +270,11 @@ asymmetric kernel has issues** that can cause considerable accuracy drops, partimodels.**AutoAWQ Format**: This format is well-suited for asymmetric 4-bit quantization on CUDA devices and is widely-adopted within the community, **only 4-bits quantization is supported**.+adopted within the community, **only 4-bits quantization is supported**.**GGUF** Format: This format is well-suited for CPU devices and is widely adopted by the community, **only q4_0 andq4_1 (W4G32) is supported in our repo**.--### Quantization CostsTesting was conducted on the Nvidia A100 80G using the nightly version of PyTorch 2.6.0.dev20241029+cu124. Please note@@ -330,8 +325,8 @@ inputs = tokenizer(text, return_tensors="pt").to(model.device)print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))```-#### Evaluation+<details><summary>Click to expand</summary>@@ -344,7 +339,6 @@ auto-round --model saved_quantized_model \</details>-### AutoGPTQ/AutoAWQ format```python@@ -370,8 +364,6 @@ Please note that an asterisk (*) indicates third-party quantized models, which mdifferent recipe. We greatly appreciate their efforts and encourage more users to share their models, as we cannotrelease most of the models ourselves.--Model | Supported ||-------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|| nvidia/Llama-3.1-Nemotron-70B-Instruct-HF | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Llama-3.1-Nemotron-70B-Instruct-HF-int4-sym-inc), [model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Llama-3.1-Nemotron-70B-Instruct-HF-int4-sym-inc), |@@ -414,12 +406,9 @@ release most of the models ourselves.| 01-ai/Yi-6B-Chat | [outdated-recipe](./docs/Yi-6B-Chat-asym-recipe.md) || facebook/opt-2.7b | [outdated-recipe](./docs/opt-2.7b-asym-recipe.md) || bigscience/bloom-3b | [outdated-recipe](./docs/bloom-3B-asym-recipe.md) |-| EleutherAI/gpt-j-6b | [outdated-recipe](./docs/gpt-j-6B-asym-recipe.md) |+| EleutherAI/gpt-j-6b | [outdated-recipe](./docs/gpt-j-6B-asym-recipe.md) |</details>---## Integration@@ -431,8 +420,6 @@ AutoRound has been integrated into multiple repositories.[pytorch/ao](https://github.com/pytorch/ao)--## ReferenceIf you find AutoRound useful for your research, please cite our paper: