PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
update readme (#550)
Wenhua Cheng59cf22edf11e
diff --git a/README.md b/README.mdindex db8ae6906..bd535602e 100644--- a/README.md+++ b/README.md@@ -17,7 +17,7 @@ AutoRoundAutoRound is an advanced quantization algorithm that delivers strong accuracy, even at 2-bit precision.It leverages sign gradient descent to fine-tune both rounding values and min-max clipping thresholds in just 200 steps.Designed for broad compatibility, it seamlessly supports a wide range of LLMs and is actively expanding to cover more VLMs as well.-It also supports quantization and inference across multiple hardware platforms, including CPU, XPU, and CUDA.+It also supports quantization and inference across multiple hardware platforms, including CPU, Intel GPU, and CUDA.AutoRound also offers a variety of useful features, including mixed-bit tuning and inference, lm-head quantization,support for exporting to formats like GPTQ/AWQ/GGUF, and flexible tuning recipes. The belowimage presents an overview of AutoRound. Check out our paper on [arxiv](https://arxiv.org/pdf/2309.05516) for more@@ -32,7 +32,7 @@ and [fbaldassarri](https://huggingface.co/fbaldassarri).<div align="left">## What's New-+* [2024/04] AutoRound has been integrated into Transformers. You can run models in the AutoRound format directly with Transformers versions later than 4.51.3.* [2024/03] The INT2-mixed R1 model (~200GB) retains 97.9% accuracy. Checkout [OPEA/DeepSeek-R1-int2-mixed-sym-inc](https://huggingface.co/OPEA/DeepSeek-R1-int2-mixed-sym-inc).* [2024/01] We provide experimental support for GGUF q4_0 and q4_1 formats.@@ -72,7 +72,7 @@ pip install auto-round-lib## Model Quantization-### Command Line Usage (Gaudi/CPU/XPU/GPU)+### Command Line Usage (Gaudi/CPU/Intel GPU/CUDA)A user guide detailing the full list of supported arguments is provided by calling ```auto-round -h``` on the terminal.Set the format you want in `format` and@@ -81,7 +81,7 @@ more details about calibration dataset or evaluation.```bashauto-round \- --model facebook/opt-125m \+ --model Qwen/Qwen3-0.6B \--bits 4 \--group_size 128 \--format "auto_gptq,auto_awq,auto_round" \@@ -96,7 +96,7 @@ respectively. Details are as follows.```bash## best accuracy, 3X slower, low_gpu_mem_usage could save ~20G but ~30% slowerauto-round-best \- --model facebook/opt-125m \+ --model Qwen/Qwen3-0.6B \--bits 4 \--group_size 128 \--low_gpu_mem_usage@@ -105,7 +105,7 @@ auto-round-best \```bash## light accuracy, 2-3X speedup, slight accuracy drop at W4 and larger accuracy drop at W2auto-round-light \- --model facebook/opt-125m \+ --model Qwen/Qwen3-0.6B \--bits 4 \--group_size 128 \@@ -114,7 +114,7 @@ auto-round-light \<!-- ```bashauto-round-fast \## fast and low memory, 2-3X speedup, slight accuracy drop at W4G128- --model facebook/opt-125m \+ --model Qwen/Qwen3-0.6B \--bits 4 \--group_size 128 \``` -->@@ -147,12 +147,12 @@ W2G64 Average Accuracy of 13 tasks and Time Cost Results(Testing was conducted o</details>-### API Usage (Gaudi/CPU/XPU/GPU)+### API Usage (HPU/CPU/XPU/CUDA)```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizer-model_name = "facebook/opt-125m"+model_name = "Qwen/Qwen3-0.6B"model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto")tokenizer = AutoTokenizer.from_pretrained(model_name)@@ -315,7 +315,7 @@ in [Gaudi Guide](https://docs.habana.ai/en/latest/).**CUDA**: no extra operations for sym quantization, for asym quantization, need to install auto-round from source-#### Gaudi/CPU/XPU/CUDA+#### HPU/CPU/XPU/CUDA**Please avoid manually moving the quantized model to a different device** (e.g., model.to('cpu')) during inference, as this may cause unexpected exceptions.@@ -486,12 +486,15 @@ model architecture or kernel limitations.AutoRound has been integrated into multiple repositories.+[huggingface/transformers](https://github.com/huggingface/transformers)+[Intel Neural Compressor](https://github.com/intel/neural-compressor)[ModelCloud/GPTQModel](https://github.com/ModelCloud/GPTQModel)[pytorch/ao](https://github.com/pytorch/ao)+## ReferenceIf you find AutoRound useful for your research, please cite our paper:diff --git a/docs/step_by_step.md b/docs/step_by_step.mdindex 4c208d4e6..9ffadaebd 100644--- a/docs/step_by_step.md+++ b/docs/step_by_step.md@@ -18,7 +18,7 @@ pip install auto-roundThe [NeelNanda/pile-10k](https://huggingface.co/datasets/NeelNanda/pile-10k) in huggingface is adopted as the defaultcalibration data and will be downloaded automatically from the datasets Hub. Other available datasets include:-- `swift/pile-val-backup` for addressing HF network issue+- `swift/pile-val-backup` from modelscope for addressing HF network issue- `BAAI/CCI3-HQ` for Chinese- `codeparrot/github-code-clean` for code- `madao33/new-title-chinese` for Chinese@@ -83,7 +83,7 @@ AutoRound supports several quantization configurations:### Hardware Compatibility-CPU, XPU, HPU,and CUDA for both quantization and inference.+CPU, Intel GPU, HPU,and CUDA for both quantization and inference.### Command Line Usage@@ -106,7 +106,7 @@ CPU, XPU, HPU,and CUDA for both quantization and inference.- **Light Settings:**- This setting offers the best speed (2 - 3X faster than AutoRound), but it may cause a significant accuracy drop for small models and 2-bit quantization. It is recommended for 4-bit settings and models larger than 3B+ This setting offers the best speed (2-3X faster than AutoRound), but it may cause a significant accuracy drop for small models and 2-bit quantization. It is recommended for 4-bit settings and models larger than 3B```bashauto-round-light --model facebook/opt-125m --bits 4 --group_size 128 --format "auto_gptq,auto_awq,auto_round"@@ -135,10 +135,14 @@ autoround = AutoRound(output_dir = "./tmp_autoround"# format= 'auto_round'(default), 'auto_gptq', 'auto_awq'-autoround.quantize_and_save(output_dir, format='auto_round')+autoround.quantize_and_save(output_dir, format='auto_gptq,auto_awq,auto_round')```#### Mixed bits Usage+Auto-GPTQ and Auto-AWQ only support a limited set of mixed-bit configurations. If you're unsure about the details, we recommend using the AutoRound format.++Also, avoid setting mixed bits to 3 for asymmetric quantization at this time, as models exported with this setting may not be compatible with future versions of the AutoRound format.+```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizerfrom auto_round import AutoRound@@ -339,7 +343,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=Fal```-### XPU+### Intel GPUSupports 4 bits only. We recommend using intel-extension-for-pytorch (IPEX) for inference.@@ -396,7 +400,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=FalAutoRound automatically selects the backend for each layer based on compatibility. In general, the priority order is Marlin > ExLLaMAV2 > Triton, but the final choice depends on factors such as group size, bit width, packing format, hardware device, and other implementation details.The backend may not always be the most suitable for certain devices.-You can specify your preferred backend such as "ipex" for CPU and XPU, "marlin/exllamav2/triton" for CUDA, according to your needs or hardware compatibility. Please note that additional corresponding libraries may be required.+You can specify your preferred backend such as "ipex" for CPU and Intel GPU, "marlin/exllamav2/triton" for CUDA, according to your needs or hardware compatibility. Please note that additional corresponding libraries may be required.```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizer@@ -410,6 +414,17 @@ text = "There is a girl who likes adventure,"inputs = tokenizer(text, return_tensors="pt").to(model.device)print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))```+| Name | Devices | Bits | Dtypes | Priority | Packing format | Requirements |+|--------------------------------------|---------|---------|-----------|----------|-----------------|-------------------------------|+| ipex | cpu/xpu | 4 | BF16/FP16 | 5 | gptq_zp+-1/awq | intel-extension-for-pytorch |+| itrex | cpu | 2,4,8 | BF16/FP16 | 0 | gptq_zp+-1/awq | intel-extension-for-transformers |+| marlin | cuda | 4,8 | BF16/FP16 | 6 | gptq/gptq_zp+-1 | gptqmodel |+| exllamav2 or<br/>gptqmodel:exllamav2 | cuda | 4 | BF16/FP16 | 5 | gptq | gptqmodel |+| exllamav2 or<br/>gptq:exllamav2 | cuda | 4 | FP16 | 5 | gptq_zp+-1 | auto-gptq |+| gptq:cuda | cuda | 2,3,4,8 | FP16 | 0 | gptq_zp+-1 | auto-gptq |+| triton | cuda | 2,4,8 | BF16/FP16 | 1 | gptq/gptq_zp+-1 | auto-round |+| awq | cuda | 4 | FP16 | 5 | awq | auto-awq |+| hpu | hpu | 4 | BF16 | 0 | gptq/gptq_zp+-1 | auto-round |### Convert GPTQ/AWQ to AutoRound