Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

update readme (#550)

Wenhua Cheng59cf22edf11e
diff --git a/README.md b/README.md
index db8ae6906..bd535602e 100644
--- a/README.md
+++ b/README.md
@@ -17,7 +17,7 @@ AutoRound
AutoRound is an advanced quantization algorithm that delivers strong accuracy, even at 2-bit precision.
It leverages sign gradient descent to fine-tune both rounding values and min-max clipping thresholds in just 200 steps.
Designed for broad compatibility, it seamlessly supports a wide range of LLMs and is actively expanding to cover more VLMs as well.
-It also supports quantization and inference across multiple hardware platforms, including CPU, XPU, and CUDA.
+It also supports quantization and inference across multiple hardware platforms, including CPU, Intel GPU, and CUDA.
AutoRound also offers a variety of useful features, including mixed-bit tuning and inference, lm-head quantization,
support for exporting to formats like GPTQ/AWQ/GGUF, and flexible tuning recipes. The below
image presents an overview of AutoRound. Check out our paper on [arxiv](https://arxiv.org/pdf/2309.05516) for more
@@ -32,7 +32,7 @@ and [fbaldassarri](https://huggingface.co/fbaldassarri).
<div align="left">
## What's New
-
+* [2024/04] AutoRound has been integrated into Transformers. You can run models in the AutoRound format directly with Transformers versions later than 4.51.3.
* [2024/03] The INT2-mixed R1 model (~200GB) retains 97.9% accuracy. Check
out [OPEA/DeepSeek-R1-int2-mixed-sym-inc](https://huggingface.co/OPEA/DeepSeek-R1-int2-mixed-sym-inc).
* [2024/01] We provide experimental support for GGUF q4_0 and q4_1 formats.
@@ -72,7 +72,7 @@ pip install auto-round-lib
## Model Quantization
-### Command Line Usage (Gaudi/CPU/XPU/GPU)
+### Command Line Usage (Gaudi/CPU/Intel GPU/CUDA)
A user guide detailing the full list of supported arguments is provided by calling ```auto-round -h``` on the terminal.
Set the format you want in `format` and
@@ -81,7 +81,7 @@ more details about calibration dataset or evaluation.
```bash
auto-round \
- --model facebook/opt-125m \
+ --model Qwen/Qwen3-0.6B \
--bits 4 \
--group_size 128 \
--format "auto_gptq,auto_awq,auto_round" \
@@ -96,7 +96,7 @@ respectively. Details are as follows.
```bash
## best accuracy, 3X slower, low_gpu_mem_usage could save ~20G but ~30% slower
auto-round-best \
- --model facebook/opt-125m \
+ --model Qwen/Qwen3-0.6B \
--bits 4 \
--group_size 128 \
--low_gpu_mem_usage
@@ -105,7 +105,7 @@ auto-round-best \
```bash
## light accuracy, 2-3X speedup, slight accuracy drop at W4 and larger accuracy drop at W2
auto-round-light \
- --model facebook/opt-125m \
+ --model Qwen/Qwen3-0.6B \
--bits 4 \
--group_size 128 \
@@ -114,7 +114,7 @@ auto-round-light \
<!-- ```bash
auto-round-fast \
## fast and low memory, 2-3X speedup, slight accuracy drop at W4G128
- --model facebook/opt-125m \
+ --model Qwen/Qwen3-0.6B \
--bits 4 \
--group_size 128 \
``` -->
@@ -147,12 +147,12 @@ W2G64 Average Accuracy of 13 tasks and Time Cost Results(Testing was conducted o
</details>
-### API Usage (Gaudi/CPU/XPU/GPU)
+### API Usage (HPU/CPU/XPU/CUDA)
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
-model_name = "facebook/opt-125m"
+model_name = "Qwen/Qwen3-0.6B"
model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
@@ -315,7 +315,7 @@ in [Gaudi Guide](https://docs.habana.ai/en/latest/).
**CUDA**: no extra operations for sym quantization, for asym quantization, need to install auto-round from source
-#### Gaudi/CPU/XPU/CUDA
+#### HPU/CPU/XPU/CUDA
**Please avoid manually moving the quantized model to a different device** (e.g., model.to('cpu')) during inference, as this may cause unexpected exceptions.
@@ -486,12 +486,15 @@ model architecture or kernel limitations.
AutoRound has been integrated into multiple repositories.
+[huggingface/transformers](https://github.com/huggingface/transformers)
+
[Intel Neural Compressor](https://github.com/intel/neural-compressor)
[ModelCloud/GPTQModel](https://github.com/ModelCloud/GPTQModel)
[pytorch/ao](https://github.com/pytorch/ao)
+
## Reference
If you find AutoRound useful for your research, please cite our paper:
diff --git a/docs/step_by_step.md b/docs/step_by_step.md
index 4c208d4e6..9ffadaebd 100644
--- a/docs/step_by_step.md
+++ b/docs/step_by_step.md
@@ -18,7 +18,7 @@ pip install auto-round
The [NeelNanda/pile-10k](https://huggingface.co/datasets/NeelNanda/pile-10k) in huggingface is adopted as the default
calibration data and will be downloaded automatically from the datasets Hub. Other available datasets include:
-- `swift/pile-val-backup` for addressing HF network issue
+- `swift/pile-val-backup` from modelscope for addressing HF network issue
- `BAAI/CCI3-HQ` for Chinese
- `codeparrot/github-code-clean` for code
- `madao33/new-title-chinese` for Chinese
@@ -83,7 +83,7 @@ AutoRound supports several quantization configurations:
### Hardware Compatibility
-CPU, XPU, HPU,and CUDA for both quantization and inference.
+CPU, Intel GPU, HPU,and CUDA for both quantization and inference.
### Command Line Usage
@@ -106,7 +106,7 @@ CPU, XPU, HPU,and CUDA for both quantization and inference.
- **Light Settings:**
- This setting offers the best speed (2 - 3X faster than AutoRound), but it may cause a significant accuracy drop for small models and 2-bit quantization. It is recommended for 4-bit settings and models larger than 3B
+ This setting offers the best speed (2-3X faster than AutoRound), but it may cause a significant accuracy drop for small models and 2-bit quantization. It is recommended for 4-bit settings and models larger than 3B
```bash
auto-round-light --model facebook/opt-125m --bits 4 --group_size 128 --format "auto_gptq,auto_awq,auto_round"
@@ -135,10 +135,14 @@ autoround = AutoRound(
output_dir = "./tmp_autoround"
# format= 'auto_round'(default), 'auto_gptq', 'auto_awq'
-autoround.quantize_and_save(output_dir, format='auto_round')
+autoround.quantize_and_save(output_dir, format='auto_gptq,auto_awq,auto_round')
```
#### Mixed bits Usage
+Auto-GPTQ and Auto-AWQ only support a limited set of mixed-bit configurations. If you're unsure about the details, we recommend using the AutoRound format.
+
+Also, avoid setting mixed bits to 3 for asymmetric quantization at this time, as models exported with this setting may not be compatible with future versions of the AutoRound format.
+
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_round import AutoRound
@@ -339,7 +343,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=Fal
```
-### XPU
+### Intel GPU
Supports 4 bits only. We recommend using intel-extension-for-pytorch (IPEX) for inference.
@@ -396,7 +400,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=Fal
AutoRound automatically selects the backend for each layer based on compatibility. In general, the priority order is Marlin > ExLLaMAV2 > Triton, but the final choice depends on factors such as group size, bit width, packing format, hardware device, and other implementation details.
The backend may not always be the most suitable for certain devices.
-You can specify your preferred backend such as "ipex" for CPU and XPU, "marlin/exllamav2/triton" for CUDA, according to your needs or hardware compatibility. Please note that additional corresponding libraries may be required.
+You can specify your preferred backend such as "ipex" for CPU and Intel GPU, "marlin/exllamav2/triton" for CUDA, according to your needs or hardware compatibility. Please note that additional corresponding libraries may be required.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
@@ -410,6 +414,17 @@ text = "There is a girl who likes adventure,"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))
```
+| Name | Devices | Bits | Dtypes | Priority | Packing format | Requirements |
+|--------------------------------------|---------|---------|-----------|----------|-----------------|-------------------------------|
+| ipex | cpu/xpu | 4 | BF16/FP16 | 5 | gptq_zp+-1/awq | intel-extension-for-pytorch |
+| itrex | cpu | 2,4,8 | BF16/FP16 | 0 | gptq_zp+-1/awq | intel-extension-for-transformers |
+| marlin | cuda | 4,8 | BF16/FP16 | 6 | gptq/gptq_zp+-1 | gptqmodel |
+| exllamav2 or<br/>gptqmodel:exllamav2 | cuda | 4 | BF16/FP16 | 5 | gptq | gptqmodel |
+| exllamav2 or<br/>gptq:exllamav2 | cuda | 4 | FP16 | 5 | gptq_zp+-1 | auto-gptq |
+| gptq:cuda | cuda | 2,3,4,8 | FP16 | 0 | gptq_zp+-1 | auto-gptq |
+| triton | cuda | 2,4,8 | BF16/FP16 | 1 | gptq/gptq_zp+-1 | auto-round |
+| awq | cuda | 4 | FP16 | 5 | awq | auto-awq |
+| hpu | hpu | 4 | BF16 | 0 | gptq/gptq_zp+-1 | auto-round |
### Convert GPTQ/AWQ to AutoRound