Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

Update readme (#695)

Wenhua Chenga598f17a7647
diff --git a/README.md b/README.md
index 35ad24583..467338a32 100644
--- a/README.md
+++ b/README.md
@@ -16,47 +16,44 @@ AutoRound
## 🚀 What is AutoRound?
-**AutoRound** is an advanced quantization library mainly for large language models (LLMs) and Visual language models (Vlms).It achieves **high accuracy at ultra-low bitwidths (2–4 bits)** with minimal tuning by leveraging sign
-gradient decent, and broad hardware support. Check out our paper on [arxiv](https://arxiv.org/pdf/2309.05516) for more details and quantized models in several
+AutoRound is an advanced quantization library designed for Large Language Models (LLMs) and Vision-Language Models (VLMs). It delivers high accuracy at ultra-low bit widths (2–4 bits) with minimal tuning by leveraging sign-gradient descent and offering broad hardware compatibility. Check out our paper on [arxiv](https://arxiv.org/pdf/2309.05516) for more details and quantized models in several
Hugging Face Spaces,
e.g. [Intel](https://huggingface.co/Intel), [OPEA](https://huggingface.co/OPEA), [Kaitchup](https://huggingface.co/kaitchup)
and [fbaldassarri](https://huggingface.co/fbaldassarri).
-<div align="center">
-
-![](docs/imgs/autoround_overview.png)
-
-<div align="left">
+<p align="center">
+ <img src="docs/imgs/autoround_overview.png" alt="AutoRound Overview" width="80%">
+</p>
## ✨ Key Features
-✅ **Top-Tier Accuracy**
+✅ **Superior Accuracy**
Delivers strong performance even at 2–3 bits [example models](https://huggingface.co/collections/OPEA/2-3-bits-67a5f0bc6b49d73c01b4753b), with leading results at 4 bits [benchmark](https://huggingface.co/spaces/Intel/low_bit_open_llm_leaderboard).
✅ **Ecosystem Integration**
Seamlessly works with Transformers, vLLM, TorchAO, sglang(on going,[pr](https://github.com/sgl-project/sglang/pull/6226)) and more.
-✅ **Multi-format Export**
-Supports **AutoRound, AutoAWQ, AutoGPTQ, and GGUF** for maximum compatibility.
+✅ **Multiple Formats Export**
+Support **AutoRound, AutoAWQ, AutoGPTQ, and GGUF** for maximum compatibility. Details are shown in [export formats](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#supported-export-formats)
-✅ **Affordable Quantization cost**
-Quantize 7B models in about 10 minutes on a single GPU.
+✅ **Affordable Quantization Cost**
+Quantize 7B models in about 10 minutes on a single GPU. Details are shown in [quantization costs](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#quantization-costs)
-✅ **10+ VLM Support**
-Out-of-the-box quantization for 10+ vision-language models [example models](https://huggingface.co/collections/OPEA/vlms-autoround-675bc712fdd6a55ebaf11bfa),[support matrix](https://github.com/intel/auto-round/tree/main/auto_round/mllm#support-matrix)
+✅ **10+ VLMs Support**
+Out-of-the-box quantization for 10+ vision-language models [example models](https://huggingface.co/collections/OPEA/vlms-autoround-675bc712fdd6a55ebaf11bfa), [support matrix](https://github.com/intel/auto-round/tree/main/auto_round/mllm#support-matrix)
-✅ **Per-layer Mixed-bit Quantization**
-Assign different bit-widths per layer for fine-grained accuracy/performance trade-offs.
+✅ **Layerwise Mixed Bits Quantization**
+Assign different bits per layer for fine-grained accuracy/performance trade-offs. Details are shown in [mixed bits quantization](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#mixed-bits-usage)
✅ **Round-to-Nearest Mode**
-Use `--iters 0` for fast, calibration-free quantization with some accuracy drop.
+Use `--iters 0` for fast, calibration-free quantization with some accuracy drop. Details are shown in [rtn mode](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#rtn-mode)
-✅ **Multiple Quantization Recipes**
-Choose from `auto-round-best`, `auto-round`, and `auto-round-light` to suit your needs.
+✅ **Multiple Recipes**
+Choose from `auto-round-best`, `auto-round`, and `auto-round-light` to suit your needs. Details are shown in [quantization recipes](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#recipe-recommendation)
✅ Advanced Utilities
-Includes immediate weight packing, multiple gpu quantization and support for 10+ runtime backends.
+Includes [multiple gpus quantization](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#devicemulti-gpu-setting-in-quantization), [multiple calibration datasets](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#default-dataset) and support for [10+ runtime backends](https://github.com/intel/auto-round/blob/main/docs/step_by_step.md#specify-inference-backend).
🟨 Beyond weight only quantization. We are actively expanding support for additional datatypes such as **MXFP**, NVFP, W8A8, and more.
@@ -107,10 +104,10 @@ pip install auto-round-lib
</details>
-## Model Quantization
+## Model Quantization (CPU/Intel GPU/Gaudi/CUDA)
Please check out [User guide](./docs/step_by_step.md) for more details
-### Command Line Usage (Gaudi/CPU/Intel GPU/CUDA)
+### Command Line Usage
Please change to `auto-round-mllm` for visual-language models (VLMs) quantization. The full list of supported arguments is provided by calling `auto-round -h` on the terminal.
```bash
@@ -157,7 +154,7 @@ auto-round-fast \
In conclusion, we recommend using **auto-round for INT4 and auto-round-best for INT2**. However, you may adjust the
configuration to suit your specific requirements and available resources.
-### API Usage (CPU/Intel GPU/HPU/CUDA)
+### API Usage
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
@@ -274,8 +271,8 @@ autoround.save_quantized(output_dir, format="auto_round", inplace=True)
## Model Inference
-### vLLM
-Please note that support for the MoE model and visual language model is currently limited.
+### vLLM (CPU/Intel GPU/CUDA)
+Please note that support for the MoE models and visual language models is currently limited.
```python
from vllm import LLM, SamplingParams
@@ -294,14 +291,16 @@ for output in outputs:
print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
```
-### Transformers
+### Transformers (CPU/Intel GPU/Gaudi/CUDA)
+
AutoRound support 10+ backends an automatically selects the best available backend based on the installed libraries and prompts the user to
install additional libraries when a better backend is found.
-
**Please avoid manually moving the quantized model to a different device** (e.g., model.to('cpu')) during inference, as
this may cause unexpected exceptions.
+
+The support for Gaudi device is limited.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
diff --git a/docs/step_by_step.md b/docs/step_by_step.md
index e2b9b48f5..2d6cf5997 100644
--- a/docs/step_by_step.md
+++ b/docs/step_by_step.md
@@ -1,8 +1,7 @@
Step-by-Step
============
-This document presents step-by-step instructions for auto-round llm quantization. For vlms quantizaition, please refer to [vlms user guide](../auto_round/mllm/README.md)
-
+This document presents step-by-step instructions for auto-round llm quantization. For vlms quantization, please refer to [vlms user guide](../auto_round/mllm/README.md)
* [1 Prerequisite](#1-prerequisite)
* [2 Prepare Calibration Dataset](#2-prepare-calibration-dataset)
@@ -118,6 +117,25 @@ AutoRound supports several quantization configurations:
- **Int2 Weight Only**
- **Mixed bits Weight only**
+### Supported export Formats
+
+**AutoRound Format**: This format is well-suited for CPU, HPU devices, 2 bits, as well as mixed-precision
+inference. **[2,3,4,8] bits are supported**.
+
+**GGUF** Format: Experimental feature. This format is well-suited for CPU devices and is widely adopted by the
+community. `q*_k`,`q*_0`,`q*_1` are supported.
+
+**AutoGPTQ Format**: This format is well-suited for symmetric quantization on CUDA devices and is widely adopted by the
+community, **[2,3,4,8] bits are supported**. However, **the
+asymmetric kernel has issues** that can cause considerable accuracy drops, particularly at 2-bit quantization and small
+models. Besides, recently 3 bits may have some accuracy issues in Transformers.
+
+**AutoAWQ Format**: This format is well-suited for asymmetric 4-bit quantization on CUDA devices and is widely
+adopted within the community, **only 4-bits quantization is supported**.
+
+**llmcompressor Format**: This format is for reusing llmcompressor format, **only INT8 W8A8 dynamic quantization is
+supported**.
+
### Hardware Compatibility
CPU, Intel GPU, HPU and CUDA for both quantization and inference.
@@ -275,8 +293,9 @@ W2G64 Average Accuracy of 13 tasks and Time Cost Results(Testing was conducted o
</details>
### RTN mode
-AutoRound also supports RTN (Round-To-Nearest) mode for fast, calibration-free baseline quantization. try setting iters=0 and use group_size=32 for better results.
+AutoRound also supports RTN (Round-To-Nearest) mode for fast, calibration-free baseline quantization. try setting `iters=0` and use `group_size=32` for better results.
+For the GGUF format, we have optimized the RTN algorithm inspired by llamacpp. To use the original (pure) RTN algorithm instead, enable the `--disable_opt_rtn` option.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_round import AutoRound
@@ -323,25 +342,6 @@ output_dir = "./tmp_autoround"
autoround.quantize_and_save(output_dir, format="gguf:q4_k_m") # gguf:q*_k_s,gguf:q*_k_0,gguf:q*_k_1,
```
-### Supported export Formats
-
-**AutoRound Format**: This format is well-suited for CPU, HPU devices, 2 bits, as well as mixed-precision
-inference. **[2,3,4,8] bits are supported**.
-
-**GGUF** Format: Experimental feature. This format is well-suited for CPU devices and is widely adopted by the
-community. `q*_k`,`q*_0`,`q*_1` are supported.
-
-**AutoGPTQ Format**: This format is well-suited for symmetric quantization on CUDA devices and is widely adopted by the
-community, **[2,3,4,8] bits are supported**. However, **the
-asymmetric kernel has issues** that can cause considerable accuracy drops, particularly at 2-bit quantization and small
-models. Besides, recently 3 bits may have some accuracy issues in Transformers.
-
-**AutoAWQ Format**: This format is well-suited for asymmetric 4-bit quantization on CUDA devices and is widely
-adopted within the community, **only 4-bits quantization is supported**.
-
-**llmcompressor Format**: This format is for reusing llmcompressor format, **only INT8 W8A8 dynamic quantization is
-supported**.
-
### Quantization Costs