Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

update format readme (#411)

Wenhua Chenga40ce88da0b3
diff --git a/README.md b/README.md
index 2f470d608..55bcb696c 100644
--- a/README.md
+++ b/README.md
@@ -26,32 +26,12 @@ details and quantized models in several Hugging Face Spaces, e.g. [OPEA](https:/
## What's New
-* [2024/12] Many quantized LLMs/VLMs using AutoRound are released in [OPEA Space](https://huggingface.co/OPEA)
+* [2024/01] We provide experimental support for GGUF q4_0 and q4_1 formats.
* [2024/11] We provide experimental support for VLM quantization, please check out
the [README](./auto_round/mllm/README.md)
* [2024/11] We provide some tips and tricks for LLM&VLM quantization, please check
out [this blog](https://medium.com/@NeuralCompressor/10-tips-for-quantizing-llms-and-vlms-with-autoround-923e733879a7)
-[//]: # (* [2024/10] AutoRound has been integrated to [torch/ao](https://github.com/pytorch/ao), check out)
-
-[//]: # ( their [release note](https://github.com/pytorch/ao/releases/tag/v0.6.1))
-
-[//]: # (* [2024/10] Important update: We now support full-range symmetric quantization and have made it the default)
-
-[//]: # ( configuration. This configuration is typically better or comparable to asymmetric quantization and significantly)
-
-[//]: # ( outperforms other symmetric variants, especially at low bit-widths like 2-bit, check)
-
-[//]: # ( out [some accuracy data](./docs/full_range_sym.md).)
-
-[//]: # (* [2024/08] AutoRound format supports Intel Gaudi2 devices. Please refer)
-
-[//]: # ( to [Intel/Qwen2-7B-int4-inc](https://huggingface.co/Intel/Qwen2-7B-int4-inc).)
-
-[//]: # (* [2024/08] AutoRound introduces several experimental features, including fast tuning of norm/bias parameters (for 2-bit)
-
-[//]: # ( and W4A4, check out [more details](./docs/tuning_norm_bias.md)), activation quantization, and the mx_fp data type.)
-
## Installation
### Install from pypi
@@ -97,7 +77,7 @@ auto-round \
--model facebook/opt-125m \
--bits 4 \
--group_size 128 \
- --format "auto_gptq,auto_round" \
+ --format "auto_gptq,auto_awq,auto_round" \
--disable_eval \
--output_dir ./tmp_autoround
```
@@ -231,10 +211,23 @@ autoround = AutoRoundMLLM(model, tokenizer, processor,
bits=bits, group_size=group_size, sym=sym)
autoround.quantize()
-# save the quantized model, set format='auto_gptq' to use AutoGPTQ format
+# save the quantized model, set format='auto_gptq' or 'auto_awq' to use other formats
output_dir = "./tmp_autoround"
autoround.save_quantized(output_dir, format='auto_round', inplace=True)
```
+#### Export Formats
+**AutoRound Format**: This format is well-suited for CPU, HPU devices, 2 bits, as well as mixed-precision
+inference. **[2,4] bits are supported**. However, it has not yet gained widespread community adoption.
+
+**AutoGPTQ Format**: This format is well-suited for symmetric quantization on CUDA devices and is widely adopted by the
+community, **[2,3,4,8] bits are supported**. However, **the
+asymmetric kernel has issues** that can cause considerable accuracy drops, particularly at 2-bit quantization and small
+models.
+
+**AutoAWQ Format**: This format is well-suited for asymmetric 4-bit quantization on CUDA devices and is widely
+adopted within the community, **only 4-bits quantization is supported**.
+
+**GGUF** Format: This format is well-suited for CPU devices and is widely adopted by the community, **only q4_0 and q4_1 (W4G32) is supported in our repo**.
### Quantization Costs