Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

support static global scale for nvfp4 and update readme (#691)

Wenhua Cheng4a3530fec349
diff --git a/README.md b/README.md
index 3416b09b0..35ad24583 100644
--- a/README.md
+++ b/README.md
@@ -14,15 +14,11 @@ AutoRound
---
<div align="left">
-AutoRound is an advanced quantization algorithm that delivers strong accuracy, even at 2-bit precision.
-It leverages sign gradient descent to fine-tune both rounding values and min-max clipping thresholds in just 200 steps.
-Designed for broad compatibility, it seamlessly supports a wide range of LLMs and is actively expanding to cover more
-VLMs as well.
-It also supports quantization and inference across multiple hardware platforms, including CPU, Intel GPU, and CUDA.
-AutoRound also offers a variety of useful features, including mixed-bit tuning and inference, lm-head quantization,
-support for exporting to formats like GPTQ/AWQ/GGUF, and flexible tuning recipes. The below
-image presents an overview of AutoRound. Check out our paper on [arxiv](https://arxiv.org/pdf/2309.05516) for more
-details and quantized models in several Hugging Face Spaces,
+## 🚀 What is AutoRound?
+
+**AutoRound** is an advanced quantization library mainly for large language models (LLMs) and Visual language models (Vlms).It achieves **high accuracy at ultra-low bitwidths (2–4 bits)** with minimal tuning by leveraging sign
+gradient decent, and broad hardware support. Check out our paper on [arxiv](https://arxiv.org/pdf/2309.05516) for more details and quantized models in several
+Hugging Face Spaces,
e.g. [Intel](https://huggingface.co/Intel), [OPEA](https://huggingface.co/OPEA), [Kaitchup](https://huggingface.co/kaitchup)
and [fbaldassarri](https://huggingface.co/fbaldassarri).
@@ -32,23 +28,58 @@ and [fbaldassarri](https://huggingface.co/fbaldassarri).
<div align="left">
-## What's New
+## ✨ Key Features
+
+
+✅ **Top-Tier Accuracy**
+Delivers strong performance even at 2–3 bits [example models](https://huggingface.co/collections/OPEA/2-3-bits-67a5f0bc6b49d73c01b4753b), with leading results at 4 bits [benchmark](https://huggingface.co/spaces/Intel/low_bit_open_llm_leaderboard).
+
+✅ **Ecosystem Integration**
+Seamlessly works with Transformers, vLLM, TorchAO, sglang(on going,[pr](https://github.com/sgl-project/sglang/pull/6226)) and more.
+
+✅ **Multi-format Export**
+Supports **AutoRound, AutoAWQ, AutoGPTQ, and GGUF** for maximum compatibility.
+
+✅ **Affordable Quantization cost**
+Quantize 7B models in about 10 minutes on a single GPU.
+
+✅ **10+ VLM Support**
+Out-of-the-box quantization for 10+ vision-language models [example models](https://huggingface.co/collections/OPEA/vlms-autoround-675bc712fdd6a55ebaf11bfa),[support matrix](https://github.com/intel/auto-round/tree/main/auto_round/mllm#support-matrix)
+
+✅ **Per-layer Mixed-bit Quantization**
+Assign different bit-widths per layer for fine-grained accuracy/performance trade-offs.
+
+✅ **Round-to-Nearest Mode**
+Use `--iters 0` for fast, calibration-free quantization with some accuracy drop.
-* [2025.07] AutoRound now offers experimental support for **GGUF** format, and recommends using optimized RTN mode (--iters 0) for
- all bits other than 3 bits. A more advanced algorithm tailored for specific configurations may be available in
+✅ **Multiple Quantization Recipes**
+Choose from `auto-round-best`, `auto-round`, and `auto-round-light` to suit your needs.
+
+✅ Advanced Utilities
+Includes immediate weight packing, multiple gpu quantization and support for 10+ runtime backends.
+
+🟨 Beyond weight only quantization. We are actively expanding support for additional datatypes such as **MXFP**, NVFP, W8A8, and more.
+
+## 🆕 What's New
+
+[2025/07] AutoRound now offers experimental support for **GGUF** format, and recommends using optimized RTN mode (--iters 0) for
+ all bits other than 3 bits. **A more advanced algorithm** tailored for specific configurations may be available in
v0.6.1. Example
- models: [Intel/Qwen3-235B-A22B-q2ks-mixed-AutoRound-inc-v1](https://huggingface.co/Intel/Qwen3-235B-A22B-q2ks-mixed-AutoRound-inc-v1)
- and [Intel/DeepSeek-R1-0528-q2ks-mixed-AutoRound-inc-v1](https://huggingface.co/Intel/DeepSeek-R1-0528-q2ks-mixed-AutoRound-inc-v1).
-* [2025.05] AutoRound provides some recipes for **DeepSeek-R1-0528**, please refer
- to [DeepSeek-R1-0528-int2-mixed-sym-inc](https://huggingface.co/Intel/DeepSeek-R1-0528-int2-mixed-sym-inc), [DeepSeek-R1-0528-int4-sym-gptq-inc](https://huggingface.co/Intel/DeepSeek-R1-0528-int4-gptq-inc-auto-round)
- and [DeepSeek-R1-0528-int4-asym-awq-inc](https://huggingface.co/Intel/DeepSeek-R1-0528-int4-awq-inc-auto-round) for
+ models: [Intel/Qwen3-235B-A22B-q2ks-mixed-ar](https://huggingface.co/Intel/Qwen3-235B-A22B-q2ks-ar)
+ and [Intel/DeepSeek-R1-0528-q2ks-mixed-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-q2ks-mixed-ar).
+
+[2025/05] AutoRound provides some recipes for **DeepSeek-R1-0528**, please refer
+ to [IntelDeepSeek-R1-0528-int2-mixed-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-int2-mixed-ar), [Intel/DeepSeek-R1-0528-int4-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-int4-ar)
+ and [Intel/DeepSeek-R1-0528-int4-asym-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-int4-asym-ar) for
more details.
-* [2025/05] AutoRound has been integrated into **vLLM**. You can now run models in the AutoRound format directly with
+
+[2025/05] AutoRound has been integrated into **vLLM**. You can now run models in the AutoRound format directly with
vLLM versions later than v0.85.post1.
-* [2025/04] AutoRound has been integrated into **Transformers**. You can run models in the AutoRound format directly
- with
- Transformers versions later than 4.51.3.
-* [2025/03] The INT2-mixed **DeepSeek-R1** model (~200GB) retains 97.9% accuracy. Check
+
+[2025/04] AutoRound has been integrated into **Transformers**. You can run models in the AutoRound format directly
+ with Transformers versions later than 4.51.3.
+
+[2025/03] The INT2-mixed **DeepSeek-R1** model (~200GB) retains 97.9% accuracy. Check
out [OPEA/DeepSeek-R1-int2-mixed-sym-inc](https://huggingface.co/OPEA/DeepSeek-R1-int2-mixed-sym-inc).
## Installation
@@ -78,12 +109,9 @@ pip install auto-round-lib
## Model Quantization
+Please check out [User guide](./docs/step_by_step.md) for more details
### Command Line Usage (Gaudi/CPU/Intel GPU/CUDA)
-
-A user guide detailing the full list of supported arguments is provided by calling ```auto-round -h``` on the terminal.
-Set the format you want in `format` and
-multiple formats exporting has been supported. Please check out [step-by-step-instruction](./docs/step_by_step.md) for
-more details about calibration dataset or evaluation.
+Please change to `auto-round-mllm` for visual-language models (VLMs) quantization. The full list of supported arguments is provided by calling `auto-round -h` on the terminal.
```bash
auto-round \
@@ -94,9 +122,7 @@ auto-round \
--output_dir ./tmp_autoround
```
-We offer another two configurations, `auto-round-best` and `auto-round-light`,
-designed for optimal accuracy and improved speed,
-respectively. Details are as follows.
+We offer another two configurations, `auto-round-best` and `auto-round-light`, designed for optimal accuracy and improved speed, respectively. Details are as follows.
<details>
<summary>Other Recipes</summary>
@@ -131,30 +157,7 @@ auto-round-fast \
In conclusion, we recommend using **auto-round for INT4 and auto-round-best for INT2**. However, you may adjust the
configuration to suit your specific requirements and available resources.
-W4G128 Average Accuracy of 13 tasks and Time Cost Results(Testing was conducted on the Nvidia A100 80G using the version
-of PyTorch 2.6.0 with enable_torch_compile):
-
-| Model | Qwen2.5-0.5B-Instruct | Falcon3-3B | Qwen2.5-7B-Instruct | Meta-Llama-3.1-8B-Instruct | Falcon3-10B | Qwen2.5-72B-Instruct |
-|---------|-----------------------|-----------------|---------------------|----------------------------|-----------------|----------------------|
-| 16bits | 0.4192 | 0.5203 | 0.6470 | 0.6212 | 0.6151 | 0.7229 |
-| Best | **0.4137**(7m) | **0.5142**(23m) | 0.6426(58m) | **0.6116**(65m) | **0.6092**(81m) | 0.7242(575m) |
-| Default | 0.4129(2m) | 0.5133(6m) | 0.6441(13m) | 0.6106(13m) | 0.6080(18m) | **0.7252**(118m) |
-| Light | 0.4052(2m) | 0.5108(3m) | **0.6453**(5m) | 0.6104(6m) | 0.6063(6m) | 0.7243(37m) |
-
-<details>
- <summary>W2G64 results</summary>
-W2G64 Average Accuracy of 13 tasks and Time Cost Results(Testing was conducted on the Nvidia A100 80G using the version of PyTorch 2.6.0 with enable_torch_compile). We recommend using higher precision for the head, tail, and non-expert modules to alleviate the significant accuracy drop.
-
-| Model | Qwen2.5-0.5B-Instruct | Falcon3-3B | Qwen2.5-7B-Instruct | Falcon3-10B | Qwen2.5-72B-Instruct |
- |---------|-----------------------|-----------------|---------------------|-----------------|----------------------|
-| 16bits | 0.4192 | 0.5203 | 0.6470 | 0.6151 | 0.7229 |
-| Best | **0.2989**(6m) | **0.4267**(24m) | **0.5343**(56m) | **0.5207**(79m) | **0.6715**(564m) |
-| Default | 0.2878(2m) | 0.4219(6m) | 0.5209(13m) | 0.5133(18m) | 0.6713(122m) |
-| Light | 0.2760(2m) | 0.4063(3m) | 0.4764(5m) | 0.4810(7m) | 0.6581(38m) |
-
-</details>
-
-### API Usage (HPU/CPU/XPU/CUDA)
+### API Usage (CPU/Intel GPU/HPU/CUDA)
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
@@ -239,10 +242,9 @@ results.
<details>
<summary>Click to expand</summary>
-**This feature is experimental and may be subject to changes**, including potential bug fixes, API modifications, or
-adjustments to default hype-parameters
+**This feature is experimental and may be subject to changes**.
-By default, AutoRoundMLLM only quantizes the text module of VLMs and uses `NeelNanda/pile-10k` for calibration. To
+By default, AutoRoundMLLM only quantize the text module of VLMs and uses `NeelNanda/pile-10k` for calibration. To
quantize the entire model, you can enable `quant_nontext_module` by setting it to True, though support for this feature
is limited. For more information, please refer to the AutoRoundMLLM [readme](./auto_round/mllm/README.md).
@@ -268,256 +270,55 @@ autoround.save_quantized(output_dir, format="auto_round", inplace=True)
</details>
-### Export Formats
-
-**AutoRound Format**: This format is well-suited for CPU, HPU devices, 2 bits, as well as mixed-precision
-inference. **[2,3,4,8] bits are supported**.
-
-**GGUF** Format: Experimental feature. This format is well-suited for CPU devices and is widely adopted by the
-community. `q*_k`,`q*_0`,`q*_1` are supported.
-
-**AutoGPTQ Format**: This format is well-suited for symmetric quantization on CUDA devices and is widely adopted by the
-community, **[2,3,4,8] bits are supported**. However, **the
-asymmetric kernel has issues** that can cause considerable accuracy drops, particularly at 2-bit quantization and small
-models. Besides, recently 3 bits may have some accuracy issues in Transformers.
-
-**AutoAWQ Format**: This format is well-suited for asymmetric 4-bit quantization on CUDA devices and is widely
-adopted within the community, **only 4-bits quantization is supported**.
-
-**llmcompressor Format**: This format is for reusing llmcompressor format, **only INT8 W8A8 dynamic quantization is
-supported**.
-
-
-
-### Quantization Costs
-
-Testing was conducted on the Nvidia A100 80G using the nightly version of PyTorch 2.6.0.dev20241029+cu124. Please note
-that data
-loading and packing costs have been excluded from the evaluation. **We recommend enabling torch.compile for PyTorch
-versions 2.6 and above.**
-
-To optimize GPU memory usage, in addition to activating `low_gpu_mem_usage`, you can set `gradient_accumulate_steps=8`
-and a
-`batch_size=1`, though this may increase tuning time.
-
-The 3B and 14B models were evaluated on Qwen 2.5, the 8X7B model is Mixtral, while the remaining models utilized LLaMA
-3.1.
-| Torch version/Config W4G128 | 3B | 8B | 14B | 70B | 8X7B |
-|---------------------------------------------------------------------------------------------|---------------|----------------|----------------|-----------------|----------------|
-| 2.6 with torch compile | 7min<br/>10GB | 12min<br/>18GB | 23min<br/>22GB | 120min<br/>42GB | 28min<br/>46GB |
-| 2.6 with torch compile <br/> low_gpu_mem_usage=True | 12min<br/>6GB | 19min<br/>10GB | 33min<br/>11GB | 140min<br/>25GB | 38min<br/>36GB |
-| 2.6 with torch compile <br/> low_gpu_mem_usage=True <br/> gradient_accumulate_steps=8,bs=1 | 15min<br/>3GB | 25min<br/>6GB | 45min<br/>7GB | 187min<br/>19GB | 75min<br/>36GB |
-| 2.5 w/o torch compile | 8min<br/>10GB | 16min<br/>20GB | 30min<br/>25GB | 140min<br/>49GB | 50min<br/>49GB |
## Model Inference
-Please run the quantization code first
-
-### AutoRound format
-
-**CPU**: pip install intel-extension-for-pytorch(much higher speed on Intel CPU) or pip
-install intel-extension-for-transformers,
-
-**HPU**: docker image with Gaudi Software Stack is recommended. More details can be found
-in [Gaudi Guide](https://docs.habana.ai/en/latest/).
-
-**CUDA**: no extra operations for sym quantization, for asym quantization, need to install auto-round from source
-
-#### HPU/CPU/XPU/CUDA
-
-**Please avoid manually moving the quantized model to a different device** (e.g., model.to('cpu')) during inference, as
-this may cause unexpected exceptions.
-
+### vLLM
+Please note that support for the MoE model and visual language model is currently limited.
```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
+from vllm import LLM, SamplingParams
-quantized_model_path = "./tmp_autoround"
-model = AutoModelForCausalLM.from_pretrained(quantized_model_path, device_map="auto", torch_dtype="auto")
-tokenizer = AutoTokenizer.from_pretrained(quantized_model_path)
-text = "There is a girl who likes adventure,"
-inputs = tokenizer(text, return_tensors="pt").to(model.device)
-print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
-```
+prompts = [
+ "Hello, my name is",
+]
+sampling_params = SamplingParams(temperature=0.6, top_p=0.95)
+model_name = "Intel/DeepSeek-R1-0528-Qwen3-8B-int4-AutoRound"
+llm = LLM(model=model_name)
-### Specify backend
-
-AutoRound automatically selects the best available backend based on the installed libraries and prompts the user to
-install additional libraries when a better backend is found. On CUDA, the default priority is Marlin > ExLLaMAV2 >
-Triton, but the final choice depends on factors such as bits, group_size, packing format compatibility, etc. And the
-backend may not always be the most suitable for certain devices. Please refer
-to the following table for the details and specify the backend you want.
-
-| Name | Devices | Bits | Dtypes | Priority | Packing format | Requirements |
-|--------------------------------------|--------------|---------|-----------|----------|-----------------|---------------------------------------|
-| ipex | cpu/xpu | 4 | BF16/FP16 | 5 | gptq_zp+-1/awq | intel-extension-for-pytorch |
-| itrex | cpu | 2,4,8 | BF16/FP16 | 1 | gptq_zp+-1/awq | <br/>intel-extension-for-transformers |
-| marlin | cuda | 4,8 | BF16/FP16 | 6 | gptq/gptq_zp+-1 | gptqmodel |
-| exllamav2 or<br/>gptqmodel:exllamav2 | cuda | 4 | BF16/FP16 | 5 | gptq | gptqmodel |
-| exllamav2 or<br/>gptq:exllamav2 | cuda | 4 | FP16 | 5 | gptq_zp+-1 | auto-gptq |
-| gptq:cuda | cuda | 2,3,4,8 | FP16 | 1 | gptq_zp+-1 | auto-gptq |
-| triton | xpu/cuda | 2,4,8 | BF16/FP16 | 2 | gptq/gptq_zp+-1 | auto-round |
-| awq | cuda | 4 | FP16 | 5 | awq | auto-awq |
-| hpu | hpu | 4 | BF16 | 0 | gptq/gptq_zp+-1 | auto-round |
-| torch | xpu/cpu/cuda | 2,3,4,8 | BF16/FP16 | 0 | gptq/gptq_zp+-1 | auto-round |
+outputs = llm.generate(prompts, sampling_params)
-```python
-from transformers import AutoModelForCausalLM, AutoTokenizer
-from transformers import AutoRoundConfig
-
-quantized_model_path = "./tmp_autoround"
-quantization_config = AutoRoundConfig(backend="auto")
-model = AutoModelForCausalLM.from_pretrained(
- quantized_model_path, device_map="auto", torch_dtype="auto", quantization_config=quantization_config
-)
-tokenizer = AutoTokenizer.from_pretrained(quantized_model_path)
-text = "There is a girl who likes adventure,"
-inputs = tokenizer(text, return_tensors="pt").to(model.device)
-print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
+for output in outputs:
+ prompt = output.prompt
+ generated_text = output.outputs[0].text
+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")
```
-### Convert GPTQ/AWQ format to AutoRound
+### Transformers
+
+AutoRound support 10+ backends an automatically selects the best available backend based on the installed libraries and prompts the user to
+install additional libraries when a better backend is found.
-Most GPTQ/AWQ models can be converted to the AutoRound format for better compatibility and support with Intel devices.
-Please note that the quantization config will be changed if the model is serialized.
+**Please avoid manually moving the quantized model to a different device** (e.g., model.to('cpu')) during inference, as
+this may cause unexpected exceptions.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
-from transformers import AutoRoundConfig
-model_name = "ybelkada/opt-125m-gptq-4bit"
-quantization_config = AutoRoundConfig()
-model = AutoModelForCausalLM.from_pretrained(
- model_name, device_map="cpu", torch_dtype="auto", quantization_config=quantization_config
-)
+model_name = "Intel/DeepSeek-R1-0528-Qwen3-8B-int4-AutoRound"
+model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
text = "There is a girl who likes adventure,"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
-print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))
-```
-
-#### Evaluation
-
-<details>
- <summary>Click to expand</summary>
-
-```bash
-auto-round --model saved_quantized_model \
- --eval \
- --task lambada_openai \
- --eval_bs 1
+print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
```
+## Acknowledgement
+Special thanks to open-source low precision libraries such as AutoGPTQ, AutoAWQ, GPTQModel, Triton, Marlin, and ExLLaMAV2 for providing low-precision CUDA kernels, which are leveraged in AutoRound.
-</details>
-
-## Support List
+## 🌟 Support Us
+If you find AutoRound helpful, please ⭐ star the repo and share it with your community!
-AutoRound supports basically all the major large language models.
-<details>
- <summary>Supported Models List</summary>
-
-Please note that an asterisk (*) indicates third-party quantized models, which may lack accuracy data and use a
-different recipe. We greatly appreciate their efforts and encourage more users to share their models, as we cannot
-release most of the models ourselves.
-
- Model | Supported |
-|-------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| nvidia/Llama-3.1-Nemotron-70B-Instruct-HF | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Llama-3.1-Nemotron-70B-Instruct-HF-int4-sym-inc), [model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Llama-3.1-Nemotron-70B-Instruct-HF-int4-sym-inc), |
-| meta-llama/Llama-3.2-90B-Vision-Instruct | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Llama-3.2-90B-Vision-Instruct-int4-sym-inc), [model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Llama-3.2-90B-Vision-Instruct-int4-sym-inc) |
-| Qwen/QwQ-32B-Preview | [model-opea-int4-sym-autoround-mixed](https://huggingface.co/OPEA/QwQ-32B-Preview-int4-sym-mixed-inc),[model-opea-int4-sym-autoawq-mixed](https://huggingface.co/OPEA/QwQ-32B-Preview-int4-sym-mixed-awq-inc) |
-| THUDM/cogvlm2-llama3-chat-19B | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/cogvlm2-llama3-chat-19B-int4-sym-inc) |
-| Qwen/Qwen2-VL-Instruct | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Qwen2-VL-7B-Instruct-int4-sym-inc),[model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Qwen2-VL-7B-Instruct-int4-sym-inc) |
-| meta-llama/Llama-3.2-11B-Vision | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-int4-sym-inc), [model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-int4-sym-inc) |
-| microsoft/Phi-3.5-vision-instruct | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-int4-sym-inc), [model-opea-int4-sym-gptq](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-int4-sym-inc) |
-| liuhaotian/llava-v1.5-7b | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/llava-v1.5-7b-int4-sym-inc),[model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/llava-v1.5-7b-int4-sym-inc) |
-| Qwen/Qwen2.5-7B-Instruct | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Qwen2.5-7B-Instruct-int4-sym-inc),[model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Qwen2.5-7B-Instruct-int4-sym-inc) [model-kaitchup-autogptq-int4*](https://huggingface.co/kaitchup/Qwen2.5-7B-Instruct-AutoRound-GPTQ-asym-4bit), [recipe](./docs/Qwen2.5-7B-Instruct-sym.md) |
-| Qwen/Qwen2.5-14B-Instruct | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Qwen2.5-14B-Instruct-int4-sym-inc),[model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Qwen2.5-14B-Instruct-int4-sym-inc) |
-| Qwen/Qwen2.5-32B-Instruct | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Qwen2.5-32B-Instruct-int4-sym-inc) |
-| Qwen/Qwen2.5-Coder-32B-Instruct | [model-kaitchup-autogptq-int4*](https://huggingface.co/kaitchup/Qwen2.5-Coder-32B-Instruct-AutoRound-GPTQ-4bit) |
-| Qwen/Qwen2.5-72B-Instruct | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Qwen2.5-72B-Instruct-int4-sym-inc),[model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Qwen2.5-72B-Instruct-int4-sym-inc), [model-kaitchup-autogptq-int4*](https://huggingface.co/kaitchup/Qwen2.5-72B-Instruct-AutoRound-GPTQ-4bit), [model-kaitchup-autogptq-int2*](https://huggingface.co/kaitchup/Qwen2.5-72B-Instruct-AutoRound-GPTQ-2bit), [recipe](./docs/Qwen2.5-72B-Instruct-sym.md) |
-| meta-llama/Meta-Llama-3.1-70B-Instruct | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Meta-Llama-3.1-70B-Instruct-int4-sym-inc), [model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Meta-Llama-3.1-70B-Instruct-int4-sym-inc),[model-opea-int4-asym-autoround](https://huggingface.co/OPEA/Meta-Llama-3.1-70B-Instruct-int4-asym-inc) |
-| meta-llama/Meta-Llama-3.1-8B-Instruct | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Meta-Llama-3.1-8B-Instruct-int4-sym-inc),[model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Meta-Llama-3.1-8B-Instruct-int4-sym-inc),[model-kaitchup-autogptq-int4*](https://huggingface.co/kaitchup/Meta-Llama-3.1-8B-Instruct-autoround-gptq-4bit-asym), [model-kaitchup-autogptq-sym-int4*](https://huggingface.co/kaitchup/Meta-Llama-3.1-8B-Instruct-autoround-gptq-4bit-sym), [recipe](https://huggingface.co/Intel/Meta-Llama-3.1-8B-Instruct-int4-inc) |
-| meta-llama/Meta-Llama-3.1-8B | [model-kaitchup-autogptq-sym-int4*](https://huggingface.co/kaitchup/Meta-Llama-3.1-8B-autoround-gptq-4bit-sym) |
-| Qwen/Qwen2-7B | [model-autoround-sym-int4](https://huggingface.co/Intel/Qwen2-7B-int4-inc), [model-autogptq-sym-int4](https://huggingface.co/Intel/Qwen2-7B-int4-inc) |
-| THUDM/glm-4-9b-chat | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/glm-4-9b-chat-int4-sym-inc),[model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/glm-4-9b-chat-int4-sym-inc) |
-| Qwen/Qwen2-57B-A14B-Instruct | [model-autoround-sym-int4](https://huggingface.co/Intel/Qwen2-57B-A14B-Instruct-int4-inc),[model-autogptq-sym-int4](https://huggingface.co/Intel/Qwen2-57B-A14B-Instruct-int4-inc) |
-| 01-ai/Yi-1.5-9B | [model-LnL-AI-autogptq-int4*](https://huggingface.co/LnL-AI/Yi-1.5-9B-4bit-gptq-autoround) |
-| 01-ai/Yi-1.5-9B-Chat | [model-LnL-AI-autogptq-int4*](https://huggingface.co/LnL-AI/Yi-1.5-9B-Chat-4bit-gptq-autoround) |
-| Intel/neural-chat-7b-v3-3 | [model-autogptq-int4](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc) |
-| Intel/neural-chat-7b-v3-1 | [model-autogptq-int4](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc) |
-| TinyLlama-1.1B-intermediate | [model-LnL-AI-autogptq-int4*](https://huggingface.co/LnL-AI/TinyLlama-1.1B-intermediate-step-1341k-3T-autoround-lm_head-symFalse) |
-| mistralai/Mistral-7B-v0.1 | [model-autogptq-lmhead-int4](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc-lmhead), [model-autogptq-int4](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) |
-| google/gemma-2b | [model-autogptq-int4](https://huggingface.co/Intel/gemma-2b-int4-inc) |
-| tiiuae/falcon-7b | [model-autogptq-int4-G64](https://huggingface.co/Intel/falcon-7b-int4-inc) |
-| sapienzanlp/modello-italia-9b | [model-fbaldassarri-autogptq-int4*](https://huggingface.co/fbaldassarri/modello-italia-9b-autoround-w4g128-cpu) |
-| microsoft/phi-2 | [model-autoround-sym-int4](https://huggingface.co/Intel/phi-2-int4-inc) [model-autogptq-sym-int4](https://huggingface.co/Intel/phi-2-int4-inc) |
-| microsoft/Phi-3.5-mini-instruct | [model-kaitchup-autogptq-sym-int4*](https://huggingface.co/kaitchup/Phi-3.5-Mini-instruct-AutoRound-4bit) |
-| mistralai/Mistral-7B-Instruct-v0.2 | [outdated-recipe](./docs/Mistral-7B-Instruct-v0.2-asym-recipe.md) |
-| mistralai/Mixtral-8x7B-Instruct-v0.1 | [outdated-recipe](./docs/Mixtral-8x7B-Instruct-v0.1-asym-recipe.md) |
-| mistralai/Mixtral-8x7B-v0.1 | [outdated-recipe](./docs/Mixtral-8x7B-v0.1-asym-acc.md) |
-| meta-llama/Meta-Llama-3-8B-Instruct | [outdated-recipe](./docs/Meta-Llama-3-8B-Instruct-asym-recipe.md) |
-| google/gemma-7b | [outdated-recipe](./docs/gemma-7b-asym-recipe.md) |
-| meta-llama/Llama-2-7b-chat-hf | [outdated-recipe](./docs/Llama-2-7b-chat-hf-asym-recipe.md) |
-| baichuan-inc/Baichuan2-7B-Chat | [outdated-recipe](./docs/baichuan2-7b-cha-asym-recipe.md) |
-| 01-ai/Yi-6B-Chat | [outdated-recipe](./docs/Yi-6B-Chat-asym-recipe.md) |
-| facebook/opt-2.7b | [outdated-recipe](./docs/opt-2.7b-asym-recipe.md) |
-| bigscience/bloom-3b | [outdated-recipe](./docs/bloom-3B-asym-recipe.md) |
-| EleutherAI/gpt-j-6b | [outdated-recipe](./docs/gpt-j-6B-asym-recipe.md) |
-
-</details>
-
-### VLM Support Matrix
-
-For most VLMs, we typically support the default quantization configuration, which involves quantizing only the language
-component while excluding the visual component. Besides, we also support quantizing non-text modules of models that
-follow the Hugging Face standard, i.e., those with a typical processor, though inference may have some issues due to
-model architecture or kernel limitations.
-
-| Model | calibration dataset | quant nontext module | Quantized Model Link |
-|-----------------------------------------------|---------------------|----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| allenai/Molmo | pile | X | [Molmo-7B-D-0924-int4-sym](https://huggingface.co/OPEA/Molmo-7B-D-0924-int4-sym-inc), [Molmo-72B-0924-int4-sym-gptq](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-gptq-inc), [Molmo-72B-0924-int4-sym](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-inc) |
-| deepseek-ai/deepseek-vl2 | pile/llava | √ | [deepseek-vl2-int4-sym-gptq](https://huggingface.co/OPEA/deepseek-vl2-int4-sym-gptq-inc) |
-| fancyfeast/llama-joycaption-beta-one-hf-llava | pile | X | [NeoChen1024-int4-gptq](https://huggingface.co/NeoChen1024/llama-joycaption-beta-one-hf-llava-GPTQ-4bit-sym-autoround)
-| google/gemma-3 | pile/llava | √ | [gemma-3-12b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-12b-it-AutoRound-gguf-q4-0), [gemma-3-27b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-27b-it-AutoRound-gguf-q4-0), [gemma-3-12b-it-int4-AutoRound](https://huggingface.co/OPEA/gemma-3-12b-it-int4-AutoRound), [gemma-3-27b-it-int4-AutoRound](https://huggingface.co/OPEA/gemma-3-27b-it-int4-AutoRound) |
-| HuggingFaceTB/SmolVLM | pile/llava | √ | [SmolVLM-Instruct-int4-sym](https://huggingface.co/OPEA/SmolVLM-Instruct-int4-sym-inc) |
-| ibm-granite/granite-vision-3.2 | pile/llava | - | |
-| liuhaotian/Llava-v1.5 | pile/llava | X | [llava-v1.5-7b-int4-sym](https://huggingface.co/OPEA/llava-v1.5-7b-int4-sym-inc) |
-| meta-llama/Llama-3.2-Vision | llava | √ | [Llama-3.2V-11B-cot-int4-sym](https://huggingface.co/OPEA/Llama-3.2V-11B-cot-int4-sym-inc), [Llama-3.2-11B-Vision-Instruct-qvision-int4-sym](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-qvision-int4-sym-inc), [Llama-3.2-90B-Vision-Instruct-int4-sym](https://huggingface.co/OPEA/Llama-3.2-90B-Vision-Instruct-int4-sym-inc), [Llama-3.2-11B-Vision-Instruct-int4-sym](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-int4-sym-inc) |
-| microsoft/Phi3.5-Vision | pile/llava | √ | [Phi-3.5-vision-instruct-int4-sym](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-int4-sym-inc), [Phi-3.5-vision-instruct-qvision-int4-sym](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-qvision-int4-sym-inc) |
-| mistralai/Mistral-Small-3.1 | pile/llava | X | [Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-gptq-sym](https://huggingface.co/OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-gptq-sym), [Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym](https://huggingface.co/OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym) |
-| moonshotai/Kimi-VL | pile/llava | √ | |
-| Qwen/Qwen2-VL | pile/llava | - | [Qwen2-VL-7B-Instruct-int4-sym](https://huggingface.co/OPEA/Qwen2-VL-7B-Instruct-int4-sym-inc), [Qwen2-VL-72B-Instruct-int4-sym](https://huggingface.co/OPEA/Qwen2-VL-72B-Instruct-int4-sym-inc), [Qwen2-VL-72B-Instruct-int2-sym](https://huggingface.co/OPEA/Qwen2-VL-72B-Instruct-int2-sym-inc) |
-| rhymes-ai/Aria | pile/llava | √ | |
-| THUDM/CogVLM2 | pile/llava | √ | [cogvlm2-llama3-chat-19B-int4-sym](https://huggingface.co/OPEA/cogvlm2-llama3-chat-19B-int4-sym-inc), [cogvlm2-llama3-chat-19B-qvision-int4-sym](https://huggingface.co/OPEA/cogvlm2-llama3-chat-19B-qvision-int4-sym-inc) |
-| THUDM/glm-4v | pile | X | [glm-4v-9b-int4-sym](https://huggingface.co/OPEA/glm-4v-9b-int4-sym-inc) |
-
-√ means support, - means support to export but cannot infer, X means not support.
-
-## Integration
-
-AutoRound has been integrated into multiple repositories.
-
-[huggingface/transformers](https://github.com/huggingface/transformers)
-
-[Intel Neural Compressor](https://github.com/intel/neural-compressor)
-
-[ModelCloud/GPTQModel](https://github.com/ModelCloud/GPTQModel)
-
-[pytorch/ao](https://github.com/pytorch/ao)
-
-## Reference
-
-If you find AutoRound useful for your research, please cite our paper:
-
-```bash
-@article{cheng2023optimize,
- title={Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs},
- author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi},
- journal={arXiv preprint arXiv:2309.05516},
- year={2023}
-}
-```
diff --git a/auto_round/autoround.py b/auto_round/autoround.py
index 87770d30c..a96978b91 100644
--- a/auto_round/autoround.py
+++ b/auto_round/autoround.py
@@ -36,6 +36,7 @@ from auto_round.utils import (
_gguf_args_check,
block_forward,
check_is_cpu,
+ check_need_act_calibration,
check_seqlen_compatible,
check_skippable_keywords,
check_to_quantized,
@@ -254,8 +255,12 @@ class AutoRound(object):
self.act_data_type = "float"
tmp_act_bits = infer_bits_by_data_type(self.act_data_type)
- if tmp_act_bits < 16:
+ if tmp_act_bits < 16 and tmp_act_bits != self.act_bits:
self.act_bits = tmp_act_bits
+ logger.warning(
+ f"act_bits set in 'act_data_type' do not"
+ f" match the specified 'act_bits' setting. Resetting 'act_bits' to {tmp_act_bits}."
+ )
self.sampler = sampler
self.not_use_best_mse = not_use_best_mse
@@ -1142,7 +1147,9 @@ class AutoRound(object):
self.model.to("cpu")
if has_gguf_k and not self.disable_opt_rtn:
self.quant_rtn_with_imatrix(all_to_quantized_module_names)
- elif self.act_bits <= 8 and self.act_dynamic is False:
+ elif self.act_bits <= 8 and check_need_act_calibration(
+ self.act_dynamic, self.act_data_type
+ ): ##TODO, mixed datatype has bug
hook_handles = self.register_act_max_hook(self.model)
try:
self.quantize_via_rtn_blockwise(all_to_quantized_module_names)
@@ -2152,7 +2159,11 @@ class AutoRound(object):
hook_handles = []
for n, m in model.named_modules():
- if hasattr(m, "act_dynamic") and not m.act_dynamic and check_to_quantized(m):
+ if (
+ hasattr(m, "act_dynamic")
+ and check_need_act_calibration(m.act_dynamic, m.act_data_type)
+ and check_to_quantized(m)
+ ):
hook = m.register_forward_hook(get_act_max_hook)
hook_handles.append(hook)
continue
@@ -2160,10 +2171,11 @@ class AutoRound(object):
# for whole model, RTN
if n in self.layer_config:
config = self.layer_config[n]
+ act_dynamic = config.get("act_dynamic", True)
+ act_data_type = config.get("act_data_type", None)
if (
config["bits"] <= 8
- and "act_dynamic" in config
- and config["act_dynamic"] is False
+ and check_need_act_calibration(act_dynamic, act_data_type)
and check_to_quantized(config)
):
hook = m.register_forward_hook(get_act_max_hook)
diff --git a/auto_round/data_type/gguf.py b/auto_round/data_type/gguf.py
index fa120938a..7ae7cfa46 100644
--- a/auto_round/data_type/gguf.py
+++ b/auto_round/data_type/gguf.py
@@ -581,12 +581,11 @@ def quant_tensor_gguf_sym_dq(
logger.warning_once(
"please use more data via setting `nsamples` to improve accuracy as calibration activations contain 0"
)
-
zero_cnt = torch.sum(quant_weights == 0, dim=-1)
replace_index = zero_cnt > group_size // 2
if torch.sum(replace_index) > 0:
if bits == 6:
- quant_weights[replace_index, :] = 1.0
+ quant_weights[replace_index, :] = tensor * tensor
else:
sigma2 = 2 * torch.sum(tensor**2, dim=-1, keepdim=True) / QK_K
tmp_quant_weights = torch.sqrt(sigma2 + tensor * tensor)
diff --git a/auto_round/data_type/mxfp.py b/auto_round/data_type/mxfp.py
index e8a9258d5..0c6ae265c 100644
--- a/auto_round/data_type/mxfp.py
+++ b/auto_round/data_type/mxfp.py
@@ -62,9 +62,11 @@ def quant_element(tensor, ebits, mbits, max_norm, mantissa_rounding="even"):
mask_tensor = ((abs_tensor - 0.5) % 2 == torch.zeros_like(abs_tensor)).type(tensor.dtype)
tensor = torch.sign(tensor) * (floor_ste(abs_tensor + 0.5) - mask_tensor)
elif mantissa_rounding == "nearest":
- tensor = round_ste(tensor)
+ tensor = torch.sign(tensor) * round_ste(torch.abs(tensor))
elif mantissa_rounding == "floor":
- tensor = floor_ste(tensor)
+ tensor = torch.sign(tensor) * floor_ste(torch.abs(tensor))
+ elif mantissa_rounding == "stochastic":
+ tensor = torch.sign(tensor) * floor_ste(torch.abs(tensor) + torch.rand_like(tensor, requires_grad=False))
else:
raise ValueError("mantissa_rounding only supports even, nearest or floor.")
diff --git a/auto_round/data_type/nvfp.py b/auto_round/data_type/nvfp.py
index b39320100..7b7d98cc8 100644
--- a/auto_round/data_type/nvfp.py
+++ b/auto_round/data_type/nvfp.py
@@ -73,8 +73,25 @@ def ref_nvfp4_quant(x, global_scale, block_size=16, v=0):
def nv_fp4(tensor, bits=4, group_size=16, v=0, **kwargs):
orig_dtype = tensor.dtype
tensor, orig_shape, pad_len = reshape_pad_tensor_by_group_size(tensor, group_size)
- tensor_amax = tensor.abs().max().to(torch.float32)
- global_scale = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX * get_reciprocal(tensor_amax)
+ tensor_max = tensor.abs().max().to(torch.float32)
+ global_scale = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX * get_reciprocal(tensor_max)
+ qdq_res, output_scale = ref_nvfp4_quant(tensor, global_scale, group_size, v)
+ qdq_res = revert_tensor_by_pad(qdq_res, orig_shape=orig_shape, pad_len=pad_len)
+ return qdq_res.to(orig_dtype), output_scale, None
+
+
+@register_dtype("nv_fp4_with_static_gs")
+def nv_fp4_with_static_gs(tensor, bits=4, group_size=16, v=0, tensor_max=None, **kwargs):
+ orig_dtype = tensor.dtype
+ tensor, orig_shape, pad_len = reshape_pad_tensor_by_group_size(tensor, group_size)
+ if tensor_max is None:
+ tensor_max = tensor.abs().max().to(torch.float32)
+ elif tensor_max is not None:
+ if not isinstance(tensor_max, torch.Tensor):
+ tensor_max = torch.tensor(tensor_max, device=tensor.device, dtype=torch.float32)
+ if tensor_max.numel() != 1:
+ tensor_max = tensor.abs().max().to(torch.float32)
+ global_scale = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX * get_reciprocal(tensor_max)
qdq_res, output_scale = ref_nvfp4_quant(tensor, global_scale, group_size, v)
qdq_res = revert_tensor_by_pad(qdq_res, orig_shape=orig_shape, pad_len=pad_len)
return qdq_res.to(orig_dtype), output_scale, None
@@ -174,12 +191,19 @@ def ref_fp4_quant(x, global_scale, block_size=16, v=0, max_scale=1.0):
@register_dtype("fp4_v2_with_global_scale")
@torch.compile()
-def fp4_v2_with_global_scale(tensor, bits=4, group_size=16, v=0, max_scale=1.0, **kwargs):
+def fp4_v2_with_global_scale(tensor, bits=4, group_size=16, v=0, tensor_max=None, max_scale=1.0, **kwargs):
assert group_size == 32 or group_size == 16
orig_dtype = tensor.dtype
tensor, orig_shape, pad_len = reshape_pad_tensor_by_group_size(tensor, group_size)
- tensor_amax = tensor.abs().max().to(torch.float32)
- global_scale = FLOAT8_UE5M3_MAX * FLOAT4_E2M1_MAX * get_reciprocal(tensor_amax)
+
+ if tensor_max is None:
+ tensor_max = tensor.abs().max().to(torch.float32)
+ elif tensor_max is not None:
+ if not isinstance(tensor_max, torch.Tensor):
+ tensor_max = torch.tensor(tensor_max, device=tensor.device, dtype=torch.float32)
+ if tensor_max.numel() != 1:
+ tensor_max = tensor.abs().max().to(torch.float32)
+ global_scale = FLOAT8_UE5M3_MAX * FLOAT4_E2M1_MAX * get_reciprocal(tensor_max)
qdq_res, output_scale = ref_fp4_quant(tensor, global_scale, group_size, v)
qdq_res = revert_tensor_by_pad(qdq_res, orig_shape=orig_shape, pad_len=pad_len)
return qdq_res.to(orig_dtype), output_scale, None
diff --git a/auto_round/export/export_to_gguf/packing.py b/auto_round/export/export_to_gguf/packing.py
index 204e3d1c0..79e635914 100644
--- a/auto_round/export/export_to_gguf/packing.py
+++ b/auto_round/export/export_to_gguf/packing.py
@@ -315,7 +315,7 @@ def q4_0_quant_block(blocks, scale=None, zp=None, **kwargs):
d = scale.reshape((-1, 1))
else:
imax = abs(blocks).argmax(axis=-1, keepdims=True)
- max = torch.take_along_dim(blocks, imax, axis=-1)
+ max = torch.take_along_dim(blocks, imax, dim=-1)
d = max / -8
id = get_reciprocal(d)
diff --git a/auto_round/mllm/README.md b/auto_round/mllm/README.md
index f33d8309b..7dfb37c3d 100644
--- a/auto_round/mllm/README.md
+++ b/auto_round/mllm/README.md
@@ -54,27 +54,34 @@ auto-round-mllm \
--output_dir ./tmp_autoround
```
-### Support Matrix
-For most VLMs, we typically support the default quantization method, which involves quantizing only the language component while excluding the visual component. Besides, we also support quantizing non-text modules of models that follow the Hugging Face standard, i.e., those with a typical processor.
-
-| Model | calibration dataset | quant nontext module |
-|-------------------------------------|---------------------|----------------------|
-| Qwen/Qwen2-VL | pile/llava | - |
-| meta-llama/Llama-3.2-Vision | llava | ✔ |
-| microsoft/Phi3-Vision | pile/llava | ✔ |
-| liuhaotian/Llava-v1.5 | pile/llava | X |
-| THUDM/CogVLM2 | pile/llava | ✔ |
-| google/gemma-3 | pile/llava | √ |
-| ibm-granite/granite-vision-3.2 | pile/llava | - |
-| mistralai/Mistral-Small-3.1 | pile/llava | X |
-| rhymes-ai/Aria | pile/llava | ✔ |
-| deepseek-ai/deepseek-vl2 | pile/llava | ✔ |
-| THUDM/glm-4v | pile | X |
-| allenai/Molmo | pile | X |
-| HuggingFaceTB/SmolVLM | pile/llava | ✔ |
-| moonshotai/Kimi-VL | pile/llava | √ |
-
-✔ means support, - means support to export but cannot infer, X means not support.
+### VLM Support Matrix
+
+For most VLMs, we typically support the default quantization configuration, which involves quantizing only the language
+component while excluding the visual component. Besides, we also support quantizing non-text modules of models that
+follow the Hugging Face standard, i.e., those with a typical processor, though inference may have some issues due to
+model architecture or kernel limitations.
+
+| Model | calibration dataset | quant nontext module | Quantized Model Link |
+|-----------------------------------------------|---------------------|----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| allenai/Molmo | pile | X | [Molmo-7B-D-0924-int4-sym](https://huggingface.co/OPEA/Molmo-7B-D-0924-int4-sym-inc), [Molmo-72B-0924-int4-sym-gptq](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-gptq-inc), [Molmo-72B-0924-int4-sym](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-inc) |
+| deepseek-ai/deepseek-vl2 | pile/llava | √ | [deepseek-vl2-int4-sym-gptq](https://huggingface.co/OPEA/deepseek-vl2-int4-sym-gptq-inc) |
+| fancyfeast/llama-joycaption-beta-one-hf-llava | pile | X | [NeoChen1024-int4-gptq](https://huggingface.co/NeoChen1024/llama-joycaption-beta-one-hf-llava-GPTQ-4bit-sym-autoround)
+| google/gemma-3 | pile/llava | √ | [gemma-3-12b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-12b-it-AutoRound-gguf-q4-0), [gemma-3-27b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-27b-it-AutoRound-gguf-q4-0), [gemma-3-12b-it-int4-AutoRound](https://huggingface.co/OPEA/gemma-3-12b-it-int4-AutoRound), [gemma-3-27b-it-int4-AutoRound](https://huggingface.co/OPEA/gemma-3-27b-it-int4-AutoRound) |
+| HuggingFaceTB/SmolVLM | pile/llava | √ | [SmolVLM-Instruct-int4-sym](https://huggingface.co/OPEA/SmolVLM-Instruct-int4-sym-inc) |
+| ibm-granite/granite-vision-3.2 | pile/llava | - | |
+| liuhaotian/Llava-v1.5 | pile/llava | X | [llava-v1.5-7b-int4-sym](https://huggingface.co/OPEA/llava-v1.5-7b-int4-sym-inc) |
+| meta-llama/Llama-3.2-Vision | llava | √ | [Llama-3.2V-11B-cot-int4-sym](https://huggingface.co/OPEA/Llama-3.2V-11B-cot-int4-sym-inc), [Llama-3.2-11B-Vision-Instruct-qvision-int4-sym](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-qvision-int4-sym-inc), [Llama-3.2-90B-Vision-Instruct-int4-sym](https://huggingface.co/OPEA/Llama-3.2-90B-Vision-Instruct-int4-sym-inc), [Llama-3.2-11B-Vision-Instruct-int4-sym](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-int4-sym-inc) |
+| microsoft/Phi3.5-Vision | pile/llava | √ | [Phi-3.5-vision-instruct-int4-sym](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-int4-sym-inc), [Phi-3.5-vision-instruct-qvision-int4-sym](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-qvision-int4-sym-inc) |
+| mistralai/Mistral-Small-3.1 | pile/llava | X | [Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-gptq-sym](https://huggingface.co/OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-gptq-sym), [Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym](https://huggingface.co/OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym) |
+| moonshotai/Kimi-VL | pile/llava | √ | |
+| Qwen/Qwen2-VL | pile/llava | - | [Qwen2-VL-7B-Instruct-int4-sym](https://huggingface.co/OPEA/Qwen2-VL-7B-Instruct-int4-sym-inc), [Qwen2-VL-72B-Instruct-int4-sym](https://huggingface.co/OPEA/Qwen2-VL-72B-Instruct-int4-sym-inc), [Qwen2-VL-72B-Instruct-int2-sym](https://huggingface.co/OPEA/Qwen2-VL-72B-Instruct-int2-sym-inc) |
+| rhymes-ai/Aria | pile/llava | √ | |
+| THUDM/CogVLM2 | pile/llava | √ | [cogvlm2-llama3-chat-19B-int4-sym](https://huggingface.co/OPEA/cogvlm2-llama3-chat-19B-int4-sym-inc), [cogvlm2-llama3-chat-19B-qvision-int4-sym](https://huggingface.co/OPEA/cogvlm2-llama3-chat-19B-qvision-int4-sym-inc) |
+| THUDM/glm-4v | pile | X | [glm-4v-9b-int4-sym](https://huggingface.co/OPEA/glm-4v-9b-int4-sym-inc) |
+
+√ means support, - means support to export but cannot infer, X means not support.
+
+
<details>
<summary style="font-size:17px;">Calibration Dataset</summary>
@@ -164,21 +171,7 @@ code in ```auto_round/mllm/processor.py```.
</details>
-## Inference
-
-For the AutoRound format, please add the following code at the beginning of the original model's inference code.
-
-```python
-from auto_round import AutoRoundConfig ## must import for auto-round format
-```
-
-For more details on quantization, inference, evaluation, and environment, see the following recipe:
-- [Qwen2-VL-7B-Instruct](../../docs/Qwen2-VL-7B-Instruct-sym.md)
-- [Llama-3.2-11B-Vision](../../docs/Llama-3.2-11B-Vision-Instruct-sym.md)
-- [Phi-3.5-vision-instruct](../../docs/Phi-3.5-vision-instruct-sym.md)
-- [llava-v1.5-7b](../../docs/llava-v1.5-7b-sym.md)
-- [cogvlm2-llama3-chat-19B](../../docs/cogvlm2-llama3-chat-19B-sym.md)
diff --git a/auto_round/script/llm.py b/auto_round/script/llm.py
index a50c877ff..827ffff2a 100644
--- a/auto_round/script/llm.py
+++ b/auto_round/script/llm.py
@@ -37,6 +37,7 @@ from auto_round.utils import (
get_device_and_parallelism,
get_fp_layer_names,
get_model_dtype,
+ infer_bits_by_data_type,
set_cuda_visible_devices,
str2bool,
)
@@ -583,7 +584,7 @@ def tune(args):
import time
eval_model_dtype = get_model_dtype(args.eval_model_dtype, "auto")
- if args.act_bits <= 8 or eval_gguf_model:
+ if args.act_bits <= 8 or infer_bits_by_data_type(args.act_data_type) <= 8 or eval_gguf_model:
if eval_gguf_model:
# for file in os.listdir(eval_folder):
# gguf_file = file
diff --git a/auto_round/utils.py b/auto_round/utils.py
index eadc090d6..36f6c70cf 100644
--- a/auto_round/utils.py
+++ b/auto_round/utils.py
@@ -70,6 +70,8 @@ SUPPORTED_DTYPES = ("int", "mx_fp", "fp", "nv_fp")
def infer_bits_by_data_type(data_type: str):
+ if data_type is None:
+ return 16
for supported_dtype in SUPPORTED_DTYPES:
if data_type.startswith(supported_dtype) and len(data_type) > len(supported_dtype):
##first check the following two bits
@@ -2043,3 +2045,11 @@ def get_reciprocal(tensor):
else:
tensor = torch.where(torch.abs(tensor) < 1e-30, 0, tensor)
return torch.where(tensor != 0, 1 / tensor, torch.zeros_like(tensor))
+
+
+def check_need_act_calibration(is_act_dynamic, act_data_type=None):
+ if not is_act_dynamic:
+ return True
+ if act_data_type is not None and "static" in act_data_type:
+ return True
+ return False
diff --git a/auto_round/version.py b/auto_round/version.py
index d9f21f781..aee1899ed 100644
--- a/auto_round/version.py
+++ b/auto_round/version.py
@@ -14,4 +14,4 @@
"""Intel® auto-round: An open-source Python library
supporting popular model weight only compression based on signround."""
-__version__ = "0.6.0"
+__version__ = "0.6.1.dev"
diff --git a/docs/step_by_step.md b/docs/step_by_step.md
index 8fe57336c..e2b9b48f5 100644
--- a/docs/step_by_step.md
+++ b/docs/step_by_step.md
@@ -1,7 +1,43 @@
Step-by-Step
============
-This document presents step-by-step instructions for auto-round llm quantization.
+This document presents step-by-step instructions for auto-round llm quantization. For vlms quantizaition, please refer to [vlms user guide](../auto_round/mllm/README.md)
+
+
+* [1 Prerequisite](#1-prerequisite)
+* [2 Prepare Calibration Dataset](#2-prepare-calibration-dataset)
+ + [Default Dataset](#default-dataset)
+ + [Customized Dataset](#customized-dataset)
+ + [Dataset operations](#dataset-operations)
+* [3 Quantization](#3-quantization)
+ + [Supported Quantization Configurations](#supported-quantization-configurations)
+ + [Hardware Compatibility](#hardware-compatibility)
+ + [Supported Export Formats](#supported-export-formats)
+ + [Command Line Usage](#command-line-usage)
+ + [API usage](#api-usage)
+ - [AutoRound API Usage](#autoround-api-usage)
+ - [Mixed bits Usage](#mixed-bits-usage)
+ - [AutoRoundBest recipe](#autoroundbest-recipe)
+ - [AutoRoundLight recipe](#autoroundlight-recipe)
+ - [Recipe recommendation](#recipe-recommendation)
+ + [RTN mode](#rtn-mode)
+ + [GGUF format](#gguf-format)
+ + [Quantization Costs](#quantization-costs)
+ + [Device/Multi-GPU setting in Quantization](#device-multi-gpu-setting-in-quantization)
+ - [Enable multiple gpus calibration in lm_head quantization](#enable-multiple-gpus-calibration-in-lm-head-quantization)
+ - [Enable multiple gpus tuning for extremely large model](#enable-multiple-gpus-tuning-for-extremely-large-model)
+ + [Adjust Hyperparameters](#adjust-hyperparameters)
+* [4 Inference](#4-inference)
+ + [CPU](#cpu)
+ + [Intel GPU](#intel-gpu)
+ + [CUDA](#cuda)
+ + [HPU](#hpu)
+ + [Specify Inference Backend](#specify-inference-backend)
+ + [Convert GPTQ/AWQ to AutoRound](#convert-gptq-awq-to-autoround)
+* [5 Evaluation](#5-evaluation)
+ + [Combine evaluation with tuning](#combine-evaluation-with-tuning)
+ + [Eval the Quantized model](#eval-the-quantized-model)
+* [6 Known Issues](#6-known-issues)
## 1 Prerequisite
@@ -210,6 +246,33 @@ autoround = AutoRound(
output_dir = "./tmp_autoround"
autoround.quantize_and_save(output_dir, format="auto_round")
```
+#### Recipe recommendation
+
+In conclusion, we recommend using **auto-round for INT4 and auto-round-best for INT2**. However, you may adjust the
+configuration to suit your specific requirements and available resources.
+
+W4G128 Average Accuracy of 13 tasks and Time Cost Results(Testing was conducted on the Nvidia A100 80G using the version
+of PyTorch 2.6.0 with enable_torch_compile):
+
+| Model | Qwen2.5-0.5B-Instruct | Falcon3-3B | Qwen2.5-7B-Instruct | Meta-Llama-3.1-8B-Instruct | Falcon3-10B | Qwen2.5-72B-Instruct |
+|---------|-----------------------|-----------------|---------------------|----------------------------|-----------------|----------------------|
+| 16bits | 0.4192 | 0.5203 | 0.6470 | 0.6212 | 0.6151 | 0.7229 |
+| Best | **0.4137**(7m) | **0.5142**(23m) | 0.6426(58m) | **0.6116**(65m) | **0.6092**(81m) | 0.7242(575m) |
+| Default | 0.4129(2m) | 0.5133(6m) | 0.6441(13m) | 0.6106(13m) | 0.6080(18m) | **0.7252**(118m) |
+| Light | 0.4052(2m) | 0.5108(3m) | **0.6453**(5m) | 0.6104(6m) | 0.6063(6m) | 0.7243(37m) |
+
+<details>
+ <summary>W2G64 results</summary>
+W2G64 Average Accuracy of 13 tasks and Time Cost Results(Testing was conducted on the Nvidia A100 80G using the version of PyTorch 2.6.0 with enable_torch_compile). We recommend using higher precision for the head, tail, and non-expert modules to alleviate the significant accuracy drop.
+
+| Model | Qwen2.5-0.5B-Instruct | Falcon3-3B | Qwen2.5-7B-Instruct | Falcon3-10B | Qwen2.5-72B-Instruct |
+ |---------|-----------------------|-----------------|---------------------|-----------------|----------------------|
+| 16bits | 0.4192 | 0.5203 | 0.6470 | 0.6151 | 0.7229 |
+| Best | **0.2989**(6m) | **0.4267**(24m) | **0.5343**(56m) | **0.5207**(79m) | **0.6715**(564m) |
+| Default | 0.2878(2m) | 0.4219(6m) | 0.5209(13m) | 0.5133(18m) | 0.6713(122m) |
+| Light | 0.2760(2m) | 0.4063(3m) | 0.4764(5m) | 0.4810(7m) | 0.6581(38m) |
+
+</details>
### RTN mode
AutoRound also supports RTN (Round-To-Nearest) mode for fast, calibration-free baseline quantization. try setting iters=0 and use group_size=32 for better results.
@@ -257,9 +320,53 @@ autoround = AutoRound(
)
output_dir = "./tmp_autoround"
-autoround.quantize_and_save(output_dir, format="gguf:q4_0") # gguf:q4_1, gguf:q*_k_s
+autoround.quantize_and_save(output_dir, format="gguf:q4_k_m") # gguf:q*_k_s,gguf:q*_k_0,gguf:q*_k_1,
```
+### Supported export Formats
+
+**AutoRound Format**: This format is well-suited for CPU, HPU devices, 2 bits, as well as mixed-precision
+inference. **[2,3,4,8] bits are supported**.
+
+**GGUF** Format: Experimental feature. This format is well-suited for CPU devices and is widely adopted by the
+community. `q*_k`,`q*_0`,`q*_1` are supported.
+
+**AutoGPTQ Format**: This format is well-suited for symmetric quantization on CUDA devices and is widely adopted by the
+community, **[2,3,4,8] bits are supported**. However, **the
+asymmetric kernel has issues** that can cause considerable accuracy drops, particularly at 2-bit quantization and small
+models. Besides, recently 3 bits may have some accuracy issues in Transformers.
+
+**AutoAWQ Format**: This format is well-suited for asymmetric 4-bit quantization on CUDA devices and is widely
+adopted within the community, **only 4-bits quantization is supported**.
+
+**llmcompressor Format**: This format is for reusing llmcompressor format, **only INT8 W8A8 dynamic quantization is
+supported**.
+
+
+### Quantization Costs
+
+Testing was conducted on the Nvidia A100 80G using the nightly version of PyTorch 2.6.0.dev20241029+cu124. Please note
+that data
+loading and packing costs have been excluded from the evaluation. **We recommend enabling torch.compile for PyTorch
+versions 2.6 and above.**
+
+To optimize GPU memory usage, in addition to activating `low_gpu_mem_usage`, you can set `gradient_accumulate_steps=8`
+and a
+`batch_size=1`, though this may increase tuning time.
+
+The 3B and 14B models were evaluated on Qwen 2.5, the 8X7B model is Mixtral, while the remaining models utilized LLaMA
+3.1.
+
+| Torch version/Config W4G128 | 3B | 8B | 14B | 70B | 8X7B |
+|---------------------------------------------------------------------------------------------|---------------|----------------|----------------|-----------------|----------------|
+| 2.6 with torch compile | 7min<br/>10GB | 12min<br/>18GB | 23min<br/>22GB | 120min<br/>42GB | 28min<br/>46GB |
+| 2.6 with torch compile <br/> low_gpu_mem_usage=True | 12min<br/>6GB | 19min<br/>10GB | 33min<br/>11GB | 140min<br/>25GB | 38min<br/>36GB |
+| 2.6 with torch compile <br/> low_gpu_mem_usage=True <br/> gradient_accumulate_steps=8,bs=1 | 15min<br/>3GB | 25min<br/>6GB | 45min<br/>7GB | 187min<br/>19GB | 75min<br/>36GB |
+| 2.5 w/o torch compile | 8min<br/>10GB | 16min<br/>20GB | 30min<br/>25GB | 140min<br/>49GB | 50min<br/>49GB |
+
+
+
+
### Device/Multi-GPU setting in Quantization
**The tuning device is specified using the `device` argument in AutoRound API, _not_ through the `device_map`
parameter used by Transformers.from_pretrained.**