PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
support static global scale for nvfp4 and update readme (#691)
Wenhua Cheng4a3530fec349
diff --git a/README.md b/README.mdindex 3416b09b0..35ad24583 100644--- a/README.md+++ b/README.md@@ -14,15 +14,11 @@ AutoRound---<div align="left">-AutoRound is an advanced quantization algorithm that delivers strong accuracy, even at 2-bit precision.-It leverages sign gradient descent to fine-tune both rounding values and min-max clipping thresholds in just 200 steps.-Designed for broad compatibility, it seamlessly supports a wide range of LLMs and is actively expanding to cover more-VLMs as well.-It also supports quantization and inference across multiple hardware platforms, including CPU, Intel GPU, and CUDA.-AutoRound also offers a variety of useful features, including mixed-bit tuning and inference, lm-head quantization,-support for exporting to formats like GPTQ/AWQ/GGUF, and flexible tuning recipes. The below-image presents an overview of AutoRound. Check out our paper on [arxiv](https://arxiv.org/pdf/2309.05516) for more-details and quantized models in several Hugging Face Spaces,+## 🚀 What is AutoRound?++**AutoRound** is an advanced quantization library mainly for large language models (LLMs) and Visual language models (Vlms).It achieves **high accuracy at ultra-low bitwidths (2–4 bits)** with minimal tuning by leveraging sign+gradient decent, and broad hardware support. Check out our paper on [arxiv](https://arxiv.org/pdf/2309.05516) for more details and quantized models in several+Hugging Face Spaces,e.g. [Intel](https://huggingface.co/Intel), [OPEA](https://huggingface.co/OPEA), [Kaitchup](https://huggingface.co/kaitchup)and [fbaldassarri](https://huggingface.co/fbaldassarri).@@ -32,23 +28,58 @@ and [fbaldassarri](https://huggingface.co/fbaldassarri).<div align="left">-## What's New+## ✨ Key Features+++✅ **Top-Tier Accuracy**+Delivers strong performance even at 2–3 bits [example models](https://huggingface.co/collections/OPEA/2-3-bits-67a5f0bc6b49d73c01b4753b), with leading results at 4 bits [benchmark](https://huggingface.co/spaces/Intel/low_bit_open_llm_leaderboard).++✅ **Ecosystem Integration**+Seamlessly works with Transformers, vLLM, TorchAO, sglang(on going,[pr](https://github.com/sgl-project/sglang/pull/6226)) and more.++✅ **Multi-format Export**+Supports **AutoRound, AutoAWQ, AutoGPTQ, and GGUF** for maximum compatibility.++✅ **Affordable Quantization cost**+Quantize 7B models in about 10 minutes on a single GPU.++✅ **10+ VLM Support**+Out-of-the-box quantization for 10+ vision-language models [example models](https://huggingface.co/collections/OPEA/vlms-autoround-675bc712fdd6a55ebaf11bfa),[support matrix](https://github.com/intel/auto-round/tree/main/auto_round/mllm#support-matrix)++✅ **Per-layer Mixed-bit Quantization**+Assign different bit-widths per layer for fine-grained accuracy/performance trade-offs.++✅ **Round-to-Nearest Mode**+Use `--iters 0` for fast, calibration-free quantization with some accuracy drop.-* [2025.07] AutoRound now offers experimental support for **GGUF** format, and recommends using optimized RTN mode (--iters 0) for- all bits other than 3 bits. A more advanced algorithm tailored for specific configurations may be available in+✅ **Multiple Quantization Recipes**+Choose from `auto-round-best`, `auto-round`, and `auto-round-light` to suit your needs.++✅ Advanced Utilities+Includes immediate weight packing, multiple gpu quantization and support for 10+ runtime backends.++🟨 Beyond weight only quantization. We are actively expanding support for additional datatypes such as **MXFP**, NVFP, W8A8, and more.++## 🆕 What's New++[2025/07] AutoRound now offers experimental support for **GGUF** format, and recommends using optimized RTN mode (--iters 0) for+ all bits other than 3 bits. **A more advanced algorithm** tailored for specific configurations may be available inv0.6.1. Example- models: [Intel/Qwen3-235B-A22B-q2ks-mixed-AutoRound-inc-v1](https://huggingface.co/Intel/Qwen3-235B-A22B-q2ks-mixed-AutoRound-inc-v1)- and [Intel/DeepSeek-R1-0528-q2ks-mixed-AutoRound-inc-v1](https://huggingface.co/Intel/DeepSeek-R1-0528-q2ks-mixed-AutoRound-inc-v1).-* [2025.05] AutoRound provides some recipes for **DeepSeek-R1-0528**, please refer- to [DeepSeek-R1-0528-int2-mixed-sym-inc](https://huggingface.co/Intel/DeepSeek-R1-0528-int2-mixed-sym-inc), [DeepSeek-R1-0528-int4-sym-gptq-inc](https://huggingface.co/Intel/DeepSeek-R1-0528-int4-gptq-inc-auto-round)- and [DeepSeek-R1-0528-int4-asym-awq-inc](https://huggingface.co/Intel/DeepSeek-R1-0528-int4-awq-inc-auto-round) for+ models: [Intel/Qwen3-235B-A22B-q2ks-mixed-ar](https://huggingface.co/Intel/Qwen3-235B-A22B-q2ks-ar)+ and [Intel/DeepSeek-R1-0528-q2ks-mixed-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-q2ks-mixed-ar).++[2025/05] AutoRound provides some recipes for **DeepSeek-R1-0528**, please refer+ to [IntelDeepSeek-R1-0528-int2-mixed-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-int2-mixed-ar), [Intel/DeepSeek-R1-0528-int4-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-int4-ar)+ and [Intel/DeepSeek-R1-0528-int4-asym-ar](https://huggingface.co/Intel/DeepSeek-R1-0528-int4-asym-ar) formore details.-* [2025/05] AutoRound has been integrated into **vLLM**. You can now run models in the AutoRound format directly with++[2025/05] AutoRound has been integrated into **vLLM**. You can now run models in the AutoRound format directly withvLLM versions later than v0.85.post1.-* [2025/04] AutoRound has been integrated into **Transformers**. You can run models in the AutoRound format directly- with- Transformers versions later than 4.51.3.-* [2025/03] The INT2-mixed **DeepSeek-R1** model (~200GB) retains 97.9% accuracy. Check++[2025/04] AutoRound has been integrated into **Transformers**. You can run models in the AutoRound format directly+ with Transformers versions later than 4.51.3.++[2025/03] The INT2-mixed **DeepSeek-R1** model (~200GB) retains 97.9% accuracy. Checkout [OPEA/DeepSeek-R1-int2-mixed-sym-inc](https://huggingface.co/OPEA/DeepSeek-R1-int2-mixed-sym-inc).## Installation@@ -78,12 +109,9 @@ pip install auto-round-lib## Model Quantization+Please check out [User guide](./docs/step_by_step.md) for more details### Command Line Usage (Gaudi/CPU/Intel GPU/CUDA)--A user guide detailing the full list of supported arguments is provided by calling ```auto-round -h``` on the terminal.-Set the format you want in `format` and-multiple formats exporting has been supported. Please check out [step-by-step-instruction](./docs/step_by_step.md) for-more details about calibration dataset or evaluation.+Please change to `auto-round-mllm` for visual-language models (VLMs) quantization. The full list of supported arguments is provided by calling `auto-round -h` on the terminal.```bashauto-round \@@ -94,9 +122,7 @@ auto-round \--output_dir ./tmp_autoround```-We offer another two configurations, `auto-round-best` and `auto-round-light`,-designed for optimal accuracy and improved speed,-respectively. Details are as follows.+We offer another two configurations, `auto-round-best` and `auto-round-light`, designed for optimal accuracy and improved speed, respectively. Details are as follows.<details><summary>Other Recipes</summary>@@ -131,30 +157,7 @@ auto-round-fast \In conclusion, we recommend using **auto-round for INT4 and auto-round-best for INT2**. However, you may adjust theconfiguration to suit your specific requirements and available resources.-W4G128 Average Accuracy of 13 tasks and Time Cost Results(Testing was conducted on the Nvidia A100 80G using the version-of PyTorch 2.6.0 with enable_torch_compile):--| Model | Qwen2.5-0.5B-Instruct | Falcon3-3B | Qwen2.5-7B-Instruct | Meta-Llama-3.1-8B-Instruct | Falcon3-10B | Qwen2.5-72B-Instruct |-|---------|-----------------------|-----------------|---------------------|----------------------------|-----------------|----------------------|-| 16bits | 0.4192 | 0.5203 | 0.6470 | 0.6212 | 0.6151 | 0.7229 |-| Best | **0.4137**(7m) | **0.5142**(23m) | 0.6426(58m) | **0.6116**(65m) | **0.6092**(81m) | 0.7242(575m) |-| Default | 0.4129(2m) | 0.5133(6m) | 0.6441(13m) | 0.6106(13m) | 0.6080(18m) | **0.7252**(118m) |-| Light | 0.4052(2m) | 0.5108(3m) | **0.6453**(5m) | 0.6104(6m) | 0.6063(6m) | 0.7243(37m) |--<details>- <summary>W2G64 results</summary>-W2G64 Average Accuracy of 13 tasks and Time Cost Results(Testing was conducted on the Nvidia A100 80G using the version of PyTorch 2.6.0 with enable_torch_compile). We recommend using higher precision for the head, tail, and non-expert modules to alleviate the significant accuracy drop.--| Model | Qwen2.5-0.5B-Instruct | Falcon3-3B | Qwen2.5-7B-Instruct | Falcon3-10B | Qwen2.5-72B-Instruct |- |---------|-----------------------|-----------------|---------------------|-----------------|----------------------|-| 16bits | 0.4192 | 0.5203 | 0.6470 | 0.6151 | 0.7229 |-| Best | **0.2989**(6m) | **0.4267**(24m) | **0.5343**(56m) | **0.5207**(79m) | **0.6715**(564m) |-| Default | 0.2878(2m) | 0.4219(6m) | 0.5209(13m) | 0.5133(18m) | 0.6713(122m) |-| Light | 0.2760(2m) | 0.4063(3m) | 0.4764(5m) | 0.4810(7m) | 0.6581(38m) |--</details>--### API Usage (HPU/CPU/XPU/CUDA)+### API Usage (CPU/Intel GPU/HPU/CUDA)```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizer@@ -239,10 +242,9 @@ results.<details><summary>Click to expand</summary>-**This feature is experimental and may be subject to changes**, including potential bug fixes, API modifications, or-adjustments to default hype-parameters+**This feature is experimental and may be subject to changes**.-By default, AutoRoundMLLM only quantizes the text module of VLMs and uses `NeelNanda/pile-10k` for calibration. To+By default, AutoRoundMLLM only quantize the text module of VLMs and uses `NeelNanda/pile-10k` for calibration. Toquantize the entire model, you can enable `quant_nontext_module` by setting it to True, though support for this featureis limited. For more information, please refer to the AutoRoundMLLM [readme](./auto_round/mllm/README.md).@@ -268,256 +270,55 @@ autoround.save_quantized(output_dir, format="auto_round", inplace=True)</details>-### Export Formats--**AutoRound Format**: This format is well-suited for CPU, HPU devices, 2 bits, as well as mixed-precision-inference. **[2,3,4,8] bits are supported**.--**GGUF** Format: Experimental feature. This format is well-suited for CPU devices and is widely adopted by the-community. `q*_k`,`q*_0`,`q*_1` are supported.--**AutoGPTQ Format**: This format is well-suited for symmetric quantization on CUDA devices and is widely adopted by the-community, **[2,3,4,8] bits are supported**. However, **the-asymmetric kernel has issues** that can cause considerable accuracy drops, particularly at 2-bit quantization and small-models. Besides, recently 3 bits may have some accuracy issues in Transformers.--**AutoAWQ Format**: This format is well-suited for asymmetric 4-bit quantization on CUDA devices and is widely-adopted within the community, **only 4-bits quantization is supported**.--**llmcompressor Format**: This format is for reusing llmcompressor format, **only INT8 W8A8 dynamic quantization is-supported**.----### Quantization Costs--Testing was conducted on the Nvidia A100 80G using the nightly version of PyTorch 2.6.0.dev20241029+cu124. Please note-that data-loading and packing costs have been excluded from the evaluation. **We recommend enabling torch.compile for PyTorch-versions 2.6 and above.**--To optimize GPU memory usage, in addition to activating `low_gpu_mem_usage`, you can set `gradient_accumulate_steps=8`-and a-`batch_size=1`, though this may increase tuning time.--The 3B and 14B models were evaluated on Qwen 2.5, the 8X7B model is Mixtral, while the remaining models utilized LLaMA-3.1.-| Torch version/Config W4G128 | 3B | 8B | 14B | 70B | 8X7B |-|---------------------------------------------------------------------------------------------|---------------|----------------|----------------|-----------------|----------------|-| 2.6 with torch compile | 7min<br/>10GB | 12min<br/>18GB | 23min<br/>22GB | 120min<br/>42GB | 28min<br/>46GB |-| 2.6 with torch compile <br/> low_gpu_mem_usage=True | 12min<br/>6GB | 19min<br/>10GB | 33min<br/>11GB | 140min<br/>25GB | 38min<br/>36GB |-| 2.6 with torch compile <br/> low_gpu_mem_usage=True <br/> gradient_accumulate_steps=8,bs=1 | 15min<br/>3GB | 25min<br/>6GB | 45min<br/>7GB | 187min<br/>19GB | 75min<br/>36GB |-| 2.5 w/o torch compile | 8min<br/>10GB | 16min<br/>20GB | 30min<br/>25GB | 140min<br/>49GB | 50min<br/>49GB |## Model Inference-Please run the quantization code first--### AutoRound format--**CPU**: pip install intel-extension-for-pytorch(much higher speed on Intel CPU) or pip-install intel-extension-for-transformers,--**HPU**: docker image with Gaudi Software Stack is recommended. More details can be found-in [Gaudi Guide](https://docs.habana.ai/en/latest/).--**CUDA**: no extra operations for sym quantization, for asym quantization, need to install auto-round from source--#### HPU/CPU/XPU/CUDA--**Please avoid manually moving the quantized model to a different device** (e.g., model.to('cpu')) during inference, as-this may cause unexpected exceptions.-+### vLLM+Please note that support for the MoE model and visual language model is currently limited.```python-from transformers import AutoModelForCausalLM, AutoTokenizer+from vllm import LLM, SamplingParams-quantized_model_path = "./tmp_autoround"-model = AutoModelForCausalLM.from_pretrained(quantized_model_path, device_map="auto", torch_dtype="auto")-tokenizer = AutoTokenizer.from_pretrained(quantized_model_path)-text = "There is a girl who likes adventure,"-inputs = tokenizer(text, return_tensors="pt").to(model.device)-print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))-```+prompts = [+ "Hello, my name is",+]+sampling_params = SamplingParams(temperature=0.6, top_p=0.95)+model_name = "Intel/DeepSeek-R1-0528-Qwen3-8B-int4-AutoRound"+llm = LLM(model=model_name)-### Specify backend--AutoRound automatically selects the best available backend based on the installed libraries and prompts the user to-install additional libraries when a better backend is found. On CUDA, the default priority is Marlin > ExLLaMAV2 >-Triton, but the final choice depends on factors such as bits, group_size, packing format compatibility, etc. And the-backend may not always be the most suitable for certain devices. Please refer-to the following table for the details and specify the backend you want.--| Name | Devices | Bits | Dtypes | Priority | Packing format | Requirements |-|--------------------------------------|--------------|---------|-----------|----------|-----------------|---------------------------------------|-| ipex | cpu/xpu | 4 | BF16/FP16 | 5 | gptq_zp+-1/awq | intel-extension-for-pytorch |-| itrex | cpu | 2,4,8 | BF16/FP16 | 1 | gptq_zp+-1/awq | <br/>intel-extension-for-transformers |-| marlin | cuda | 4,8 | BF16/FP16 | 6 | gptq/gptq_zp+-1 | gptqmodel |-| exllamav2 or<br/>gptqmodel:exllamav2 | cuda | 4 | BF16/FP16 | 5 | gptq | gptqmodel |-| exllamav2 or<br/>gptq:exllamav2 | cuda | 4 | FP16 | 5 | gptq_zp+-1 | auto-gptq |-| gptq:cuda | cuda | 2,3,4,8 | FP16 | 1 | gptq_zp+-1 | auto-gptq |-| triton | xpu/cuda | 2,4,8 | BF16/FP16 | 2 | gptq/gptq_zp+-1 | auto-round |-| awq | cuda | 4 | FP16 | 5 | awq | auto-awq |-| hpu | hpu | 4 | BF16 | 0 | gptq/gptq_zp+-1 | auto-round |-| torch | xpu/cpu/cuda | 2,3,4,8 | BF16/FP16 | 0 | gptq/gptq_zp+-1 | auto-round |+outputs = llm.generate(prompts, sampling_params)-```python-from transformers import AutoModelForCausalLM, AutoTokenizer-from transformers import AutoRoundConfig--quantized_model_path = "./tmp_autoround"-quantization_config = AutoRoundConfig(backend="auto")-model = AutoModelForCausalLM.from_pretrained(- quantized_model_path, device_map="auto", torch_dtype="auto", quantization_config=quantization_config-)-tokenizer = AutoTokenizer.from_pretrained(quantized_model_path)-text = "There is a girl who likes adventure,"-inputs = tokenizer(text, return_tensors="pt").to(model.device)-print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))+for output in outputs:+ prompt = output.prompt+ generated_text = output.outputs[0].text+ print(f"Prompt: {prompt!r}, Generated text: {generated_text!r}")```-### Convert GPTQ/AWQ format to AutoRound+### Transformers++AutoRound support 10+ backends an automatically selects the best available backend based on the installed libraries and prompts the user to+install additional libraries when a better backend is found.-Most GPTQ/AWQ models can be converted to the AutoRound format for better compatibility and support with Intel devices.-Please note that the quantization config will be changed if the model is serialized.+**Please avoid manually moving the quantized model to a different device** (e.g., model.to('cpu')) during inference, as+this may cause unexpected exceptions.```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizer-from transformers import AutoRoundConfig-model_name = "ybelkada/opt-125m-gptq-4bit"-quantization_config = AutoRoundConfig()-model = AutoModelForCausalLM.from_pretrained(- model_name, device_map="cpu", torch_dtype="auto", quantization_config=quantization_config-)+model_name = "Intel/DeepSeek-R1-0528-Qwen3-8B-int4-AutoRound"+model = AutoModelForCausalLM.from_pretrained(model_name, device_map="auto", torch_dtype="auto")tokenizer = AutoTokenizer.from_pretrained(model_name)text = "There is a girl who likes adventure,"inputs = tokenizer(text, return_tensors="pt").to(model.device)-print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))-```--#### Evaluation--<details>- <summary>Click to expand</summary>--```bash-auto-round --model saved_quantized_model \- --eval \- --task lambada_openai \- --eval_bs 1+print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))```+## Acknowledgement+Special thanks to open-source low precision libraries such as AutoGPTQ, AutoAWQ, GPTQModel, Triton, Marlin, and ExLLaMAV2 for providing low-precision CUDA kernels, which are leveraged in AutoRound.-</details>--## Support List+## 🌟 Support Us+If you find AutoRound helpful, please ⭐ star the repo and share it with your community!-AutoRound supports basically all the major large language models.-<details>- <summary>Supported Models List</summary>--Please note that an asterisk (*) indicates third-party quantized models, which may lack accuracy data and use a-different recipe. We greatly appreciate their efforts and encourage more users to share their models, as we cannot-release most of the models ourselves.-- Model | Supported |-|-------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-| nvidia/Llama-3.1-Nemotron-70B-Instruct-HF | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Llama-3.1-Nemotron-70B-Instruct-HF-int4-sym-inc), [model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Llama-3.1-Nemotron-70B-Instruct-HF-int4-sym-inc), |-| meta-llama/Llama-3.2-90B-Vision-Instruct | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Llama-3.2-90B-Vision-Instruct-int4-sym-inc), [model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Llama-3.2-90B-Vision-Instruct-int4-sym-inc) |-| Qwen/QwQ-32B-Preview | [model-opea-int4-sym-autoround-mixed](https://huggingface.co/OPEA/QwQ-32B-Preview-int4-sym-mixed-inc),[model-opea-int4-sym-autoawq-mixed](https://huggingface.co/OPEA/QwQ-32B-Preview-int4-sym-mixed-awq-inc) |-| THUDM/cogvlm2-llama3-chat-19B | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/cogvlm2-llama3-chat-19B-int4-sym-inc) |-| Qwen/Qwen2-VL-Instruct | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Qwen2-VL-7B-Instruct-int4-sym-inc),[model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Qwen2-VL-7B-Instruct-int4-sym-inc) |-| meta-llama/Llama-3.2-11B-Vision | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-int4-sym-inc), [model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-int4-sym-inc) |-| microsoft/Phi-3.5-vision-instruct | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-int4-sym-inc), [model-opea-int4-sym-gptq](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-int4-sym-inc) |-| liuhaotian/llava-v1.5-7b | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/llava-v1.5-7b-int4-sym-inc),[model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/llava-v1.5-7b-int4-sym-inc) |-| Qwen/Qwen2.5-7B-Instruct | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Qwen2.5-7B-Instruct-int4-sym-inc),[model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Qwen2.5-7B-Instruct-int4-sym-inc) [model-kaitchup-autogptq-int4*](https://huggingface.co/kaitchup/Qwen2.5-7B-Instruct-AutoRound-GPTQ-asym-4bit), [recipe](./docs/Qwen2.5-7B-Instruct-sym.md) |-| Qwen/Qwen2.5-14B-Instruct | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Qwen2.5-14B-Instruct-int4-sym-inc),[model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Qwen2.5-14B-Instruct-int4-sym-inc) |-| Qwen/Qwen2.5-32B-Instruct | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Qwen2.5-32B-Instruct-int4-sym-inc) |-| Qwen/Qwen2.5-Coder-32B-Instruct | [model-kaitchup-autogptq-int4*](https://huggingface.co/kaitchup/Qwen2.5-Coder-32B-Instruct-AutoRound-GPTQ-4bit) |-| Qwen/Qwen2.5-72B-Instruct | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Qwen2.5-72B-Instruct-int4-sym-inc),[model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Qwen2.5-72B-Instruct-int4-sym-inc), [model-kaitchup-autogptq-int4*](https://huggingface.co/kaitchup/Qwen2.5-72B-Instruct-AutoRound-GPTQ-4bit), [model-kaitchup-autogptq-int2*](https://huggingface.co/kaitchup/Qwen2.5-72B-Instruct-AutoRound-GPTQ-2bit), [recipe](./docs/Qwen2.5-72B-Instruct-sym.md) |-| meta-llama/Meta-Llama-3.1-70B-Instruct | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Meta-Llama-3.1-70B-Instruct-int4-sym-inc), [model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Meta-Llama-3.1-70B-Instruct-int4-sym-inc),[model-opea-int4-asym-autoround](https://huggingface.co/OPEA/Meta-Llama-3.1-70B-Instruct-int4-asym-inc) |-| meta-llama/Meta-Llama-3.1-8B-Instruct | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Meta-Llama-3.1-8B-Instruct-int4-sym-inc),[model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Meta-Llama-3.1-8B-Instruct-int4-sym-inc),[model-kaitchup-autogptq-int4*](https://huggingface.co/kaitchup/Meta-Llama-3.1-8B-Instruct-autoround-gptq-4bit-asym), [model-kaitchup-autogptq-sym-int4*](https://huggingface.co/kaitchup/Meta-Llama-3.1-8B-Instruct-autoround-gptq-4bit-sym), [recipe](https://huggingface.co/Intel/Meta-Llama-3.1-8B-Instruct-int4-inc) |-| meta-llama/Meta-Llama-3.1-8B | [model-kaitchup-autogptq-sym-int4*](https://huggingface.co/kaitchup/Meta-Llama-3.1-8B-autoround-gptq-4bit-sym) |-| Qwen/Qwen2-7B | [model-autoround-sym-int4](https://huggingface.co/Intel/Qwen2-7B-int4-inc), [model-autogptq-sym-int4](https://huggingface.co/Intel/Qwen2-7B-int4-inc) |-| THUDM/glm-4-9b-chat | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/glm-4-9b-chat-int4-sym-inc),[model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/glm-4-9b-chat-int4-sym-inc) |-| Qwen/Qwen2-57B-A14B-Instruct | [model-autoround-sym-int4](https://huggingface.co/Intel/Qwen2-57B-A14B-Instruct-int4-inc),[model-autogptq-sym-int4](https://huggingface.co/Intel/Qwen2-57B-A14B-Instruct-int4-inc) |-| 01-ai/Yi-1.5-9B | [model-LnL-AI-autogptq-int4*](https://huggingface.co/LnL-AI/Yi-1.5-9B-4bit-gptq-autoround) |-| 01-ai/Yi-1.5-9B-Chat | [model-LnL-AI-autogptq-int4*](https://huggingface.co/LnL-AI/Yi-1.5-9B-Chat-4bit-gptq-autoround) |-| Intel/neural-chat-7b-v3-3 | [model-autogptq-int4](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc) |-| Intel/neural-chat-7b-v3-1 | [model-autogptq-int4](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc) |-| TinyLlama-1.1B-intermediate | [model-LnL-AI-autogptq-int4*](https://huggingface.co/LnL-AI/TinyLlama-1.1B-intermediate-step-1341k-3T-autoround-lm_head-symFalse) |-| mistralai/Mistral-7B-v0.1 | [model-autogptq-lmhead-int4](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc-lmhead), [model-autogptq-int4](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) |-| google/gemma-2b | [model-autogptq-int4](https://huggingface.co/Intel/gemma-2b-int4-inc) |-| tiiuae/falcon-7b | [model-autogptq-int4-G64](https://huggingface.co/Intel/falcon-7b-int4-inc) |-| sapienzanlp/modello-italia-9b | [model-fbaldassarri-autogptq-int4*](https://huggingface.co/fbaldassarri/modello-italia-9b-autoround-w4g128-cpu) |-| microsoft/phi-2 | [model-autoround-sym-int4](https://huggingface.co/Intel/phi-2-int4-inc) [model-autogptq-sym-int4](https://huggingface.co/Intel/phi-2-int4-inc) |-| microsoft/Phi-3.5-mini-instruct | [model-kaitchup-autogptq-sym-int4*](https://huggingface.co/kaitchup/Phi-3.5-Mini-instruct-AutoRound-4bit) |-| mistralai/Mistral-7B-Instruct-v0.2 | [outdated-recipe](./docs/Mistral-7B-Instruct-v0.2-asym-recipe.md) |-| mistralai/Mixtral-8x7B-Instruct-v0.1 | [outdated-recipe](./docs/Mixtral-8x7B-Instruct-v0.1-asym-recipe.md) |-| mistralai/Mixtral-8x7B-v0.1 | [outdated-recipe](./docs/Mixtral-8x7B-v0.1-asym-acc.md) |-| meta-llama/Meta-Llama-3-8B-Instruct | [outdated-recipe](./docs/Meta-Llama-3-8B-Instruct-asym-recipe.md) |-| google/gemma-7b | [outdated-recipe](./docs/gemma-7b-asym-recipe.md) |-| meta-llama/Llama-2-7b-chat-hf | [outdated-recipe](./docs/Llama-2-7b-chat-hf-asym-recipe.md) |-| baichuan-inc/Baichuan2-7B-Chat | [outdated-recipe](./docs/baichuan2-7b-cha-asym-recipe.md) |-| 01-ai/Yi-6B-Chat | [outdated-recipe](./docs/Yi-6B-Chat-asym-recipe.md) |-| facebook/opt-2.7b | [outdated-recipe](./docs/opt-2.7b-asym-recipe.md) |-| bigscience/bloom-3b | [outdated-recipe](./docs/bloom-3B-asym-recipe.md) |-| EleutherAI/gpt-j-6b | [outdated-recipe](./docs/gpt-j-6B-asym-recipe.md) |--</details>--### VLM Support Matrix--For most VLMs, we typically support the default quantization configuration, which involves quantizing only the language-component while excluding the visual component. Besides, we also support quantizing non-text modules of models that-follow the Hugging Face standard, i.e., those with a typical processor, though inference may have some issues due to-model architecture or kernel limitations.--| Model | calibration dataset | quant nontext module | Quantized Model Link |-|-----------------------------------------------|---------------------|----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-| allenai/Molmo | pile | X | [Molmo-7B-D-0924-int4-sym](https://huggingface.co/OPEA/Molmo-7B-D-0924-int4-sym-inc), [Molmo-72B-0924-int4-sym-gptq](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-gptq-inc), [Molmo-72B-0924-int4-sym](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-inc) |-| deepseek-ai/deepseek-vl2 | pile/llava | √ | [deepseek-vl2-int4-sym-gptq](https://huggingface.co/OPEA/deepseek-vl2-int4-sym-gptq-inc) |-| fancyfeast/llama-joycaption-beta-one-hf-llava | pile | X | [NeoChen1024-int4-gptq](https://huggingface.co/NeoChen1024/llama-joycaption-beta-one-hf-llava-GPTQ-4bit-sym-autoround)-| google/gemma-3 | pile/llava | √ | [gemma-3-12b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-12b-it-AutoRound-gguf-q4-0), [gemma-3-27b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-27b-it-AutoRound-gguf-q4-0), [gemma-3-12b-it-int4-AutoRound](https://huggingface.co/OPEA/gemma-3-12b-it-int4-AutoRound), [gemma-3-27b-it-int4-AutoRound](https://huggingface.co/OPEA/gemma-3-27b-it-int4-AutoRound) |-| HuggingFaceTB/SmolVLM | pile/llava | √ | [SmolVLM-Instruct-int4-sym](https://huggingface.co/OPEA/SmolVLM-Instruct-int4-sym-inc) |-| ibm-granite/granite-vision-3.2 | pile/llava | - | |-| liuhaotian/Llava-v1.5 | pile/llava | X | [llava-v1.5-7b-int4-sym](https://huggingface.co/OPEA/llava-v1.5-7b-int4-sym-inc) |-| meta-llama/Llama-3.2-Vision | llava | √ | [Llama-3.2V-11B-cot-int4-sym](https://huggingface.co/OPEA/Llama-3.2V-11B-cot-int4-sym-inc), [Llama-3.2-11B-Vision-Instruct-qvision-int4-sym](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-qvision-int4-sym-inc), [Llama-3.2-90B-Vision-Instruct-int4-sym](https://huggingface.co/OPEA/Llama-3.2-90B-Vision-Instruct-int4-sym-inc), [Llama-3.2-11B-Vision-Instruct-int4-sym](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-int4-sym-inc) |-| microsoft/Phi3.5-Vision | pile/llava | √ | [Phi-3.5-vision-instruct-int4-sym](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-int4-sym-inc), [Phi-3.5-vision-instruct-qvision-int4-sym](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-qvision-int4-sym-inc) |-| mistralai/Mistral-Small-3.1 | pile/llava | X | [Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-gptq-sym](https://huggingface.co/OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-gptq-sym), [Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym](https://huggingface.co/OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym) |-| moonshotai/Kimi-VL | pile/llava | √ | |-| Qwen/Qwen2-VL | pile/llava | - | [Qwen2-VL-7B-Instruct-int4-sym](https://huggingface.co/OPEA/Qwen2-VL-7B-Instruct-int4-sym-inc), [Qwen2-VL-72B-Instruct-int4-sym](https://huggingface.co/OPEA/Qwen2-VL-72B-Instruct-int4-sym-inc), [Qwen2-VL-72B-Instruct-int2-sym](https://huggingface.co/OPEA/Qwen2-VL-72B-Instruct-int2-sym-inc) |-| rhymes-ai/Aria | pile/llava | √ | |-| THUDM/CogVLM2 | pile/llava | √ | [cogvlm2-llama3-chat-19B-int4-sym](https://huggingface.co/OPEA/cogvlm2-llama3-chat-19B-int4-sym-inc), [cogvlm2-llama3-chat-19B-qvision-int4-sym](https://huggingface.co/OPEA/cogvlm2-llama3-chat-19B-qvision-int4-sym-inc) |-| THUDM/glm-4v | pile | X | [glm-4v-9b-int4-sym](https://huggingface.co/OPEA/glm-4v-9b-int4-sym-inc) |--√ means support, - means support to export but cannot infer, X means not support.--## Integration--AutoRound has been integrated into multiple repositories.--[huggingface/transformers](https://github.com/huggingface/transformers)--[Intel Neural Compressor](https://github.com/intel/neural-compressor)--[ModelCloud/GPTQModel](https://github.com/ModelCloud/GPTQModel)--[pytorch/ao](https://github.com/pytorch/ao)--## Reference--If you find AutoRound useful for your research, please cite our paper:--```bash-@article{cheng2023optimize,- title={Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs},- author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi},- journal={arXiv preprint arXiv:2309.05516},- year={2023}-}-```diff --git a/auto_round/autoround.py b/auto_round/autoround.pyindex 87770d30c..a96978b91 100644--- a/auto_round/autoround.py+++ b/auto_round/autoround.py@@ -36,6 +36,7 @@ from auto_round.utils import (_gguf_args_check,block_forward,check_is_cpu,+ check_need_act_calibration,check_seqlen_compatible,check_skippable_keywords,check_to_quantized,@@ -254,8 +255,12 @@ class AutoRound(object):self.act_data_type = "float"tmp_act_bits = infer_bits_by_data_type(self.act_data_type)- if tmp_act_bits < 16:+ if tmp_act_bits < 16 and tmp_act_bits != self.act_bits:self.act_bits = tmp_act_bits+ logger.warning(+ f"act_bits set in 'act_data_type' do not"+ f" match the specified 'act_bits' setting. Resetting 'act_bits' to {tmp_act_bits}."+ )self.sampler = samplerself.not_use_best_mse = not_use_best_mse@@ -1142,7 +1147,9 @@ class AutoRound(object):self.model.to("cpu")if has_gguf_k and not self.disable_opt_rtn:self.quant_rtn_with_imatrix(all_to_quantized_module_names)- elif self.act_bits <= 8 and self.act_dynamic is False:+ elif self.act_bits <= 8 and check_need_act_calibration(+ self.act_dynamic, self.act_data_type+ ): ##TODO, mixed datatype has bughook_handles = self.register_act_max_hook(self.model)try:self.quantize_via_rtn_blockwise(all_to_quantized_module_names)@@ -2152,7 +2159,11 @@ class AutoRound(object):hook_handles = []for n, m in model.named_modules():- if hasattr(m, "act_dynamic") and not m.act_dynamic and check_to_quantized(m):+ if (+ hasattr(m, "act_dynamic")+ and check_need_act_calibration(m.act_dynamic, m.act_data_type)+ and check_to_quantized(m)+ ):hook = m.register_forward_hook(get_act_max_hook)hook_handles.append(hook)continue@@ -2160,10 +2171,11 @@ class AutoRound(object):# for whole model, RTNif n in self.layer_config:config = self.layer_config[n]+ act_dynamic = config.get("act_dynamic", True)+ act_data_type = config.get("act_data_type", None)if (config["bits"] <= 8- and "act_dynamic" in config- and config["act_dynamic"] is False+ and check_need_act_calibration(act_dynamic, act_data_type)and check_to_quantized(config)):hook = m.register_forward_hook(get_act_max_hook)diff --git a/auto_round/data_type/gguf.py b/auto_round/data_type/gguf.pyindex fa120938a..7ae7cfa46 100644--- a/auto_round/data_type/gguf.py+++ b/auto_round/data_type/gguf.py@@ -581,12 +581,11 @@ def quant_tensor_gguf_sym_dq(logger.warning_once("please use more data via setting `nsamples` to improve accuracy as calibration activations contain 0")-zero_cnt = torch.sum(quant_weights == 0, dim=-1)replace_index = zero_cnt > group_size // 2if torch.sum(replace_index) > 0:if bits == 6:- quant_weights[replace_index, :] = 1.0+ quant_weights[replace_index, :] = tensor * tensorelse:sigma2 = 2 * torch.sum(tensor**2, dim=-1, keepdim=True) / QK_Ktmp_quant_weights = torch.sqrt(sigma2 + tensor * tensor)diff --git a/auto_round/data_type/mxfp.py b/auto_round/data_type/mxfp.pyindex e8a9258d5..0c6ae265c 100644--- a/auto_round/data_type/mxfp.py+++ b/auto_round/data_type/mxfp.py@@ -62,9 +62,11 @@ def quant_element(tensor, ebits, mbits, max_norm, mantissa_rounding="even"):mask_tensor = ((abs_tensor - 0.5) % 2 == torch.zeros_like(abs_tensor)).type(tensor.dtype)tensor = torch.sign(tensor) * (floor_ste(abs_tensor + 0.5) - mask_tensor)elif mantissa_rounding == "nearest":- tensor = round_ste(tensor)+ tensor = torch.sign(tensor) * round_ste(torch.abs(tensor))elif mantissa_rounding == "floor":- tensor = floor_ste(tensor)+ tensor = torch.sign(tensor) * floor_ste(torch.abs(tensor))+ elif mantissa_rounding == "stochastic":+ tensor = torch.sign(tensor) * floor_ste(torch.abs(tensor) + torch.rand_like(tensor, requires_grad=False))else:raise ValueError("mantissa_rounding only supports even, nearest or floor.")diff --git a/auto_round/data_type/nvfp.py b/auto_round/data_type/nvfp.pyindex b39320100..7b7d98cc8 100644--- a/auto_round/data_type/nvfp.py+++ b/auto_round/data_type/nvfp.py@@ -73,8 +73,25 @@ def ref_nvfp4_quant(x, global_scale, block_size=16, v=0):def nv_fp4(tensor, bits=4, group_size=16, v=0, **kwargs):orig_dtype = tensor.dtypetensor, orig_shape, pad_len = reshape_pad_tensor_by_group_size(tensor, group_size)- tensor_amax = tensor.abs().max().to(torch.float32)- global_scale = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX * get_reciprocal(tensor_amax)+ tensor_max = tensor.abs().max().to(torch.float32)+ global_scale = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX * get_reciprocal(tensor_max)+ qdq_res, output_scale = ref_nvfp4_quant(tensor, global_scale, group_size, v)+ qdq_res = revert_tensor_by_pad(qdq_res, orig_shape=orig_shape, pad_len=pad_len)+ return qdq_res.to(orig_dtype), output_scale, None+++@register_dtype("nv_fp4_with_static_gs")+def nv_fp4_with_static_gs(tensor, bits=4, group_size=16, v=0, tensor_max=None, **kwargs):+ orig_dtype = tensor.dtype+ tensor, orig_shape, pad_len = reshape_pad_tensor_by_group_size(tensor, group_size)+ if tensor_max is None:+ tensor_max = tensor.abs().max().to(torch.float32)+ elif tensor_max is not None:+ if not isinstance(tensor_max, torch.Tensor):+ tensor_max = torch.tensor(tensor_max, device=tensor.device, dtype=torch.float32)+ if tensor_max.numel() != 1:+ tensor_max = tensor.abs().max().to(torch.float32)+ global_scale = FLOAT8_E4M3_MAX * FLOAT4_E2M1_MAX * get_reciprocal(tensor_max)qdq_res, output_scale = ref_nvfp4_quant(tensor, global_scale, group_size, v)qdq_res = revert_tensor_by_pad(qdq_res, orig_shape=orig_shape, pad_len=pad_len)return qdq_res.to(orig_dtype), output_scale, None@@ -174,12 +191,19 @@ def ref_fp4_quant(x, global_scale, block_size=16, v=0, max_scale=1.0):@register_dtype("fp4_v2_with_global_scale")@torch.compile()-def fp4_v2_with_global_scale(tensor, bits=4, group_size=16, v=0, max_scale=1.0, **kwargs):+def fp4_v2_with_global_scale(tensor, bits=4, group_size=16, v=0, tensor_max=None, max_scale=1.0, **kwargs):assert group_size == 32 or group_size == 16orig_dtype = tensor.dtypetensor, orig_shape, pad_len = reshape_pad_tensor_by_group_size(tensor, group_size)- tensor_amax = tensor.abs().max().to(torch.float32)- global_scale = FLOAT8_UE5M3_MAX * FLOAT4_E2M1_MAX * get_reciprocal(tensor_amax)++ if tensor_max is None:+ tensor_max = tensor.abs().max().to(torch.float32)+ elif tensor_max is not None:+ if not isinstance(tensor_max, torch.Tensor):+ tensor_max = torch.tensor(tensor_max, device=tensor.device, dtype=torch.float32)+ if tensor_max.numel() != 1:+ tensor_max = tensor.abs().max().to(torch.float32)+ global_scale = FLOAT8_UE5M3_MAX * FLOAT4_E2M1_MAX * get_reciprocal(tensor_max)qdq_res, output_scale = ref_fp4_quant(tensor, global_scale, group_size, v)qdq_res = revert_tensor_by_pad(qdq_res, orig_shape=orig_shape, pad_len=pad_len)return qdq_res.to(orig_dtype), output_scale, Nonediff --git a/auto_round/export/export_to_gguf/packing.py b/auto_round/export/export_to_gguf/packing.pyindex 204e3d1c0..79e635914 100644--- a/auto_round/export/export_to_gguf/packing.py+++ b/auto_round/export/export_to_gguf/packing.py@@ -315,7 +315,7 @@ def q4_0_quant_block(blocks, scale=None, zp=None, **kwargs):d = scale.reshape((-1, 1))else:imax = abs(blocks).argmax(axis=-1, keepdims=True)- max = torch.take_along_dim(blocks, imax, axis=-1)+ max = torch.take_along_dim(blocks, imax, dim=-1)d = max / -8id = get_reciprocal(d)diff --git a/auto_round/mllm/README.md b/auto_round/mllm/README.mdindex f33d8309b..7dfb37c3d 100644--- a/auto_round/mllm/README.md+++ b/auto_round/mllm/README.md@@ -54,27 +54,34 @@ auto-round-mllm \--output_dir ./tmp_autoround```-### Support Matrix-For most VLMs, we typically support the default quantization method, which involves quantizing only the language component while excluding the visual component. Besides, we also support quantizing non-text modules of models that follow the Hugging Face standard, i.e., those with a typical processor.--| Model | calibration dataset | quant nontext module |-|-------------------------------------|---------------------|----------------------|-| Qwen/Qwen2-VL | pile/llava | - |-| meta-llama/Llama-3.2-Vision | llava | ✔ |-| microsoft/Phi3-Vision | pile/llava | ✔ |-| liuhaotian/Llava-v1.5 | pile/llava | X |-| THUDM/CogVLM2 | pile/llava | ✔ |-| google/gemma-3 | pile/llava | √ |-| ibm-granite/granite-vision-3.2 | pile/llava | - |-| mistralai/Mistral-Small-3.1 | pile/llava | X |-| rhymes-ai/Aria | pile/llava | ✔ |-| deepseek-ai/deepseek-vl2 | pile/llava | ✔ |-| THUDM/glm-4v | pile | X |-| allenai/Molmo | pile | X |-| HuggingFaceTB/SmolVLM | pile/llava | ✔ |-| moonshotai/Kimi-VL | pile/llava | √ |--✔ means support, - means support to export but cannot infer, X means not support.+### VLM Support Matrix++For most VLMs, we typically support the default quantization configuration, which involves quantizing only the language+component while excluding the visual component. Besides, we also support quantizing non-text modules of models that+follow the Hugging Face standard, i.e., those with a typical processor, though inference may have some issues due to+model architecture or kernel limitations.++| Model | calibration dataset | quant nontext module | Quantized Model Link |+|-----------------------------------------------|---------------------|----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|+| allenai/Molmo | pile | X | [Molmo-7B-D-0924-int4-sym](https://huggingface.co/OPEA/Molmo-7B-D-0924-int4-sym-inc), [Molmo-72B-0924-int4-sym-gptq](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-gptq-inc), [Molmo-72B-0924-int4-sym](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-inc) |+| deepseek-ai/deepseek-vl2 | pile/llava | √ | [deepseek-vl2-int4-sym-gptq](https://huggingface.co/OPEA/deepseek-vl2-int4-sym-gptq-inc) |+| fancyfeast/llama-joycaption-beta-one-hf-llava | pile | X | [NeoChen1024-int4-gptq](https://huggingface.co/NeoChen1024/llama-joycaption-beta-one-hf-llava-GPTQ-4bit-sym-autoround)+| google/gemma-3 | pile/llava | √ | [gemma-3-12b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-12b-it-AutoRound-gguf-q4-0), [gemma-3-27b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-27b-it-AutoRound-gguf-q4-0), [gemma-3-12b-it-int4-AutoRound](https://huggingface.co/OPEA/gemma-3-12b-it-int4-AutoRound), [gemma-3-27b-it-int4-AutoRound](https://huggingface.co/OPEA/gemma-3-27b-it-int4-AutoRound) |+| HuggingFaceTB/SmolVLM | pile/llava | √ | [SmolVLM-Instruct-int4-sym](https://huggingface.co/OPEA/SmolVLM-Instruct-int4-sym-inc) |+| ibm-granite/granite-vision-3.2 | pile/llava | - | |+| liuhaotian/Llava-v1.5 | pile/llava | X | [llava-v1.5-7b-int4-sym](https://huggingface.co/OPEA/llava-v1.5-7b-int4-sym-inc) |+| meta-llama/Llama-3.2-Vision | llava | √ | [Llama-3.2V-11B-cot-int4-sym](https://huggingface.co/OPEA/Llama-3.2V-11B-cot-int4-sym-inc), [Llama-3.2-11B-Vision-Instruct-qvision-int4-sym](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-qvision-int4-sym-inc), [Llama-3.2-90B-Vision-Instruct-int4-sym](https://huggingface.co/OPEA/Llama-3.2-90B-Vision-Instruct-int4-sym-inc), [Llama-3.2-11B-Vision-Instruct-int4-sym](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-int4-sym-inc) |+| microsoft/Phi3.5-Vision | pile/llava | √ | [Phi-3.5-vision-instruct-int4-sym](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-int4-sym-inc), [Phi-3.5-vision-instruct-qvision-int4-sym](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-qvision-int4-sym-inc) |+| mistralai/Mistral-Small-3.1 | pile/llava | X | [Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-gptq-sym](https://huggingface.co/OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-gptq-sym), [Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym](https://huggingface.co/OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym) |+| moonshotai/Kimi-VL | pile/llava | √ | |+| Qwen/Qwen2-VL | pile/llava | - | [Qwen2-VL-7B-Instruct-int4-sym](https://huggingface.co/OPEA/Qwen2-VL-7B-Instruct-int4-sym-inc), [Qwen2-VL-72B-Instruct-int4-sym](https://huggingface.co/OPEA/Qwen2-VL-72B-Instruct-int4-sym-inc), [Qwen2-VL-72B-Instruct-int2-sym](https://huggingface.co/OPEA/Qwen2-VL-72B-Instruct-int2-sym-inc) |+| rhymes-ai/Aria | pile/llava | √ | |+| THUDM/CogVLM2 | pile/llava | √ | [cogvlm2-llama3-chat-19B-int4-sym](https://huggingface.co/OPEA/cogvlm2-llama3-chat-19B-int4-sym-inc), [cogvlm2-llama3-chat-19B-qvision-int4-sym](https://huggingface.co/OPEA/cogvlm2-llama3-chat-19B-qvision-int4-sym-inc) |+| THUDM/glm-4v | pile | X | [glm-4v-9b-int4-sym](https://huggingface.co/OPEA/glm-4v-9b-int4-sym-inc) |++√ means support, - means support to export but cannot infer, X means not support.++<details><summary style="font-size:17px;">Calibration Dataset</summary>@@ -164,21 +171,7 @@ code in ```auto_round/mllm/processor.py```.</details>-## Inference--For the AutoRound format, please add the following code at the beginning of the original model's inference code.--```python-from auto_round import AutoRoundConfig ## must import for auto-round format-```--For more details on quantization, inference, evaluation, and environment, see the following recipe:-- [Qwen2-VL-7B-Instruct](../../docs/Qwen2-VL-7B-Instruct-sym.md)-- [Llama-3.2-11B-Vision](../../docs/Llama-3.2-11B-Vision-Instruct-sym.md)-- [Phi-3.5-vision-instruct](../../docs/Phi-3.5-vision-instruct-sym.md)-- [llava-v1.5-7b](../../docs/llava-v1.5-7b-sym.md)-- [cogvlm2-llama3-chat-19B](../../docs/cogvlm2-llama3-chat-19B-sym.md)diff --git a/auto_round/script/llm.py b/auto_round/script/llm.pyindex a50c877ff..827ffff2a 100644--- a/auto_round/script/llm.py+++ b/auto_round/script/llm.py@@ -37,6 +37,7 @@ from auto_round.utils import (get_device_and_parallelism,get_fp_layer_names,get_model_dtype,+ infer_bits_by_data_type,set_cuda_visible_devices,str2bool,)@@ -583,7 +584,7 @@ def tune(args):import timeeval_model_dtype = get_model_dtype(args.eval_model_dtype, "auto")- if args.act_bits <= 8 or eval_gguf_model:+ if args.act_bits <= 8 or infer_bits_by_data_type(args.act_data_type) <= 8 or eval_gguf_model:if eval_gguf_model:# for file in os.listdir(eval_folder):# gguf_file = filediff --git a/auto_round/utils.py b/auto_round/utils.pyindex eadc090d6..36f6c70cf 100644--- a/auto_round/utils.py+++ b/auto_round/utils.py@@ -70,6 +70,8 @@ SUPPORTED_DTYPES = ("int", "mx_fp", "fp", "nv_fp")def infer_bits_by_data_type(data_type: str):+ if data_type is None:+ return 16for supported_dtype in SUPPORTED_DTYPES:if data_type.startswith(supported_dtype) and len(data_type) > len(supported_dtype):##first check the following two bits@@ -2043,3 +2045,11 @@ def get_reciprocal(tensor):else:tensor = torch.where(torch.abs(tensor) < 1e-30, 0, tensor)return torch.where(tensor != 0, 1 / tensor, torch.zeros_like(tensor))+++def check_need_act_calibration(is_act_dynamic, act_data_type=None):+ if not is_act_dynamic:+ return True+ if act_data_type is not None and "static" in act_data_type:+ return True+ return Falsediff --git a/auto_round/version.py b/auto_round/version.pyindex d9f21f781..aee1899ed 100644--- a/auto_round/version.py+++ b/auto_round/version.py@@ -14,4 +14,4 @@"""Intel® auto-round: An open-source Python librarysupporting popular model weight only compression based on signround."""-__version__ = "0.6.0"+__version__ = "0.6.1.dev"diff --git a/docs/step_by_step.md b/docs/step_by_step.mdindex 8fe57336c..e2b9b48f5 100644--- a/docs/step_by_step.md+++ b/docs/step_by_step.md@@ -1,7 +1,43 @@Step-by-Step============-This document presents step-by-step instructions for auto-round llm quantization.+This document presents step-by-step instructions for auto-round llm quantization. For vlms quantizaition, please refer to [vlms user guide](../auto_round/mllm/README.md)+++* [1 Prerequisite](#1-prerequisite)+* [2 Prepare Calibration Dataset](#2-prepare-calibration-dataset)+ + [Default Dataset](#default-dataset)+ + [Customized Dataset](#customized-dataset)+ + [Dataset operations](#dataset-operations)+* [3 Quantization](#3-quantization)+ + [Supported Quantization Configurations](#supported-quantization-configurations)+ + [Hardware Compatibility](#hardware-compatibility)+ + [Supported Export Formats](#supported-export-formats)+ + [Command Line Usage](#command-line-usage)+ + [API usage](#api-usage)+ - [AutoRound API Usage](#autoround-api-usage)+ - [Mixed bits Usage](#mixed-bits-usage)+ - [AutoRoundBest recipe](#autoroundbest-recipe)+ - [AutoRoundLight recipe](#autoroundlight-recipe)+ - [Recipe recommendation](#recipe-recommendation)+ + [RTN mode](#rtn-mode)+ + [GGUF format](#gguf-format)+ + [Quantization Costs](#quantization-costs)+ + [Device/Multi-GPU setting in Quantization](#device-multi-gpu-setting-in-quantization)+ - [Enable multiple gpus calibration in lm_head quantization](#enable-multiple-gpus-calibration-in-lm-head-quantization)+ - [Enable multiple gpus tuning for extremely large model](#enable-multiple-gpus-tuning-for-extremely-large-model)+ + [Adjust Hyperparameters](#adjust-hyperparameters)+* [4 Inference](#4-inference)+ + [CPU](#cpu)+ + [Intel GPU](#intel-gpu)+ + [CUDA](#cuda)+ + [HPU](#hpu)+ + [Specify Inference Backend](#specify-inference-backend)+ + [Convert GPTQ/AWQ to AutoRound](#convert-gptq-awq-to-autoround)+* [5 Evaluation](#5-evaluation)+ + [Combine evaluation with tuning](#combine-evaluation-with-tuning)+ + [Eval the Quantized model](#eval-the-quantized-model)+* [6 Known Issues](#6-known-issues)## 1 Prerequisite@@ -210,6 +246,33 @@ autoround = AutoRound(output_dir = "./tmp_autoround"autoround.quantize_and_save(output_dir, format="auto_round")```+#### Recipe recommendation++In conclusion, we recommend using **auto-round for INT4 and auto-round-best for INT2**. However, you may adjust the+configuration to suit your specific requirements and available resources.++W4G128 Average Accuracy of 13 tasks and Time Cost Results(Testing was conducted on the Nvidia A100 80G using the version+of PyTorch 2.6.0 with enable_torch_compile):++| Model | Qwen2.5-0.5B-Instruct | Falcon3-3B | Qwen2.5-7B-Instruct | Meta-Llama-3.1-8B-Instruct | Falcon3-10B | Qwen2.5-72B-Instruct |+|---------|-----------------------|-----------------|---------------------|----------------------------|-----------------|----------------------|+| 16bits | 0.4192 | 0.5203 | 0.6470 | 0.6212 | 0.6151 | 0.7229 |+| Best | **0.4137**(7m) | **0.5142**(23m) | 0.6426(58m) | **0.6116**(65m) | **0.6092**(81m) | 0.7242(575m) |+| Default | 0.4129(2m) | 0.5133(6m) | 0.6441(13m) | 0.6106(13m) | 0.6080(18m) | **0.7252**(118m) |+| Light | 0.4052(2m) | 0.5108(3m) | **0.6453**(5m) | 0.6104(6m) | 0.6063(6m) | 0.7243(37m) |++<details>+ <summary>W2G64 results</summary>+W2G64 Average Accuracy of 13 tasks and Time Cost Results(Testing was conducted on the Nvidia A100 80G using the version of PyTorch 2.6.0 with enable_torch_compile). We recommend using higher precision for the head, tail, and non-expert modules to alleviate the significant accuracy drop.++| Model | Qwen2.5-0.5B-Instruct | Falcon3-3B | Qwen2.5-7B-Instruct | Falcon3-10B | Qwen2.5-72B-Instruct |+ |---------|-----------------------|-----------------|---------------------|-----------------|----------------------|+| 16bits | 0.4192 | 0.5203 | 0.6470 | 0.6151 | 0.7229 |+| Best | **0.2989**(6m) | **0.4267**(24m) | **0.5343**(56m) | **0.5207**(79m) | **0.6715**(564m) |+| Default | 0.2878(2m) | 0.4219(6m) | 0.5209(13m) | 0.5133(18m) | 0.6713(122m) |+| Light | 0.2760(2m) | 0.4063(3m) | 0.4764(5m) | 0.4810(7m) | 0.6581(38m) |++</details>### RTN modeAutoRound also supports RTN (Round-To-Nearest) mode for fast, calibration-free baseline quantization. try setting iters=0 and use group_size=32 for better results.@@ -257,9 +320,53 @@ autoround = AutoRound()output_dir = "./tmp_autoround"-autoround.quantize_and_save(output_dir, format="gguf:q4_0") # gguf:q4_1, gguf:q*_k_s+autoround.quantize_and_save(output_dir, format="gguf:q4_k_m") # gguf:q*_k_s,gguf:q*_k_0,gguf:q*_k_1,```+### Supported export Formats++**AutoRound Format**: This format is well-suited for CPU, HPU devices, 2 bits, as well as mixed-precision+inference. **[2,3,4,8] bits are supported**.++**GGUF** Format: Experimental feature. This format is well-suited for CPU devices and is widely adopted by the+community. `q*_k`,`q*_0`,`q*_1` are supported.++**AutoGPTQ Format**: This format is well-suited for symmetric quantization on CUDA devices and is widely adopted by the+community, **[2,3,4,8] bits are supported**. However, **the+asymmetric kernel has issues** that can cause considerable accuracy drops, particularly at 2-bit quantization and small+models. Besides, recently 3 bits may have some accuracy issues in Transformers.++**AutoAWQ Format**: This format is well-suited for asymmetric 4-bit quantization on CUDA devices and is widely+adopted within the community, **only 4-bits quantization is supported**.++**llmcompressor Format**: This format is for reusing llmcompressor format, **only INT8 W8A8 dynamic quantization is+supported**.+++### Quantization Costs++Testing was conducted on the Nvidia A100 80G using the nightly version of PyTorch 2.6.0.dev20241029+cu124. Please note+that data+loading and packing costs have been excluded from the evaluation. **We recommend enabling torch.compile for PyTorch+versions 2.6 and above.**++To optimize GPU memory usage, in addition to activating `low_gpu_mem_usage`, you can set `gradient_accumulate_steps=8`+and a+`batch_size=1`, though this may increase tuning time.++The 3B and 14B models were evaluated on Qwen 2.5, the 8X7B model is Mixtral, while the remaining models utilized LLaMA+3.1.++| Torch version/Config W4G128 | 3B | 8B | 14B | 70B | 8X7B |+|---------------------------------------------------------------------------------------------|---------------|----------------|----------------|-----------------|----------------|+| 2.6 with torch compile | 7min<br/>10GB | 12min<br/>18GB | 23min<br/>22GB | 120min<br/>42GB | 28min<br/>46GB |+| 2.6 with torch compile <br/> low_gpu_mem_usage=True | 12min<br/>6GB | 19min<br/>10GB | 33min<br/>11GB | 140min<br/>25GB | 38min<br/>36GB |+| 2.6 with torch compile <br/> low_gpu_mem_usage=True <br/> gradient_accumulate_steps=8,bs=1 | 15min<br/>3GB | 25min<br/>6GB | 45min<br/>7GB | 187min<br/>19GB | 75min<br/>36GB |+| 2.5 w/o torch compile | 8min<br/>10GB | 16min<br/>20GB | 30min<br/>25GB | 140min<br/>49GB | 50min<br/>49GB |++++### Device/Multi-GPU setting in Quantization**The tuning device is specified using the `device` argument in AutoRound API, _not_ through the `device_map`parameter used by Transformers.from_pretrained.**