Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

update mllm readme (#530)

WeiweiZhang15e1836810a07
diff --git a/README.md b/README.md
index 0e87a457c..9f343ad3a 100644
--- a/README.md
+++ b/README.md
@@ -14,11 +14,12 @@ AutoRound
---
<div align="left">
-AutoRound is an advanced quantization algorithm for low-bits LLM/VLM inference. It's tailored for a wide range
-of models. AutoRound adopts sign gradient descent to fine-tune rounding values and minmax values of weights in just 200
-steps,
-which competes impressively against recent methods without introducing any additional inference overhead and keeping low
-tuning cost. The below
+AutoRound is an advanced quantization algorithm that delivers strong accuracy, even at 2-bit precision.
+It leverages sign gradient descent to fine-tune both rounding values and min-max clipping thresholds in just 200 steps.
+Designed for broad compatibility, it seamlessly supports a wide range of LLMs and is actively expanding to cover more VLMs as well.
+It also supports quantization and inference across multiple hardware platforms, including CPU, XPU, and CUDA.
+AutoRound also offers a variety of useful features, including mixed-bit tuning and inference, lm-head quantization,
+support for exporting to formats like GPTQ/AWQ/GGUF, and flexible tuning recipes. The below
image presents an overview of AutoRound. Check out our paper on [arxiv](https://arxiv.org/pdf/2309.05516) for more
details and quantized models in several Hugging Face Spaces,
e.g. [OPEA](https://huggingface.co/OPEA), [Kaitchup](https://huggingface.co/kaitchup)
@@ -509,3 +510,4 @@ If you find AutoRound useful for your research, please cite our paper:
+
diff --git a/auto_round/mllm/README.md b/auto_round/mllm/README.md
index b62c48534..2f961df1c 100644
--- a/auto_round/mllm/README.md
+++ b/auto_round/mllm/README.md
@@ -96,24 +96,28 @@ liuhaotian/llava_instruct_80k", "liuhaotian/llava_instruct_150k" or a file path
### Support Matrix
-For most VLMs, we typically support the default quantization configuration, which involves quantizing only the language component while excluding the visual component. Besides, we also support quantizing non-text modules of models that follow the Hugging Face standard, i.e., those with a typical processor, though inference may have some issues due to model architecture or kernel limitations.
-
-| Model | calibration dataset | quant nontext module |
-|-------------------------------------|---------------------|----------------------|
-| allenai/Molmo | pile | X |
-| deepseek-ai/deepseek-vl2 | pile/llava | √ |
-| google/gemma-3 | pile/llava | √ |
-| HuggingFaceTB/SmolVLM | pile/llava | √ |
-| ibm-granite/granite-vision-3.2 | pile/llava | - |
-| liuhaotian/Llava-v1.5 | pile/llava | X |
-| meta-llama/Llama-3.2-Vision | llava | √ |
-| microsoft/Phi3-Vision | pile/llava | √ |
-| mistralai/Mistral-Small-3.1 | pile/llava | X |
-| moonshotai/Kimi-VL | pile/llava | √ |
-| Qwen/Qwen2-VL | pile/llava | - |
-| rhymes-ai/Aria | pile/llava | √ |
-| THUDM/CogVLM2 | pile/llava | √ |
-| THUDM/glm-4v | pile | X |
+For most VLMs, we typically support the default quantization configuration, which involves quantizing only the language
+component while excluding the visual component. Besides, we also support quantizing non-text modules of models that
+follow the Hugging Face standard, i.e., those with a typical processor, though inference may have some issues due to
+model architecture or kernel limitations.
+
+
+| Model | calibration dataset | quant nontext module | Quantized Model Link |
+|--------------------------------|---------------------|----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| allenai/Molmo | pile | X | [Molmo-7B-D-0924-int4-sym](https://huggingface.co/OPEA/Molmo-7B-D-0924-int4-sym-inc), [Molmo-72B-0924-int4-sym-gptq](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-gptq-inc), [Molmo-72B-0924-int4-sym](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-inc) |
+| deepseek-ai/deepseek-vl2 | pile/llava | √ | [deepseek-vl2-int4-sym-gptq](https://huggingface.co/OPEA/deepseek-vl2-int4-sym-gptq-inc) |
+| google/gemma-3 | pile/llava | √ | [gemma-3-12b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-12b-it-AutoRound-gguf-q4-0), [gemma-3-27b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-27b-it-AutoRound-gguf-q4-0), [gemma-3-12b-it-int4-AutoRound-cpu](https://huggingface.co/OPEA/gemma-3-12b-it-int4-AutoRound-cpu), [gemma-3-27b-it-int4-AutoRound-cpu](https://huggingface.co/OPEA/gemma-3-27b-it-int4-AutoRound-cpu) |
+| HuggingFaceTB/SmolVLM | pile/llava | √ | [SmolVLM-Instruct-int4-sym](https://huggingface.co/OPEA/SmolVLM-Instruct-int4-sym-inc) |
+| ibm-granite/granite-vision-3.2 | pile/llava | - | |
+| liuhaotian/Llava-v1.5 | pile/llava | X | [llava-v1.5-7b-int4-sym](https://huggingface.co/OPEA/llava-v1.5-7b-int4-sym-inc) |
+| meta-llama/Llama-3.2-Vision | llava | √ | [Llama-3.2V-11B-cot-int4-sym](https://huggingface.co/OPEA/Llama-3.2V-11B-cot-int4-sym-inc), [Llama-3.2-11B-Vision-Instruct-qvision-int4-sym](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-qvision-int4-sym-inc), [Llama-3.2-90B-Vision-Instruct-int4-sym](https://huggingface.co/OPEA/Llama-3.2-90B-Vision-Instruct-int4-sym-inc), [Llama-3.2-11B-Vision-Instruct-int4-sym](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-int4-sym-inc) |
+| microsoft/Phi3.5-Vision | pile/llava | √ | [Phi-3.5-vision-instruct-int4-sym](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-int4-sym-inc), [Phi-3.5-vision-instruct-qvision-int4-sym](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-qvision-int4-sym-inc) |
+| mistralai/Mistral-Small-3.1 | pile/llava | X | [Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-gptq-sym](https://huggingface.co/OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-gptq-sym), [Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym](https://huggingface.co/OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym) |
+| moonshotai/Kimi-VL | pile/llava | √ | |
+| Qwen/Qwen2-VL | pile/llava | - | [Qwen2-VL-7B-Instruct-int4-sym](https://huggingface.co/OPEA/Qwen2-VL-7B-Instruct-int4-sym-inc), [Qwen2-VL-72B-Instruct-int4-sym](https://huggingface.co/OPEA/Qwen2-VL-72B-Instruct-int4-sym-inc), [Qwen2-VL-72B-Instruct-int2-sym](https://huggingface.co/OPEA/Qwen2-VL-72B-Instruct-int2-sym-inc) |
+| rhymes-ai/Aria | pile/llava | √ | |
+| THUDM/CogVLM2 | pile/llava | √ | [cogvlm2-llama3-chat-19B-int4-sym](https://huggingface.co/OPEA/cogvlm2-llama3-chat-19B-int4-sym-inc), [cogvlm2-llama3-chat-19B-qvision-int4-sym](https://huggingface.co/OPEA/cogvlm2-llama3-chat-19B-qvision-int4-sym-inc) |
+| THUDM/glm-4v | pile | X | [glm-4v-9b-int4-sym](https://huggingface.co/OPEA/glm-4v-9b-int4-sym-inc) |
√ means support, - means support to export but cannot infer, X means not support.
@@ -180,3 +184,4 @@ For more details on quantization, inference, evaluation, and environment, see th
+