PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
update mllm readme (#530)
WeiweiZhang15e1836810a07
diff --git a/README.md b/README.mdindex 0e87a457c..9f343ad3a 100644--- a/README.md+++ b/README.md@@ -14,11 +14,12 @@ AutoRound---<div align="left">-AutoRound is an advanced quantization algorithm for low-bits LLM/VLM inference. It's tailored for a wide range-of models. AutoRound adopts sign gradient descent to fine-tune rounding values and minmax values of weights in just 200-steps,-which competes impressively against recent methods without introducing any additional inference overhead and keeping low-tuning cost. The below+AutoRound is an advanced quantization algorithm that delivers strong accuracy, even at 2-bit precision.+It leverages sign gradient descent to fine-tune both rounding values and min-max clipping thresholds in just 200 steps.+Designed for broad compatibility, it seamlessly supports a wide range of LLMs and is actively expanding to cover more VLMs as well.+It also supports quantization and inference across multiple hardware platforms, including CPU, XPU, and CUDA.+AutoRound also offers a variety of useful features, including mixed-bit tuning and inference, lm-head quantization,+support for exporting to formats like GPTQ/AWQ/GGUF, and flexible tuning recipes. The belowimage presents an overview of AutoRound. Check out our paper on [arxiv](https://arxiv.org/pdf/2309.05516) for moredetails and quantized models in several Hugging Face Spaces,e.g. [OPEA](https://huggingface.co/OPEA), [Kaitchup](https://huggingface.co/kaitchup)@@ -509,3 +510,4 @@ If you find AutoRound useful for your research, please cite our paper:+diff --git a/auto_round/mllm/README.md b/auto_round/mllm/README.mdindex b62c48534..2f961df1c 100644--- a/auto_round/mllm/README.md+++ b/auto_round/mllm/README.md@@ -96,24 +96,28 @@ liuhaotian/llava_instruct_80k", "liuhaotian/llava_instruct_150k" or a file path### Support Matrix-For most VLMs, we typically support the default quantization configuration, which involves quantizing only the language component while excluding the visual component. Besides, we also support quantizing non-text modules of models that follow the Hugging Face standard, i.e., those with a typical processor, though inference may have some issues due to model architecture or kernel limitations.--| Model | calibration dataset | quant nontext module |-|-------------------------------------|---------------------|----------------------|-| allenai/Molmo | pile | X |-| deepseek-ai/deepseek-vl2 | pile/llava | √ |-| google/gemma-3 | pile/llava | √ |-| HuggingFaceTB/SmolVLM | pile/llava | √ |-| ibm-granite/granite-vision-3.2 | pile/llava | - |-| liuhaotian/Llava-v1.5 | pile/llava | X |-| meta-llama/Llama-3.2-Vision | llava | √ |-| microsoft/Phi3-Vision | pile/llava | √ |-| mistralai/Mistral-Small-3.1 | pile/llava | X |-| moonshotai/Kimi-VL | pile/llava | √ |-| Qwen/Qwen2-VL | pile/llava | - |-| rhymes-ai/Aria | pile/llava | √ |-| THUDM/CogVLM2 | pile/llava | √ |-| THUDM/glm-4v | pile | X |+For most VLMs, we typically support the default quantization configuration, which involves quantizing only the language+component while excluding the visual component. Besides, we also support quantizing non-text modules of models that+follow the Hugging Face standard, i.e., those with a typical processor, though inference may have some issues due to+model architecture or kernel limitations.+++| Model | calibration dataset | quant nontext module | Quantized Model Link |+|--------------------------------|---------------------|----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|+| allenai/Molmo | pile | X | [Molmo-7B-D-0924-int4-sym](https://huggingface.co/OPEA/Molmo-7B-D-0924-int4-sym-inc), [Molmo-72B-0924-int4-sym-gptq](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-gptq-inc), [Molmo-72B-0924-int4-sym](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-inc) |+| deepseek-ai/deepseek-vl2 | pile/llava | √ | [deepseek-vl2-int4-sym-gptq](https://huggingface.co/OPEA/deepseek-vl2-int4-sym-gptq-inc) |+| google/gemma-3 | pile/llava | √ | [gemma-3-12b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-12b-it-AutoRound-gguf-q4-0), [gemma-3-27b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-27b-it-AutoRound-gguf-q4-0), [gemma-3-12b-it-int4-AutoRound-cpu](https://huggingface.co/OPEA/gemma-3-12b-it-int4-AutoRound-cpu), [gemma-3-27b-it-int4-AutoRound-cpu](https://huggingface.co/OPEA/gemma-3-27b-it-int4-AutoRound-cpu) |+| HuggingFaceTB/SmolVLM | pile/llava | √ | [SmolVLM-Instruct-int4-sym](https://huggingface.co/OPEA/SmolVLM-Instruct-int4-sym-inc) |+| ibm-granite/granite-vision-3.2 | pile/llava | - | |+| liuhaotian/Llava-v1.5 | pile/llava | X | [llava-v1.5-7b-int4-sym](https://huggingface.co/OPEA/llava-v1.5-7b-int4-sym-inc) |+| meta-llama/Llama-3.2-Vision | llava | √ | [Llama-3.2V-11B-cot-int4-sym](https://huggingface.co/OPEA/Llama-3.2V-11B-cot-int4-sym-inc), [Llama-3.2-11B-Vision-Instruct-qvision-int4-sym](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-qvision-int4-sym-inc), [Llama-3.2-90B-Vision-Instruct-int4-sym](https://huggingface.co/OPEA/Llama-3.2-90B-Vision-Instruct-int4-sym-inc), [Llama-3.2-11B-Vision-Instruct-int4-sym](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-int4-sym-inc) |+| microsoft/Phi3.5-Vision | pile/llava | √ | [Phi-3.5-vision-instruct-int4-sym](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-int4-sym-inc), [Phi-3.5-vision-instruct-qvision-int4-sym](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-qvision-int4-sym-inc) |+| mistralai/Mistral-Small-3.1 | pile/llava | X | [Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-gptq-sym](https://huggingface.co/OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-gptq-sym), [Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym](https://huggingface.co/OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym) |+| moonshotai/Kimi-VL | pile/llava | √ | |+| Qwen/Qwen2-VL | pile/llava | - | [Qwen2-VL-7B-Instruct-int4-sym](https://huggingface.co/OPEA/Qwen2-VL-7B-Instruct-int4-sym-inc), [Qwen2-VL-72B-Instruct-int4-sym](https://huggingface.co/OPEA/Qwen2-VL-72B-Instruct-int4-sym-inc), [Qwen2-VL-72B-Instruct-int2-sym](https://huggingface.co/OPEA/Qwen2-VL-72B-Instruct-int2-sym-inc) |+| rhymes-ai/Aria | pile/llava | √ | |+| THUDM/CogVLM2 | pile/llava | √ | [cogvlm2-llama3-chat-19B-int4-sym](https://huggingface.co/OPEA/cogvlm2-llama3-chat-19B-int4-sym-inc), [cogvlm2-llama3-chat-19B-qvision-int4-sym](https://huggingface.co/OPEA/cogvlm2-llama3-chat-19B-qvision-int4-sym-inc) |+| THUDM/glm-4v | pile | X | [glm-4v-9b-int4-sym](https://huggingface.co/OPEA/glm-4v-9b-int4-sym-inc) |√ means support, - means support to export but cannot infer, X means not support.@@ -180,3 +184,4 @@ For more details on quantization, inference, evaluation, and environment, see th+