PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
add backend readme (#528)
Wenhua Chengc62cd0fb0ca2
diff --git a/README.md b/README.mdindex f8ffca01f..0e87a457c 100644--- a/README.md+++ b/README.md@@ -224,7 +224,9 @@ autoround.quantize_and_save(output_dir, format='auto_round')</details>### API Usage for VLMs-If you encounter issues during quantization, try setting iters=0 (to enable RTN) and use group_size=32 for better results.++If you encounter issues during quantization, try setting iters=0 (to enable RTN) and use group_size=32 for better+results.<details>@@ -316,7 +318,7 @@ in [Gaudi Guide](https://docs.habana.ai/en/latest/).```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizer-from auto_round import AutoRoundConfig+from auto_round import AutoRoundConfig ## must import for auto-round formatquantized_model_path = "./tmp_autoround"model = AutoModelForCausalLM.from_pretrained(quantized_model_path,@@ -327,6 +329,58 @@ inputs = tokenizer(text, return_tensors="pt").to(model.device)print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))```+### Specify backend++AutoRound automatically selects the best available backend based on the installed libraries and prompts the user to+install additional libraries when a better backend is found. On CUDA, the default priority is Marlin > ExLLaMAV2 >+Triton, but the final choice depends on factors such as bits group_size packing format compatibility, etc. Please refer+to the following table for the details.++| Name | Devices | Bits | Dtypes | Priority | Packing format | Requirements |+|--------------------------------------|---------|---------|-----------|----------|-----------------|-------------------------------|+| ipex | cpu/xpu | 4 | BF16/FP16 | 5 | gptq_zp+-1/awq | intel-extension-for-pytorch |+| itrex | cpu | 2,4,8 | BF16/FP16 | 0 | gptq_zp+-1/awq | intel-extension-for-transformers |+| marlin | cuda | 4,8 | BF16/FP16 | 6 | gptq/gptq_zp+-1 | gptqmodel |+| exllamav2 or<br/>gptqmodel:exllamav2 | cuda | 4 | BF16/FP16 | 5 | gptq | gptqmodel |+| exllamav2 or<br/>gptq:exllamav2 | cuda | 4 | FP16 | 5 | gptq_zp+-1 | auto-gptq |+| gptq:cuda | cuda | 2,3,4,8 | FP16 | 0 | gptq_zp+-1 | auto-gptq |+| triton | cuda | 2,3,8 | BF16/FP16 | 1 | gptq/gptq_zp+-1 | auto-round |+| awq | cuda | 4 | FP16 | 5 | awq | auto-awq |+| hpu | hpu | 4 | BF16 | 0 | gptq/gptq_zp+-1 | auto-round |++```python+from transformers import AutoModelForCausalLM, AutoTokenizer++quantized_model_path = "./tmp_autoround"+quantization_config = AutoRoundConfig(backend="auto")+model = AutoModelForCausalLM.from_pretrained(quantized_model_path, device_map="auto",+ torch_dtype="auto",+ quantization_config=quantization_config)+tokenizer = AutoTokenizer.from_pretrained(quantized_model_path)+text = "There is a girl who likes adventure,"+inputs = tokenizer(text, return_tensors="pt").to(model.device)+print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))+```++### Convert GPTQ/AWQ format to AutoRound++Most GPTQ/AWQ models can be converted to the AutoRound format for better compatibility and support with Intel devices.+Please note that the quantization config will be changed if the model is serialized.++```python+from transformers import AutoModelForCausalLM, AutoTokenizer+from auto_round import AutoRoundConfig ## must import for auto-round format++model_name = "ybelkada/opt-125m-gptq-4bit"+quantization_config = AutoRoundConfig()+model = AutoModelForCausalLM.from_pretrained(model_name, device_map="cpu", torch_dtype="auto",+ quantization_config=quantization_config)+tokenizer = AutoTokenizer.from_pretrained(model_name)+text = "There is a girl who likes adventure,"+inputs = tokenizer(text, return_tensors="pt").to(model.device)+print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))+```+#### Evaluation<details>@@ -341,20 +395,6 @@ auto-round --model saved_quantized_model \</details>-### AutoGPTQ/AutoAWQ format--```python-from transformers import AutoModelForCausalLM, AutoTokenizer--quantized_model_path = "./tmp_autoround"-model = AutoModelForCausalLM.from_pretrained(quantized_model_path, torch_dtype="auto",- device_map="auto")-tokenizer = AutoTokenizer.from_pretrained(quantized_model_path)-text = "There is a girl who likes adventure,"-inputs = tokenizer(text, return_tensors="pt").to(model.device)-print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))-```-## Support ListAutoRound supports basically all the major large language models.@@ -412,27 +452,29 @@ release most of the models ourselves.</details>-### VLM Support Matrix-For most VLMs, we typically support the default quantization configuration, which involves quantizing only the language component while excluding the visual component. Besides, we also support quantizing non-text modules of models that follow the Hugging Face standard, i.e., those with a typical processor, though inference may have some issues due to model architecture or kernel limitations.--| Model | calibration dataset | quant nontext module | Quantized Model Link |-|-------------------------------------|---------------------|----------------------| ----------------------|-| allenai/Molmo | pile | X | [Molmo-7B-D-0924-int4-sym](https://huggingface.co/OPEA/Molmo-7B-D-0924-int4-sym-inc), [Molmo-72B-0924-int4-sym-gptq](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-gptq-inc), [Molmo-72B-0924-int4-sym](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-inc) |-| deepseek-ai/deepseek-vl2 | pile/llava | √ | [deepseek-vl2-int4-sym-gptq](https://huggingface.co/OPEA/deepseek-vl2-int4-sym-gptq-inc) |-| google/gemma-3 | pile/llava | √ | [gemma-3-12b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-12b-it-AutoRound-gguf-q4-0), [gemma-3-27b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-27b-it-AutoRound-gguf-q4-0), [gemma-3-12b-it-int4-AutoRound-cpu](https://huggingface.co/OPEA/gemma-3-12b-it-int4-AutoRound-cpu), [gemma-3-27b-it-int4-AutoRound-cpu](https://huggingface.co/OPEA/gemma-3-27b-it-int4-AutoRound-cpu) |-| HuggingFaceTB/SmolVLM | pile/llava | √ | [SmolVLM-Instruct-int4-sym](https://huggingface.co/OPEA/SmolVLM-Instruct-int4-sym-inc) |-| ibm-granite/granite-vision-3.2 | pile/llava | - | |-| liuhaotian/Llava-v1.5 | pile/llava | X |[llava-v1.5-7b-int4-sym](https://huggingface.co/OPEA/llava-v1.5-7b-int4-sym-inc) |-| meta-llama/Llama-3.2-Vision | llava | √ | [Llama-3.2V-11B-cot-int4-sym](https://huggingface.co/OPEA/Llama-3.2V-11B-cot-int4-sym-inc), [Llama-3.2-11B-Vision-Instruct-qvision-int4-sym](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-qvision-int4-sym-inc), [Llama-3.2-90B-Vision-Instruct-int4-sym](https://huggingface.co/OPEA/Llama-3.2-90B-Vision-Instruct-int4-sym-inc), [Llama-3.2-11B-Vision-Instruct-int4-sym](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-int4-sym-inc) |-| microsoft/Phi3.5-Vision | pile/llava | √ | [Phi-3.5-vision-instruct-int4-sym](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-int4-sym-inc), [Phi-3.5-vision-instruct-qvision-int4-sym](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-qvision-int4-sym-inc) |-| mistralai/Mistral-Small-3.1 | pile/llava | X | [Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-gptq-sym](https://huggingface.co/OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-gptq-sym), [Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym](https://huggingface.co/OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym) |-| moonshotai/Kimi-VL | pile/llava | √ | |-| Qwen/Qwen2-VL | pile/llava | - | [Qwen2-VL-7B-Instruct-int4-sym](https://huggingface.co/OPEA/Qwen2-VL-7B-Instruct-int4-sym-inc), [Qwen2-VL-72B-Instruct-int4-sym](https://huggingface.co/OPEA/Qwen2-VL-72B-Instruct-int4-sym-inc), [Qwen2-VL-72B-Instruct-int2-sym](https://huggingface.co/OPEA/Qwen2-VL-72B-Instruct-int2-sym-inc) |-| rhymes-ai/Aria | pile/llava | √ | |-| THUDM/CogVLM2 | pile/llava | √ | [cogvlm2-llama3-chat-19B-int4-sym](https://huggingface.co/OPEA/cogvlm2-llama3-chat-19B-int4-sym-inc), [cogvlm2-llama3-chat-19B-qvision-int4-sym](https://huggingface.co/OPEA/cogvlm2-llama3-chat-19B-qvision-int4-sym-inc) |-| THUDM/glm-4v | pile | X | [glm-4v-9b-int4-sym](https://huggingface.co/OPEA/glm-4v-9b-int4-sym-inc) |+For most VLMs, we typically support the default quantization configuration, which involves quantizing only the language+component while excluding the visual component. Besides, we also support quantizing non-text modules of models that+follow the Hugging Face standard, i.e., those with a typical processor, though inference may have some issues due to+model architecture or kernel limitations.++| Model | calibration dataset | quant nontext module | Quantized Model Link |+|--------------------------------|---------------------|----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|+| allenai/Molmo | pile | X | [Molmo-7B-D-0924-int4-sym](https://huggingface.co/OPEA/Molmo-7B-D-0924-int4-sym-inc), [Molmo-72B-0924-int4-sym-gptq](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-gptq-inc), [Molmo-72B-0924-int4-sym](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-inc) |+| deepseek-ai/deepseek-vl2 | pile/llava | √ | [deepseek-vl2-int4-sym-gptq](https://huggingface.co/OPEA/deepseek-vl2-int4-sym-gptq-inc) |+| google/gemma-3 | pile/llava | √ | [gemma-3-12b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-12b-it-AutoRound-gguf-q4-0), [gemma-3-27b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-27b-it-AutoRound-gguf-q4-0), [gemma-3-12b-it-int4-AutoRound-cpu](https://huggingface.co/OPEA/gemma-3-12b-it-int4-AutoRound-cpu), [gemma-3-27b-it-int4-AutoRound-cpu](https://huggingface.co/OPEA/gemma-3-27b-it-int4-AutoRound-cpu) |+| HuggingFaceTB/SmolVLM | pile/llava | √ | [SmolVLM-Instruct-int4-sym](https://huggingface.co/OPEA/SmolVLM-Instruct-int4-sym-inc) |+| ibm-granite/granite-vision-3.2 | pile/llava | - | |+| liuhaotian/Llava-v1.5 | pile/llava | X | [llava-v1.5-7b-int4-sym](https://huggingface.co/OPEA/llava-v1.5-7b-int4-sym-inc) |+| meta-llama/Llama-3.2-Vision | llava | √ | [Llama-3.2V-11B-cot-int4-sym](https://huggingface.co/OPEA/Llama-3.2V-11B-cot-int4-sym-inc), [Llama-3.2-11B-Vision-Instruct-qvision-int4-sym](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-qvision-int4-sym-inc), [Llama-3.2-90B-Vision-Instruct-int4-sym](https://huggingface.co/OPEA/Llama-3.2-90B-Vision-Instruct-int4-sym-inc), [Llama-3.2-11B-Vision-Instruct-int4-sym](https://huggingface.co/OPEA/Llama-3.2-11B-Vision-Instruct-int4-sym-inc) |+| microsoft/Phi3.5-Vision | pile/llava | √ | [Phi-3.5-vision-instruct-int4-sym](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-int4-sym-inc), [Phi-3.5-vision-instruct-qvision-int4-sym](https://huggingface.co/OPEA/Phi-3.5-vision-instruct-qvision-int4-sym-inc) |+| mistralai/Mistral-Small-3.1 | pile/llava | X | [Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-gptq-sym](https://huggingface.co/OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-gptq-sym), [Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym](https://huggingface.co/OPEA/Mistral-Small-3.1-24B-Instruct-2503-int4-AutoRound-awq-sym) |+| moonshotai/Kimi-VL | pile/llava | √ | |+| Qwen/Qwen2-VL | pile/llava | - | [Qwen2-VL-7B-Instruct-int4-sym](https://huggingface.co/OPEA/Qwen2-VL-7B-Instruct-int4-sym-inc), [Qwen2-VL-72B-Instruct-int4-sym](https://huggingface.co/OPEA/Qwen2-VL-72B-Instruct-int4-sym-inc), [Qwen2-VL-72B-Instruct-int2-sym](https://huggingface.co/OPEA/Qwen2-VL-72B-Instruct-int2-sym-inc) |+| rhymes-ai/Aria | pile/llava | √ | |+| THUDM/CogVLM2 | pile/llava | √ | [cogvlm2-llama3-chat-19B-int4-sym](https://huggingface.co/OPEA/cogvlm2-llama3-chat-19B-int4-sym-inc), [cogvlm2-llama3-chat-19B-qvision-int4-sym](https://huggingface.co/OPEA/cogvlm2-llama3-chat-19B-qvision-int4-sym-inc) |+| THUDM/glm-4v | pile | X | [glm-4v-9b-int4-sym](https://huggingface.co/OPEA/glm-4v-9b-int4-sym-inc) |√ means support, - means support to export but cannot infer, X means not support.diff --git a/auto_round/inference/backend.py b/auto_round/inference/backend.pyindex ed2d3fac1..99e9417f2 100644--- a/auto_round/inference/backend.py+++ b/auto_round/inference/backend.py@@ -178,17 +178,17 @@ BackendInfos['gptqmodel:exllamav2'] = BackendInfo(device=["cuda"], sym=[True, FaBackendInfos['auto_awq:gemm'] = BackendInfo(device=["cuda"], sym=[True, False], ##actually is gemmpacking_format="awq",bits=[4], group_size=None,- priority=4,+ priority=5,dtype=["float16"],alias=["auto_awq:gemm", "awq","awq:gemm","auto_awq"],requirements=["autoawq"])-BackendInfos['qbits_no_zp'] = BackendInfo(device=["cpu"], sym=[True, False],+BackendInfos['qbits'] = BackendInfo(device=["cpu"], sym=[True, False],packing_format="qbits",bits=[2, 4, 8], group_size=None,- priority=0 if "intel" in get_cpu_manufacturer() else 5,+ priority=0,feature_checks=[],alias=["itrex", "qbits"],dtype=["float16", "bfloat16"],@@ -199,25 +199,25 @@ BackendInfos['qbits_zp'] = BackendInfo(device=["cpu"], sym=[True, False],packing_format="qbits_zp",bits=[2, 4, 8], group_size=None,dtype=["float16", "bfloat16"],- priority=0 if "intel" in get_cpu_manufacturer() else 5,+ priority=0 ,feature_checks=[],alias=["itrex", "qbits"],convertable_format=["int32_zp"],requirements=["intel-extension-for-transformers"])-# BackendInfos['auto_round:qbits_awq'] = BackendInfo(device=["cpu"], sym=[True], ## for awq, not robust-# packing_format="awq",-# bits=[2, 4, 8], group_size=None,-# priority=0 if "intel" in get_cpu_manufacturer() else 5,-# feature_checks=[],-# requirements=["intel-extension-for-transformers"]-# )+BackendInfos['auto_round:qbits_awq'] = BackendInfo(device=["cpu"], sym=[True,False], ## for awq, not robust+ packing_format="awq",+ bits=[2, 4, 8], group_size=None,+ priority=0,+ feature_checks=[],+ requirements=["intel-extension-for-transformers"]+ )BackendInfos['ipex_gptq'] = BackendInfo(device=["cpu", "xpu"], sym=[True, False],packing_format="ipex_gptq",bits=[4], group_size=None,- priority=5 if "intel" in get_cpu_manufacturer() else 5,+ priority=5,feature_checks=[],dtype=["float16", "bfloat16"],convertable_format=["int32_zp"],@@ -240,6 +240,7 @@ BackendInfos['hpu'] = BackendInfo(device=["hpu"], sym=[True, False],packing_format="hpu",bits=[4],dtype=["bfloat16"],+ alias=["hpu"],priority=0,convertable_format=["int32"])