PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
using multicard when device set to "auto" (#368)
Heng Guob2944b910688
diff --git a/README.md b/README.mdindex c482fdc70..d45ddcbc7 100644--- a/README.md+++ b/README.md@@ -26,7 +26,7 @@ more accuracy data and recipes across various models.<div align="left">## What's New-* [2024/12] Many quantized LLMs/VLMs AutoRound are released in [OPEA Space](https://huggingface.co/OPEA)+* [2024/12] Many quantized LLMs/VLMs using AutoRound are released in [OPEA Space](https://huggingface.co/OPEA)* [2024/11] We provide experimental support for VLM quantization, please check outthe [README](./auto_round/mllm/README.md)* [2024/11] We provide some tips and tricks for LLM&VLM quantization, please checkdiff --git a/auto_round/mllm/README.md b/auto_round/mllm/README.mdindex fb206c532..bb58b8e2d 100644--- a/auto_round/mllm/README.md+++ b/auto_round/mllm/README.md@@ -12,7 +12,7 @@ quantize the entire model, you can enable `quant_nontext_module` by setting it tis limited.```python- from auto_round import AutoRoundMLLM+from auto_round import AutoRoundMLLMfrom transformers import Qwen2VLForConditionalGeneration, AutoProcessor, AutoTokenizer## load the modeldiff --git a/auto_round/script/llm.py b/auto_round/script/llm.pyindex f7cd48b0f..092ff2159 100644--- a/auto_round/script/llm.py+++ b/auto_round/script/llm.py@@ -506,6 +506,9 @@ def eval(args):if len(devices) > 1:parallelism = Truedevice_str = None+ elif args.device == "auto":+ device_str = None+ parallelism = Trueelse:device_str = detect_device(args.device.replace(" ", ""))