Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

using multicard when device set to "auto" (#368)

Heng Guob2944b910688
diff --git a/README.md b/README.md
index c482fdc70..d45ddcbc7 100644
--- a/README.md
+++ b/README.md
@@ -26,7 +26,7 @@ more accuracy data and recipes across various models.
<div align="left">
## What's New
-* [2024/12] Many quantized LLMs/VLMs AutoRound are released in [OPEA Space](https://huggingface.co/OPEA)
+* [2024/12] Many quantized LLMs/VLMs using AutoRound are released in [OPEA Space](https://huggingface.co/OPEA)
* [2024/11] We provide experimental support for VLM quantization, please check out
the [README](./auto_round/mllm/README.md)
* [2024/11] We provide some tips and tricks for LLM&VLM quantization, please check
diff --git a/auto_round/mllm/README.md b/auto_round/mllm/README.md
index fb206c532..bb58b8e2d 100644
--- a/auto_round/mllm/README.md
+++ b/auto_round/mllm/README.md
@@ -12,7 +12,7 @@ quantize the entire model, you can enable `quant_nontext_module` by setting it t
is limited.
```python
- from auto_round import AutoRoundMLLM
+from auto_round import AutoRoundMLLM
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor, AutoTokenizer
## load the model
diff --git a/auto_round/script/llm.py b/auto_round/script/llm.py
index f7cd48b0f..092ff2159 100644
--- a/auto_round/script/llm.py
+++ b/auto_round/script/llm.py
@@ -506,6 +506,9 @@ def eval(args):
if len(devices) > 1:
parallelism = True
device_str = None
+ elif args.device == "auto":
+ device_str = None
+ parallelism = True
else:
device_str = detect_device(args.device.replace(" ", ""))