Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

fix model_dtype issue and reformat mllm code (#335)

Wenhua Chengfcaaac917c7d
diff --git a/auto_round/mllm/README.md b/auto_round/mllm/README.md
index cd342f70d..50efca1e6 100644
--- a/auto_round/mllm/README.md
+++ b/auto_round/mllm/README.md
@@ -1,6 +1,10 @@
# AutoRound for MLLMs
+
## Basic Usage (Gaudi2/CPU/GPU)
-A user guide detailing the full list of supported arguments is provided by calling ```auto-round-mllm -h``` on the terminal.Alternatively, you can use ```auto_round_mllm``` instead of ```auto-round-mllm```. Set the format you want in `format` and
+
+A user guide detailing the full list of supported arguments is provided by calling ```auto-round-mllm -h``` on the
+terminal.Alternatively, you can use ```auto_round_mllm``` instead of ```auto-round-mllm```. Set the format you want
+in `format` and
multiple formats exporting has been supported.
```bash
@@ -14,7 +18,9 @@ auto—round-mllm \
--format "auto_round" \
--output_dir ./tmp_autoround
```
+
## API Usage (Gaudi2/CPU/GPU)
+
```python
from auto_round import AutoRoundMLLM
from transformers import Qwen2VLForConditionalGeneration, AutoProcessor, AutoTokenizer
@@ -24,12 +30,13 @@ tokenizer = AutoTokenizer.from_pretrained(model_name)
processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=trust_remote_code)
tokenizer.processor = processor
model = Qwen2VLForConditionalGeneration.from_pretrained(
- model_name, trust_remote_code=True)
+ model_name, trust_remote_code=True)
dataset = "/path/to/llava.json"
-extra_data_dir = "/path/to/images/dir"
+extra_data_dir = "/path/to/images/dir"
bits, group_size = 4, 128
-autoround = AutoRoundMLLM(model, tokenizer, bits=bits, group_size=group_size, dataset=dataset, extra_data_dir=extra_data_dir)
+autoround = AutoRoundMLLM(model, tokenizer, bits=bits, group_size=group_size, dataset=dataset,
+ extra_data_dir=extra_data_dir)
autoround.quantize()
output_dir = "./tmp_autoround"
@@ -37,45 +44,62 @@ autoround.save_quantized(output_dir, format='auto_round', inplace=True)
```
### Dataset
-For mllm, we used liuhaotian/llava_conv_58k as our default calib datasets. Through argument ```--dataset```, user can use other datasets such as "liuhaotian/llava_instruct_80k", "liuhaotian/llava_instruct_150k" or a file path to use local file.
+
+For mllm, we used liuhaotian/llava_conv_58k as our default calib datasets. Through argument ```--dataset```, user can
+use other datasets such as "liuhaotian/llava_instruct_80k", "liuhaotian/llava_instruct_150k" or a file path to use local
+file.
### Support Matrix
-So far, auto-round for mllm supports five model families, include Qwen2-VL, Llama-Vision, Phi3-Vision, Llava-v1.5 and CogVLM2.
-|Model |Eval Lib |calibration dataset|quant nontext module|
-|---------------|-----------|-------------------|--------------------|
-|Qwen2-VL |vlmeval |pile/llava |- |
-|Llama-Vision |lmms_eval |llava |✔ |
-|Phi3-Vision |vlmeval |pile/llava |✔ |
-|Llava-v1.5 |lmms_eval |pile/llava |- |
-|CogVLM2 |lmms_eval |pile/llava |✔ |
+So far, auto-round for mllm supports five model families, include Qwen2-VL, Llama-Vision, Phi3-Vision, Llava-v1.5 and
+CogVLM2.
+
+| Model | Eval Lib | calibration dataset | quant nontext module |
+|--------------|-----------|---------------------|----------------------|
+| Qwen2-VL | vlmeval | pile/llava | - |
+| Llama-Vision | lmms_eval | llava | ✔ |
+| Phi3-Vision | vlmeval | pile/llava | ✔ |
+| Llava-v1.5 | lmms_eval | pile/llava | - |
+| CogVLM2 | lmms_eval | pile/llava | ✔ |
## New Models Support
+
### Template
-For autoround MLLMs, using Template to customize different operations for different models. User can add a custom chat template through json file as below.
+
+For autoround MLLMs, using Template to customize different operations for different models. User can add a custom chat
+template through json file as below.
+
```json
{
- "model_type": "qwen2_vl",
- "format_user": "<|im_start|>user\n{{content}}<|im_end|>\n",
- "format_assistant": "<|im_start|>assistant\n{{content}}<|im_end|>\n",
- "format_system": "<|im_start|>system\n{{content}}<|im_end|>\n",
- "format_observation": "<|im_start|>tool\n{{content}}<|im_end|>\n<|im_start|>assistant\n",
- "format_separator": "\n",
- "default_system": "You are a helpful assistant.",
- "replace_tokens": ["<image>", "<|vision_start|><|image_pad|><|vision_end|>"],
- "extra_encode": "True",
- "processor": "qwen2_vl"
+ "model_type": "qwen2_vl",
+ "format_user": "<|im_start|>user\n{{content}}<|im_end|>\n",
+ "format_assistant": "<|im_start|>assistant\n{{content}}<|im_end|>\n",
+ "format_system": "<|im_start|>system\n{{content}}<|im_end|>\n",
+ "format_observation": "<|im_start|>tool\n{{content}}<|im_end|>\n<|im_start|>assistant\n",
+ "format_separator": "\n",
+ "default_system": "You are a helpful assistant.",
+ "replace_tokens": [
+ "<image>",
+ "<|vision_start|><|image_pad|><|vision_end|>"
+ ],
+ "extra_encode": "True",
+ "processor": "qwen2_vl"
}
```
-The special token ```{{content}}``` is a placeholder to tell the preprocessor where to fill in the corresponding dialogue content.
+
+The special token ```{{content}}``` is a placeholder to tell the preprocessor where to fill in the corresponding
+dialogue content.
```format_*```: Add specific token to chat content depends on different role names.
For example, the input conversations:<br>
- ```[{'role': 'user', 'value': '<image>\nWhat are the colors of the bus in the image?'}, {'role': 'assistant', 'value': 'The bus in the image is white and red.'}]```
+```[{'role': 'user', 'value': '<image>\nWhat are the colors of the bus in the image?'}, {'role': 'assistant', 'value': 'The bus in the image is white and red.'}]```
Using the above template, the input will be converted to the specified format required by Qwen2-vl as below: <br>
- ```'<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>\nWhat are the colors of the bus in the image?<|im_end|>\n<|im_start|>assistant\nThe bus in the image is white and red.<|im_end|>\n<|im_start|>user\nWhat feature can be seen on the back of the bus?<|im_end|>\n<|im_start|>assistant\nThe back of the bus features an advertisement.<|im_end|>\n<|im_start|>user\nIs the bus driving down the street or pulled off to the side?<|im_end|>\n<|im_start|>assistant\nThe bus is driving down the street, which is crowded with people and other vehicles.<|im_end|>\n'```.
+```'<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>\nWhat are the colors of the bus in the image?<|im_end|>\n<|im_start|>assistant\nThe bus in the image is white and red.<|im_end|>\n<|im_start|>user\nWhat feature can be seen on the back of the bus?<|im_end|>\n<|im_start|>assistant\nThe back of the bus features an advertisement.<|im_end|>\n<|im_start|>user\nIs the bus driving down the street or pulled off to the side?<|im_end|>\n<|im_start|>assistant\nThe bus is driving down the street, which is crowded with people and other vehicles.<|im_end|>\n'```.
### Processor
-Processor is callback interface for calling different processors, such as texts or images processors, for MLLMs. User can define own processor and use registration function to declare. For more information, please refer to the relevant code in ```auto_round/mllm/processor.py```.
\ No newline at end of file
+
+Processor is callback interface for calling different processors, such as texts or images processors, for MLLMs. User
+can define own processor and use registration function to declare. For more information, please refer to the relevant
+code in ```auto_round/mllm/processor.py```.
\ No newline at end of file
diff --git a/auto_round/mllm/autoround_mllm.py b/auto_round/mllm/autoround_mllm.py
index 1f23cf4a7..3c9d54b8b 100644
--- a/auto_round/mllm/autoround_mllm.py
+++ b/auto_round/mllm/autoround_mllm.py
@@ -30,12 +30,11 @@ from .mllm_dataset import get_mllm_dataloader
from ..low_cpu_mem.utils import get_layers_before_block
-
def _only_text_test(model, tokenizer):
"""Test if the model whether can use text-only datasets."""
try:
- text = ["only text", "test"]
- tokenizer.padding_side = 'left'
+ text = ["only text", "test"]
+ tokenizer.padding_side = 'left'
if tokenizer.pad_token is None:
tokenizer.pad_token = tokenizer.eos_token
inputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True).to(model.device)
@@ -97,7 +96,7 @@ class AutoRoundMLLM(AutoRound):
self,
model,
tokenizer,
- image_processor = None,
+ image_processor=None,
bits: int = 4,
group_size: int = 128,
sym: bool = False,
@@ -145,7 +144,7 @@ class AutoRoundMLLM(AutoRound):
self.template = template if template is not None else model.config.model_type
self.template = get_template(
self.template, model=model, tokenizer=tokenizer, image_processor=image_processor)
-
+
dataset = self.template.default_dataset if dataset is None else dataset
from ..calib_dataset import CALIB_DATASETS
if truncation is None:
@@ -162,14 +161,13 @@ class AutoRoundMLLM(AutoRound):
"will use liuhaotian/llava_conv_58k with default config as an alternative.")
else:
logger.warning(f"{model.config.model_type} not support for {dataset},"
- " will use liuhaotian/llava_conv_58k with default config as an alternative.")
+ " will use liuhaotian/llava_conv_58k with default config as an alternative.")
dataset = "liuhaotian/llava_conv_58k"
self.truncation = False
batch_size = 1
gradient_accumulate_steps = 4
seqlen = 512
-
super(AutoRoundMLLM, self).__init__(
model=model,
tokenizer=tokenizer,
@@ -209,10 +207,6 @@ class AutoRoundMLLM(AutoRound):
**kwargs,
)
-
-
-
-
def calib(self, nsamples, bs):
"""Perform calibration for quantization.
@@ -228,19 +222,19 @@ class AutoRoundMLLM(AutoRound):
if isinstance(self.dataset, str):
dataset = self.dataset.replace(" ", "")
self.dataloader, self.batch_size, self.gradient_accumulate_steps = get_mllm_dataloader(
- template=self.template,
- model=self.model,
- tokenizer=self.tokenizer,
- image_processor=self.image_processor,
- dataset=dataset,
- extra_data_dir=self.extra_data_dir,
- seqlen=self.seqlen,
- bs=self.batch_size,
- seed=self.seed,
- truncation=self.truncation,
- nsamples=self.nsamples,
- gradient_accumulate_steps=self.gradient_accumulate_steps,
- quant_nontext_module=self.quant_nontext_module
+ template=self.template,
+ model=self.model,
+ tokenizer=self.tokenizer,
+ image_processor=self.image_processor,
+ dataset=dataset,
+ extra_data_dir=self.extra_data_dir,
+ seqlen=self.seqlen,
+ bs=self.batch_size,
+ seed=self.seed,
+ truncation=self.truncation,
+ nsamples=self.nsamples,
+ gradient_accumulate_steps=self.gradient_accumulate_steps,
+ quant_nontext_module=self.quant_nontext_module
)
else:
self.dataloader = self.dataset
@@ -338,10 +332,8 @@ class AutoRoundMLLM(AutoRound):
if isinstance(v[key], list) and len(v[key]) == total_cnt:
self.inputs[k][key] = v[key][:max_len]
-
# clean embed weight to save memory
if self.low_cpu_mem_usage:
for n, m in embed_layers:
m = m.to("meta")
# torch.cuda.empty_cache()
-
diff --git a/auto_round/mllm/eval.py b/auto_round/mllm/eval.py
index 887286de0..2cb27d1d8 100644
--- a/auto_round/mllm/eval.py
+++ b/auto_round/mllm/eval.py
@@ -57,12 +57,11 @@ import numpy as np
vlmeval = LazyImport("vlmeval")
-
MODEL_TYPE_TO_VLMEVAL_MODEL = {
- #model_name
+ # model_name
"Qwen-VL": dict(cls="QwenVL"),
"Qwen-VL-Chat": dict(cls="QwenVLChat"),
- "Qwen2-VL": dict(cls="Qwen2VLChat", min_pixels=1280*28*28, max_pixels=16384*28*28, verbose=False),
+ "Qwen2-VL": dict(cls="Qwen2VLChat", min_pixels=1280 * 28 * 28, max_pixels=16384 * 28 * 28, verbose=False),
"Llama-3.2": dict(cls="llama_vision"),
"Phi-3-vision": dict(cls="Phi3Vision"),
"Phi-3.5-vision": dict(cls="Phi3_5Vision"),
@@ -75,7 +74,7 @@ MODEL_TYPE_TO_VLMEVAL_MODEL = {
"Molmo": dict(cls="molmo"),
# config.model_type
- "qwen2_vl": dict(cls="Qwen2VLChat", min_pixels=1280*28*28, max_pixels=16384*28*28),
+ "qwen2_vl": dict(cls="Qwen2VLChat", min_pixels=1280 * 28 * 28, max_pixels=16384 * 28 * 28),
"qwen": dict(cls="QwenVL"),
"qwen_chat": dict(cls="QwenVLChat"),
"llava": dict(cls="LLaVA"),
@@ -84,6 +83,7 @@ MODEL_TYPE_TO_VLMEVAL_MODEL = {
"mllama": dict(cls="llama_vision"),
}
+
def mllm_eval(
pretrained_model_name_or_path: str,
work_dir: str,
@@ -98,8 +98,7 @@ def mllm_eval(
verbose: bool = False,
mode: str = 'all',
ignore: bool = False
- ):
-
+):
try:
from auto_round import AutoRoundConfig
except:
@@ -158,7 +157,7 @@ def mllm_eval(
dataset_kwargs['pack'] = pack
if dataset_name == 'Video-MME':
dataset_kwargs['use_subtitle'] = use_subtitle
-
+
dataset = vlmeval.dataset.build_dataset(dataset_name, **dataset_kwargs)
if dataset is None:
logger.error(f'Dataset {dataset_name} is not valid, will be skipped. ')
@@ -229,7 +228,7 @@ def mllm_eval(
dataset=dataset,
verbose=verbose,
ignore_failed=ignore)
-
+
# Set the judge kwargs first before evaluation or dumping
judge_kwargs = {
'verbose': verbose,
@@ -240,16 +239,16 @@ def mllm_eval(
if dataset.TYPE in ['MCQ', 'Y/N'] or vlmeval.smp.listinstr(['MathVerse'], dataset_name):
judge_kwargs['model'] = 'chatgpt-0125'
elif vlmeval.smp.listinstr(['MMVet', 'MathVista', 'LLaVABench', 'MMBench-Video', 'MathVision'],
- dataset_name):
+ dataset_name):
judge_kwargs['model'] = 'gpt-4-turbo'
elif vlmeval.smp.listinstr(['MMLongBench', 'MMDU', 'DUDE', 'DUDE_MINI', 'SLIDEVQA', 'SLIDEVQA_MINI'],
- dataset_name):
+ dataset_name):
judge_kwargs['model'] = 'gpt-4o'
if 'OPENAI_API_KEY_JUDGE' in os.environ and len(os.environ['OPENAI_API_KEY_JUDGE']):
judge_kwargs['key'] = os.environ['OPENAI_API_KEY_JUDGE']
if 'OPENAI_API_BASE_JUDGE' in os.environ and len(os.environ['OPENAI_API_BASE_JUDGE']):
judge_kwargs['api_base'] = os.environ['OPENAI_API_BASE_JUDGE']
-
+
if dataset_name in ['MMMU_TEST']:
result_json = vlmeval.utils.result_transfer.MMMU_result_transfer(result_file)
logger.info(f'Transfer MMMU_TEST result to json for official evaluation, '
@@ -271,7 +270,7 @@ def mllm_eval(
elif dataset_name in ['DocVQA_TEST', 'InfoVQA_TEST', 'Q-Bench1_TEST', 'A-Bench_TEST']:
logger.info(f'{dataset_name} is a test split without ground-truth. '
'Thus only the inference part is supported for those datasets. ') # noqa: E501
-
+
if dataset_name in [
'MMBench_TEST_CN', 'MMBench_TEST_EN', 'MMBench', 'MMBench_CN',
'MMBench_TEST_CN_V11', 'MMBench_TEST_EN_V11', 'MMBench_V11', 'MMBench_CN_V11'
@@ -282,7 +281,7 @@ def mllm_eval(
'will skip the evaluation. '
)
continue
-
+
if mode == 'all':
eval_results = dataset.evaluate(result_file, **judge_kwargs)
if eval_results is not None:
@@ -299,12 +298,12 @@ def mllm_eval(
logger.info('\n' + tabulate.tabulate(eval_results))
except:
logger.info(eval_results.to_string())
- rt_file.write('%s cost: %.4fs\n' % (dataset_name, time.time() - task_st))
+ rt_file.write('%s cost: %.4fs\n' % (dataset_name, time.time() - task_st))
except Exception as e:
logger.exception(f'Model {model_name} x Dataset {dataset_name} combination failed: {e}, '
- 'skipping this combination.')
+ 'skipping this combination.')
continue
- rt_file.write('%d tasks cost: %.4fs\n' % (len(dataset), time.time() - st))
+ rt_file.write('%d tasks cost: %.4fs\n' % (len(dataset), time.time() - st))
rt_file.close()
@@ -328,6 +327,7 @@ MODEL_TYPE_TO_LMMS_MODEL = {
_lmms_eval = LazyImport("lmms_eval")
+
def _handle_non_serializable(o):
if isinstance(o, np.int64) or isinstance(o, np.int32):
return int(o)
@@ -336,10 +336,11 @@ def _handle_non_serializable(o):
else:
return str(o)
+
def lmms_eval(
model,
tasks,
- output_dir = None,
+ output_dir=None,
num_fewshot=None,
limit=None,
batch_size=1,
@@ -347,11 +348,11 @@ def lmms_eval(
device='cpu',
use_cache=None,
apply_chat_template=False
- ):
+):
from auto_round import AutoRoundConfig
if isinstance(tasks, str):
- tasks = tasks.replace(' ', '').split(',')
+ tasks = tasks.replace(' ', '').split(',')
model_name = model
if model_name[-1] == "/":
@@ -369,7 +370,7 @@ def lmms_eval(
from transformers import AutoConfig
config = AutoConfig.from_pretrained(model, trust_remote_code=True)
model_type = config.model_type
-
+
assert model_type in MODEL_TYPE_TO_LMMS_MODEL, f"{model_type} is not support by lmms."
if MODEL_TYPE_TO_LMMS_MODEL[model_type] == "phi3v":
@@ -378,6 +379,7 @@ def lmms_eval(
model_args = f"pretrained={model}"
if MODEL_TYPE_TO_LMMS_MODEL[model_type] == "llama_vision":
model_args += f",device_map={device}"
+
class CliArgs:
output_path = output_dir
@@ -399,7 +401,7 @@ def lmms_eval(
print(_lmms_eval.utils.make_table(results))
if output_dir:
os.makedirs(output_dir, exist_ok=True)
-
+
from datetime import datetime
now = datetime.now().strftime("%Y%m%d_%H%M%S")
output_file = os.path.join(output_dir, f"{model_name}_{now}_result.json")
diff --git a/auto_round/mllm/mllm_dataset.py b/auto_round/mllm/mllm_dataset.py
index 4d5e7f30b..890559348 100644
--- a/auto_round/mllm/mllm_dataset.py
+++ b/auto_round/mllm/mllm_dataset.py
@@ -16,7 +16,6 @@ import os
import json
from typing import Dict
-
import torch
from torch.utils.data import Dataset, DataLoader
from transformers import set_seed
@@ -26,8 +25,8 @@ from .template import Template
from ..utils import logger
from ..special_model_handler import check_mllm_model_batch
+MLLM_DATASET: Dict[str, Dataset] = {}
-MLLM_DATASET : Dict[str, Dataset] = {}
def register_dataset(name_list):
"""Class decorator to register a DATASET subclass to the registry.
@@ -44,8 +43,8 @@ def register_dataset(name_list):
def register(dataset):
for name in name_list.replace(' ', '').split(','):
MLLM_DATASET[name] = dataset
- return register
+ return register
@register_dataset(
@@ -73,7 +72,7 @@ class LlavaDataset(Dataset):
padding=True,
truncation=True,
nsamples=512
- ) -> None:
+ ) -> None:
super().__init__()
self.model = model
self.model_type = template.model_type
@@ -109,7 +108,6 @@ class LlavaDataset(Dataset):
image_fold = image_fold['image']
self.image_fold = image_fold
-
def check(self, questions, seqlen, nsamples):
def _check(questions, min_seqlen, max_seqlen, nsamples):
new_questions = []
@@ -129,19 +127,19 @@ class LlavaDataset(Dataset):
else:
if seqlen > max_len:
logger.warning(f"seqlen={seqlen} is greater than the max length of dataset {max_len},"
- f" will change seqlen to {max_len - 128}")
+ f" will change seqlen to {max_len - 128}")
new_min_seqlen = max_len - 128
else:
logger.warning(f"no enough sample for seqlen greater than {min_seqlen},"
- f" will decrease to {min_seqlen-128}")
+ f" will decrease to {min_seqlen - 128}")
new_min_seqlen = min_seqlen - 128
- return new_questions + _check(questions, new_min_seqlen, min_seqlen, nsamples-len(new_questions))
+ return new_questions + _check(questions, new_min_seqlen, min_seqlen, nsamples - len(new_questions))
+
return _check(questions, seqlen, float("inf"), nsamples)
-
def __len__(self):
return len(self.questions)
-
+
def __getitem__(self, i) -> Dict[str, torch.Tensor]:
if i in self.cached_data_dict:
return self.cached_data_dict[i]
@@ -164,17 +162,17 @@ class LlavaDataset(Dataset):
max_length = self.seqlen
truncation_strategy = "text"
ret = self.template.processor.get_input(
- text=text,
+ text=text,
images=image_path,
padding=self.padding,
truncation=self.truncation,
return_tensors="pt",
- max_length = max_length,
+ max_length=max_length,
truncation_strategy=truncation_strategy
- )
+ )
self.cached_data_dict[i] = ret
return ret
-
+
def covert_conversations(self, data):
new_data = []
for d in data:
@@ -192,12 +190,12 @@ class LlavaDataset(Dataset):
def get_mllm_dataloader(
template,
model,
- tokenizer,
+ tokenizer,
image_processor=None,
dataset="liuhaotian/llava_conv_58k",
extra_data_dir=None,
seqlen=512,
- bs=1,
+ bs=1,
split=None,
apply_template=None,
truncation=False,
@@ -226,16 +224,15 @@ def get_mllm_dataloader(
if isinstance(template, str):
from .template import get_template
template = get_template(template, model=model, tokenizer=tokenizer, image_processor=image_processor)
-
if os.path.isfile(dataset) or dataset in MLLM_DATASET.keys():
dataset = MLLM_DATASET['liuhaotian/llava'](
- template, model, tokenizer, dataset, extra_data_dir,
+ template, model, tokenizer, dataset, extra_data_dir,
seqlen=seqlen, truncation=truncation, nsamples=nsamples)
bs, gradient_accumulate_steps = check_mllm_model_batch(
model, batch_size=bs, gradient_accumulate_steps=gradient_accumulate_steps)
-
+
set_seed(seed)
dataloader_params = {
"batch_size": bs,
@@ -251,10 +248,7 @@ def get_mllm_dataloader(
tokenizer, seqlen, dataset, seed, bs, nsamples)
if quant_nontext_module:
logger.error(
- f"Quantitative nontext module is not supported for plain text datasets," \
+ f"Quantitative nontext module is not supported for plain text datasets," \
" please disable arg '--quant_nontext_module'")
exit(-1)
return dataloader, bs, gradient_accumulate_steps
-
-
-
diff --git a/auto_round/mllm/processor.py b/auto_round/mllm/processor.py
index 7de74d7f3..cd6518f30 100644
--- a/auto_round/mllm/processor.py
+++ b/auto_round/mllm/processor.py
@@ -19,24 +19,27 @@ from .utils import fetch_image
PROCESSORS = {}
+
def regist_processor(name):
def register(processor):
PROCESSORS[name] = processor
return processor
+
return register
+
@regist_processor("basic")
class BasicProcessor:
def __init__(self):
pass
-
+
def post_init(self, model, tokenizer, image_processor=None, **kwargs):
self.model = model
self.tokenizer = tokenizer
if image_processor is not None:
self.image_processor = image_processor
else:
- self.image_processor = self.default_image_processor
+ self.image_processor = self.default_image_processor
def get_input(
self,
@@ -48,7 +51,7 @@ class BasicProcessor:
truncation=False,
truncation_strategy="text",
**kwargs):
-
+
if isinstance(text, list):
for message in text:
if not ("role" in message and "content" in message):
@@ -60,7 +63,7 @@ class BasicProcessor:
try:
text = self.tokenizer.apply_chat_template(
text, tokenize=False, add_generation_prompt=not continue_final_message,
- continue_final_message=continue_final_message,)
+ continue_final_message=continue_final_message, )
except:
raise NotImplementedError("current not support for non-instructed model.")
@@ -69,7 +72,7 @@ class BasicProcessor:
if images is not None:
images = self.image_processor(images)
-
+
if truncation is True and truncation_strategy == "text":
text = self.tokenizer.decode(self.tokenizer(text).input_ids[:max_length])
@@ -84,10 +87,10 @@ class BasicProcessor:
for key in ret:
shape_ = ret[key].shape
if len(shape_) == 2 and shape_[-1] == seqlen:
- ret[key] = ret[key][:,:max_length]
+ ret[key] = ret[key][:, :max_length]
elif len(shape_) == 4 and shape_[1] == seqlen:
- ret[key] = ret[key][:,:max_length]
-
+ ret[key] = ret[key][:, :max_length]
+
if squeeze:
ret = self.squeeze_result(ret)
return ret
@@ -96,32 +99,34 @@ class BasicProcessor:
def data_collator(batch):
return default_data_collator(batch)
- @staticmethod
+ @staticmethod
def default_image_processor(image_path_or_url):
return fetch_image(image_path_or_url)
-
+
@staticmethod
def squeeze_result(ret):
for key in ret:
ret[key] = ret[key][0]
return ret
+
@regist_processor("qwen2_vl")
class Qwen2VLProcessor(BasicProcessor):
@staticmethod
def squeeze_result(ret):
for key in ret:
if key == "pixel_values":
- continue
+ continue
ret[key] = ret[key][0]
return ret
+
@regist_processor("cogvlm2")
class CogVLM2Processor(BasicProcessor):
def get_input(
self, text, images, truncation=False,
squeeze=True, **kwargs):
-
+
if images is not None:
images = self.image_processor(images)
@@ -130,12 +135,13 @@ class CogVLM2Processor(BasicProcessor):
max_length += padding_len
padding = False
input_data = self.model.build_conversation_input_ids(
- self.tokenizer,
- query=text,
- history=None,
- images=[images],
- template_version='base'
- )
+ self.tokenizer,
+ query=text,
+ history=None,
+ images=[images],
+ template_version='base'
+ )
+
def pad_to_len(unpadded_tensor, pad_to_length, pad_value=0):
current_length = len(unpadded_tensor)
if current_length >= pad_to_length:
@@ -145,13 +151,14 @@ class CogVLM2Processor(BasicProcessor):
return unpadded_tensor
if padding:
return torch.cat(
- (unpadded_tensor,
- torch.full([pad_to_length - current_length],
- fill_value=pad_value,
- dtype=unpadded_tensor.dtype,
- device=unpadded_tensor.device)), dim=0)
+ (unpadded_tensor,
+ torch.full([pad_to_length - current_length],
+ fill_value=pad_value,
+ dtype=unpadded_tensor.dtype,
+ device=unpadded_tensor.device)), dim=0)
else:
return unpadded_tensor
+
input_data['input_ids'] = pad_to_len(
input_data['input_ids'],
max_length,
@@ -167,14 +174,14 @@ class CogVLM2Processor(BasicProcessor):
max_length,
pad_value=0
)
- if input_data['labels']:
+ if input_data['labels']:
input_data['labels'] = pad_to_len(
input_data['labels'],
max_length,
pad_value=-100
)
return input_data
-
+
@staticmethod
def data_collator(batch):
batched_data = {}
@@ -186,15 +193,17 @@ class CogVLM2Processor(BasicProcessor):
# else:
# raise ValueError("Unsupported datatype in custom collate_fn")
return batched_data
-
+
@staticmethod
def default_image_processor(image_path_or_url):
return fetch_image(image_path_or_url).convert('RGB')
from ..utils import LazyImport
+
llava_train = LazyImport("llava.train.train")
+
@regist_processor("llava")
class LlavaProcessor(BasicProcessor):
def post_init(self, model, tokenizer, image_processor=None, **kwargs):
@@ -204,11 +213,10 @@ class LlavaProcessor(BasicProcessor):
self.image_processor = image_processor
self.collator_func = llava_train.DataCollatorForSupervisedDataset(tokenizer=self.tokenizer)
-
def get_input(
- self, text, images,max_length=None,
+ self, text, images, max_length=None,
squeeze=True, truncation=False, truncation_strategy="text", **kwargs):
-
+
if images is not None:
images = fetch_image(images).convert('RGB')
images = self.image_processor.preprocess(images, return_tensors='pt')['pixel_values'][0]
@@ -216,7 +224,7 @@ class LlavaProcessor(BasicProcessor):
class DataArgs:
is_multimodal = True
mm_use_im_start_end = False
-
+
if truncation is True and truncation_strategy == "text":
text = self.tokenizer.decode(self.tokenizer(text).input_ids[:max_length])
@@ -227,12 +235,12 @@ class LlavaProcessor(BasicProcessor):
seqlen = ret['input_ids'].shape[-1]
for key in ret:
if ret[key].shape[-1] == seqlen:
- ret[key] = ret[key][:,:max_length]
+ ret[key] = ret[key][:, :max_length]
if squeeze:
ret = self.squeeze_result(ret)
ret['image'] = images
return ret
-
+
def data_collator(self, batch):
- return self.collator_func(batch)
\ No newline at end of file
+ return self.collator_func(batch)
diff --git a/auto_round/mllm/template.py b/auto_round/mllm/template.py
index 16bb4e21f..99ad1fc46 100644
--- a/auto_round/mllm/template.py
+++ b/auto_round/mllm/template.py
@@ -23,6 +23,7 @@ from .processor import BasicProcessor, PROCESSORS
TEMPLATES: Dict[str, "Template"] = {}
+
def fill_content(target, **kwargs):
for name, value in kwargs.items():
target = target.replace("{{" + name + "}}", value, 1)
@@ -37,6 +38,7 @@ class Role(str, Enum):
FUNCTION = "function"
OBSERVATION = "observation"
+
@dataclass
class Template:
model_type: str
@@ -61,7 +63,7 @@ class Template:
element += fill_content(self.format_system, content=self.default_system)
# if i > 0 and i % 2 ==0:
# element += fill_content(self.format_separator)
-
+
if source['role'] == Role.USER.value:
element += fill_content(self.format_user, content=source["content"])
elif source['role'] == Role.ASSISTANT.value:
@@ -73,21 +75,21 @@ class Template:
return element
else:
return sources
-
+
def _register_template(
- model_type: str,
- format_user: Optional[str] = None,
- format_assistant: Optional[str] = None,
- format_system: Optional[str] = None,
- format_function: Optional[str] = None,
- format_observation: Optional[str] = None,
- format_separator: Optional[str] = None,
- default_system: str = "",
- replace_tokens: List[tuple] = None,
- extra_encode: Optional[bool] = False,
- default_dataset: Optional[bool] = "NeelNanda/pile-10k",
- processor: "BasicProcessor" = PROCESSORS["basic"],
+ model_type: str,
+ format_user: Optional[str] = None,
+ format_assistant: Optional[str] = None,
+ format_system: Optional[str] = None,
+ format_function: Optional[str] = None,
+ format_observation: Optional[str] = None,
+ format_separator: Optional[str] = None,
+ default_system: str = "",
+ replace_tokens: List[tuple] = None,
+ extra_encode: Optional[bool] = False,
+ default_dataset: Optional[bool] = "NeelNanda/pile-10k",
+ processor: "BasicProcessor" = PROCESSORS["basic"],
):
"""Registers a chat template."""
template_class = Template
@@ -98,18 +100,18 @@ def _register_template(
default_format_observation = ""
default_format_separator = "\n"
TEMPLATES[model_type] = template_class(
- model_type = model_type,
- format_user = format_user or default_format_user,
- format_assistant = format_assistant or default_format_assistant,
- format_system = format_system or default_format_system,
- format_function = format_function or default_format_function,
- format_observation = format_observation or default_format_observation,
- format_separator = format_separator or default_format_separator,
- default_system = default_system,
- replace_tokens = replace_tokens,
- extra_encode = extra_encode,
- default_dataset = default_dataset,
- processor = processor()
+ model_type=model_type,
+ format_user=format_user or default_format_user,
+ format_assistant=format_assistant or default_format_assistant,
+ format_system=format_system or default_format_system,
+ format_function=format_function or default_format_function,
+ format_observation=format_observation or default_format_observation,
+ format_separator=format_separator or default_format_separator,
+ default_system=default_system,
+ replace_tokens=replace_tokens,
+ extra_encode=extra_encode,
+ default_dataset=default_dataset,
+ processor=processor()
)
return TEMPLATES[model_type]
@@ -124,7 +126,7 @@ def load_template(path: str):
"the format of replace_tokens should be [old_tag1, replace_tag1, old_tag2, replace_tag2]"
temp = []
for i in range(0, len(data["replace_tokens"]), 2):
- temp.append((data["replace_tokens"][i], data["replace_tokens"][i+1]))
+ temp.append((data["replace_tokens"][i], data["replace_tokens"][i + 1]))
data["replace_tokens"] = temp
if "processor" in data:
assert data["processor"] in PROCESSORS.keys(), \
@@ -141,6 +143,7 @@ def _load_preset_template():
for file_name in os.listdir(dir_path):
load_template(os.path.join(dir_path, file_name))
+
_load_preset_template()
@@ -165,4 +168,4 @@ def get_template(template_or_path: str, model=None, tokenizer=None, image_proces
template.processor.post_init(model=model, tokenizer=tokenizer, image_processor=image_processor)
- return template
\ No newline at end of file
+ return template
diff --git a/auto_round/mllm/utils.py b/auto_round/mllm/utils.py
index 118ac1974..eebb2a189 100644
--- a/auto_round/mllm/utils.py
+++ b/auto_round/mllm/utils.py
@@ -44,4 +44,4 @@ def fetch_image(path_or_url):
else:
raise TypeError(f"{path_or_url} neither a path or url.")
- return image_obj
\ No newline at end of file
+ return image_obj
diff --git a/auto_round/script/llm.py b/auto_round/script/llm.py
index 8f721d46d..d06134e56 100644
--- a/auto_round/script/llm.py
+++ b/auto_round/script/llm.py
@@ -338,8 +338,10 @@ def tune(args):
try:
if args.model_dtype == "float16" or args.model_dtype == "fp16":
model = model.to(torch.float16)
- if args.model_dtype == "bfloat16" or args.model_dtype == "bfp16":
+ elif args.model_dtype == "bfloat16" or args.model_dtype == "bfp16" or args.model_dtype=="bf16":
model = model.to(torch.bfloat16)
+ elif args.model_dtype=="float32" or args.model_dtype=="fp32":
+ model = model.to(torch.float32)
except:
logger.error("please use more device to fit the device or just use one device")
exit()
diff --git a/auto_round/script/mllm.py b/auto_round/script/mllm.py
index d3fbb3995..a47c76056 100644
--- a/auto_round/script/mllm.py
+++ b/auto_round/script/mllm.py
@@ -137,7 +137,7 @@ class BasicArgumentParser(argparse.ArgumentParser):
help="whether to use the iter of best mes loss in the tuning phase")
self.add_argument("--enable_torch_compile", default=None, type=bool,
- help="whether to enable torch compile")
+ help="whether to enable torch compile")
## ======================= VLM =======================
self.add_argument("--quant_nontext_module", action='store_true',
@@ -155,8 +155,8 @@ class BasicArgumentParser(argparse.ArgumentParser):
self.add_argument("--truncation", action="store_true",
help="whether to truncate sequences at the maximum length."
- " Default True for pile and False for llava dataset.")
-
+ " Default True for pile and False for llava dataset.")
+
self.add_argument("--to_quant_block_names", default=None, type=str,
help="Names of quantitative blocks, please use commas to separate them.")
@@ -295,10 +295,16 @@ def tune(args):
seqlen = args.seqlen
if args.model_dtype != None:
- if args.model_dtype == "float16" or args.model_dtype == "fp16":
- model = model.to(torch.float16)
- if args.model_dtype == "bfloat16" or args.model_dtype == "bfp16":
- model = model.to(torch.bfloat16)
+ try:
+ if args.model_dtype == "float16" or args.model_dtype == "fp16":
+ model = model.to(torch.float16)
+ elif args.model_dtype == "bfloat16" or args.model_dtype == "bfp16" or args.model_dtype == "bf16":
+ model = model.to(torch.bfloat16)
+ elif args.model_dtype == "float32" or args.model_dtype == "fp32":
+ model = model.to(torch.float32)
+ except:
+ logger.error("please use more device to fit the device or just use one device")
+ exit()
round = AutoRoundMLLM
@@ -410,17 +416,18 @@ def eval(args):
ignore=args.ignore
)
+
def setup_lmms_parser():
parser = argparse.ArgumentParser()
parser.add_argument("--model", "--model_name", "--model_name_or_path",
- help="model name or path")
+ help="model name or path")
parser.add_argument(
"--tasks",
default="pope,textvqa_val,scienceqa,mmbench_en",
help="To get full list of tasks, use the command lmms-eval --tasks list",
)
parser.add_argument("--output_dir", default="./tmp_autoround", type=str,
- help="the directory to save quantized model")
+ help="the directory to save quantized model")
parser.add_argument(
"--num_fewshot",
type=int,
@@ -453,11 +460,12 @@ def setup_lmms_parser():
type=float,
default=None,
help="Limit the number of examples per task. " "If <1, limit is a percentage of the total"
- " number of examples.",
+ " number of examples.",
)
args = parser.parse_args()
return args
+
def lmms_eval(args):
from auto_round.mllm import lmms_eval
@@ -474,5 +482,3 @@ def lmms_eval(args):
apply_chat_template=False,
)
return results
-
-
diff --git a/examples/language-modeling/main.py b/examples/language-modeling/main.py
index d39283781..ac21e2cd7 100644
--- a/examples/language-modeling/main.py
+++ b/examples/language-modeling/main.py
@@ -267,11 +267,13 @@ if __name__ == '__main__':
try:
if args.model_dtype == "float16" or args.model_dtype == "fp16":
model = model.to(torch.float16)
- if args.model_dtype == "bfloat16" or args.model_dtype == "bfp16":
+ elif args.model_dtype == "bfloat16" or args.model_dtype == "bfp16" or args.model_dtype == "bf16":
model = model.to(torch.bfloat16)
+ elif args.model_dtype == "float32" or args.model_dtype == "fp32":
+ model = model.to(torch.float32)
except:
- print("please use more device,e.g `--devices 0,1,2,3` to fit the device or just use one device")
- exit()
+ print("please use more device to fit the device or just use one device")
+ exit(-1)
tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=not args.disable_trust_remote_code)