PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
fix model_dtype issue and reformat mllm code (#335)
Wenhua Chengfcaaac917c7d
diff --git a/auto_round/mllm/README.md b/auto_round/mllm/README.mdindex cd342f70d..50efca1e6 100644--- a/auto_round/mllm/README.md+++ b/auto_round/mllm/README.md@@ -1,6 +1,10 @@# AutoRound for MLLMs+## Basic Usage (Gaudi2/CPU/GPU)-A user guide detailing the full list of supported arguments is provided by calling ```auto-round-mllm -h``` on the terminal.Alternatively, you can use ```auto_round_mllm``` instead of ```auto-round-mllm```. Set the format you want in `format` and++A user guide detailing the full list of supported arguments is provided by calling ```auto-round-mllm -h``` on the+terminal.Alternatively, you can use ```auto_round_mllm``` instead of ```auto-round-mllm```. Set the format you want+in `format` andmultiple formats exporting has been supported.```bash@@ -14,7 +18,9 @@ auto—round-mllm \--format "auto_round" \--output_dir ./tmp_autoround```+## API Usage (Gaudi2/CPU/GPU)+```pythonfrom auto_round import AutoRoundMLLMfrom transformers import Qwen2VLForConditionalGeneration, AutoProcessor, AutoTokenizer@@ -24,12 +30,13 @@ tokenizer = AutoTokenizer.from_pretrained(model_name)processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=trust_remote_code)tokenizer.processor = processormodel = Qwen2VLForConditionalGeneration.from_pretrained(- model_name, trust_remote_code=True)+ model_name, trust_remote_code=True)dataset = "/path/to/llava.json"-extra_data_dir = "/path/to/images/dir"+extra_data_dir = "/path/to/images/dir"bits, group_size = 4, 128-autoround = AutoRoundMLLM(model, tokenizer, bits=bits, group_size=group_size, dataset=dataset, extra_data_dir=extra_data_dir)+autoround = AutoRoundMLLM(model, tokenizer, bits=bits, group_size=group_size, dataset=dataset,+ extra_data_dir=extra_data_dir)autoround.quantize()output_dir = "./tmp_autoround"@@ -37,45 +44,62 @@ autoround.save_quantized(output_dir, format='auto_round', inplace=True)```### Dataset-For mllm, we used liuhaotian/llava_conv_58k as our default calib datasets. Through argument ```--dataset```, user can use other datasets such as "liuhaotian/llava_instruct_80k", "liuhaotian/llava_instruct_150k" or a file path to use local file.++For mllm, we used liuhaotian/llava_conv_58k as our default calib datasets. Through argument ```--dataset```, user can+use other datasets such as "liuhaotian/llava_instruct_80k", "liuhaotian/llava_instruct_150k" or a file path to use local+file.### Support Matrix-So far, auto-round for mllm supports five model families, include Qwen2-VL, Llama-Vision, Phi3-Vision, Llava-v1.5 and CogVLM2.-|Model |Eval Lib |calibration dataset|quant nontext module|-|---------------|-----------|-------------------|--------------------|-|Qwen2-VL |vlmeval |pile/llava |- |-|Llama-Vision |lmms_eval |llava |✔ |-|Phi3-Vision |vlmeval |pile/llava |✔ |-|Llava-v1.5 |lmms_eval |pile/llava |- |-|CogVLM2 |lmms_eval |pile/llava |✔ |+So far, auto-round for mllm supports five model families, include Qwen2-VL, Llama-Vision, Phi3-Vision, Llava-v1.5 and+CogVLM2.++| Model | Eval Lib | calibration dataset | quant nontext module |+|--------------|-----------|---------------------|----------------------|+| Qwen2-VL | vlmeval | pile/llava | - |+| Llama-Vision | lmms_eval | llava | ✔ |+| Phi3-Vision | vlmeval | pile/llava | ✔ |+| Llava-v1.5 | lmms_eval | pile/llava | - |+| CogVLM2 | lmms_eval | pile/llava | ✔ |## New Models Support+### Template-For autoround MLLMs, using Template to customize different operations for different models. User can add a custom chat template through json file as below.++For autoround MLLMs, using Template to customize different operations for different models. User can add a custom chat+template through json file as below.+```json{- "model_type": "qwen2_vl",- "format_user": "<|im_start|>user\n{{content}}<|im_end|>\n",- "format_assistant": "<|im_start|>assistant\n{{content}}<|im_end|>\n",- "format_system": "<|im_start|>system\n{{content}}<|im_end|>\n",- "format_observation": "<|im_start|>tool\n{{content}}<|im_end|>\n<|im_start|>assistant\n",- "format_separator": "\n",- "default_system": "You are a helpful assistant.",- "replace_tokens": ["<image>", "<|vision_start|><|image_pad|><|vision_end|>"],- "extra_encode": "True",- "processor": "qwen2_vl"+ "model_type": "qwen2_vl",+ "format_user": "<|im_start|>user\n{{content}}<|im_end|>\n",+ "format_assistant": "<|im_start|>assistant\n{{content}}<|im_end|>\n",+ "format_system": "<|im_start|>system\n{{content}}<|im_end|>\n",+ "format_observation": "<|im_start|>tool\n{{content}}<|im_end|>\n<|im_start|>assistant\n",+ "format_separator": "\n",+ "default_system": "You are a helpful assistant.",+ "replace_tokens": [+ "<image>",+ "<|vision_start|><|image_pad|><|vision_end|>"+ ],+ "extra_encode": "True",+ "processor": "qwen2_vl"}```-The special token ```{{content}}``` is a placeholder to tell the preprocessor where to fill in the corresponding dialogue content.++The special token ```{{content}}``` is a placeholder to tell the preprocessor where to fill in the corresponding+dialogue content.```format_*```: Add specific token to chat content depends on different role names.For example, the input conversations:<br>- ```[{'role': 'user', 'value': '<image>\nWhat are the colors of the bus in the image?'}, {'role': 'assistant', 'value': 'The bus in the image is white and red.'}]```+```[{'role': 'user', 'value': '<image>\nWhat are the colors of the bus in the image?'}, {'role': 'assistant', 'value': 'The bus in the image is white and red.'}]```Using the above template, the input will be converted to the specified format required by Qwen2-vl as below: <br>- ```'<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>\nWhat are the colors of the bus in the image?<|im_end|>\n<|im_start|>assistant\nThe bus in the image is white and red.<|im_end|>\n<|im_start|>user\nWhat feature can be seen on the back of the bus?<|im_end|>\n<|im_start|>assistant\nThe back of the bus features an advertisement.<|im_end|>\n<|im_start|>user\nIs the bus driving down the street or pulled off to the side?<|im_end|>\n<|im_start|>assistant\nThe bus is driving down the street, which is crowded with people and other vehicles.<|im_end|>\n'```.+```'<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>\nWhat are the colors of the bus in the image?<|im_end|>\n<|im_start|>assistant\nThe bus in the image is white and red.<|im_end|>\n<|im_start|>user\nWhat feature can be seen on the back of the bus?<|im_end|>\n<|im_start|>assistant\nThe back of the bus features an advertisement.<|im_end|>\n<|im_start|>user\nIs the bus driving down the street or pulled off to the side?<|im_end|>\n<|im_start|>assistant\nThe bus is driving down the street, which is crowded with people and other vehicles.<|im_end|>\n'```.### Processor-Processor is callback interface for calling different processors, such as texts or images processors, for MLLMs. User can define own processor and use registration function to declare. For more information, please refer to the relevant code in ```auto_round/mllm/processor.py```.\ No newline at end of file++Processor is callback interface for calling different processors, such as texts or images processors, for MLLMs. User+can define own processor and use registration function to declare. For more information, please refer to the relevant+code in ```auto_round/mllm/processor.py```.\ No newline at end of filediff --git a/auto_round/mllm/autoround_mllm.py b/auto_round/mllm/autoround_mllm.pyindex 1f23cf4a7..3c9d54b8b 100644--- a/auto_round/mllm/autoround_mllm.py+++ b/auto_round/mllm/autoround_mllm.py@@ -30,12 +30,11 @@ from .mllm_dataset import get_mllm_dataloaderfrom ..low_cpu_mem.utils import get_layers_before_block-def _only_text_test(model, tokenizer):"""Test if the model whether can use text-only datasets."""try:- text = ["only text", "test"]- tokenizer.padding_side = 'left'+ text = ["only text", "test"]+ tokenizer.padding_side = 'left'if tokenizer.pad_token is None:tokenizer.pad_token = tokenizer.eos_tokeninputs = tokenizer(text, return_tensors="pt", padding=True, truncation=True).to(model.device)@@ -97,7 +96,7 @@ class AutoRoundMLLM(AutoRound):self,model,tokenizer,- image_processor = None,+ image_processor=None,bits: int = 4,group_size: int = 128,sym: bool = False,@@ -145,7 +144,7 @@ class AutoRoundMLLM(AutoRound):self.template = template if template is not None else model.config.model_typeself.template = get_template(self.template, model=model, tokenizer=tokenizer, image_processor=image_processor)-+dataset = self.template.default_dataset if dataset is None else datasetfrom ..calib_dataset import CALIB_DATASETSif truncation is None:@@ -162,14 +161,13 @@ class AutoRoundMLLM(AutoRound):"will use liuhaotian/llava_conv_58k with default config as an alternative.")else:logger.warning(f"{model.config.model_type} not support for {dataset},"- " will use liuhaotian/llava_conv_58k with default config as an alternative.")+ " will use liuhaotian/llava_conv_58k with default config as an alternative.")dataset = "liuhaotian/llava_conv_58k"self.truncation = Falsebatch_size = 1gradient_accumulate_steps = 4seqlen = 512-super(AutoRoundMLLM, self).__init__(model=model,tokenizer=tokenizer,@@ -209,10 +207,6 @@ class AutoRoundMLLM(AutoRound):**kwargs,)----def calib(self, nsamples, bs):"""Perform calibration for quantization.@@ -228,19 +222,19 @@ class AutoRoundMLLM(AutoRound):if isinstance(self.dataset, str):dataset = self.dataset.replace(" ", "")self.dataloader, self.batch_size, self.gradient_accumulate_steps = get_mllm_dataloader(- template=self.template,- model=self.model,- tokenizer=self.tokenizer,- image_processor=self.image_processor,- dataset=dataset,- extra_data_dir=self.extra_data_dir,- seqlen=self.seqlen,- bs=self.batch_size,- seed=self.seed,- truncation=self.truncation,- nsamples=self.nsamples,- gradient_accumulate_steps=self.gradient_accumulate_steps,- quant_nontext_module=self.quant_nontext_module+ template=self.template,+ model=self.model,+ tokenizer=self.tokenizer,+ image_processor=self.image_processor,+ dataset=dataset,+ extra_data_dir=self.extra_data_dir,+ seqlen=self.seqlen,+ bs=self.batch_size,+ seed=self.seed,+ truncation=self.truncation,+ nsamples=self.nsamples,+ gradient_accumulate_steps=self.gradient_accumulate_steps,+ quant_nontext_module=self.quant_nontext_module)else:self.dataloader = self.dataset@@ -338,10 +332,8 @@ class AutoRoundMLLM(AutoRound):if isinstance(v[key], list) and len(v[key]) == total_cnt:self.inputs[k][key] = v[key][:max_len]-# clean embed weight to save memoryif self.low_cpu_mem_usage:for n, m in embed_layers:m = m.to("meta")# torch.cuda.empty_cache()-diff --git a/auto_round/mllm/eval.py b/auto_round/mllm/eval.pyindex 887286de0..2cb27d1d8 100644--- a/auto_round/mllm/eval.py+++ b/auto_round/mllm/eval.py@@ -57,12 +57,11 @@ import numpy as npvlmeval = LazyImport("vlmeval")-MODEL_TYPE_TO_VLMEVAL_MODEL = {- #model_name+ # model_name"Qwen-VL": dict(cls="QwenVL"),"Qwen-VL-Chat": dict(cls="QwenVLChat"),- "Qwen2-VL": dict(cls="Qwen2VLChat", min_pixels=1280*28*28, max_pixels=16384*28*28, verbose=False),+ "Qwen2-VL": dict(cls="Qwen2VLChat", min_pixels=1280 * 28 * 28, max_pixels=16384 * 28 * 28, verbose=False),"Llama-3.2": dict(cls="llama_vision"),"Phi-3-vision": dict(cls="Phi3Vision"),"Phi-3.5-vision": dict(cls="Phi3_5Vision"),@@ -75,7 +74,7 @@ MODEL_TYPE_TO_VLMEVAL_MODEL = {"Molmo": dict(cls="molmo"),# config.model_type- "qwen2_vl": dict(cls="Qwen2VLChat", min_pixels=1280*28*28, max_pixels=16384*28*28),+ "qwen2_vl": dict(cls="Qwen2VLChat", min_pixels=1280 * 28 * 28, max_pixels=16384 * 28 * 28),"qwen": dict(cls="QwenVL"),"qwen_chat": dict(cls="QwenVLChat"),"llava": dict(cls="LLaVA"),@@ -84,6 +83,7 @@ MODEL_TYPE_TO_VLMEVAL_MODEL = {"mllama": dict(cls="llama_vision"),}+def mllm_eval(pretrained_model_name_or_path: str,work_dir: str,@@ -98,8 +98,7 @@ def mllm_eval(verbose: bool = False,mode: str = 'all',ignore: bool = False- ):-+):try:from auto_round import AutoRoundConfigexcept:@@ -158,7 +157,7 @@ def mllm_eval(dataset_kwargs['pack'] = packif dataset_name == 'Video-MME':dataset_kwargs['use_subtitle'] = use_subtitle-+dataset = vlmeval.dataset.build_dataset(dataset_name, **dataset_kwargs)if dataset is None:logger.error(f'Dataset {dataset_name} is not valid, will be skipped. ')@@ -229,7 +228,7 @@ def mllm_eval(dataset=dataset,verbose=verbose,ignore_failed=ignore)-+# Set the judge kwargs first before evaluation or dumpingjudge_kwargs = {'verbose': verbose,@@ -240,16 +239,16 @@ def mllm_eval(if dataset.TYPE in ['MCQ', 'Y/N'] or vlmeval.smp.listinstr(['MathVerse'], dataset_name):judge_kwargs['model'] = 'chatgpt-0125'elif vlmeval.smp.listinstr(['MMVet', 'MathVista', 'LLaVABench', 'MMBench-Video', 'MathVision'],- dataset_name):+ dataset_name):judge_kwargs['model'] = 'gpt-4-turbo'elif vlmeval.smp.listinstr(['MMLongBench', 'MMDU', 'DUDE', 'DUDE_MINI', 'SLIDEVQA', 'SLIDEVQA_MINI'],- dataset_name):+ dataset_name):judge_kwargs['model'] = 'gpt-4o'if 'OPENAI_API_KEY_JUDGE' in os.environ and len(os.environ['OPENAI_API_KEY_JUDGE']):judge_kwargs['key'] = os.environ['OPENAI_API_KEY_JUDGE']if 'OPENAI_API_BASE_JUDGE' in os.environ and len(os.environ['OPENAI_API_BASE_JUDGE']):judge_kwargs['api_base'] = os.environ['OPENAI_API_BASE_JUDGE']-+if dataset_name in ['MMMU_TEST']:result_json = vlmeval.utils.result_transfer.MMMU_result_transfer(result_file)logger.info(f'Transfer MMMU_TEST result to json for official evaluation, '@@ -271,7 +270,7 @@ def mllm_eval(elif dataset_name in ['DocVQA_TEST', 'InfoVQA_TEST', 'Q-Bench1_TEST', 'A-Bench_TEST']:logger.info(f'{dataset_name} is a test split without ground-truth. ''Thus only the inference part is supported for those datasets. ') # noqa: E501-+if dataset_name in ['MMBench_TEST_CN', 'MMBench_TEST_EN', 'MMBench', 'MMBench_CN','MMBench_TEST_CN_V11', 'MMBench_TEST_EN_V11', 'MMBench_V11', 'MMBench_CN_V11'@@ -282,7 +281,7 @@ def mllm_eval('will skip the evaluation. ')continue-+if mode == 'all':eval_results = dataset.evaluate(result_file, **judge_kwargs)if eval_results is not None:@@ -299,12 +298,12 @@ def mllm_eval(logger.info('\n' + tabulate.tabulate(eval_results))except:logger.info(eval_results.to_string())- rt_file.write('%s cost: %.4fs\n' % (dataset_name, time.time() - task_st))+ rt_file.write('%s cost: %.4fs\n' % (dataset_name, time.time() - task_st))except Exception as e:logger.exception(f'Model {model_name} x Dataset {dataset_name} combination failed: {e}, '- 'skipping this combination.')+ 'skipping this combination.')continue- rt_file.write('%d tasks cost: %.4fs\n' % (len(dataset), time.time() - st))+ rt_file.write('%d tasks cost: %.4fs\n' % (len(dataset), time.time() - st))rt_file.close()@@ -328,6 +327,7 @@ MODEL_TYPE_TO_LMMS_MODEL = {_lmms_eval = LazyImport("lmms_eval")+def _handle_non_serializable(o):if isinstance(o, np.int64) or isinstance(o, np.int32):return int(o)@@ -336,10 +336,11 @@ def _handle_non_serializable(o):else:return str(o)+def lmms_eval(model,tasks,- output_dir = None,+ output_dir=None,num_fewshot=None,limit=None,batch_size=1,@@ -347,11 +348,11 @@ def lmms_eval(device='cpu',use_cache=None,apply_chat_template=False- ):+):from auto_round import AutoRoundConfigif isinstance(tasks, str):- tasks = tasks.replace(' ', '').split(',')+ tasks = tasks.replace(' ', '').split(',')model_name = modelif model_name[-1] == "/":@@ -369,7 +370,7 @@ def lmms_eval(from transformers import AutoConfigconfig = AutoConfig.from_pretrained(model, trust_remote_code=True)model_type = config.model_type-+assert model_type in MODEL_TYPE_TO_LMMS_MODEL, f"{model_type} is not support by lmms."if MODEL_TYPE_TO_LMMS_MODEL[model_type] == "phi3v":@@ -378,6 +379,7 @@ def lmms_eval(model_args = f"pretrained={model}"if MODEL_TYPE_TO_LMMS_MODEL[model_type] == "llama_vision":model_args += f",device_map={device}"+class CliArgs:output_path = output_dir@@ -399,7 +401,7 @@ def lmms_eval(print(_lmms_eval.utils.make_table(results))if output_dir:os.makedirs(output_dir, exist_ok=True)-+from datetime import datetimenow = datetime.now().strftime("%Y%m%d_%H%M%S")output_file = os.path.join(output_dir, f"{model_name}_{now}_result.json")diff --git a/auto_round/mllm/mllm_dataset.py b/auto_round/mllm/mllm_dataset.pyindex 4d5e7f30b..890559348 100644--- a/auto_round/mllm/mllm_dataset.py+++ b/auto_round/mllm/mllm_dataset.py@@ -16,7 +16,6 @@ import osimport jsonfrom typing import Dict-import torchfrom torch.utils.data import Dataset, DataLoaderfrom transformers import set_seed@@ -26,8 +25,8 @@ from .template import Templatefrom ..utils import loggerfrom ..special_model_handler import check_mllm_model_batch+MLLM_DATASET: Dict[str, Dataset] = {}-MLLM_DATASET : Dict[str, Dataset] = {}def register_dataset(name_list):"""Class decorator to register a DATASET subclass to the registry.@@ -44,8 +43,8 @@ def register_dataset(name_list):def register(dataset):for name in name_list.replace(' ', '').split(','):MLLM_DATASET[name] = dataset- return register+ return register@register_dataset(@@ -73,7 +72,7 @@ class LlavaDataset(Dataset):padding=True,truncation=True,nsamples=512- ) -> None:+ ) -> None:super().__init__()self.model = modelself.model_type = template.model_type@@ -109,7 +108,6 @@ class LlavaDataset(Dataset):image_fold = image_fold['image']self.image_fold = image_fold-def check(self, questions, seqlen, nsamples):def _check(questions, min_seqlen, max_seqlen, nsamples):new_questions = []@@ -129,19 +127,19 @@ class LlavaDataset(Dataset):else:if seqlen > max_len:logger.warning(f"seqlen={seqlen} is greater than the max length of dataset {max_len},"- f" will change seqlen to {max_len - 128}")+ f" will change seqlen to {max_len - 128}")new_min_seqlen = max_len - 128else:logger.warning(f"no enough sample for seqlen greater than {min_seqlen},"- f" will decrease to {min_seqlen-128}")+ f" will decrease to {min_seqlen - 128}")new_min_seqlen = min_seqlen - 128- return new_questions + _check(questions, new_min_seqlen, min_seqlen, nsamples-len(new_questions))+ return new_questions + _check(questions, new_min_seqlen, min_seqlen, nsamples - len(new_questions))+return _check(questions, seqlen, float("inf"), nsamples)-def __len__(self):return len(self.questions)-+def __getitem__(self, i) -> Dict[str, torch.Tensor]:if i in self.cached_data_dict:return self.cached_data_dict[i]@@ -164,17 +162,17 @@ class LlavaDataset(Dataset):max_length = self.seqlentruncation_strategy = "text"ret = self.template.processor.get_input(- text=text,+ text=text,images=image_path,padding=self.padding,truncation=self.truncation,return_tensors="pt",- max_length = max_length,+ max_length=max_length,truncation_strategy=truncation_strategy- )+ )self.cached_data_dict[i] = retreturn ret-+def covert_conversations(self, data):new_data = []for d in data:@@ -192,12 +190,12 @@ class LlavaDataset(Dataset):def get_mllm_dataloader(template,model,- tokenizer,+ tokenizer,image_processor=None,dataset="liuhaotian/llava_conv_58k",extra_data_dir=None,seqlen=512,- bs=1,+ bs=1,split=None,apply_template=None,truncation=False,@@ -226,16 +224,15 @@ def get_mllm_dataloader(if isinstance(template, str):from .template import get_templatetemplate = get_template(template, model=model, tokenizer=tokenizer, image_processor=image_processor)-if os.path.isfile(dataset) or dataset in MLLM_DATASET.keys():dataset = MLLM_DATASET['liuhaotian/llava'](- template, model, tokenizer, dataset, extra_data_dir,+ template, model, tokenizer, dataset, extra_data_dir,seqlen=seqlen, truncation=truncation, nsamples=nsamples)bs, gradient_accumulate_steps = check_mllm_model_batch(model, batch_size=bs, gradient_accumulate_steps=gradient_accumulate_steps)-+set_seed(seed)dataloader_params = {"batch_size": bs,@@ -251,10 +248,7 @@ def get_mllm_dataloader(tokenizer, seqlen, dataset, seed, bs, nsamples)if quant_nontext_module:logger.error(- f"Quantitative nontext module is not supported for plain text datasets," \+ f"Quantitative nontext module is not supported for plain text datasets," \" please disable arg '--quant_nontext_module'")exit(-1)return dataloader, bs, gradient_accumulate_steps---diff --git a/auto_round/mllm/processor.py b/auto_round/mllm/processor.pyindex 7de74d7f3..cd6518f30 100644--- a/auto_round/mllm/processor.py+++ b/auto_round/mllm/processor.py@@ -19,24 +19,27 @@ from .utils import fetch_imagePROCESSORS = {}+def regist_processor(name):def register(processor):PROCESSORS[name] = processorreturn processor+return register+@regist_processor("basic")class BasicProcessor:def __init__(self):pass-+def post_init(self, model, tokenizer, image_processor=None, **kwargs):self.model = modelself.tokenizer = tokenizerif image_processor is not None:self.image_processor = image_processorelse:- self.image_processor = self.default_image_processor+ self.image_processor = self.default_image_processordef get_input(self,@@ -48,7 +51,7 @@ class BasicProcessor:truncation=False,truncation_strategy="text",**kwargs):-+if isinstance(text, list):for message in text:if not ("role" in message and "content" in message):@@ -60,7 +63,7 @@ class BasicProcessor:try:text = self.tokenizer.apply_chat_template(text, tokenize=False, add_generation_prompt=not continue_final_message,- continue_final_message=continue_final_message,)+ continue_final_message=continue_final_message, )except:raise NotImplementedError("current not support for non-instructed model.")@@ -69,7 +72,7 @@ class BasicProcessor:if images is not None:images = self.image_processor(images)-+if truncation is True and truncation_strategy == "text":text = self.tokenizer.decode(self.tokenizer(text).input_ids[:max_length])@@ -84,10 +87,10 @@ class BasicProcessor:for key in ret:shape_ = ret[key].shapeif len(shape_) == 2 and shape_[-1] == seqlen:- ret[key] = ret[key][:,:max_length]+ ret[key] = ret[key][:, :max_length]elif len(shape_) == 4 and shape_[1] == seqlen:- ret[key] = ret[key][:,:max_length]-+ ret[key] = ret[key][:, :max_length]+if squeeze:ret = self.squeeze_result(ret)return ret@@ -96,32 +99,34 @@ class BasicProcessor:def data_collator(batch):return default_data_collator(batch)- @staticmethod+ @staticmethoddef default_image_processor(image_path_or_url):return fetch_image(image_path_or_url)-+@staticmethoddef squeeze_result(ret):for key in ret:ret[key] = ret[key][0]return ret+@regist_processor("qwen2_vl")class Qwen2VLProcessor(BasicProcessor):@staticmethoddef squeeze_result(ret):for key in ret:if key == "pixel_values":- continue+ continueret[key] = ret[key][0]return ret+@regist_processor("cogvlm2")class CogVLM2Processor(BasicProcessor):def get_input(self, text, images, truncation=False,squeeze=True, **kwargs):-+if images is not None:images = self.image_processor(images)@@ -130,12 +135,13 @@ class CogVLM2Processor(BasicProcessor):max_length += padding_lenpadding = Falseinput_data = self.model.build_conversation_input_ids(- self.tokenizer,- query=text,- history=None,- images=[images],- template_version='base'- )+ self.tokenizer,+ query=text,+ history=None,+ images=[images],+ template_version='base'+ )+def pad_to_len(unpadded_tensor, pad_to_length, pad_value=0):current_length = len(unpadded_tensor)if current_length >= pad_to_length:@@ -145,13 +151,14 @@ class CogVLM2Processor(BasicProcessor):return unpadded_tensorif padding:return torch.cat(- (unpadded_tensor,- torch.full([pad_to_length - current_length],- fill_value=pad_value,- dtype=unpadded_tensor.dtype,- device=unpadded_tensor.device)), dim=0)+ (unpadded_tensor,+ torch.full([pad_to_length - current_length],+ fill_value=pad_value,+ dtype=unpadded_tensor.dtype,+ device=unpadded_tensor.device)), dim=0)else:return unpadded_tensor+input_data['input_ids'] = pad_to_len(input_data['input_ids'],max_length,@@ -167,14 +174,14 @@ class CogVLM2Processor(BasicProcessor):max_length,pad_value=0)- if input_data['labels']:+ if input_data['labels']:input_data['labels'] = pad_to_len(input_data['labels'],max_length,pad_value=-100)return input_data-+@staticmethoddef data_collator(batch):batched_data = {}@@ -186,15 +193,17 @@ class CogVLM2Processor(BasicProcessor):# else:# raise ValueError("Unsupported datatype in custom collate_fn")return batched_data-+@staticmethoddef default_image_processor(image_path_or_url):return fetch_image(image_path_or_url).convert('RGB')from ..utils import LazyImport+llava_train = LazyImport("llava.train.train")+@regist_processor("llava")class LlavaProcessor(BasicProcessor):def post_init(self, model, tokenizer, image_processor=None, **kwargs):@@ -204,11 +213,10 @@ class LlavaProcessor(BasicProcessor):self.image_processor = image_processorself.collator_func = llava_train.DataCollatorForSupervisedDataset(tokenizer=self.tokenizer)-def get_input(- self, text, images,max_length=None,+ self, text, images, max_length=None,squeeze=True, truncation=False, truncation_strategy="text", **kwargs):-+if images is not None:images = fetch_image(images).convert('RGB')images = self.image_processor.preprocess(images, return_tensors='pt')['pixel_values'][0]@@ -216,7 +224,7 @@ class LlavaProcessor(BasicProcessor):class DataArgs:is_multimodal = Truemm_use_im_start_end = False-+if truncation is True and truncation_strategy == "text":text = self.tokenizer.decode(self.tokenizer(text).input_ids[:max_length])@@ -227,12 +235,12 @@ class LlavaProcessor(BasicProcessor):seqlen = ret['input_ids'].shape[-1]for key in ret:if ret[key].shape[-1] == seqlen:- ret[key] = ret[key][:,:max_length]+ ret[key] = ret[key][:, :max_length]if squeeze:ret = self.squeeze_result(ret)ret['image'] = imagesreturn ret-+def data_collator(self, batch):- return self.collator_func(batch)\ No newline at end of file+ return self.collator_func(batch)diff --git a/auto_round/mllm/template.py b/auto_round/mllm/template.pyindex 16bb4e21f..99ad1fc46 100644--- a/auto_round/mllm/template.py+++ b/auto_round/mllm/template.py@@ -23,6 +23,7 @@ from .processor import BasicProcessor, PROCESSORSTEMPLATES: Dict[str, "Template"] = {}+def fill_content(target, **kwargs):for name, value in kwargs.items():target = target.replace("{{" + name + "}}", value, 1)@@ -37,6 +38,7 @@ class Role(str, Enum):FUNCTION = "function"OBSERVATION = "observation"+@dataclassclass Template:model_type: str@@ -61,7 +63,7 @@ class Template:element += fill_content(self.format_system, content=self.default_system)# if i > 0 and i % 2 ==0:# element += fill_content(self.format_separator)-+if source['role'] == Role.USER.value:element += fill_content(self.format_user, content=source["content"])elif source['role'] == Role.ASSISTANT.value:@@ -73,21 +75,21 @@ class Template:return elementelse:return sources-+def _register_template(- model_type: str,- format_user: Optional[str] = None,- format_assistant: Optional[str] = None,- format_system: Optional[str] = None,- format_function: Optional[str] = None,- format_observation: Optional[str] = None,- format_separator: Optional[str] = None,- default_system: str = "",- replace_tokens: List[tuple] = None,- extra_encode: Optional[bool] = False,- default_dataset: Optional[bool] = "NeelNanda/pile-10k",- processor: "BasicProcessor" = PROCESSORS["basic"],+ model_type: str,+ format_user: Optional[str] = None,+ format_assistant: Optional[str] = None,+ format_system: Optional[str] = None,+ format_function: Optional[str] = None,+ format_observation: Optional[str] = None,+ format_separator: Optional[str] = None,+ default_system: str = "",+ replace_tokens: List[tuple] = None,+ extra_encode: Optional[bool] = False,+ default_dataset: Optional[bool] = "NeelNanda/pile-10k",+ processor: "BasicProcessor" = PROCESSORS["basic"],):"""Registers a chat template."""template_class = Template@@ -98,18 +100,18 @@ def _register_template(default_format_observation = ""default_format_separator = "\n"TEMPLATES[model_type] = template_class(- model_type = model_type,- format_user = format_user or default_format_user,- format_assistant = format_assistant or default_format_assistant,- format_system = format_system or default_format_system,- format_function = format_function or default_format_function,- format_observation = format_observation or default_format_observation,- format_separator = format_separator or default_format_separator,- default_system = default_system,- replace_tokens = replace_tokens,- extra_encode = extra_encode,- default_dataset = default_dataset,- processor = processor()+ model_type=model_type,+ format_user=format_user or default_format_user,+ format_assistant=format_assistant or default_format_assistant,+ format_system=format_system or default_format_system,+ format_function=format_function or default_format_function,+ format_observation=format_observation or default_format_observation,+ format_separator=format_separator or default_format_separator,+ default_system=default_system,+ replace_tokens=replace_tokens,+ extra_encode=extra_encode,+ default_dataset=default_dataset,+ processor=processor())return TEMPLATES[model_type]@@ -124,7 +126,7 @@ def load_template(path: str):"the format of replace_tokens should be [old_tag1, replace_tag1, old_tag2, replace_tag2]"temp = []for i in range(0, len(data["replace_tokens"]), 2):- temp.append((data["replace_tokens"][i], data["replace_tokens"][i+1]))+ temp.append((data["replace_tokens"][i], data["replace_tokens"][i + 1]))data["replace_tokens"] = tempif "processor" in data:assert data["processor"] in PROCESSORS.keys(), \@@ -141,6 +143,7 @@ def _load_preset_template():for file_name in os.listdir(dir_path):load_template(os.path.join(dir_path, file_name))+_load_preset_template()@@ -165,4 +168,4 @@ def get_template(template_or_path: str, model=None, tokenizer=None, image_procestemplate.processor.post_init(model=model, tokenizer=tokenizer, image_processor=image_processor)- return template\ No newline at end of file+ return templatediff --git a/auto_round/mllm/utils.py b/auto_round/mllm/utils.pyindex 118ac1974..eebb2a189 100644--- a/auto_round/mllm/utils.py+++ b/auto_round/mllm/utils.py@@ -44,4 +44,4 @@ def fetch_image(path_or_url):else:raise TypeError(f"{path_or_url} neither a path or url.")- return image_obj\ No newline at end of file+ return image_objdiff --git a/auto_round/script/llm.py b/auto_round/script/llm.pyindex 8f721d46d..d06134e56 100644--- a/auto_round/script/llm.py+++ b/auto_round/script/llm.py@@ -338,8 +338,10 @@ def tune(args):try:if args.model_dtype == "float16" or args.model_dtype == "fp16":model = model.to(torch.float16)- if args.model_dtype == "bfloat16" or args.model_dtype == "bfp16":+ elif args.model_dtype == "bfloat16" or args.model_dtype == "bfp16" or args.model_dtype=="bf16":model = model.to(torch.bfloat16)+ elif args.model_dtype=="float32" or args.model_dtype=="fp32":+ model = model.to(torch.float32)except:logger.error("please use more device to fit the device or just use one device")exit()diff --git a/auto_round/script/mllm.py b/auto_round/script/mllm.pyindex d3fbb3995..a47c76056 100644--- a/auto_round/script/mllm.py+++ b/auto_round/script/mllm.py@@ -137,7 +137,7 @@ class BasicArgumentParser(argparse.ArgumentParser):help="whether to use the iter of best mes loss in the tuning phase")self.add_argument("--enable_torch_compile", default=None, type=bool,- help="whether to enable torch compile")+ help="whether to enable torch compile")## ======================= VLM =======================self.add_argument("--quant_nontext_module", action='store_true',@@ -155,8 +155,8 @@ class BasicArgumentParser(argparse.ArgumentParser):self.add_argument("--truncation", action="store_true",help="whether to truncate sequences at the maximum length."- " Default True for pile and False for llava dataset.")-+ " Default True for pile and False for llava dataset.")+self.add_argument("--to_quant_block_names", default=None, type=str,help="Names of quantitative blocks, please use commas to separate them.")@@ -295,10 +295,16 @@ def tune(args):seqlen = args.seqlenif args.model_dtype != None:- if args.model_dtype == "float16" or args.model_dtype == "fp16":- model = model.to(torch.float16)- if args.model_dtype == "bfloat16" or args.model_dtype == "bfp16":- model = model.to(torch.bfloat16)+ try:+ if args.model_dtype == "float16" or args.model_dtype == "fp16":+ model = model.to(torch.float16)+ elif args.model_dtype == "bfloat16" or args.model_dtype == "bfp16" or args.model_dtype == "bf16":+ model = model.to(torch.bfloat16)+ elif args.model_dtype == "float32" or args.model_dtype == "fp32":+ model = model.to(torch.float32)+ except:+ logger.error("please use more device to fit the device or just use one device")+ exit()round = AutoRoundMLLM@@ -410,17 +416,18 @@ def eval(args):ignore=args.ignore)+def setup_lmms_parser():parser = argparse.ArgumentParser()parser.add_argument("--model", "--model_name", "--model_name_or_path",- help="model name or path")+ help="model name or path")parser.add_argument("--tasks",default="pope,textvqa_val,scienceqa,mmbench_en",help="To get full list of tasks, use the command lmms-eval --tasks list",)parser.add_argument("--output_dir", default="./tmp_autoround", type=str,- help="the directory to save quantized model")+ help="the directory to save quantized model")parser.add_argument("--num_fewshot",type=int,@@ -453,11 +460,12 @@ def setup_lmms_parser():type=float,default=None,help="Limit the number of examples per task. " "If <1, limit is a percentage of the total"- " number of examples.",+ " number of examples.",)args = parser.parse_args()return args+def lmms_eval(args):from auto_round.mllm import lmms_eval@@ -474,5 +482,3 @@ def lmms_eval(args):apply_chat_template=False,)return results--diff --git a/examples/language-modeling/main.py b/examples/language-modeling/main.pyindex d39283781..ac21e2cd7 100644--- a/examples/language-modeling/main.py+++ b/examples/language-modeling/main.py@@ -267,11 +267,13 @@ if __name__ == '__main__':try:if args.model_dtype == "float16" or args.model_dtype == "fp16":model = model.to(torch.float16)- if args.model_dtype == "bfloat16" or args.model_dtype == "bfp16":+ elif args.model_dtype == "bfloat16" or args.model_dtype == "bfp16" or args.model_dtype == "bf16":model = model.to(torch.bfloat16)+ elif args.model_dtype == "float32" or args.model_dtype == "fp32":+ model = model.to(torch.float32)except:- print("please use more device,e.g `--devices 0,1,2,3` to fit the device or just use one device")- exit()+ print("please use more device to fit the device or just use one device")+ exit(-1)tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=not args.disable_trust_remote_code)