PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
support real lm-head quantization and mixed precision inference (#114)
wenhuach214d1caebba03c
diff --git a/README.md b/README.mdindex fc75033b1..9e67cbbc7 100644--- a/README.md+++ b/README.md@@ -151,35 +151,35 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))## Support List-| Model | Supported |-|--------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-| Intel/neural-chat-7b-v3-3 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc), [accuracy](./docs/neural-chat-7b-v3-3-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-3.sh), [example](./examples/language-modeling/) |-| Intel/neural-chat-7b-v3-1 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc), [accuracy](./docs/neural-chat-7b-v3-1-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-1.sh), [example](./examples/language-modeling/) |-| mistralai/Mistral-7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc), [accuracy](./docs/Mistral-7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-v0.1.sh), [example](./examples/language-modeling/) |-| microsoft/phi-2 | [HF-int4-model](https://huggingface.co/Intel/phi-2-int4-inc), [accuracy](./docs/phi-2-acc.md), [recipe](./examples/language-modeling/scripts/phi-2.sh), [example](./examples/language-modeling/)-| tiiuae/falcon-7b | [HF-int4-model](https://huggingface.co/Intel/falcon-7b-int4-inc), [accuracy](./docs/falcon-7b-acc.md), [recipe](./examples/language-modeling/scripts/falcon-7b.sh), [example](./examples/language-modeling/) |-| google/gemma-2b | [HF-int4-model](https://huggingface.co/Intel/gemma-2b-int4-inc), [accuracy](./docs/gemma-2b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-2b.sh), [example](./examples/language-modeling/)-| mistralai/Mistral-7B-Instruct-v0.2 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-Instruct-v0.2-int4-inc) (under review), [accuracy](./docs/Mistral-7B-Instruct-v0.2-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.sh), [example](./examples/language-modeling/) |-| google/gemma-7b | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-int4-inc) (under review), [accuracy](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh), [example](./examples/language-modeling/) |-| google/gemma-7b-it | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-it-int4-inc) (under review), [accuracy](./docs/gemma-7b-it-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b-it.sh), [example](./examples/language-modeling/) |-| mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) |-| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |-| meta-llama/Meta-Llama-3-8B-Instruct | [accuracy](./docs/Meta-Llama-3-8B-Instruct-acc.md), [recipe](./examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.sh), [example](./examples/language-modeling/) |-| meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) |-| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [sym recipe](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh), [asym recipe ](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh), [example](./examples/language-modeling/) |-| baichuan-inc/Baichuan2-7B-Chat | [accuracy](./docs/baichuan2-7b-chat-acc.md), [recipe](./examples/language-modeling/scripts/baichuan2-7b-chat.sh), [example](./examples/language-modeling/) |-| 01-ai/Yi-6B-Chat | [accuracy](./docs/Yi-6B-Chat-acc.md), [recipe](./examples/language-modeling/scripts/Yi-6B-Chat.sh), [example](./examples/language-modeling/) |-| facebook/opt-2.7b | [accuracy](./docs/opt-2.7b-acc.md), [recipe](./examples/language-modeling/scripts/opt-2.7b.sh), [example](./examples/language-modeling/) |-| bigscience/bloom-3b | [accuracy](./docs/bloom-3B-acc.md), [recipe](./examples/language-modeling/scripts/bloom-3b.sh), [example](./examples/language-modeling/) |-| EleutherAI/gpt-j-6b | [accuracy](./docs/gpt-j-6B-acc.md), [recipe](./examples/language-modeling/scripts/gpt-j-6b.sh), [example](./examples/language-modeling/) |-| Salesforce/codegen25-7b-multi | [example](./examples/language-modeling/) |-| huggyllama/llama-7b | [example](./examples/language-modeling/) |-| mosaicml/mpt-7b | [example](./examples/language-modeling/) |-| THUDM/chatglm3-6b | [example](./examples/language-modeling/) |-| MBZUAI/LaMini-GPT-124M | [example](./examples/language-modeling/) |-| EleutherAI/gpt-neo-125m | [example](./examples/language-modeling/) |-| databricks/dolly-v2-3b | [example](./examples/language-modeling/) |-| stabilityai/stablelm-base-alpha-3b | [example](./examples/language-modeling/)+| Model | Supported |+|--------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|+| Intel/neural-chat-7b-v3-3 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc), [accuracy](./docs/neural-chat-7b-v3-3-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-3.sh), [example](./examples/language-modeling/) |+| Intel/neural-chat-7b-v3-1 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc), [accuracy](./docs/neural-chat-7b-v3-1-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-1.sh), [example](./examples/language-modeling/) |+| mistralai/Mistral-7B-v0.1 | [HF-int4-model-lmhead](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc-lmhead),[HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc), [accuracy](./docs/Mistral-7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-v0.1.sh), [example](./examples/language-modeling/) |+| microsoft/phi-2 | [HF-int4-model](https://huggingface.co/Intel/phi-2-int4-inc), [accuracy](./docs/phi-2-acc.md), [recipe](./examples/language-modeling/scripts/phi-2.sh), [example](./examples/language-modeling/)+| tiiuae/falcon-7b | [HF-int4-model](https://huggingface.co/Intel/falcon-7b-int4-inc), [accuracy](./docs/falcon-7b-acc.md), [recipe](./examples/language-modeling/scripts/falcon-7b.sh), [example](./examples/language-modeling/) |+| google/gemma-2b | [HF-int4-model](https://huggingface.co/Intel/gemma-2b-int4-inc), [accuracy](./docs/gemma-2b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-2b.sh), [example](./examples/language-modeling/)+| mistralai/Mistral-7B-Instruct-v0.2 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-Instruct-v0.2-int4-inc) (under review), [accuracy](./docs/Mistral-7B-Instruct-v0.2-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.sh), [example](./examples/language-modeling/) |+| google/gemma-7b | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-int4-inc) (under review), [accuracy](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh), [example](./examples/language-modeling/) |+| google/gemma-7b-it | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-it-int4-inc) (under review), [accuracy](./docs/gemma-7b-it-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b-it.sh), [example](./examples/language-modeling/) |+| mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) |+| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |+| meta-llama/Meta-Llama-3-8B-Instruct | [accuracy](./docs/Meta-Llama-3-8B-Instruct-acc.md), [recipe](./examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.sh), [example](./examples/language-modeling/) |+| meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) |+| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [sym recipe](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh), [asym recipe ](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh), [example](./examples/language-modeling/) |+| baichuan-inc/Baichuan2-7B-Chat | [accuracy](./docs/baichuan2-7b-chat-acc.md), [recipe](./examples/language-modeling/scripts/baichuan2-7b-chat.sh), [example](./examples/language-modeling/) |+| 01-ai/Yi-6B-Chat | [accuracy](./docs/Yi-6B-Chat-acc.md), [recipe](./examples/language-modeling/scripts/Yi-6B-Chat.sh), [example](./examples/language-modeling/) |+| facebook/opt-2.7b | [accuracy](./docs/opt-2.7b-acc.md), [recipe](./examples/language-modeling/scripts/opt-2.7b.sh), [example](./examples/language-modeling/) |+| bigscience/bloom-3b | [accuracy](./docs/bloom-3B-acc.md), [recipe](./examples/language-modeling/scripts/bloom-3b.sh), [example](./examples/language-modeling/) |+| EleutherAI/gpt-j-6b | [accuracy](./docs/gpt-j-6B-acc.md), [recipe](./examples/language-modeling/scripts/gpt-j-6b.sh), [example](./examples/language-modeling/) |+| Salesforce/codegen25-7b-multi | [example](./examples/language-modeling/) |+| huggyllama/llama-7b | [example](./examples/language-modeling/) |+| mosaicml/mpt-7b | [example](./examples/language-modeling/) |+| THUDM/chatglm3-6b | [example](./examples/language-modeling/) |+| MBZUAI/LaMini-GPT-124M | [example](./examples/language-modeling/) |+| EleutherAI/gpt-neo-125m | [example](./examples/language-modeling/) |+| databricks/dolly-v2-3b | [example](./examples/language-modeling/) |+| stabilityai/stablelm-base-alpha-3b | [example](./examples/language-modeling/)## Comparison with other methodsdiff --git a/auto_round/autoround.py b/auto_round/autoround.pyindex 361e3e668..dcb08bfbb 100644--- a/auto_round/autoround.py+++ b/auto_round/autoround.py@@ -485,11 +485,39 @@ class AutoRound(object):self.share_attention_mask_flag = Noneself.hidden_dim_flag = Nonetorch.set_printoptions(precision=3, sci_mode=True)++ self.check_configs()+ serialization_keys = [+ "bits",+ "group_size",+ "sym",+ "data_type",+ "enable_quanted_input",+ "enable_minmax_tuning",+ "data_type",+ "seqlen",+ "train_bs",+ "scale_dtype",+ "lr",+ "minmax_lr",+ "gradient_accumulate_steps",+ "iters",+ "amp",+ ]+ if isinstance(dataset, str):+ serialization_keys.append("dataset")+ self.serialization_dict = {}+ for key in serialization_keys:+ self.serialization_dict[key] = getattr(self, key)+ from .version import __version__++ self.serialization_dict["autoround_version"] = __version__+ if "scale_dtype" in self.serialization_dict.keys():+ self.serialization_dict["scale_dtype"] = str(self.serialization_dict["scale_dtype"])if is_optimum_habana_available():logger.info("Optimum Habana is available, import htcore explicitly.")import habana_frameworks.torch.core as htcore # pylint: disable=E0401import habana_frameworks.torch.hpu as hthpu # pylint: disable=E0401- self.check_configs()def check_configs(self):"""Checks if the configurations are valid.@@ -815,6 +843,7 @@ class AutoRound(object):block_names, n_samples, layer_names=layer_names, last_cache_name=last_cache_name)self.model = self.model.to("cpu")+ torch.cuda.empty_cache()except:logger.info("switch to cpu to cache inputs")self.model = self.model.to("cpu")@@ -1014,10 +1043,7 @@ class AutoRound(object):lr_schedule = copy.deepcopy(self.lr_scheduler)train_bs = self.train_bs- pick_samples = train_bsn_samples = len(inputs)- if self.sampler != "rand":- indices = torch.randperm(n_samples)[:pick_samples]last_best_iter = 0best_loss = torch.finfo(torch.float).maxmse_loss = torch.nn.MSELoss().to(device)@@ -1025,6 +1051,11 @@ class AutoRound(object):init_loss = Nonebest_v, best_min_scale, best_max_scale = torch.tensor(0), torch.tensor(0), torch.tensor(0)gradient_accumulate_steps = self.gradient_accumulate_steps+ gradient_accumulate_steps = self.train_bs ##Force to low gpu+ train_bs = 1 ##Force to low gpu+ pick_samples = train_bs+ if self.sampler != "rand":+ indices = torch.randperm(n_samples)[:pick_samples]for i in range(self.iters):total_loss = 0for _ in range(gradient_accumulate_steps):@@ -1320,7 +1351,7 @@ class AutoRound(object):logger.error(f"export format only supports {EXPORT_FORMAT.keys()}")exit()save_quantized_as_format = EXPORT_FORMAT.get(format)- compressed_model = save_quantized_as_format(+ compressed_model = save_quantized_as_format( ##TODO refine the codeoutput_dir,model=self.model,weight_config=self.weight_config,@@ -1336,6 +1367,8 @@ class AutoRound(object):scale_dtype=self.scale_dtype,tokenizer=self.tokenizer,supported_types=self.supported_types,+ data_type=self.data_type,+ serialization_dict=self.serialization_dict,**kwargs,)return compressed_modeldiff --git a/auto_round/export/__init__.py b/auto_round/export/__init__.pyindex 9b4384ff9..5a48d41f9 100644--- a/auto_round/export/__init__.py+++ b/auto_round/export/__init__.py@@ -15,3 +15,5 @@from .register import EXPORT_FORMATfrom .export_to_autogptq import save_quantized_as_autogptqfrom .export_to_itrex import save_quantized_as_itrex, QuantConfig+from .export_to_autoround.export_to_autoround import save_quantized_as_autoround+from .export_to_autoround import AutoHfQuantizerdiff --git a/auto_round/export/export_to_autoround/__init__.py b/auto_round/export/export_to_autoround/__init__.pynew file mode 100644index 000000000..be98f7b1b--- /dev/null+++ b/auto_round/export/export_to_autoround/__init__.py@@ -0,0 +1,8 @@+import transformers+from .autoround_quantizer import AutoHfQuantizer++transformers.quantizers.auto.AutoHfQuantizer = AutoHfQuantizer+transformers.quantizers.auto.AutoQuantizationConfig = AutoHfQuantizer+transformers.modeling_utils.AutoHfQuantizer = AutoHfQuantizer+from transformers import AutoModelForCausalLM as AutoRoundModelForCausalLM+from .export_to_autoround import save_quantized_as_autorounddiff --git a/auto_round/export/export_to_autoround/autoround_quantizer.py b/auto_round/export/export_to_autoround/autoround_quantizer.pynew file mode 100644index 000000000..9a2651042--- /dev/null+++ b/auto_round/export/export_to_autoround/autoround_quantizer.py@@ -0,0 +1,427 @@+# Copyright (c) 2024 Intel Corporation+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.++# coding=utf-8+# Copyright 2023 HuggingFace Inc. team and GPTQ and AutoGPTQ authors.+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+import importlib.util+import warnings+from dataclasses import dataclass+from logging import getLogger+from typing import Any, Dict, List, Optional, Tuple, Union++import torch+import torch.nn as nn+from packaging import version+from transformers.modeling_utils import PreTrainedModel+from transformers.pytorch_utils import Conv1D+from transformers.quantizers import AutoQuantizationConfig, HfQuantizer+from transformers.quantizers.auto import AUTO_QUANTIZER_MAPPING+from transformers.utils.quantization_config import AwqConfig, GPTQConfig, QuantizationConfigMixin, QuantizationMethod++from auto_round.utils import get_module, set_module++logger = getLogger(__name__)+import sys++if sys.version_info < (3, 8):+ import importlib_metadata+else:+ import importlib.metadata as importlib_metadata++AUTOGPTQ_MINIMUM_VERSION = version.parse("0.4.99") # Allows 0.5.0.dev0+++def _is_package_available(pkg_name: str, return_version: bool = False) -> Union[Tuple[bool, str], bool]:+ # Check we're not importing a "pkg_name" directory somewhere but the actual library by trying to grab the version+ package_exists = importlib.util.find_spec(pkg_name) is not None+ package_version = "N/A"+ if package_exists:+ try:+ package_version = importlib.metadata.version(pkg_name)+ package_exists = True+ except importlib.metadata.PackageNotFoundError:+ package_exists = False+ if return_version:+ return package_exists, package_version+ else:+ return package_exists+++_auto_gptq_available = _is_package_available("auto_gptq")+++def is_auto_gptq_available():+ if _auto_gptq_available:+ version_autogptq = version.parse(importlib_metadata.version("auto_gptq"))+ if AUTOGPTQ_MINIMUM_VERSION < version_autogptq:+ return True+ else:+ raise ImportError(+ f"Found an incompatible version of auto-gptq. Found version {version_autogptq},"+ f" but only version above {AUTOGPTQ_MINIMUM_VERSION} are supported"+ )+++if is_auto_gptq_available():+ from auto_gptq import exllama_set_max_input_length+ from auto_gptq.modeling._utils import autogptq_post_init+ from auto_gptq.quantization import GPTQ+ from auto_gptq.utils.import_utils import dynamically_import_QuantLinear+++#+def get_device(obj: Union[torch.Tensor, nn.Module]):+ if isinstance(obj, torch.Tensor):+ return obj.device+ return next(obj.parameters()).device+++class AutoHfQuantizer:+ """The Auto-HF quantizer class that takes care of automatically instantiating to the correct+ `HfQuantizer` given the `QuantizationConfig`."""++ @classmethod+ def from_config(cls, quantization_config: Union[QuantizationConfigMixin, Dict], **kwargs):+ # Convert it to a QuantizationConfig if the q_config is a dict+ if isinstance(quantization_config, dict):+ if "auto-round" in quantization_config["quant_method"]:+ quantization_config = AutoRoundConfig.from_dict(quantization_config)+ else:+ quantization_config = AutoQuantizationConfig.from_dict(quantization_config) # pylint: disable=E1101+ quant_method = quantization_config.quant_method++ # Again, we need a special care for bnb as we have a single quantization config+ # class for both 4-bit and 8-bit quantization+ if quant_method == QuantizationMethod.BITS_AND_BYTES:+ if quantization_config.load_in_8bit:+ quant_method += "_8bit"+ else:+ quant_method += "_4bit"++ if quant_method not in AUTO_QUANTIZER_MAPPING.keys() and "auto-round" not in quant_method:+ raise ValueError(+ f"Unknown quantization type, got {quant_method} - supported types are:"+ f" {list(AUTO_QUANTIZER_MAPPING.keys())}"+ )+ if "auto-round" in quant_method:+ target_cls = AutoRoundQuantizer+ else:+ target_cls = AUTO_QUANTIZER_MAPPING[quant_method]++ return target_cls(quantization_config, **kwargs)++ @classmethod+ def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):+ quantization_config = AutoQuantizationConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)+ return cls.from_config(quantization_config)++ @classmethod+ def merge_quantization_configs(+ cls,+ quantization_config: Union[dict, QuantizationConfigMixin],+ quantization_config_from_args: Optional[QuantizationConfigMixin],+ ):+ """Handles situations where both quantization_config+ from args and quantization_config from model config are present."""+ if quantization_config_from_args is not None:+ warning_msg = (+ "You passed `quantization_config` or equivalent parameters to "+ "`from_pretrained` but the model you're loading"+ " already has a `quantization_config` attribute. The `quantization_config` from the model will be used."+ )+ else:+ warning_msg = ""++ if isinstance(quantization_config, dict):+ quantization_config = AutoRoundConfig.from_dict(quantization_config)++ if isinstance(quantization_config, (GPTQConfig, AwqConfig)) and quantization_config_from_args is not None:+ # special case for GPTQ / AWQ config collision+ loading_attr_dict = quantization_config_from_args.get_loading_attributes()+ for attr, val in loading_attr_dict.items():+ setattr(quantization_config, attr, val)+ warning_msg += (+ f"However, loading attributes (e.g. {list(loading_attr_dict.keys())}) "+ f"will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored."+ )++ if warning_msg != "":+ warnings.warn(warning_msg)++ return quantization_config+++@dataclass+class AutoRoundConfig(QuantizationConfigMixin):+ """This is a wrapper class about all possible attributes and features that you can play with a model that has been+ loaded using `optimum` api for gptq quantization relying on auto_gptq backend.++ Args:+ bits (`int`):+ The number of bits to quantize to, supported numbers are (2, 3, 4, 8).+ tokenizer (`str` or `PreTrainedTokenizerBase`, *optional*):+ The tokenizer used to process the dataset. You can pass either:+ - A custom tokenizer object.+ - A string, the *model id* of a predefined tokenizer hosted inside a model repo on huggingface.co.+ - A path to a *directory* containing vocabulary files required by the tokenizer, for instance saved+ using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`.+ """++ def __init__(+ self,+ bits: int,+ tokenizer: Any = None,+ dataset: str = None,+ group_size: int = 128,+ sym: bool = False,+ backend="gptq:exllamav2",+ iters: int = 200,+ weight_config: dict = None,+ enable_quanted_input=True,+ enable_minmax_tuning=True,+ lr=None,+ minmax_lr=None,+ n_samples=512,+ seqlen=2048,+ **kwargs,+ ):+ self.bits = bits+ self.tokenizer = tokenizer+ self.dataset = dataset+ self.group_size = group_size+ self.sym = sym+ self.backend = backend+ self.inters = iters+ self.weight_config = weight_config+ self.enable_quanted_input = enable_quanted_input+ self.enable_minmax_tuning = enable_minmax_tuning+ self.lr = lr+ self.minmax_lr = minmax_lr+ self.n_samples = n_samples+ self.seqlen = seqlen+ if kwargs is not None:+ for key in kwargs.keys():+ setattr(self, key, kwargs[key])++ self.post_init()++ def get_loading_attributes(self):+ pass+ # attibutes_dict = copy.deepcopy(self.__dict__)+ # loading_attibutes = ["disable_exllama", "use_exllama", "exllama_config", "use_cuda_fp16", "max_input_length"]+ # loading_attibutes_dict = {i: j for i, j in attibutes_dict.items() if i in loading_attibutes}+ # return loading_attibutes_dict++ def post_init(self):+ r"""Safety checker that arguments are correct."""+ if self.bits not in [2, 3, 4, 8]:+ raise ValueError(f"Only support quantization to [2,3,4,8] bits but found {self.bits}")+ if self.group_size != -1 and self.group_size <= 0:+ raise ValueError("group_size must be greater than 0 or equal to -1")+ ##TODO add more check++ def to_dict(self):+ config_dict = super().to_dict()+ config_dict.pop("disable_exllama", None)+ return config_dict+++class AutoRoundQuantizer(HfQuantizer):+ """+ Quantizer of the GPTQ method - for GPTQ the quantizer support calibration of the model through+ `auto_gptq` package. Quantization is done under the hood for users if they load a non-prequantized model.+ """++ requires_calibration = False+ required_packages = ["auto_gptq"]+ optimum_quantizer = None++ def __init__(self, quantization_config: QuantizationConfigMixin, **kwargs):+ super().__init__(quantization_config, **kwargs)++ def validate_environment(self, *args, **kwargs):+ gptq_supports_cpu = version.parse(importlib.metadata.version("auto-gptq")) > version.parse("0.4.2")+ if not gptq_supports_cpu and not torch.cuda.is_available():+ raise RuntimeError("GPU is required to quantize or run quantize model.")+ elif not is_auto_gptq_available():+ raise ImportError("Loading a GPTQ quantized model requires auto-gptq library (`pip install auto-gptq`)")+ elif version.parse(importlib.metadata.version("auto_gptq")) < version.parse("0.4.2"):+ raise ImportError("You need a version of auto_gptq >= 0.4.2 to use GPTQ: `pip install --upgrade auto-gptq`")++ def update_torch_dtype(self, torch_dtype: "torch.dtype") -> "torch.dtype":+ if torch_dtype is None:+ torch_dtype = torch.float16+ elif torch_dtype != torch.float16:+ logger.info("We suggest you to set `torch_dtype=torch.float16` for better efficiency with AutoRound.")+ return torch_dtype++ def convert_model(self, model: nn.Module):+ """Convert the model to a GPTQ model by getting and replacing the layers.++ Args:+ model (`nn.Module`):+ Model to be converted+ """+ from .export_to_autoround import get_layer_names_in_block++ layer_names = get_layer_names_in_block(model)+ quantization_config = model.config.quantization_config+ bits = quantization_config.bits+ group_size = quantization_config.group_size+ data_type = quantization_config.data_type+ sym = quantization_config.sym+ extra_config = {}+ if hasattr(quantization_config, "extra_config"):+ extra_config = quantization_config.extra_config+ layer_names += extra_config.keys()+ layer_names = list(set(layer_names))+ layer_configs = {}+ for layer_name in layer_names:+ layer_configs[layer_name] = {}+ if layer_name not in extra_config:+ layer_configs[layer_name]["bits"] = bits+ layer_configs[layer_name]["group_size"] = group_size+ layer_configs[layer_name]["data_type"] = data_type+ layer_configs[layer_name]["sym"] = sym+ else:+ layer_configs[layer_name]["bits"] = extra_config.get("bits", bits)+ layer_configs[layer_name]["group_size"] = extra_config.get("group_size", group_size)+ layer_configs[layer_name]["data_type"] = extra_config.get("data_type", data_type)+ layer_configs[layer_name]["sym"] = extra_config.get("sym", sym)+ backend = quantization_config.backend++ self._replace_by_quant_layers(model, layer_configs, backend)+ return model++ def _replace_by_quant_layers(self, module: nn.Module, layer_configs, backend):+ """Replaces linear layers in `module` by `QuantLinear`++ Args:+ module (`nn.Module`):+ Module to quantize+ names (`List[str]`):+ List of names of the module to quantize+ name (`str`, defaults to `""`):+ To keep track of the name of the current module+ """+ for layer_name in layer_configs.keys():+ config = layer_configs[layer_name]+ bits = config["bits"]+ group_size = config["group_size"]+ data_type = config["data_type"]+ if not (bits <= 8 and data_type == "int"):+ continue+ from .export_to_autoround import get_autogptq_backend_config++ use_triton, disable_exllama, disable_exllamav2, use_qigen, disable_marlin = get_autogptq_backend_config(+ backend, bits+ )+ QuantLinear = dynamically_import_QuantLinear(+ use_triton=False,+ desc_act=False,+ group_size=group_size,+ bits=bits,+ disable_exllama=True,+ disable_exllamav2=False,+ use_qigen=use_qigen,+ disable_marlin=disable_marlin,+ )+ layer = get_module(module, layer_name)+ device = get_device(layer)+ if isinstance(layer, nn.Linear):+ in_features = layer.in_features+ out_features = layer.out_features+ elif isinstance(layer, nn.Conv2d): ##not supported now+ in_features = layer.in_channels+ out_features = layer.out_channels+ elif isinstance(layer, Conv1D): ##TODO need to have a check+ in_features = layer.weight.shape[0]+ out_features = layer.weight.shape[1]+ bias = layer.bias is not None+ new_layer = QuantLinear(+ bits,+ group_size,+ in_features,+ out_features,+ bias,+ weight_dtype=layer.weight.dtype, # pylint: disable=E1123+ )++ new_layer.device = device+ set_module(module, layer_name, new_layer)++ def post_init_model(self, model):+ """Post-initialization that require device information, for example buffers initialization on device.++ Args:+ model (`nn.Module`):+ The input model+ """++ # if self.bits == 4 and not self.disable_exllama:+ # if get_device(model) == torch.device("cpu") or (+ # hasattr(model, "hf_device_map") and any(d in model.hf_device_map for d in ["cpu", "disk"])+ # ):+ # raise ValueError(+ # "Found modules on cpu/disk. Using Exllama+ # or Exllamav2 backend requires all the modules to be on GPU."+ # "You can deactivate exllama backend by+ # setting `disable_exllama=True` in the quantization config object"+ # )++ class StoreAttr(object):+ pass++ model.quantize_config = StoreAttr()+ model.quantize_config.desc_act = False+ model = autogptq_post_init(model, use_act_order=False)+ return model++ def _process_model_before_weight_loading(self, model: "PreTrainedModel", **kwargs):+ if model.__class__.main_input_name != "input_ids":+ raise RuntimeError("We can only quantize pure text model.")++ if self.pre_quantized:+ model = self.convert_model(model)++ def _process_model_after_weight_loading(self, model: "PreTrainedModel", **kwargs):+ if self.pre_quantized:+ model = self.post_init_model(model)+ else:+ raise NotImplementedError+ # if self.quantization_config.tokenizer is None:+ # self.quantization_config.tokenizer = model.name_or_path+ #+ # self.optimum_quantizer.quantize_model(model, self.quantization_config.tokenizer)+ # model.config.quantization_config = GPTQConfig.from_dict(self.optimum_quantizer.to_dict())++ @property+ def is_trainable(self, model: Optional["PreTrainedModel"] = None):+ return True++ @property+ def is_serializable(self):+ return Truediff --git a/auto_round/export/export_to_autoround/export_to_autoround.py b/auto_round/export/export_to_autoround/export_to_autoround.pynew file mode 100644index 000000000..fab55a26c--- /dev/null+++ b/auto_round/export/export_to_autoround/export_to_autoround.py@@ -0,0 +1,217 @@+# Copyright (c) 2024 Intel Corporation+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0+#+# Unless required by applicable law or agreed to in writing, software+# distributed under the License is distributed on an "AS IS" BASIS,+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.+# See the License for the specific language governing permissions and+# limitations under the License.+++import copy+import json+import os++import torch+import torch.nn as nn+import transformers++from auto_round.export.register import register_format+from auto_round.utils import get_block_names, get_module, logger, set_module+++def get_layer_names_in_block(model, supported_types=[torch.nn.Linear, transformers.modeling_utils.Conv1D]):+ """Retrieves the names of layers within each block of the model.++ Returns:+ list: A list of strings, where each string is the name of a layer+ within a block of the model.+ """+ for n, m in model.named_modules():+ if isinstance(m, tuple(supported_types)):+ m.tmp_name = n+ layers_in_block = []+ block_names = get_block_names(model)+ for block_name in block_names:+ block = get_module(model, block_name)+ for n, m in block.named_modules():+ if hasattr(m, "tmp_name"):+ layers_in_block.append(m.tmp_name)+ for n, m in model.named_modules():+ if hasattr(m, "tmp_name"):+ delattr(m, "tmp_name")+ return layers_in_block+++def check_neq_config(config, data_type, bits, group_size, sym):+ res = []+ if data_type != config["data_type"]:+ res.append("data_type")+ return res+ if bits != config["bits"]:+ res.append("bits")+ if group_size != config["group_size"]:+ res.append("group_size")+ if sym != config["sym"]:+ res.append("sym")+ return res+++def get_autogptq_backend_config(backend, bits=4):+ use_triton = False+ disable_exllamav2 = False+ disable_exllamav1 = False+ disable_marlin = True+ use_qigen = False+ if backend == "gptq:qigen":+ use_qigen = True+ if backend == "gptq:triton": ##TODO refine the code+ use_triton = True+ if backend == "gptq:marlin":+ use_triton = False+ disable_marlin = True+ if backend == "gptq:exllamav2":+ use_triton = False+ disable_marlin = True+ if backend == "gptq:exllamav1":+ use_triton = False+ disable_marlin = True+ if backend == "gptq:cuda":+ use_triton = False+ disable_marlin = True+ disable_exllamav2 = True+ disable_exllamav1 = True+ if bits not in [2, 4, 8]:+ use_qigen = False+ if bits not in [2, 4]:+ use_triton = False+ return use_triton, disable_exllamav1, disable_exllamav2, use_qigen, disable_marlin+++@register_format("autoround")+def save_quantized_as_autoround(output_dir, inplace=True, backend="gptq:exllamav2", **kwargs):+ from auto_gptq.utils.import_utils import dynamically_import_QuantLinear++ model = kwargs["model"]+ if not inplace:+ model = copy.deepcopy(model.to("cpu"))+ layer_names_in_block = get_layer_names_in_block(model)++ weight_config = kwargs["weight_config"]+ for name in weight_config.keys():++ config = kwargs["weight_config"][name]+ if config["data_type"] != "int" and config["bits"] >= 16:+ continue+ logger.info(f"packing {name}")++ bits = config["bits"]+ group_size = config["group_size"]+ use_triton, disable_exllamav1, disable_exllamav2, use_qigen, disable_marlin = get_autogptq_backend_config(+ backend, bits+ )++ layer = get_module(model, name)+ device = "cpu"+ QuantLinear = dynamically_import_QuantLinear(+ use_triton=use_triton,+ desc_act=False,+ group_size=group_size,+ bits=bits,+ disable_exllama=disable_exllamav1,+ disable_exllamav2=disable_exllamav2,+ use_qigen=use_qigen,+ disable_marlin=disable_marlin,+ )++ if isinstance(layer, nn.Linear):+ in_features = layer.in_features+ out_features = layer.out_features+ elif isinstance(layer, nn.Conv2d):+ in_features = layer.in_channels+ out_features = layer.out_channels+ elif isinstance(layer, transformers.pytorch_utils.Conv1D):+ in_features = layer.weight.shape[0]+ out_features = layer.weight.shape[1]+ bias = layer.bias is not None and torch.any(layer.bias)++ new_layer = QuantLinear(+ bits, group_size, in_features, out_features, bias, weight_dtype=layer.weight.dtype ##pylint: disable=E1123+ )++ new_layer.device = device+ set_module(model, name, new_layer)+ qlayer = new_layer+ scale = weight_config[name]["scale"]+ zero = weight_config[name]["zp"]+ # so far can only pack layer on CPU+ qlayer.to("cpu")+ layer, scale, zero = layer.to("cpu"), scale.to("cpu"), zero.to("cpu")+ qlayer.pack(layer, scale, zero, None)+ qlayer.to(device)+ quantization_config = kwargs["serialization_dict"]+ quantization_config["quant_method"] = "intel/auto-round"+ quantization_config["backend"] = backend+ extra_config = {}+ for layer_name in weight_config:+ if weight_config[layer_name]["data_type"] != "int" and weight_config[layer_name]["bits"] >= 16:+ continue+ if layer_name not in layer_names_in_block:+ extra_config[layer_name] = {}+ extra_config[layer_name]["bits"] = weight_config[layer_name]["bits"]+ extra_config[layer_name]["data_type"] = weight_config[layer_name]["data_type"]+ extra_config[layer_name]["group_size"] = weight_config[layer_name]["group_size"]+ extra_config[layer_name]["sym"] = weight_config[layer_name]["sym"]+ else:+ neq_keys = check_neq_config(+ weight_config[layer_name],+ data_type=quantization_config["data_type"],+ bits=quantization_config["bits"],+ group_size=quantization_config["group_size"],+ sym=quantization_config["sym"],+ )+ if len(neq_keys) > 0:+ extra_config[layer_name] = {}+ for key in neq_keys:+ extra_config[layer_name][key] = weight_config[layer_name][key]+ if len(extra_config) > 0:+ quantization_config["extra_config"] = extra_config+ if hasattr(model, "config"):+ model.config.quantization_config = quantization_config+ tokenizer = kwargs["tokenizer"]+ if tokenizer is not None:+ tokenizer.save_pretrained(output_dir)+ save(model, output_dir)+++def save(model: nn.Module, save_dir: str, max_shard_size: str = "10GB", safe_serialization: bool = True):+ """Save model state dict and configs.++ Args:+ model (`nn.Module`):+ Model to be saved. The model can be wrapped or unwrapped.+ save_dir (`str`):+ Directory to which to save. Will be created if it doesn't exist.+ max_shard_size (`str`, defaults to `"10GB"`):+ The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size+ lower than this size. If expressed as a string, needs to be digits followed by a unit (like `"5MB"`).+ <Tip warning={true}>++ If a single weight of the model is bigger than `max_shard_size`, it will be in its own checkpoint shard+ which will be bigger than `max_shard_size`.++ </Tip>+ safe_serialization (`bool`, defaults to `True`):+ Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).+ """+ os.makedirs(save_dir, exist_ok=True)+ model.save_pretrained(save_dir, max_shard_size=max_shard_size, safe_serialization=safe_serialization)+ config_file = "quantize_config.json"+ if hasattr(model, "config") and hasattr(model.config, "quantize_config"):+ with open(os.path.join(save_dir, config_file), "w", encoding="utf-8") as f:+ json.dump(model.config.quantization_config, f, indent=2)diff --git a/docs/Mistral-7B-v0.1-acc.md b/docs/Mistral-7B-v0.1-acc.mdindex 58ddfd1d9..8f2a93c6a 100644--- a/docs/Mistral-7B-v0.1-acc.md+++ b/docs/Mistral-7B-v0.1-acc.md@@ -1,4 +1,4 @@-Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id f3b7917091afba325af3980a35d8a6dcba03dc3f+Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used 0.4.2```bashlm_eval --model hf --model_args pretrained="Intel/Mistral-7B-v0.1-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32@@ -6,17 +6,17 @@ lm_eval --model hf --model_args pretrained="Intel/Mistral-7B-v0.1-int4-inc",auto-| Metric | BF16 | INT4 |-| -------------- |--------| ------ |-| Avg. | 0.6306 | 0.6308 |-| mmlu | 0.5961 | 0.5880 |-| lambada_openai | 0.7561 | 0.7551 |-| hellaswag | 0.6128 | 0.6079 |-| winogrande | 0.7443 | 0.7451 |-| piqa | 0.8079 | 0.8014 |-| truthfulqa_mc1 | 0.2803 | 0.2889 |-| openbookqa | 0.3280 | 0.3300 |-| boolq | 0.8373 | 0.8278 |-| rte | 0.6643 | 0.6968 |-| arc_easy | 0.8085 | 0.8060 |-| arc_challenge | 0.5009 | 0.4915 |++| Metric | BF16 | [INT4-lmhead](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc-lmhead) | [INT4](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) |+| -------------- | ------ |-----------------| ------------------------------------------------------------ |+| Avg. | 0.6260 | 0.6228 | 0.6218 |+| mmlu | 0.5868 | 0.5760 | 0.5772 |+| lambada_openai | 0.7555 | 0.7539 | 0.7543 |+| hellaswag | 0.6125 | 0.6055 | 0.6072 |+| winogrande | 0.7395 | 0.7380 | 0.7388 |+| piqa | 0.8069 | 0.8009 | 0.8030 |+| truthfulqa_mc1 | 0.2803 | 0.2876 | 0.2864 |+| openbookqa | 0.3280 | 0.3300 | 0.3260 |+| boolq | 0.8379 | 0.8291 | 0.8281 |+| arc_easy | 0.8089 | 0.8043 | 0.8035 |+| arc_challenge | 0.5034 | 0.5026 | 0.4932 |diff --git a/examples/language-modeling/eval_042/__init__.py b/examples/language-modeling/eval_042/__init__.pynew file mode 100644index 000000000..e69de29bbdiff --git a/examples/language-modeling/eval_042/evaluation.py b/examples/language-modeling/eval_042/evaluation.pynew file mode 100644index 000000000..d05847316--- /dev/null+++ b/examples/language-modeling/eval_042/evaluation.py@@ -0,0 +1,602 @@+import itertools+import logging+import random+import time+from collections import defaultdict+from typing import TYPE_CHECKING, List, Optional, Union++import numpy as np+import torch++import lm_eval.api.metrics+import lm_eval.api.registry+import lm_eval.models+from lm_eval.caching.cache import delete_cache+from lm_eval.evaluator_utils import (+ consolidate_results,+ get_sample_size,+ get_task_list,+ prepare_print_tasks,+ print_writeout,+ run_task_tests,+)+from lm_eval.logging_utils import add_env_info, get_git_commit_hash+from lm_eval.tasks import TaskManager, get_task_dict+from lm_eval.utils import eval_logger, positional_deprecated, simple_parse_args_string++if TYPE_CHECKING:+ from lm_eval.api.model import LM+ from lm_eval.tasks import Task+++@positional_deprecated+def simple_evaluate(+ model,+ model_args: Optional[Union[str, dict]] = None,+ tasks: Optional[List[Union[str, dict, object]]] = None,+ num_fewshot: Optional[int] = None,+ batch_size: Optional[int] = None,+ max_batch_size: Optional[int] = None,+ device: Optional[str] = None,+ use_cache: Optional[str] = None,+ cache_requests: bool = False,+ rewrite_requests_cache: bool = False,+ delete_requests_cache: bool = False,+ limit: Optional[Union[int, float]] = None,+ bootstrap_iters: int = 100000,+ check_integrity: bool = False,+ write_out: bool = False,+ log_samples: bool = True,+ gen_kwargs: Optional[str] = None,+ task_manager: Optional[TaskManager] = None,+ verbosity: str = "INFO",+ predict_only: bool = False,+ random_seed: int = 0,+ numpy_random_seed: int = 1234,+ torch_random_seed: int = 1234,+):+ """Instantiate and evaluate a model on a list of tasks.++ :param model: Union[str, LM]+ Name of model or LM object, see lm_eval.models.get_model+ :param model_args: Optional[str, dict]+ String or dict arguments for each model class, see LM.create_from_arg_string and LM.create_from_arg_object.+ Ignored if `model` argument is a LM object.+ :param tasks: list[Union[str, dict, Task]]+ List of task names or Task objects. Task objects will be taken to have name task.EVAL_HARNESS_NAME if defined and type(task).__name__ otherwise.+ :param num_fewshot: int+ Number of examples in few-shot context+ :param batch_size: int or str, optional+ Batch size for model+ :param max_batch_size: int, optional+ Maximal batch size to try with automatic batch size detection+ :param device: str, optional+ PyTorch device (e.g. "cpu" or "cuda:0") for running models+ :param use_cache: str, optional+ A path to a sqlite db file for caching model responses. `None` if not caching.+ :param cache_requests: bool, optional+ Speed up evaluation by caching the building of dataset requests. `None` if not caching.+ :param rewrite_requests_cache: bool, optional+ Rewrites all of the request cache if set to `True`. `None` if not desired.+ :param delete_requests_cache: bool, optional+ Deletes all of the request cache if set to `True`. `None` if not desired.+ :param limit: int or float, optional+ Limit the number of examples per task (only use this for testing), If <1, limit is a percentage of the total number of examples.+ :param bootstrap_iters:+ Number of iterations for bootstrap statistics+ :param check_integrity: bool+ Whether to run the relevant part of the test suite for the tasks+ :param write_out: bool+ If True, write out an example document and model input for checking task integrity+ :param log_samples: bool+ If True, write out all model outputs and documents for per-sample measurement and post-hoc analysis+ :param gen_kwargs: str+ String arguments for model generation+ Ignored for all tasks with loglikelihood output_type+ :param predict_only: bool+ If true only model outputs will be generated and returned. Metrics will not be evaluated+ :param random_seed: int+ Random seed for python's random module. If set to None, the seed will not be set.+ :param numpy_random_seed: int+ Random seed for numpy. If set to None, the seed will not be set.+ :param torch_random_seed: int+ Random seed for torch. If set to None, the seed will not be set.++ :return+ Dictionary of results+ """+ eval_logger.setLevel(getattr(logging, f"{verbosity}"))+ start_date = time.time()++ if delete_requests_cache:+ eval_logger.info("Deleting requests cache...")+ delete_cache()++ seed_message = []+ if random_seed is not None:+ # See https://github.com/EleutherAI/lm-evaluation-harness/pull/1412+ seed_message.append(f"Setting random seed to {random_seed}")+ random.seed(random_seed)++ if numpy_random_seed is not None:+ seed_message.append(f"Setting numpy seed to {numpy_random_seed}")+ np.random.seed(numpy_random_seed)++ if torch_random_seed is not None:+ seed_message.append(f"Setting torch manual seed to {torch_random_seed}")+ torch.manual_seed(torch_random_seed)++ if seed_message:+ eval_logger.info(" | ".join(seed_message))++ if tasks is None:+ tasks = []+ if len(tasks) == 0:+ raise ValueError(+ "No tasks specified, or no tasks found. Please verify the task names."+ )++ if gen_kwargs is not None:+ gen_kwargs = simple_parse_args_string(gen_kwargs)+ eval_logger.warning(+ "generation_kwargs specified through cli, these settings will update set parameters in yaml tasks. "+ "Ensure 'do_sample=True' for non-greedy decoding!"+ )+ if gen_kwargs == "":+ gen_kwargs = None++ if isinstance(model, str):+ if model_args is None:+ model_args = ""++ if isinstance(model_args, dict):+ lm = lm_eval.api.registry.get_model(model).create_from_arg_obj(+ model_args,+ {+ "batch_size": batch_size,+ "max_batch_size": max_batch_size,+ "device": device,+ },+ )++ else:+ lm = lm_eval.api.registry.get_model(model).create_from_arg_string(+ model_args,+ {+ "batch_size": batch_size,+ "max_batch_size": max_batch_size,+ "device": device,+ },+ )+ else:+ if not isinstance(model, lm_eval.api.model.LM):+ raise TypeError+ lm = model++ if use_cache is not None:+ eval_logger.info(f"Using cache at {use_cache + '_rank' + str(lm.rank) + '.db'}")+ lm = lm_eval.api.model.CachingLM(+ lm,+ use_cache+ # each rank receives a different cache db.+ # necessary to avoid multiple writes to cache at once+ + "_rank"+ + str(lm.rank)+ + ".db",+ )++ if task_manager is None:+ task_manager = TaskManager(verbosity)++ task_dict = get_task_dict(tasks, task_manager)+ for task_name in task_dict.keys():+ task_obj = task_dict[task_name]+ if isinstance(task_obj, tuple):+ _, task_obj = task_obj+ if task_obj is None:+ continue++ if task_obj.get_config("output_type") == "generate_until":+ if gen_kwargs is not None:+ task_obj.set_config(+ key="generation_kwargs", value=gen_kwargs, update=True+ )++ if predict_only:+ log_samples = True+ eval_logger.info(+ f"Processing {task_name} in output-only mode. Metrics will not be calculated!"+ )+ # we have to change the class properties post-hoc. This is pretty hacky.+ task_obj.override_metric(metric_name="bypass")++ # override tasks' fewshot values to the provided num_fewshot arg value+ # except if tasks have it set to 0 manually in their configs--then we should never overwrite that+ if num_fewshot is not None:+ if (default_num_fewshot := task_obj.get_config("num_fewshot")) == 0:+ eval_logger.info(+ f"num_fewshot has been set to 0 for {task_name} in its config. Manual configuration will be ignored."+ )+ else:+ eval_logger.warning(+ f"Overwriting default num_fewshot of {task_name} from {default_num_fewshot} to {num_fewshot}"+ )+ task_obj.set_config(key="num_fewshot", value=num_fewshot)+ else:+ # if num_fewshot not provided, and the task does not define a default one, default to 0+ if (default_num_fewshot := task_obj.get_config("num_fewshot")) is None:+ task_obj.set_config(key="num_fewshot", value=0)++ if check_integrity:+ run_task_tests(task_list=tasks)++ results = evaluate(+ lm=lm,+ task_dict=task_dict,+ limit=limit,+ cache_requests=cache_requests,+ rewrite_requests_cache=rewrite_requests_cache,+ bootstrap_iters=bootstrap_iters,+ write_out=write_out,+ log_samples=log_samples,+ verbosity=verbosity,+ )++ if lm.rank == 0:+ if isinstance(model, str):+ model_name = model+ elif hasattr(model, "config") and hasattr(model.config, "_name_or_path"):+ model_name = model.config._name_or_path+ else:+ model_name = type(model).__name__++ # add info about the model and few shot config+ results["config"] = {+ "model": model_name,+ "model_args": model_args,+ "batch_size": batch_size,+ "batch_sizes": (+ list(lm.batch_sizes.values()) if hasattr(lm, "batch_sizes") else []+ ),+ "device": device,+ "use_cache": use_cache,+ "limit": limit,+ "bootstrap_iters": bootstrap_iters,+ "gen_kwargs": gen_kwargs,+ }+ results["git_hash"] = get_git_commit_hash()+ results["date"] = start_date+ add_env_info(results) # additional environment info to results+ return results+ else:+ return None+++@positional_deprecated+def evaluate(+ lm: "LM",+ task_dict,+ limit: Optional[int] = None,+ cache_requests: bool = False,+ rewrite_requests_cache: bool = False,+ bootstrap_iters: Optional[int] = 100000,+ write_out: bool = False,+ log_samples: bool = True,+ verbosity: str = "INFO",+):+ """Instantiate and evaluate a model on a list of tasks.++ :param lm: obj+ Language Model+ :param task_dict: dict[str, Task]+ Dictionary of tasks. Tasks will be taken to have name type(task).config.task .+ :param limit: int, optional+ Limit the number of examples per task (only use this for testing)+ :param bootstrap_iters:+ Number of iterations for bootstrap statistics+ :param write_out: bool+ If True, write out an example document and model input for checking task integrity+ :param log_samples: bool+ If True, write out all model outputs and documents for per-sample measurement and post-hoc analysis+ :return+ Dictionary of results+ """++ eval_logger.setLevel(getattr(logging, f"{verbosity}"))++ # tracks all Instances/requests a model must generate output on.+ requests = defaultdict(list)+ # stores the amount to pad out reqs per req. type so that+ # number of fwd passes per distributed rank is equal+ padding_requests = defaultdict(int)++ # get lists of group hierarchy and each type of request+ task_hierarchy, eval_tasks = get_task_list(task_dict)+ if not log_samples:+ if not all(+ "bypass" not in getattr(task_output.task, "_metric_fn_list", {}).keys()+ for task_output in eval_tasks+ ):+ raise ValueError("log_samples must be True for 'bypass' metric-only tasks")+ for task_output in eval_tasks:+ task: Task = task_output.task+ limit = get_sample_size(task, limit)+ task.build_all_requests(+ limit=limit,+ rank=lm.rank,+ world_size=lm.world_size,+ cache_requests=cache_requests,+ rewrite_requests_cache=rewrite_requests_cache,+ )+ eval_logger.debug(+ f"Task: {task_output.task_name}; number of requests on this rank: {len(task.instances)}"+ )++ if write_out:+ print_writeout(task)+ # aggregate Instances by LM method requested to get output.+ for instance in task.instances:+ reqtype = instance.request_type+ requests[reqtype].append(instance)++ if lm.world_size > 1:+ instances_rnk = torch.tensor(len(task._instances), device=lm.device)+ gathered_item = (+ lm.accelerator.gather(instances_rnk).cpu().detach().numpy().tolist()+ )+ # "multiple_choice" task types dispatch (several) "loglikelihood" request types+ reqtype = (+ "loglikelihood"+ if task.OUTPUT_TYPE == "multiple_choice"+ else task.OUTPUT_TYPE+ )+ # compute number of pseudo-batches to pad with (FSDP/DDP require even batches among ranks)+ numpad = max(gathered_item) - gathered_item[lm.rank]+ # todo: may not account for padding in cases like SquadV2 which has multiple req types+ padding_requests[reqtype] += numpad++ ### Run LM on inputs, get all outputs ###+ # execute each type of request+ for reqtype, reqs in requests.items():+ eval_logger.info(f"Running {reqtype} requests")+ # create `K` copies of each request `req` based off `K = req.repeats`+ cloned_reqs = []+ for req in reqs:+ cloned_reqs.extend([req] * req.repeats)++ if (lm.world_size > 1) and (padding_requests[reqtype] > 0):+ for _ in range(padding_requests[reqtype]):+ cloned_reqs.extend([req] * req.repeats)++ # run requests through model+ resps = getattr(lm, reqtype)(cloned_reqs)++ # put responses from model into a list of length K for each request.+ for x, req in zip(resps, cloned_reqs):+ req.resps.append(x)++ if lm.world_size > 1:+ lm.accelerator.wait_for_everyone()++ RANK = lm.rank+ WORLD_SIZE = lm.world_size+ ### Postprocess outputs ###+ # TODO: del model here, maybe (idea: allow user to specify device of e.g. reward model separately)+ for task_output in eval_tasks:+ task = task_output.task+ task.apply_filters()++ ### Collect values of metrics on all datapoints ###+ # # unpack results and sort back in order and return control to Task+ # TODO: make it possible to use a different metric per filter+ # Pre-process task.instances to group by doc_id+ instances_by_doc_id = defaultdict(list)+ for instance in task.instances:+ instances_by_doc_id[instance.doc_id].append(instance)+ # Sort instances within each group+ for instances in instances_by_doc_id.values():+ instances.sort(key=lambda x: x.idx)+ # iterate over different filters used+ for filter_key in task.instances[0].filtered_resps.keys():+ doc_iterator = task.doc_iterator(+ rank=RANK, limit=limit, world_size=WORLD_SIZE+ )+ for doc_id, doc in doc_iterator:+ requests = instances_by_doc_id[doc_id]+ metrics = task.process_results(+ doc, [req.filtered_resps[filter_key] for req in requests]+ )+ if log_samples:+ target = task.doc_to_target(doc)+ example = {+ "doc_id": doc_id,+ "doc": doc,+ "target": target,+ "arguments": [req.args for req in requests],+ "resps": [req.resps for req in requests],+ "filtered_resps": [+ req.filtered_resps[filter_key] for req in requests+ ],+ }+ example.update(metrics)+ task_output.logged_samples.append(example)+ for metric, value in metrics.items():+ task_output.sample_metrics[(metric, filter_key)].append(value)++ if WORLD_SIZE > 1:+ # if multigpu, then gather data across all ranks to rank 0+ # first gather logged samples across all ranks+ for task_output in eval_tasks:+ if log_samples:+ # for task_name, task_samples in list(samples.items()):+ full_samples = [None] * WORLD_SIZE if RANK == 0 else None+ torch.distributed.gather_object(+ obj=task_output.logged_samples,+ object_gather_list=full_samples,+ dst=0,+ )++ if RANK == 0:+ task_output.logged_samples = list(+ itertools.chain.from_iterable(full_samples)+ )++ # then collect metrics across all ranks+ for metrics in task_output.sample_metrics:+ metric_list = [None] * WORLD_SIZE if RANK == 0 else None+ torch.distributed.gather_object(+ obj=task_output.sample_metrics[metrics],+ object_gather_list=metric_list,+ dst=0,+ )+ if RANK == 0:+ task_output.sample_metrics[metrics] = list(+ itertools.chain.from_iterable(metric_list)+ )++ if RANK == 0:+ ### Aggregate results over all datapoints ###+ # aggregate results ; run bootstrap CIs+ for task_output in eval_tasks:+ task_output.calculate_aggregate_metric(bootstrap_iters=bootstrap_iters)+ results, samples, configs, versions, num_fewshot = consolidate_results(+ eval_tasks+ )++ ### Calculate group metrics ###+ if bool(results):+ for group, task_list in reversed(task_hierarchy.items()):+ if len(task_list) == 0:+ # task_hierarchy entries are either+ # `group_name: [subtask1, subtask2, ...]`+ # or `task_name: []`.+ # we only want to operate on groups here.+ continue+ metric_list = list(+ {+ key+ for task in task_list+ for key in results[task].keys()+ if "_stderr" not in key and key not in ["alias", "samples"]+ }+ )+ for metric in metric_list:+ stderr = "_stderr,".join(metric.split(","))++ # gather metrics, sizes, and stderrs from subtasks+ metrics = [+ results[task][metric]+ for task in task_list+ if metric in results[task]+ ] # TODO: copy?+ stderrs = [+ results[task][stderr]+ for task in task_list+ if stderr in results[task]+ ]+ sizes = [+ results[task]["samples"]+ for task in task_list+ if metric in results[task]+ ]++ # compute group's pooled metric and stderr+ results[group][+ metric+ ] = lm_eval.api.metrics.aggregate_subtask_metrics(metrics, sizes)+ # TODO: calculate grouped metric using aggregation fn+ if "N/A" in stderrs:+ results[group][stderr] = "N/A"+ else:+ results[group][+ stderr+ ] = lm_eval.api.metrics.pooled_sample_stderr(stderrs, sizes)+ # TODO: allow GroupConfigs to choose which variance formula is used, for back-compatibility+ # To use the old (likely incorrect) variance formula, comment out the above and uncomment this line:+ # results[group][stderr] = lm_eval.api.metrics.combined_sample_stderr(stderrs, sizes, metrics=metrics)++ results[group]["samples"] = sum(sizes)++ results_agg = defaultdict(dict)+ groups_agg = defaultdict(dict)+ all_tasks_list = list(task_hierarchy.keys())+ while True:+ add_tasks_list = list(k for k in results_agg.keys())+ left_tasks_list = sorted(list(set(all_tasks_list) - set(add_tasks_list)))+ if len(left_tasks_list) == 0:+ break++ _task_hierarchy = {+ k: v for k, v in task_hierarchy.items() if k in left_tasks_list+ }+ _results_agg, _groups_agg = prepare_print_tasks(_task_hierarchy, results)++ results_agg = {**results_agg, **_results_agg}+ groups_agg = {**groups_agg, **_groups_agg}++ for group_name, task_list in task_hierarchy.items():+ if task_list:+ num_fewshot[group_name] = num_fewshot[+ task_list[0]+ ] # TODO: validate this++ results_dict = {+ "results": dict(results_agg.items()),+ **({"groups": dict(groups_agg.items())} if bool(groups_agg) else {}),+ "group_subtasks": dict(reversed(task_hierarchy.items())),+ "configs": dict(sorted(configs.items())),+ "versions": dict(sorted(versions.items())),+ "n-shot": dict(sorted(num_fewshot.items())),+ }+ if log_samples:+ results_dict["samples"] = dict(samples)++ return results_dict++ else:+ return None+++if __name__ == "__main__":++ import sys++ sys.path.insert(0, '../../')+ import time+ import argparse++ parser = argparse.ArgumentParser()+ parser.add_argument(+ "--model_name", default="/models/opt-125m/"+ )+ parser.add_argument(+ "--eval_bs", default=1,+ )+ parser.add_argument("--tasks",+ default="lambada_openai,hellaswag,winogrande,piqa,mmlu,truthfulqa_mc1," \+ "truthfulqa_mc2,openbookqa,boolq,rte,arc_easy,arc_challenge",+ help="lm-eval tasks for lm_eval version 0.4.2")++ args = parser.parse_args()+ s = time.time()+ from transformers import AutoConfig++ config = AutoConfig.from_pretrained(args.model_name)+ if hasattr(config, "quantization_config"):+ quantization_config = config.quantization_config+ if "quant_method" in quantization_config and "auto-round" in quantization_config["quant_method"]:+ from auto_round.export.export_to_autoround import AutoHfQuantizer++ test_tasks = args.tasks+ if isinstance(test_tasks, str):+ test_tasks = test_tasks.split(',')+ model_name = args.model_name.rstrip('/')+ from lm_eval.utils import make_table++ result = simple_evaluate(model="hf",+ model_args=f"pretrained={args.model_name}",+ tasks=test_tasks,+ batch_size=args.eval_bs)+ print(make_table(result))++ print("cost time: ", time.time() - s)diff --git a/examples/language-modeling/main.py b/examples/language-modeling/main.pyindex 90a83d93a..546d61cfb 100644--- a/examples/language-modeling/main.py+++ b/examples/language-modeling/main.py@@ -265,6 +265,8 @@ if __name__ == '__main__':print(f"{n} will not be quantized due to its shape not being divisible by 32, resulting in an exporting issue to autogptq")lm_head_layer_name = "lm_head"+ for n, _ in model.named_modules():+ lm_head_layer_name = nif args.quant_lm_head:from transformers import AutoConfig@@ -303,8 +305,11 @@ if __name__ == '__main__':output_dir = args.output_dir + "/" + model_name.split('/')[-1] + f"-autoround-w{args.bits}g{args.group_size}-qdq"deployment_device = args.deployment_device.split(',')inplace = True if len(deployment_device) < 2 else False- if 'gpu' in deployment_device:- autoround.save_quantized(f'{export_dir}-gpu', format="auto_gptq", use_triton=True, inplace=inplace)+ if 'gpu' in deployment_device :+ if lm_head_layer_name in weight_config.keys():+ autoround.save_quantized(f'{export_dir}-gpu', format="autoround", use_triton=True, inplace=inplace)+ else:+ autoround.save_quantized(f'{export_dir}-gpu', format="auto_gptq", use_triton=True, inplace=inplace)if 'xpu' in deployment_device:autoround.save_quantized(f'{export_dir}-xpu', format="itrex_xpu", use_triton=True, inplace=inplace,compression_dtype=torch.int8, compression_dim=0, use_optimum_format=False,