Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

support real lm-head quantization and mixed precision inference (#114)

wenhuach214d1caebba03c
diff --git a/README.md b/README.md
index fc75033b1..9e67cbbc7 100644
--- a/README.md
+++ b/README.md
@@ -151,35 +151,35 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
## Support List
-| Model | Supported |
-|--------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Intel/neural-chat-7b-v3-3 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc), [accuracy](./docs/neural-chat-7b-v3-3-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-3.sh), [example](./examples/language-modeling/) |
-| Intel/neural-chat-7b-v3-1 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc), [accuracy](./docs/neural-chat-7b-v3-1-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-1.sh), [example](./examples/language-modeling/) |
-| mistralai/Mistral-7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc), [accuracy](./docs/Mistral-7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-v0.1.sh), [example](./examples/language-modeling/) |
-| microsoft/phi-2 | [HF-int4-model](https://huggingface.co/Intel/phi-2-int4-inc), [accuracy](./docs/phi-2-acc.md), [recipe](./examples/language-modeling/scripts/phi-2.sh), [example](./examples/language-modeling/)
-| tiiuae/falcon-7b | [HF-int4-model](https://huggingface.co/Intel/falcon-7b-int4-inc), [accuracy](./docs/falcon-7b-acc.md), [recipe](./examples/language-modeling/scripts/falcon-7b.sh), [example](./examples/language-modeling/) |
-| google/gemma-2b | [HF-int4-model](https://huggingface.co/Intel/gemma-2b-int4-inc), [accuracy](./docs/gemma-2b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-2b.sh), [example](./examples/language-modeling/)
-| mistralai/Mistral-7B-Instruct-v0.2 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-Instruct-v0.2-int4-inc) (under review), [accuracy](./docs/Mistral-7B-Instruct-v0.2-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.sh), [example](./examples/language-modeling/) |
-| google/gemma-7b | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-int4-inc) (under review), [accuracy](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh), [example](./examples/language-modeling/) |
-| google/gemma-7b-it | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-it-int4-inc) (under review), [accuracy](./docs/gemma-7b-it-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b-it.sh), [example](./examples/language-modeling/) |
-| mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) |
-| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |
-| meta-llama/Meta-Llama-3-8B-Instruct | [accuracy](./docs/Meta-Llama-3-8B-Instruct-acc.md), [recipe](./examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.sh), [example](./examples/language-modeling/) |
-| meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) |
-| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [sym recipe](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh), [asym recipe ](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh), [example](./examples/language-modeling/) |
-| baichuan-inc/Baichuan2-7B-Chat | [accuracy](./docs/baichuan2-7b-chat-acc.md), [recipe](./examples/language-modeling/scripts/baichuan2-7b-chat.sh), [example](./examples/language-modeling/) |
-| 01-ai/Yi-6B-Chat | [accuracy](./docs/Yi-6B-Chat-acc.md), [recipe](./examples/language-modeling/scripts/Yi-6B-Chat.sh), [example](./examples/language-modeling/) |
-| facebook/opt-2.7b | [accuracy](./docs/opt-2.7b-acc.md), [recipe](./examples/language-modeling/scripts/opt-2.7b.sh), [example](./examples/language-modeling/) |
-| bigscience/bloom-3b | [accuracy](./docs/bloom-3B-acc.md), [recipe](./examples/language-modeling/scripts/bloom-3b.sh), [example](./examples/language-modeling/) |
-| EleutherAI/gpt-j-6b | [accuracy](./docs/gpt-j-6B-acc.md), [recipe](./examples/language-modeling/scripts/gpt-j-6b.sh), [example](./examples/language-modeling/) |
-| Salesforce/codegen25-7b-multi | [example](./examples/language-modeling/) |
-| huggyllama/llama-7b | [example](./examples/language-modeling/) |
-| mosaicml/mpt-7b | [example](./examples/language-modeling/) |
-| THUDM/chatglm3-6b | [example](./examples/language-modeling/) |
-| MBZUAI/LaMini-GPT-124M | [example](./examples/language-modeling/) |
-| EleutherAI/gpt-neo-125m | [example](./examples/language-modeling/) |
-| databricks/dolly-v2-3b | [example](./examples/language-modeling/) |
-| stabilityai/stablelm-base-alpha-3b | [example](./examples/language-modeling/)
+| Model | Supported |
+|--------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Intel/neural-chat-7b-v3-3 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc), [accuracy](./docs/neural-chat-7b-v3-3-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-3.sh), [example](./examples/language-modeling/) |
+| Intel/neural-chat-7b-v3-1 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc), [accuracy](./docs/neural-chat-7b-v3-1-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-1.sh), [example](./examples/language-modeling/) |
+| mistralai/Mistral-7B-v0.1 | [HF-int4-model-lmhead](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc-lmhead),[HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc), [accuracy](./docs/Mistral-7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-v0.1.sh), [example](./examples/language-modeling/) |
+| microsoft/phi-2 | [HF-int4-model](https://huggingface.co/Intel/phi-2-int4-inc), [accuracy](./docs/phi-2-acc.md), [recipe](./examples/language-modeling/scripts/phi-2.sh), [example](./examples/language-modeling/)
+| tiiuae/falcon-7b | [HF-int4-model](https://huggingface.co/Intel/falcon-7b-int4-inc), [accuracy](./docs/falcon-7b-acc.md), [recipe](./examples/language-modeling/scripts/falcon-7b.sh), [example](./examples/language-modeling/) |
+| google/gemma-2b | [HF-int4-model](https://huggingface.co/Intel/gemma-2b-int4-inc), [accuracy](./docs/gemma-2b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-2b.sh), [example](./examples/language-modeling/)
+| mistralai/Mistral-7B-Instruct-v0.2 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-Instruct-v0.2-int4-inc) (under review), [accuracy](./docs/Mistral-7B-Instruct-v0.2-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.sh), [example](./examples/language-modeling/) |
+| google/gemma-7b | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-int4-inc) (under review), [accuracy](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh), [example](./examples/language-modeling/) |
+| google/gemma-7b-it | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-it-int4-inc) (under review), [accuracy](./docs/gemma-7b-it-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b-it.sh), [example](./examples/language-modeling/) |
+| mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) |
+| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |
+| meta-llama/Meta-Llama-3-8B-Instruct | [accuracy](./docs/Meta-Llama-3-8B-Instruct-acc.md), [recipe](./examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.sh), [example](./examples/language-modeling/) |
+| meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) |
+| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [sym recipe](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh), [asym recipe ](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh), [example](./examples/language-modeling/) |
+| baichuan-inc/Baichuan2-7B-Chat | [accuracy](./docs/baichuan2-7b-chat-acc.md), [recipe](./examples/language-modeling/scripts/baichuan2-7b-chat.sh), [example](./examples/language-modeling/) |
+| 01-ai/Yi-6B-Chat | [accuracy](./docs/Yi-6B-Chat-acc.md), [recipe](./examples/language-modeling/scripts/Yi-6B-Chat.sh), [example](./examples/language-modeling/) |
+| facebook/opt-2.7b | [accuracy](./docs/opt-2.7b-acc.md), [recipe](./examples/language-modeling/scripts/opt-2.7b.sh), [example](./examples/language-modeling/) |
+| bigscience/bloom-3b | [accuracy](./docs/bloom-3B-acc.md), [recipe](./examples/language-modeling/scripts/bloom-3b.sh), [example](./examples/language-modeling/) |
+| EleutherAI/gpt-j-6b | [accuracy](./docs/gpt-j-6B-acc.md), [recipe](./examples/language-modeling/scripts/gpt-j-6b.sh), [example](./examples/language-modeling/) |
+| Salesforce/codegen25-7b-multi | [example](./examples/language-modeling/) |
+| huggyllama/llama-7b | [example](./examples/language-modeling/) |
+| mosaicml/mpt-7b | [example](./examples/language-modeling/) |
+| THUDM/chatglm3-6b | [example](./examples/language-modeling/) |
+| MBZUAI/LaMini-GPT-124M | [example](./examples/language-modeling/) |
+| EleutherAI/gpt-neo-125m | [example](./examples/language-modeling/) |
+| databricks/dolly-v2-3b | [example](./examples/language-modeling/) |
+| stabilityai/stablelm-base-alpha-3b | [example](./examples/language-modeling/)
## Comparison with other methods
diff --git a/auto_round/autoround.py b/auto_round/autoround.py
index 361e3e668..dcb08bfbb 100644
--- a/auto_round/autoround.py
+++ b/auto_round/autoround.py
@@ -485,11 +485,39 @@ class AutoRound(object):
self.share_attention_mask_flag = None
self.hidden_dim_flag = None
torch.set_printoptions(precision=3, sci_mode=True)
+
+ self.check_configs()
+ serialization_keys = [
+ "bits",
+ "group_size",
+ "sym",
+ "data_type",
+ "enable_quanted_input",
+ "enable_minmax_tuning",
+ "data_type",
+ "seqlen",
+ "train_bs",
+ "scale_dtype",
+ "lr",
+ "minmax_lr",
+ "gradient_accumulate_steps",
+ "iters",
+ "amp",
+ ]
+ if isinstance(dataset, str):
+ serialization_keys.append("dataset")
+ self.serialization_dict = {}
+ for key in serialization_keys:
+ self.serialization_dict[key] = getattr(self, key)
+ from .version import __version__
+
+ self.serialization_dict["autoround_version"] = __version__
+ if "scale_dtype" in self.serialization_dict.keys():
+ self.serialization_dict["scale_dtype"] = str(self.serialization_dict["scale_dtype"])
if is_optimum_habana_available():
logger.info("Optimum Habana is available, import htcore explicitly.")
import habana_frameworks.torch.core as htcore # pylint: disable=E0401
import habana_frameworks.torch.hpu as hthpu # pylint: disable=E0401
- self.check_configs()
def check_configs(self):
"""Checks if the configurations are valid.
@@ -815,6 +843,7 @@ class AutoRound(object):
block_names, n_samples, layer_names=layer_names, last_cache_name=last_cache_name
)
self.model = self.model.to("cpu")
+ torch.cuda.empty_cache()
except:
logger.info("switch to cpu to cache inputs")
self.model = self.model.to("cpu")
@@ -1014,10 +1043,7 @@ class AutoRound(object):
lr_schedule = copy.deepcopy(self.lr_scheduler)
train_bs = self.train_bs
- pick_samples = train_bs
n_samples = len(inputs)
- if self.sampler != "rand":
- indices = torch.randperm(n_samples)[:pick_samples]
last_best_iter = 0
best_loss = torch.finfo(torch.float).max
mse_loss = torch.nn.MSELoss().to(device)
@@ -1025,6 +1051,11 @@ class AutoRound(object):
init_loss = None
best_v, best_min_scale, best_max_scale = torch.tensor(0), torch.tensor(0), torch.tensor(0)
gradient_accumulate_steps = self.gradient_accumulate_steps
+ gradient_accumulate_steps = self.train_bs ##Force to low gpu
+ train_bs = 1 ##Force to low gpu
+ pick_samples = train_bs
+ if self.sampler != "rand":
+ indices = torch.randperm(n_samples)[:pick_samples]
for i in range(self.iters):
total_loss = 0
for _ in range(gradient_accumulate_steps):
@@ -1320,7 +1351,7 @@ class AutoRound(object):
logger.error(f"export format only supports {EXPORT_FORMAT.keys()}")
exit()
save_quantized_as_format = EXPORT_FORMAT.get(format)
- compressed_model = save_quantized_as_format(
+ compressed_model = save_quantized_as_format( ##TODO refine the code
output_dir,
model=self.model,
weight_config=self.weight_config,
@@ -1336,6 +1367,8 @@ class AutoRound(object):
scale_dtype=self.scale_dtype,
tokenizer=self.tokenizer,
supported_types=self.supported_types,
+ data_type=self.data_type,
+ serialization_dict=self.serialization_dict,
**kwargs,
)
return compressed_model
diff --git a/auto_round/export/__init__.py b/auto_round/export/__init__.py
index 9b4384ff9..5a48d41f9 100644
--- a/auto_round/export/__init__.py
+++ b/auto_round/export/__init__.py
@@ -15,3 +15,5 @@
from .register import EXPORT_FORMAT
from .export_to_autogptq import save_quantized_as_autogptq
from .export_to_itrex import save_quantized_as_itrex, QuantConfig
+from .export_to_autoround.export_to_autoround import save_quantized_as_autoround
+from .export_to_autoround import AutoHfQuantizer
diff --git a/auto_round/export/export_to_autoround/__init__.py b/auto_round/export/export_to_autoround/__init__.py
new file mode 100644
index 000000000..be98f7b1b
--- /dev/null
+++ b/auto_round/export/export_to_autoround/__init__.py
@@ -0,0 +1,8 @@
+import transformers
+from .autoround_quantizer import AutoHfQuantizer
+
+transformers.quantizers.auto.AutoHfQuantizer = AutoHfQuantizer
+transformers.quantizers.auto.AutoQuantizationConfig = AutoHfQuantizer
+transformers.modeling_utils.AutoHfQuantizer = AutoHfQuantizer
+from transformers import AutoModelForCausalLM as AutoRoundModelForCausalLM
+from .export_to_autoround import save_quantized_as_autoround
diff --git a/auto_round/export/export_to_autoround/autoround_quantizer.py b/auto_round/export/export_to_autoround/autoround_quantizer.py
new file mode 100644
index 000000000..9a2651042
--- /dev/null
+++ b/auto_round/export/export_to_autoround/autoround_quantizer.py
@@ -0,0 +1,427 @@
+# Copyright (c) 2024 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+# coding=utf-8
+# Copyright 2023 HuggingFace Inc. team and GPTQ and AutoGPTQ authors.
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+import importlib.util
+import warnings
+from dataclasses import dataclass
+from logging import getLogger
+from typing import Any, Dict, List, Optional, Tuple, Union
+
+import torch
+import torch.nn as nn
+from packaging import version
+from transformers.modeling_utils import PreTrainedModel
+from transformers.pytorch_utils import Conv1D
+from transformers.quantizers import AutoQuantizationConfig, HfQuantizer
+from transformers.quantizers.auto import AUTO_QUANTIZER_MAPPING
+from transformers.utils.quantization_config import AwqConfig, GPTQConfig, QuantizationConfigMixin, QuantizationMethod
+
+from auto_round.utils import get_module, set_module
+
+logger = getLogger(__name__)
+import sys
+
+if sys.version_info < (3, 8):
+ import importlib_metadata
+else:
+ import importlib.metadata as importlib_metadata
+
+AUTOGPTQ_MINIMUM_VERSION = version.parse("0.4.99") # Allows 0.5.0.dev0
+
+
+def _is_package_available(pkg_name: str, return_version: bool = False) -> Union[Tuple[bool, str], bool]:
+ # Check we're not importing a "pkg_name" directory somewhere but the actual library by trying to grab the version
+ package_exists = importlib.util.find_spec(pkg_name) is not None
+ package_version = "N/A"
+ if package_exists:
+ try:
+ package_version = importlib.metadata.version(pkg_name)
+ package_exists = True
+ except importlib.metadata.PackageNotFoundError:
+ package_exists = False
+ if return_version:
+ return package_exists, package_version
+ else:
+ return package_exists
+
+
+_auto_gptq_available = _is_package_available("auto_gptq")
+
+
+def is_auto_gptq_available():
+ if _auto_gptq_available:
+ version_autogptq = version.parse(importlib_metadata.version("auto_gptq"))
+ if AUTOGPTQ_MINIMUM_VERSION < version_autogptq:
+ return True
+ else:
+ raise ImportError(
+ f"Found an incompatible version of auto-gptq. Found version {version_autogptq},"
+ f" but only version above {AUTOGPTQ_MINIMUM_VERSION} are supported"
+ )
+
+
+if is_auto_gptq_available():
+ from auto_gptq import exllama_set_max_input_length
+ from auto_gptq.modeling._utils import autogptq_post_init
+ from auto_gptq.quantization import GPTQ
+ from auto_gptq.utils.import_utils import dynamically_import_QuantLinear
+
+
+#
+def get_device(obj: Union[torch.Tensor, nn.Module]):
+ if isinstance(obj, torch.Tensor):
+ return obj.device
+ return next(obj.parameters()).device
+
+
+class AutoHfQuantizer:
+ """The Auto-HF quantizer class that takes care of automatically instantiating to the correct
+ `HfQuantizer` given the `QuantizationConfig`."""
+
+ @classmethod
+ def from_config(cls, quantization_config: Union[QuantizationConfigMixin, Dict], **kwargs):
+ # Convert it to a QuantizationConfig if the q_config is a dict
+ if isinstance(quantization_config, dict):
+ if "auto-round" in quantization_config["quant_method"]:
+ quantization_config = AutoRoundConfig.from_dict(quantization_config)
+ else:
+ quantization_config = AutoQuantizationConfig.from_dict(quantization_config) # pylint: disable=E1101
+ quant_method = quantization_config.quant_method
+
+ # Again, we need a special care for bnb as we have a single quantization config
+ # class for both 4-bit and 8-bit quantization
+ if quant_method == QuantizationMethod.BITS_AND_BYTES:
+ if quantization_config.load_in_8bit:
+ quant_method += "_8bit"
+ else:
+ quant_method += "_4bit"
+
+ if quant_method not in AUTO_QUANTIZER_MAPPING.keys() and "auto-round" not in quant_method:
+ raise ValueError(
+ f"Unknown quantization type, got {quant_method} - supported types are:"
+ f" {list(AUTO_QUANTIZER_MAPPING.keys())}"
+ )
+ if "auto-round" in quant_method:
+ target_cls = AutoRoundQuantizer
+ else:
+ target_cls = AUTO_QUANTIZER_MAPPING[quant_method]
+
+ return target_cls(quantization_config, **kwargs)
+
+ @classmethod
+ def from_pretrained(cls, pretrained_model_name_or_path, **kwargs):
+ quantization_config = AutoQuantizationConfig.from_pretrained(pretrained_model_name_or_path, **kwargs)
+ return cls.from_config(quantization_config)
+
+ @classmethod
+ def merge_quantization_configs(
+ cls,
+ quantization_config: Union[dict, QuantizationConfigMixin],
+ quantization_config_from_args: Optional[QuantizationConfigMixin],
+ ):
+ """Handles situations where both quantization_config
+ from args and quantization_config from model config are present."""
+ if quantization_config_from_args is not None:
+ warning_msg = (
+ "You passed `quantization_config` or equivalent parameters to "
+ "`from_pretrained` but the model you're loading"
+ " already has a `quantization_config` attribute. The `quantization_config` from the model will be used."
+ )
+ else:
+ warning_msg = ""
+
+ if isinstance(quantization_config, dict):
+ quantization_config = AutoRoundConfig.from_dict(quantization_config)
+
+ if isinstance(quantization_config, (GPTQConfig, AwqConfig)) and quantization_config_from_args is not None:
+ # special case for GPTQ / AWQ config collision
+ loading_attr_dict = quantization_config_from_args.get_loading_attributes()
+ for attr, val in loading_attr_dict.items():
+ setattr(quantization_config, attr, val)
+ warning_msg += (
+ f"However, loading attributes (e.g. {list(loading_attr_dict.keys())}) "
+ f"will be overwritten with the one you passed to `from_pretrained`. The rest will be ignored."
+ )
+
+ if warning_msg != "":
+ warnings.warn(warning_msg)
+
+ return quantization_config
+
+
+@dataclass
+class AutoRoundConfig(QuantizationConfigMixin):
+ """This is a wrapper class about all possible attributes and features that you can play with a model that has been
+ loaded using `optimum` api for gptq quantization relying on auto_gptq backend.
+
+ Args:
+ bits (`int`):
+ The number of bits to quantize to, supported numbers are (2, 3, 4, 8).
+ tokenizer (`str` or `PreTrainedTokenizerBase`, *optional*):
+ The tokenizer used to process the dataset. You can pass either:
+ - A custom tokenizer object.
+ - A string, the *model id* of a predefined tokenizer hosted inside a model repo on huggingface.co.
+ - A path to a *directory* containing vocabulary files required by the tokenizer, for instance saved
+ using the [`~PreTrainedTokenizer.save_pretrained`] method, e.g., `./my_model_directory/`.
+ """
+
+ def __init__(
+ self,
+ bits: int,
+ tokenizer: Any = None,
+ dataset: str = None,
+ group_size: int = 128,
+ sym: bool = False,
+ backend="gptq:exllamav2",
+ iters: int = 200,
+ weight_config: dict = None,
+ enable_quanted_input=True,
+ enable_minmax_tuning=True,
+ lr=None,
+ minmax_lr=None,
+ n_samples=512,
+ seqlen=2048,
+ **kwargs,
+ ):
+ self.bits = bits
+ self.tokenizer = tokenizer
+ self.dataset = dataset
+ self.group_size = group_size
+ self.sym = sym
+ self.backend = backend
+ self.inters = iters
+ self.weight_config = weight_config
+ self.enable_quanted_input = enable_quanted_input
+ self.enable_minmax_tuning = enable_minmax_tuning
+ self.lr = lr
+ self.minmax_lr = minmax_lr
+ self.n_samples = n_samples
+ self.seqlen = seqlen
+ if kwargs is not None:
+ for key in kwargs.keys():
+ setattr(self, key, kwargs[key])
+
+ self.post_init()
+
+ def get_loading_attributes(self):
+ pass
+ # attibutes_dict = copy.deepcopy(self.__dict__)
+ # loading_attibutes = ["disable_exllama", "use_exllama", "exllama_config", "use_cuda_fp16", "max_input_length"]
+ # loading_attibutes_dict = {i: j for i, j in attibutes_dict.items() if i in loading_attibutes}
+ # return loading_attibutes_dict
+
+ def post_init(self):
+ r"""Safety checker that arguments are correct."""
+ if self.bits not in [2, 3, 4, 8]:
+ raise ValueError(f"Only support quantization to [2,3,4,8] bits but found {self.bits}")
+ if self.group_size != -1 and self.group_size <= 0:
+ raise ValueError("group_size must be greater than 0 or equal to -1")
+ ##TODO add more check
+
+ def to_dict(self):
+ config_dict = super().to_dict()
+ config_dict.pop("disable_exllama", None)
+ return config_dict
+
+
+class AutoRoundQuantizer(HfQuantizer):
+ """
+ Quantizer of the GPTQ method - for GPTQ the quantizer support calibration of the model through
+ `auto_gptq` package. Quantization is done under the hood for users if they load a non-prequantized model.
+ """
+
+ requires_calibration = False
+ required_packages = ["auto_gptq"]
+ optimum_quantizer = None
+
+ def __init__(self, quantization_config: QuantizationConfigMixin, **kwargs):
+ super().__init__(quantization_config, **kwargs)
+
+ def validate_environment(self, *args, **kwargs):
+ gptq_supports_cpu = version.parse(importlib.metadata.version("auto-gptq")) > version.parse("0.4.2")
+ if not gptq_supports_cpu and not torch.cuda.is_available():
+ raise RuntimeError("GPU is required to quantize or run quantize model.")
+ elif not is_auto_gptq_available():
+ raise ImportError("Loading a GPTQ quantized model requires auto-gptq library (`pip install auto-gptq`)")
+ elif version.parse(importlib.metadata.version("auto_gptq")) < version.parse("0.4.2"):
+ raise ImportError("You need a version of auto_gptq >= 0.4.2 to use GPTQ: `pip install --upgrade auto-gptq`")
+
+ def update_torch_dtype(self, torch_dtype: "torch.dtype") -> "torch.dtype":
+ if torch_dtype is None:
+ torch_dtype = torch.float16
+ elif torch_dtype != torch.float16:
+ logger.info("We suggest you to set `torch_dtype=torch.float16` for better efficiency with AutoRound.")
+ return torch_dtype
+
+ def convert_model(self, model: nn.Module):
+ """Convert the model to a GPTQ model by getting and replacing the layers.
+
+ Args:
+ model (`nn.Module`):
+ Model to be converted
+ """
+ from .export_to_autoround import get_layer_names_in_block
+
+ layer_names = get_layer_names_in_block(model)
+ quantization_config = model.config.quantization_config
+ bits = quantization_config.bits
+ group_size = quantization_config.group_size
+ data_type = quantization_config.data_type
+ sym = quantization_config.sym
+ extra_config = {}
+ if hasattr(quantization_config, "extra_config"):
+ extra_config = quantization_config.extra_config
+ layer_names += extra_config.keys()
+ layer_names = list(set(layer_names))
+ layer_configs = {}
+ for layer_name in layer_names:
+ layer_configs[layer_name] = {}
+ if layer_name not in extra_config:
+ layer_configs[layer_name]["bits"] = bits
+ layer_configs[layer_name]["group_size"] = group_size
+ layer_configs[layer_name]["data_type"] = data_type
+ layer_configs[layer_name]["sym"] = sym
+ else:
+ layer_configs[layer_name]["bits"] = extra_config.get("bits", bits)
+ layer_configs[layer_name]["group_size"] = extra_config.get("group_size", group_size)
+ layer_configs[layer_name]["data_type"] = extra_config.get("data_type", data_type)
+ layer_configs[layer_name]["sym"] = extra_config.get("sym", sym)
+ backend = quantization_config.backend
+
+ self._replace_by_quant_layers(model, layer_configs, backend)
+ return model
+
+ def _replace_by_quant_layers(self, module: nn.Module, layer_configs, backend):
+ """Replaces linear layers in `module` by `QuantLinear`
+
+ Args:
+ module (`nn.Module`):
+ Module to quantize
+ names (`List[str]`):
+ List of names of the module to quantize
+ name (`str`, defaults to `""`):
+ To keep track of the name of the current module
+ """
+ for layer_name in layer_configs.keys():
+ config = layer_configs[layer_name]
+ bits = config["bits"]
+ group_size = config["group_size"]
+ data_type = config["data_type"]
+ if not (bits <= 8 and data_type == "int"):
+ continue
+ from .export_to_autoround import get_autogptq_backend_config
+
+ use_triton, disable_exllama, disable_exllamav2, use_qigen, disable_marlin = get_autogptq_backend_config(
+ backend, bits
+ )
+ QuantLinear = dynamically_import_QuantLinear(
+ use_triton=False,
+ desc_act=False,
+ group_size=group_size,
+ bits=bits,
+ disable_exllama=True,
+ disable_exllamav2=False,
+ use_qigen=use_qigen,
+ disable_marlin=disable_marlin,
+ )
+ layer = get_module(module, layer_name)
+ device = get_device(layer)
+ if isinstance(layer, nn.Linear):
+ in_features = layer.in_features
+ out_features = layer.out_features
+ elif isinstance(layer, nn.Conv2d): ##not supported now
+ in_features = layer.in_channels
+ out_features = layer.out_channels
+ elif isinstance(layer, Conv1D): ##TODO need to have a check
+ in_features = layer.weight.shape[0]
+ out_features = layer.weight.shape[1]
+ bias = layer.bias is not None
+ new_layer = QuantLinear(
+ bits,
+ group_size,
+ in_features,
+ out_features,
+ bias,
+ weight_dtype=layer.weight.dtype, # pylint: disable=E1123
+ )
+
+ new_layer.device = device
+ set_module(module, layer_name, new_layer)
+
+ def post_init_model(self, model):
+ """Post-initialization that require device information, for example buffers initialization on device.
+
+ Args:
+ model (`nn.Module`):
+ The input model
+ """
+
+ # if self.bits == 4 and not self.disable_exllama:
+ # if get_device(model) == torch.device("cpu") or (
+ # hasattr(model, "hf_device_map") and any(d in model.hf_device_map for d in ["cpu", "disk"])
+ # ):
+ # raise ValueError(
+ # "Found modules on cpu/disk. Using Exllama
+ # or Exllamav2 backend requires all the modules to be on GPU."
+ # "You can deactivate exllama backend by
+ # setting `disable_exllama=True` in the quantization config object"
+ # )
+
+ class StoreAttr(object):
+ pass
+
+ model.quantize_config = StoreAttr()
+ model.quantize_config.desc_act = False
+ model = autogptq_post_init(model, use_act_order=False)
+ return model
+
+ def _process_model_before_weight_loading(self, model: "PreTrainedModel", **kwargs):
+ if model.__class__.main_input_name != "input_ids":
+ raise RuntimeError("We can only quantize pure text model.")
+
+ if self.pre_quantized:
+ model = self.convert_model(model)
+
+ def _process_model_after_weight_loading(self, model: "PreTrainedModel", **kwargs):
+ if self.pre_quantized:
+ model = self.post_init_model(model)
+ else:
+ raise NotImplementedError
+ # if self.quantization_config.tokenizer is None:
+ # self.quantization_config.tokenizer = model.name_or_path
+ #
+ # self.optimum_quantizer.quantize_model(model, self.quantization_config.tokenizer)
+ # model.config.quantization_config = GPTQConfig.from_dict(self.optimum_quantizer.to_dict())
+
+ @property
+ def is_trainable(self, model: Optional["PreTrainedModel"] = None):
+ return True
+
+ @property
+ def is_serializable(self):
+ return True
diff --git a/auto_round/export/export_to_autoround/export_to_autoround.py b/auto_round/export/export_to_autoround/export_to_autoround.py
new file mode 100644
index 000000000..fab55a26c
--- /dev/null
+++ b/auto_round/export/export_to_autoround/export_to_autoround.py
@@ -0,0 +1,217 @@
+# Copyright (c) 2024 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+
+import copy
+import json
+import os
+
+import torch
+import torch.nn as nn
+import transformers
+
+from auto_round.export.register import register_format
+from auto_round.utils import get_block_names, get_module, logger, set_module
+
+
+def get_layer_names_in_block(model, supported_types=[torch.nn.Linear, transformers.modeling_utils.Conv1D]):
+ """Retrieves the names of layers within each block of the model.
+
+ Returns:
+ list: A list of strings, where each string is the name of a layer
+ within a block of the model.
+ """
+ for n, m in model.named_modules():
+ if isinstance(m, tuple(supported_types)):
+ m.tmp_name = n
+ layers_in_block = []
+ block_names = get_block_names(model)
+ for block_name in block_names:
+ block = get_module(model, block_name)
+ for n, m in block.named_modules():
+ if hasattr(m, "tmp_name"):
+ layers_in_block.append(m.tmp_name)
+ for n, m in model.named_modules():
+ if hasattr(m, "tmp_name"):
+ delattr(m, "tmp_name")
+ return layers_in_block
+
+
+def check_neq_config(config, data_type, bits, group_size, sym):
+ res = []
+ if data_type != config["data_type"]:
+ res.append("data_type")
+ return res
+ if bits != config["bits"]:
+ res.append("bits")
+ if group_size != config["group_size"]:
+ res.append("group_size")
+ if sym != config["sym"]:
+ res.append("sym")
+ return res
+
+
+def get_autogptq_backend_config(backend, bits=4):
+ use_triton = False
+ disable_exllamav2 = False
+ disable_exllamav1 = False
+ disable_marlin = True
+ use_qigen = False
+ if backend == "gptq:qigen":
+ use_qigen = True
+ if backend == "gptq:triton": ##TODO refine the code
+ use_triton = True
+ if backend == "gptq:marlin":
+ use_triton = False
+ disable_marlin = True
+ if backend == "gptq:exllamav2":
+ use_triton = False
+ disable_marlin = True
+ if backend == "gptq:exllamav1":
+ use_triton = False
+ disable_marlin = True
+ if backend == "gptq:cuda":
+ use_triton = False
+ disable_marlin = True
+ disable_exllamav2 = True
+ disable_exllamav1 = True
+ if bits not in [2, 4, 8]:
+ use_qigen = False
+ if bits not in [2, 4]:
+ use_triton = False
+ return use_triton, disable_exllamav1, disable_exllamav2, use_qigen, disable_marlin
+
+
+@register_format("autoround")
+def save_quantized_as_autoround(output_dir, inplace=True, backend="gptq:exllamav2", **kwargs):
+ from auto_gptq.utils.import_utils import dynamically_import_QuantLinear
+
+ model = kwargs["model"]
+ if not inplace:
+ model = copy.deepcopy(model.to("cpu"))
+ layer_names_in_block = get_layer_names_in_block(model)
+
+ weight_config = kwargs["weight_config"]
+ for name in weight_config.keys():
+
+ config = kwargs["weight_config"][name]
+ if config["data_type"] != "int" and config["bits"] >= 16:
+ continue
+ logger.info(f"packing {name}")
+
+ bits = config["bits"]
+ group_size = config["group_size"]
+ use_triton, disable_exllamav1, disable_exllamav2, use_qigen, disable_marlin = get_autogptq_backend_config(
+ backend, bits
+ )
+
+ layer = get_module(model, name)
+ device = "cpu"
+ QuantLinear = dynamically_import_QuantLinear(
+ use_triton=use_triton,
+ desc_act=False,
+ group_size=group_size,
+ bits=bits,
+ disable_exllama=disable_exllamav1,
+ disable_exllamav2=disable_exllamav2,
+ use_qigen=use_qigen,
+ disable_marlin=disable_marlin,
+ )
+
+ if isinstance(layer, nn.Linear):
+ in_features = layer.in_features
+ out_features = layer.out_features
+ elif isinstance(layer, nn.Conv2d):
+ in_features = layer.in_channels
+ out_features = layer.out_channels
+ elif isinstance(layer, transformers.pytorch_utils.Conv1D):
+ in_features = layer.weight.shape[0]
+ out_features = layer.weight.shape[1]
+ bias = layer.bias is not None and torch.any(layer.bias)
+
+ new_layer = QuantLinear(
+ bits, group_size, in_features, out_features, bias, weight_dtype=layer.weight.dtype ##pylint: disable=E1123
+ )
+
+ new_layer.device = device
+ set_module(model, name, new_layer)
+ qlayer = new_layer
+ scale = weight_config[name]["scale"]
+ zero = weight_config[name]["zp"]
+ # so far can only pack layer on CPU
+ qlayer.to("cpu")
+ layer, scale, zero = layer.to("cpu"), scale.to("cpu"), zero.to("cpu")
+ qlayer.pack(layer, scale, zero, None)
+ qlayer.to(device)
+ quantization_config = kwargs["serialization_dict"]
+ quantization_config["quant_method"] = "intel/auto-round"
+ quantization_config["backend"] = backend
+ extra_config = {}
+ for layer_name in weight_config:
+ if weight_config[layer_name]["data_type"] != "int" and weight_config[layer_name]["bits"] >= 16:
+ continue
+ if layer_name not in layer_names_in_block:
+ extra_config[layer_name] = {}
+ extra_config[layer_name]["bits"] = weight_config[layer_name]["bits"]
+ extra_config[layer_name]["data_type"] = weight_config[layer_name]["data_type"]
+ extra_config[layer_name]["group_size"] = weight_config[layer_name]["group_size"]
+ extra_config[layer_name]["sym"] = weight_config[layer_name]["sym"]
+ else:
+ neq_keys = check_neq_config(
+ weight_config[layer_name],
+ data_type=quantization_config["data_type"],
+ bits=quantization_config["bits"],
+ group_size=quantization_config["group_size"],
+ sym=quantization_config["sym"],
+ )
+ if len(neq_keys) > 0:
+ extra_config[layer_name] = {}
+ for key in neq_keys:
+ extra_config[layer_name][key] = weight_config[layer_name][key]
+ if len(extra_config) > 0:
+ quantization_config["extra_config"] = extra_config
+ if hasattr(model, "config"):
+ model.config.quantization_config = quantization_config
+ tokenizer = kwargs["tokenizer"]
+ if tokenizer is not None:
+ tokenizer.save_pretrained(output_dir)
+ save(model, output_dir)
+
+
+def save(model: nn.Module, save_dir: str, max_shard_size: str = "10GB", safe_serialization: bool = True):
+ """Save model state dict and configs.
+
+ Args:
+ model (`nn.Module`):
+ Model to be saved. The model can be wrapped or unwrapped.
+ save_dir (`str`):
+ Directory to which to save. Will be created if it doesn't exist.
+ max_shard_size (`str`, defaults to `"10GB"`):
+ The maximum size for a checkpoint before being sharded. Checkpoints shard will then be each of size
+ lower than this size. If expressed as a string, needs to be digits followed by a unit (like `"5MB"`).
+ <Tip warning={true}>
+
+ If a single weight of the model is bigger than `max_shard_size`, it will be in its own checkpoint shard
+ which will be bigger than `max_shard_size`.
+
+ </Tip>
+ safe_serialization (`bool`, defaults to `True`):
+ Whether to save the model using `safetensors` or the traditional PyTorch way (that uses `pickle`).
+ """
+ os.makedirs(save_dir, exist_ok=True)
+ model.save_pretrained(save_dir, max_shard_size=max_shard_size, safe_serialization=safe_serialization)
+ config_file = "quantize_config.json"
+ if hasattr(model, "config") and hasattr(model.config, "quantize_config"):
+ with open(os.path.join(save_dir, config_file), "w", encoding="utf-8") as f:
+ json.dump(model.config.quantization_config, f, indent=2)
diff --git a/docs/Mistral-7B-v0.1-acc.md b/docs/Mistral-7B-v0.1-acc.md
index 58ddfd1d9..8f2a93c6a 100644
--- a/docs/Mistral-7B-v0.1-acc.md
+++ b/docs/Mistral-7B-v0.1-acc.md
@@ -1,4 +1,4 @@
-Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id f3b7917091afba325af3980a35d8a6dcba03dc3f
+Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used 0.4.2
```bash
lm_eval --model hf --model_args pretrained="Intel/Mistral-7B-v0.1-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32
@@ -6,17 +6,17 @@ lm_eval --model hf --model_args pretrained="Intel/Mistral-7B-v0.1-int4-inc",auto
-| Metric | BF16 | INT4 |
-| -------------- |--------| ------ |
-| Avg. | 0.6306 | 0.6308 |
-| mmlu | 0.5961 | 0.5880 |
-| lambada_openai | 0.7561 | 0.7551 |
-| hellaswag | 0.6128 | 0.6079 |
-| winogrande | 0.7443 | 0.7451 |
-| piqa | 0.8079 | 0.8014 |
-| truthfulqa_mc1 | 0.2803 | 0.2889 |
-| openbookqa | 0.3280 | 0.3300 |
-| boolq | 0.8373 | 0.8278 |
-| rte | 0.6643 | 0.6968 |
-| arc_easy | 0.8085 | 0.8060 |
-| arc_challenge | 0.5009 | 0.4915 |
+
+| Metric | BF16 | [INT4-lmhead](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc-lmhead) | [INT4](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) |
+| -------------- | ------ |-----------------| ------------------------------------------------------------ |
+| Avg. | 0.6260 | 0.6228 | 0.6218 |
+| mmlu | 0.5868 | 0.5760 | 0.5772 |
+| lambada_openai | 0.7555 | 0.7539 | 0.7543 |
+| hellaswag | 0.6125 | 0.6055 | 0.6072 |
+| winogrande | 0.7395 | 0.7380 | 0.7388 |
+| piqa | 0.8069 | 0.8009 | 0.8030 |
+| truthfulqa_mc1 | 0.2803 | 0.2876 | 0.2864 |
+| openbookqa | 0.3280 | 0.3300 | 0.3260 |
+| boolq | 0.8379 | 0.8291 | 0.8281 |
+| arc_easy | 0.8089 | 0.8043 | 0.8035 |
+| arc_challenge | 0.5034 | 0.5026 | 0.4932 |
diff --git a/examples/language-modeling/eval_042/__init__.py b/examples/language-modeling/eval_042/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/examples/language-modeling/eval_042/evaluation.py b/examples/language-modeling/eval_042/evaluation.py
new file mode 100644
index 000000000..d05847316
--- /dev/null
+++ b/examples/language-modeling/eval_042/evaluation.py
@@ -0,0 +1,602 @@
+import itertools
+import logging
+import random
+import time
+from collections import defaultdict
+from typing import TYPE_CHECKING, List, Optional, Union
+
+import numpy as np
+import torch
+
+import lm_eval.api.metrics
+import lm_eval.api.registry
+import lm_eval.models
+from lm_eval.caching.cache import delete_cache
+from lm_eval.evaluator_utils import (
+ consolidate_results,
+ get_sample_size,
+ get_task_list,
+ prepare_print_tasks,
+ print_writeout,
+ run_task_tests,
+)
+from lm_eval.logging_utils import add_env_info, get_git_commit_hash
+from lm_eval.tasks import TaskManager, get_task_dict
+from lm_eval.utils import eval_logger, positional_deprecated, simple_parse_args_string
+
+if TYPE_CHECKING:
+ from lm_eval.api.model import LM
+ from lm_eval.tasks import Task
+
+
+@positional_deprecated
+def simple_evaluate(
+ model,
+ model_args: Optional[Union[str, dict]] = None,
+ tasks: Optional[List[Union[str, dict, object]]] = None,
+ num_fewshot: Optional[int] = None,
+ batch_size: Optional[int] = None,
+ max_batch_size: Optional[int] = None,
+ device: Optional[str] = None,
+ use_cache: Optional[str] = None,
+ cache_requests: bool = False,
+ rewrite_requests_cache: bool = False,
+ delete_requests_cache: bool = False,
+ limit: Optional[Union[int, float]] = None,
+ bootstrap_iters: int = 100000,
+ check_integrity: bool = False,
+ write_out: bool = False,
+ log_samples: bool = True,
+ gen_kwargs: Optional[str] = None,
+ task_manager: Optional[TaskManager] = None,
+ verbosity: str = "INFO",
+ predict_only: bool = False,
+ random_seed: int = 0,
+ numpy_random_seed: int = 1234,
+ torch_random_seed: int = 1234,
+):
+ """Instantiate and evaluate a model on a list of tasks.
+
+ :param model: Union[str, LM]
+ Name of model or LM object, see lm_eval.models.get_model
+ :param model_args: Optional[str, dict]
+ String or dict arguments for each model class, see LM.create_from_arg_string and LM.create_from_arg_object.
+ Ignored if `model` argument is a LM object.
+ :param tasks: list[Union[str, dict, Task]]
+ List of task names or Task objects. Task objects will be taken to have name task.EVAL_HARNESS_NAME if defined and type(task).__name__ otherwise.
+ :param num_fewshot: int
+ Number of examples in few-shot context
+ :param batch_size: int or str, optional
+ Batch size for model
+ :param max_batch_size: int, optional
+ Maximal batch size to try with automatic batch size detection
+ :param device: str, optional
+ PyTorch device (e.g. "cpu" or "cuda:0") for running models
+ :param use_cache: str, optional
+ A path to a sqlite db file for caching model responses. `None` if not caching.
+ :param cache_requests: bool, optional
+ Speed up evaluation by caching the building of dataset requests. `None` if not caching.
+ :param rewrite_requests_cache: bool, optional
+ Rewrites all of the request cache if set to `True`. `None` if not desired.
+ :param delete_requests_cache: bool, optional
+ Deletes all of the request cache if set to `True`. `None` if not desired.
+ :param limit: int or float, optional
+ Limit the number of examples per task (only use this for testing), If <1, limit is a percentage of the total number of examples.
+ :param bootstrap_iters:
+ Number of iterations for bootstrap statistics
+ :param check_integrity: bool
+ Whether to run the relevant part of the test suite for the tasks
+ :param write_out: bool
+ If True, write out an example document and model input for checking task integrity
+ :param log_samples: bool
+ If True, write out all model outputs and documents for per-sample measurement and post-hoc analysis
+ :param gen_kwargs: str
+ String arguments for model generation
+ Ignored for all tasks with loglikelihood output_type
+ :param predict_only: bool
+ If true only model outputs will be generated and returned. Metrics will not be evaluated
+ :param random_seed: int
+ Random seed for python's random module. If set to None, the seed will not be set.
+ :param numpy_random_seed: int
+ Random seed for numpy. If set to None, the seed will not be set.
+ :param torch_random_seed: int
+ Random seed for torch. If set to None, the seed will not be set.
+
+ :return
+ Dictionary of results
+ """
+ eval_logger.setLevel(getattr(logging, f"{verbosity}"))
+ start_date = time.time()
+
+ if delete_requests_cache:
+ eval_logger.info("Deleting requests cache...")
+ delete_cache()
+
+ seed_message = []
+ if random_seed is not None:
+ # See https://github.com/EleutherAI/lm-evaluation-harness/pull/1412
+ seed_message.append(f"Setting random seed to {random_seed}")
+ random.seed(random_seed)
+
+ if numpy_random_seed is not None:
+ seed_message.append(f"Setting numpy seed to {numpy_random_seed}")
+ np.random.seed(numpy_random_seed)
+
+ if torch_random_seed is not None:
+ seed_message.append(f"Setting torch manual seed to {torch_random_seed}")
+ torch.manual_seed(torch_random_seed)
+
+ if seed_message:
+ eval_logger.info(" | ".join(seed_message))
+
+ if tasks is None:
+ tasks = []
+ if len(tasks) == 0:
+ raise ValueError(
+ "No tasks specified, or no tasks found. Please verify the task names."
+ )
+
+ if gen_kwargs is not None:
+ gen_kwargs = simple_parse_args_string(gen_kwargs)
+ eval_logger.warning(
+ "generation_kwargs specified through cli, these settings will update set parameters in yaml tasks. "
+ "Ensure 'do_sample=True' for non-greedy decoding!"
+ )
+ if gen_kwargs == "":
+ gen_kwargs = None
+
+ if isinstance(model, str):
+ if model_args is None:
+ model_args = ""
+
+ if isinstance(model_args, dict):
+ lm = lm_eval.api.registry.get_model(model).create_from_arg_obj(
+ model_args,
+ {
+ "batch_size": batch_size,
+ "max_batch_size": max_batch_size,
+ "device": device,
+ },
+ )
+
+ else:
+ lm = lm_eval.api.registry.get_model(model).create_from_arg_string(
+ model_args,
+ {
+ "batch_size": batch_size,
+ "max_batch_size": max_batch_size,
+ "device": device,
+ },
+ )
+ else:
+ if not isinstance(model, lm_eval.api.model.LM):
+ raise TypeError
+ lm = model
+
+ if use_cache is not None:
+ eval_logger.info(f"Using cache at {use_cache + '_rank' + str(lm.rank) + '.db'}")
+ lm = lm_eval.api.model.CachingLM(
+ lm,
+ use_cache
+ # each rank receives a different cache db.
+ # necessary to avoid multiple writes to cache at once
+ + "_rank"
+ + str(lm.rank)
+ + ".db",
+ )
+
+ if task_manager is None:
+ task_manager = TaskManager(verbosity)
+
+ task_dict = get_task_dict(tasks, task_manager)
+ for task_name in task_dict.keys():
+ task_obj = task_dict[task_name]
+ if isinstance(task_obj, tuple):
+ _, task_obj = task_obj
+ if task_obj is None:
+ continue
+
+ if task_obj.get_config("output_type") == "generate_until":
+ if gen_kwargs is not None:
+ task_obj.set_config(
+ key="generation_kwargs", value=gen_kwargs, update=True
+ )
+
+ if predict_only:
+ log_samples = True
+ eval_logger.info(
+ f"Processing {task_name} in output-only mode. Metrics will not be calculated!"
+ )
+ # we have to change the class properties post-hoc. This is pretty hacky.
+ task_obj.override_metric(metric_name="bypass")
+
+ # override tasks' fewshot values to the provided num_fewshot arg value
+ # except if tasks have it set to 0 manually in their configs--then we should never overwrite that
+ if num_fewshot is not None:
+ if (default_num_fewshot := task_obj.get_config("num_fewshot")) == 0:
+ eval_logger.info(
+ f"num_fewshot has been set to 0 for {task_name} in its config. Manual configuration will be ignored."
+ )
+ else:
+ eval_logger.warning(
+ f"Overwriting default num_fewshot of {task_name} from {default_num_fewshot} to {num_fewshot}"
+ )
+ task_obj.set_config(key="num_fewshot", value=num_fewshot)
+ else:
+ # if num_fewshot not provided, and the task does not define a default one, default to 0
+ if (default_num_fewshot := task_obj.get_config("num_fewshot")) is None:
+ task_obj.set_config(key="num_fewshot", value=0)
+
+ if check_integrity:
+ run_task_tests(task_list=tasks)
+
+ results = evaluate(
+ lm=lm,
+ task_dict=task_dict,
+ limit=limit,
+ cache_requests=cache_requests,
+ rewrite_requests_cache=rewrite_requests_cache,
+ bootstrap_iters=bootstrap_iters,
+ write_out=write_out,
+ log_samples=log_samples,
+ verbosity=verbosity,
+ )
+
+ if lm.rank == 0:
+ if isinstance(model, str):
+ model_name = model
+ elif hasattr(model, "config") and hasattr(model.config, "_name_or_path"):
+ model_name = model.config._name_or_path
+ else:
+ model_name = type(model).__name__
+
+ # add info about the model and few shot config
+ results["config"] = {
+ "model": model_name,
+ "model_args": model_args,
+ "batch_size": batch_size,
+ "batch_sizes": (
+ list(lm.batch_sizes.values()) if hasattr(lm, "batch_sizes") else []
+ ),
+ "device": device,
+ "use_cache": use_cache,
+ "limit": limit,
+ "bootstrap_iters": bootstrap_iters,
+ "gen_kwargs": gen_kwargs,
+ }
+ results["git_hash"] = get_git_commit_hash()
+ results["date"] = start_date
+ add_env_info(results) # additional environment info to results
+ return results
+ else:
+ return None
+
+
+@positional_deprecated
+def evaluate(
+ lm: "LM",
+ task_dict,
+ limit: Optional[int] = None,
+ cache_requests: bool = False,
+ rewrite_requests_cache: bool = False,
+ bootstrap_iters: Optional[int] = 100000,
+ write_out: bool = False,
+ log_samples: bool = True,
+ verbosity: str = "INFO",
+):
+ """Instantiate and evaluate a model on a list of tasks.
+
+ :param lm: obj
+ Language Model
+ :param task_dict: dict[str, Task]
+ Dictionary of tasks. Tasks will be taken to have name type(task).config.task .
+ :param limit: int, optional
+ Limit the number of examples per task (only use this for testing)
+ :param bootstrap_iters:
+ Number of iterations for bootstrap statistics
+ :param write_out: bool
+ If True, write out an example document and model input for checking task integrity
+ :param log_samples: bool
+ If True, write out all model outputs and documents for per-sample measurement and post-hoc analysis
+ :return
+ Dictionary of results
+ """
+
+ eval_logger.setLevel(getattr(logging, f"{verbosity}"))
+
+ # tracks all Instances/requests a model must generate output on.
+ requests = defaultdict(list)
+ # stores the amount to pad out reqs per req. type so that
+ # number of fwd passes per distributed rank is equal
+ padding_requests = defaultdict(int)
+
+ # get lists of group hierarchy and each type of request
+ task_hierarchy, eval_tasks = get_task_list(task_dict)
+ if not log_samples:
+ if not all(
+ "bypass" not in getattr(task_output.task, "_metric_fn_list", {}).keys()
+ for task_output in eval_tasks
+ ):
+ raise ValueError("log_samples must be True for 'bypass' metric-only tasks")
+ for task_output in eval_tasks:
+ task: Task = task_output.task
+ limit = get_sample_size(task, limit)
+ task.build_all_requests(
+ limit=limit,
+ rank=lm.rank,
+ world_size=lm.world_size,
+ cache_requests=cache_requests,
+ rewrite_requests_cache=rewrite_requests_cache,
+ )
+ eval_logger.debug(
+ f"Task: {task_output.task_name}; number of requests on this rank: {len(task.instances)}"
+ )
+
+ if write_out:
+ print_writeout(task)
+ # aggregate Instances by LM method requested to get output.
+ for instance in task.instances:
+ reqtype = instance.request_type
+ requests[reqtype].append(instance)
+
+ if lm.world_size > 1:
+ instances_rnk = torch.tensor(len(task._instances), device=lm.device)
+ gathered_item = (
+ lm.accelerator.gather(instances_rnk).cpu().detach().numpy().tolist()
+ )
+ # "multiple_choice" task types dispatch (several) "loglikelihood" request types
+ reqtype = (
+ "loglikelihood"
+ if task.OUTPUT_TYPE == "multiple_choice"
+ else task.OUTPUT_TYPE
+ )
+ # compute number of pseudo-batches to pad with (FSDP/DDP require even batches among ranks)
+ numpad = max(gathered_item) - gathered_item[lm.rank]
+ # todo: may not account for padding in cases like SquadV2 which has multiple req types
+ padding_requests[reqtype] += numpad
+
+ ### Run LM on inputs, get all outputs ###
+ # execute each type of request
+ for reqtype, reqs in requests.items():
+ eval_logger.info(f"Running {reqtype} requests")
+ # create `K` copies of each request `req` based off `K = req.repeats`
+ cloned_reqs = []
+ for req in reqs:
+ cloned_reqs.extend([req] * req.repeats)
+
+ if (lm.world_size > 1) and (padding_requests[reqtype] > 0):
+ for _ in range(padding_requests[reqtype]):
+ cloned_reqs.extend([req] * req.repeats)
+
+ # run requests through model
+ resps = getattr(lm, reqtype)(cloned_reqs)
+
+ # put responses from model into a list of length K for each request.
+ for x, req in zip(resps, cloned_reqs):
+ req.resps.append(x)
+
+ if lm.world_size > 1:
+ lm.accelerator.wait_for_everyone()
+
+ RANK = lm.rank
+ WORLD_SIZE = lm.world_size
+ ### Postprocess outputs ###
+ # TODO: del model here, maybe (idea: allow user to specify device of e.g. reward model separately)
+ for task_output in eval_tasks:
+ task = task_output.task
+ task.apply_filters()
+
+ ### Collect values of metrics on all datapoints ###
+ # # unpack results and sort back in order and return control to Task
+ # TODO: make it possible to use a different metric per filter
+ # Pre-process task.instances to group by doc_id
+ instances_by_doc_id = defaultdict(list)
+ for instance in task.instances:
+ instances_by_doc_id[instance.doc_id].append(instance)
+ # Sort instances within each group
+ for instances in instances_by_doc_id.values():
+ instances.sort(key=lambda x: x.idx)
+ # iterate over different filters used
+ for filter_key in task.instances[0].filtered_resps.keys():
+ doc_iterator = task.doc_iterator(
+ rank=RANK, limit=limit, world_size=WORLD_SIZE
+ )
+ for doc_id, doc in doc_iterator:
+ requests = instances_by_doc_id[doc_id]
+ metrics = task.process_results(
+ doc, [req.filtered_resps[filter_key] for req in requests]
+ )
+ if log_samples:
+ target = task.doc_to_target(doc)
+ example = {
+ "doc_id": doc_id,
+ "doc": doc,
+ "target": target,
+ "arguments": [req.args for req in requests],
+ "resps": [req.resps for req in requests],
+ "filtered_resps": [
+ req.filtered_resps[filter_key] for req in requests
+ ],
+ }
+ example.update(metrics)
+ task_output.logged_samples.append(example)
+ for metric, value in metrics.items():
+ task_output.sample_metrics[(metric, filter_key)].append(value)
+
+ if WORLD_SIZE > 1:
+ # if multigpu, then gather data across all ranks to rank 0
+ # first gather logged samples across all ranks
+ for task_output in eval_tasks:
+ if log_samples:
+ # for task_name, task_samples in list(samples.items()):
+ full_samples = [None] * WORLD_SIZE if RANK == 0 else None
+ torch.distributed.gather_object(
+ obj=task_output.logged_samples,
+ object_gather_list=full_samples,
+ dst=0,
+ )
+
+ if RANK == 0:
+ task_output.logged_samples = list(
+ itertools.chain.from_iterable(full_samples)
+ )
+
+ # then collect metrics across all ranks
+ for metrics in task_output.sample_metrics:
+ metric_list = [None] * WORLD_SIZE if RANK == 0 else None
+ torch.distributed.gather_object(
+ obj=task_output.sample_metrics[metrics],
+ object_gather_list=metric_list,
+ dst=0,
+ )
+ if RANK == 0:
+ task_output.sample_metrics[metrics] = list(
+ itertools.chain.from_iterable(metric_list)
+ )
+
+ if RANK == 0:
+ ### Aggregate results over all datapoints ###
+ # aggregate results ; run bootstrap CIs
+ for task_output in eval_tasks:
+ task_output.calculate_aggregate_metric(bootstrap_iters=bootstrap_iters)
+ results, samples, configs, versions, num_fewshot = consolidate_results(
+ eval_tasks
+ )
+
+ ### Calculate group metrics ###
+ if bool(results):
+ for group, task_list in reversed(task_hierarchy.items()):
+ if len(task_list) == 0:
+ # task_hierarchy entries are either
+ # `group_name: [subtask1, subtask2, ...]`
+ # or `task_name: []`.
+ # we only want to operate on groups here.
+ continue
+ metric_list = list(
+ {
+ key
+ for task in task_list
+ for key in results[task].keys()
+ if "_stderr" not in key and key not in ["alias", "samples"]
+ }
+ )
+ for metric in metric_list:
+ stderr = "_stderr,".join(metric.split(","))
+
+ # gather metrics, sizes, and stderrs from subtasks
+ metrics = [
+ results[task][metric]
+ for task in task_list
+ if metric in results[task]
+ ] # TODO: copy?
+ stderrs = [
+ results[task][stderr]
+ for task in task_list
+ if stderr in results[task]
+ ]
+ sizes = [
+ results[task]["samples"]
+ for task in task_list
+ if metric in results[task]
+ ]
+
+ # compute group's pooled metric and stderr
+ results[group][
+ metric
+ ] = lm_eval.api.metrics.aggregate_subtask_metrics(metrics, sizes)
+ # TODO: calculate grouped metric using aggregation fn
+ if "N/A" in stderrs:
+ results[group][stderr] = "N/A"
+ else:
+ results[group][
+ stderr
+ ] = lm_eval.api.metrics.pooled_sample_stderr(stderrs, sizes)
+ # TODO: allow GroupConfigs to choose which variance formula is used, for back-compatibility
+ # To use the old (likely incorrect) variance formula, comment out the above and uncomment this line:
+ # results[group][stderr] = lm_eval.api.metrics.combined_sample_stderr(stderrs, sizes, metrics=metrics)
+
+ results[group]["samples"] = sum(sizes)
+
+ results_agg = defaultdict(dict)
+ groups_agg = defaultdict(dict)
+ all_tasks_list = list(task_hierarchy.keys())
+ while True:
+ add_tasks_list = list(k for k in results_agg.keys())
+ left_tasks_list = sorted(list(set(all_tasks_list) - set(add_tasks_list)))
+ if len(left_tasks_list) == 0:
+ break
+
+ _task_hierarchy = {
+ k: v for k, v in task_hierarchy.items() if k in left_tasks_list
+ }
+ _results_agg, _groups_agg = prepare_print_tasks(_task_hierarchy, results)
+
+ results_agg = {**results_agg, **_results_agg}
+ groups_agg = {**groups_agg, **_groups_agg}
+
+ for group_name, task_list in task_hierarchy.items():
+ if task_list:
+ num_fewshot[group_name] = num_fewshot[
+ task_list[0]
+ ] # TODO: validate this
+
+ results_dict = {
+ "results": dict(results_agg.items()),
+ **({"groups": dict(groups_agg.items())} if bool(groups_agg) else {}),
+ "group_subtasks": dict(reversed(task_hierarchy.items())),
+ "configs": dict(sorted(configs.items())),
+ "versions": dict(sorted(versions.items())),
+ "n-shot": dict(sorted(num_fewshot.items())),
+ }
+ if log_samples:
+ results_dict["samples"] = dict(samples)
+
+ return results_dict
+
+ else:
+ return None
+
+
+if __name__ == "__main__":
+
+ import sys
+
+ sys.path.insert(0, '../../')
+ import time
+ import argparse
+
+ parser = argparse.ArgumentParser()
+ parser.add_argument(
+ "--model_name", default="/models/opt-125m/"
+ )
+ parser.add_argument(
+ "--eval_bs", default=1,
+ )
+ parser.add_argument("--tasks",
+ default="lambada_openai,hellaswag,winogrande,piqa,mmlu,truthfulqa_mc1," \
+ "truthfulqa_mc2,openbookqa,boolq,rte,arc_easy,arc_challenge",
+ help="lm-eval tasks for lm_eval version 0.4.2")
+
+ args = parser.parse_args()
+ s = time.time()
+ from transformers import AutoConfig
+
+ config = AutoConfig.from_pretrained(args.model_name)
+ if hasattr(config, "quantization_config"):
+ quantization_config = config.quantization_config
+ if "quant_method" in quantization_config and "auto-round" in quantization_config["quant_method"]:
+ from auto_round.export.export_to_autoround import AutoHfQuantizer
+
+ test_tasks = args.tasks
+ if isinstance(test_tasks, str):
+ test_tasks = test_tasks.split(',')
+ model_name = args.model_name.rstrip('/')
+ from lm_eval.utils import make_table
+
+ result = simple_evaluate(model="hf",
+ model_args=f"pretrained={args.model_name}",
+ tasks=test_tasks,
+ batch_size=args.eval_bs)
+ print(make_table(result))
+
+ print("cost time: ", time.time() - s)
diff --git a/examples/language-modeling/main.py b/examples/language-modeling/main.py
index 90a83d93a..546d61cfb 100644
--- a/examples/language-modeling/main.py
+++ b/examples/language-modeling/main.py
@@ -265,6 +265,8 @@ if __name__ == '__main__':
print(
f"{n} will not be quantized due to its shape not being divisible by 32, resulting in an exporting issue to autogptq")
lm_head_layer_name = "lm_head"
+ for n, _ in model.named_modules():
+ lm_head_layer_name = n
if args.quant_lm_head:
from transformers import AutoConfig
@@ -303,8 +305,11 @@ if __name__ == '__main__':
output_dir = args.output_dir + "/" + model_name.split('/')[-1] + f"-autoround-w{args.bits}g{args.group_size}-qdq"
deployment_device = args.deployment_device.split(',')
inplace = True if len(deployment_device) < 2 else False
- if 'gpu' in deployment_device:
- autoround.save_quantized(f'{export_dir}-gpu', format="auto_gptq", use_triton=True, inplace=inplace)
+ if 'gpu' in deployment_device :
+ if lm_head_layer_name in weight_config.keys():
+ autoround.save_quantized(f'{export_dir}-gpu', format="autoround", use_triton=True, inplace=inplace)
+ else:
+ autoround.save_quantized(f'{export_dir}-gpu', format="auto_gptq", use_triton=True, inplace=inplace)
if 'xpu' in deployment_device:
autoround.save_quantized(f'{export_dir}-xpu', format="itrex_xpu", use_triton=True, inplace=inplace,
compression_dtype=torch.int8, compression_dim=0, use_optimum_format=False,