PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
[Experimental Feature]fast tuning norm/bias at 2 bits (#208)
wenhuach21459c0c5c3dc5
diff --git a/README.md b/README.mdindex 5bc67d3ea..3acc426a9 100644--- a/README.md+++ b/README.md@@ -11,9 +11,10 @@ AutoRound<div align="left">AutoRound is an advanced quantization algorithm for low-bits LLM inference. It's tailored for a wide range-of models. Our method adopts sign gradient descent to fine-tune rounding values and minmax values of weights in just 200 steps,+of models. Our method adopts sign gradient descent to fine-tune rounding values and minmax values of weights in just 200+steps,which competes impressively against recent methods without introducing any additional inference overhead. The below-image presents an overview of AutoRound. Check out our updated paper on [arxiv](https://arxiv.org/pdf/2309.05516v4)+image presents an overview of AutoRound. Check out our updated paper on [arxiv](https://arxiv.org/pdf/2309.05516v4)<div align="center">@@ -22,7 +23,8 @@ image presents an overview of AutoRound. Check out our updated paper on [arxiv]<div align="left">## What's New-* [2024/08] Enabled the export and inference of the quantized model to the AutoRound format on HPU devices, please refer to [Intel/Qwen2-7B-int4-inc](https://huggingface.co/Intel/Qwen2-7B-int4-inc) and [Intel/Qwen2-57B-A14B-Instruct-int4-inc](https://huggingface.co/Intel/Qwen2-57B-A14B-Instruct-int4-inc).++* [2024/08] Enabled the export and inference of the quantized model to the AutoRound format on Intel Gaudi2 devices, please refer to [Intel/Qwen2-7B-int4-inc](https://huggingface.co/Intel/Qwen2-7B-int4-inc).* [2024/07] Important change: the default value of nsamples has been changed from 512 to 128 to reduce the memory usages, which may cause a slight accuracy drop in some scenarios* [2024/06] AutoRound format supports mixed bit-widths and group sizes for inference, resolving the significant performance drop issue with the asymmetric kernel* [2024/05] AutoRound supports lm-head quantization, saving 0.7G for LLaMA3-8B at W4G128.@@ -69,7 +71,7 @@ bits, group_size, sym = 4, 128, Falseautoround = AutoRound(model, tokenizer, bits=bits, group_size=group_size, sym=sym)autoround.quantize()output_dir = "./tmp_autoround"-autoround.save_quantized(output_dir) ##save_quantized(output_dir,format="auto_gptq)+autoround.save_quantized(output_dir) ##save_quantized(output_dir,format="auto_gptq")```<details>@@ -133,38 +135,42 @@ autoround.save_quantized(output_dir) ##save_quantized(output_dir,format="auto_gp3 Setting 'minmax_lr' to 2.0/iters has been observed to occasionally yield improved results.## Model inference+Please run the quantization code first### Setup env for AutoRound format+**cuda**: git clone https://github.com/intel/auto-round.git && cd auto-round && pip install -vvv --no-build-isolation+-e .-**cuda**: git clone https://github.com/intel/auto-round.git && cd auto-round && pip install -vvv --no-build-isolation -e .+**cpu**:-**cpu**:* option 1: pip install auto-round && pip install intel-extension-for-transformers-* option 2: git clone https://github.com/intel/auto-round.git && cd auto-round && pip install -vvv --no-build-isolation -e .--**hpu**: docker image with Gaudi Software Stack is recommended. More details can be found in [Gaudi Guide](https://docs.habana.ai/en/latest/).+* option 2: git clone https://github.com/intel/auto-round.git && cd auto-round && pip install -vvv --no-build-isolation+ -e .+**hpu**: docker image with Gaudi Software Stack is recommended. More details can be found+in [Gaudi Guide](https://docs.habana.ai/en/latest/).### Gaudi2/ CPU/ GPU```python-from transformers import AutoModelForCausalLM,AutoTokenizer+from transformers import AutoModelForCausalLM, AutoTokenizerfrom auto_round import AutoRoundConfig-device = "auto" ##cpu, hpu, cuda++device = "auto" ##cpu, hpu, cudaquantization_config = AutoRoundConfig(- backend=device+ backend=device)quantized_model_path = "./tmp_autoround"-model = AutoModelForCausalLM.from_pretrained(quantized_model_path, device_map=device,quantization_config=quantization_config)+model = AutoModelForCausalLM.from_pretrained(quantized_model_path,+ device_map=device, quantization_config=quantization_config)tokenizer = AutoTokenizer.from_pretrained(quantized_model_path)text = "There is a girl who likes adventure,"inputs = tokenizer(text, return_tensors="pt").to(model.device)print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))```-## Support List| Model | Supported |@@ -181,7 +187,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))| mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) || mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) || meta-llama/Meta-Llama-3-8B-Instruct | [accuracy](./docs/Meta-Llama-3-8B-Instruct-acc.md), [recipe](./examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.sh), [example](./examples/language-modeling/) |-| google/gemma-7b | [accuracy](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh), [example](./examples/language-modeling/) |+| google/gemma-7b | [accuracy](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh), [example](./examples/language-modeling/) || meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) || Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [sym recipe](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh), [asym recipe ](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh), [example](./examples/language-modeling/) || baichuan-inc/Baichuan2-7B-Chat | [accuracy](./docs/baichuan2-7b-chat-acc.md), [recipe](./examples/language-modeling/scripts/baichuan2-7b-chat.sh), [example](./examples/language-modeling/) |@@ -198,15 +204,6 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))| databricks/dolly-v2-3b | [example](./examples/language-modeling/) || stabilityai/stablelm-base-alpha-3b | [example](./examples/language-modeling/)-## Comparison with other methods--We provide a [comprehensive analysis](docs/acc.md) with other methods in our accuracy data section. In summary, our-approach achieved superior performance compared to GPTQ, scoring 30/32, AWQ with 27/32, HQQ with 15/16, and OmniQuant-with a perfect score of 16/16 across llamv1/llamav2/mistral-7b on W4G-1, W4G128, W3G128, and W2G128, based on the-average accuracies of 11 zero-shot tasks.---## ReferenceIf you find AutoRound useful for your research, please cite our paper:diff --git a/auto_round/autoround.py b/auto_round/autoround.pyindex 72ce8a235..43766ad23 100644--- a/auto_round/autoround.py+++ b/auto_round/autoround.py@@ -23,15 +23,15 @@ from transformers import set_seedfrom torch import autocastfrom .calib_dataset import get_dataloader-from .quantizer import WrapperMultiblock, wrapper_block, unwrapper_block, WrapperLinear, unwrapper_layer+from .quantizer import WrapperMultiblock, wrapper_block, unwrapper_block, WrapperLinear, unwrapper_layer, \+ WrapperTransformerConv1dfrom .special_model_handler import check_hidden_state_dim, check_share_attention_mask, check_not_share_position_idsfrom .utils import (CpuInfo,block_forward,check_is_cpu,check_to_quantized,- collect_minmax_scale,- collect_round_v,+ collect_best_params,convert_dtype_str2torch,detect_device,get_block_names,@@ -147,6 +147,7 @@ class AutoRound(object):act_sym: bool = None,act_dynamic: bool = True,quant_block_list: list = None,+ enable_norm_bias_tuning: bool = False,**kwargs,):self.quantized = False@@ -164,6 +165,12 @@ class AutoRound(object):self.nsamples = nsamplesself.nblocks = nblocksself.bits = bits+ self.enable_norm_bias_tuning = enable_norm_bias_tuning+ if self.enable_norm_bias_tuning is None:+ if self.bits == 2:+ self.enable_norm_bias_tuning = True+ else:+ self.enable_norm_bias_tuning = Falseself.group_size = group_sizeself.sym = symself.low_gpu_mem_usage = low_gpu_mem_usage@@ -744,7 +751,12 @@ class AutoRound(object):if q_inputs is not None:q_inputs[i] = q_inputs[i].to(layer.weight.dtype)- wrapper_linear = WrapperLinear(layer, self.enable_minmax_tuning, device).to(device)+ if isinstance(layer, torch.nn.Linear):+ wrapper_linear = WrapperLinear(layer, enable_minmax_tuning=self.enable_minmax_tuning, device=device).to(+ device)+ else:+ wrapper_linear = WrapperTransformerConv1d(layer, enable_minmax_tuning=self.enable_minmax_tuning,+ device=device).to(device)round_params = []minmax_params = []round_params.append(wrapper_linear.value)@@ -769,7 +781,7 @@ class AutoRound(object):mse_loss = torch.nn.MSELoss().to(device)scaler = self.get_scaler() # pylint: disable=assignment-from-noneinit_loss = None- best_v, best_min_scale, best_max_scale = torch.tensor(0), torch.tensor(1.0), torch.tensor(1.0)+ # best_v, best_min_scale, best_max_scale = torch.tensor(0), torch.tensor(1.0), torch.tensor(1.0)gradient_accumulate_steps = self.train_bs ##Force to low gputrain_bs = 1 ##Force to low gpupick_samples = train_bs * gradient_accumulate_steps@@ -811,15 +823,10 @@ class AutoRound(object):if total_loss < best_loss:best_loss = total_lossif not self.not_use_best_mse:- best_v = copy.deepcopy(wrapper_linear.value.data)- best_min_scale = copy.deepcopy(torch.clamp(wrapper_linear.min_scale.data, 0, 1.0))- best_max_scale = copy.deepcopy(torch.clamp(wrapper_linear.max_scale.data, 0, 1.0))-+ best_params = collect_best_params(wrapper_linear)last_best_iter = iif self.not_use_best_mse and i == self.iters - 1:- best_v = copy.deepcopy(wrapper_linear.value.data)- best_min_scale = copy.deepcopy(torch.clamp(wrapper_linear.min_scale.data, 0, 1.0))- best_max_scale = copy.deepcopy(torch.clamp(wrapper_linear.max_scale.data, 0, 1.0))+ best_params = collect_best_params(wrapper_linear)if not self.not_use_best_mse:if self.dynamic_max_gap > 0 and i - last_best_iter >= self.dynamic_max_gap:@@ -832,7 +839,7 @@ class AutoRound(object):last_loss = best_lossbest_iter = last_best_iterwith torch.no_grad():- unwrapper_layer(self.model, wrapper_linear, layer_name, best_v, best_min_scale, best_max_scale)+ unwrapper_layer(self.model, wrapper_linear, layer_name, best_params)layer = mv_module_from_gpu(layer, self.low_cpu_mem_usage)dump_info = f"quantized {layer_name}, loss iter 0: {init_loss:.6f} -> iter {best_iter}: {last_loss:.6f}"logger.info(dump_info)@@ -858,15 +865,19 @@ class AutoRound(object):input_ids = q_inputquantized_layer_names, unquantized_layer_names = wrapper_block(- block, self.enable_minmax_tuning, device=self.device)+ block, self.enable_minmax_tuning, self.enable_norm_bias_tuning, device=self.device)round_params = []minmax_params = []for n, m in block.named_modules():if hasattr(m, "orig_layer"):- round_params.append(m.value)- minmax_params.append(m.min_scale)- minmax_params.append(m.max_scale)+ if "v" in m.params.keys():+ round_params.append(m.params['v'])+ if "max_scale" in m.params.keys():+ minmax_params.append(m.params["min_scale"])+ minmax_params.append(m.params["max_scale"])+ if "bias_v" in m.params.keys():+ round_params.append(m.params["bias_v"])if self.enable_minmax_tuning:optimizer = self.optimizer(@@ -899,7 +910,7 @@ class AutoRound(object):mse_loss = torch.nn.MSELoss().to(device)scaler = self.get_scaler() # pylint: disable=assignment-from-noneinit_loss = None- best_v, best_min_scale, best_max_scale = torch.tensor(0), torch.tensor(1.0), torch.tensor(1.0)+ best_params = {}for i in range(self.iters):total_loss = 0if self.sampler == "rand":@@ -940,13 +951,12 @@ class AutoRound(object):if total_loss < best_loss:best_loss = total_lossif not self.not_use_best_mse:+ best_params = collect_best_params(block)# print(f"get better result at iter {i}, the loss is {total_loss}", flush=True)- best_v = collect_round_v(block)- best_min_scale, best_max_scale = collect_minmax_scale(block)+last_best_iter = iif self.not_use_best_mse and i == self.iters - 1:- best_v = collect_round_v(block)- best_min_scale, best_max_scale = collect_minmax_scale(block)+ best_params = collect_best_params(block)if not self.not_use_best_mse:if self.dynamic_max_gap > 0 and i - last_best_iter >= self.dynamic_max_gap:@@ -966,7 +976,7 @@ class AutoRound(object):if len(unquantized_layer_names) != 0:logger.info(f"{unquantized_layer_names} have not been quantized")with torch.no_grad():- unwrapper_block(block, best_v, best_min_scale, best_max_scale)+ unwrapper_block(block, best_params)if self.enable_quanted_input:if self.low_cpu_mem_usage:block = block.to(device)@@ -1114,6 +1124,7 @@ class AutoRound(object):"amp","nsamples","low_gpu_mem_usage",+ "enable_norm_bias_tuning"]if isinstance(self.dataset, str):serialization_keys.append("dataset")@@ -1326,6 +1337,7 @@ class AutoOPTRound(AutoRound):act_group_size: int = None,act_sym: bool = None,act_dynamic: bool = True,+ enable_norm_bias_tuning: bool = False,optimizer="AdamW",**kwargs,):@@ -1363,6 +1375,7 @@ class AutoOPTRound(AutoRound):act_group_size,act_sym,act_dynamic,+ enable_norm_bias_tuning,**kwargs,)@@ -1489,6 +1502,7 @@ class AutoAdamRound(AutoOPTRound):act_group_size: int = None,act_sym: bool = None,act_dynamic: bool = True,+ enable_norm_bias_tuning: bool = False,optimizer="AdamW",**kwargs,):@@ -1526,6 +1540,7 @@ class AutoAdamRound(AutoOPTRound):act_group_size,act_sym,act_dynamic,+ enable_norm_bias_tuning,optimizer,**kwargs,)diff --git a/auto_round/data_type/int.py b/auto_round/data_type/int.pyindex 19a2a6865..545eebd0a 100644--- a/auto_round/data_type/int.py+++ b/auto_round/data_type/int.py@@ -18,13 +18,13 @@ from auto_round.data_type.register import register_dtype, QUANT_FUNC_WITH_DTYPE@register_dtype("int_asym")-def quant_tensor_asym(weight, num_bits=4, v=0, min_scale=1.0, max_scale=1.0, scale_dtype=torch.float16,+def quant_tensor_asym(weight, bits=4, v=0, min_scale=1.0, max_scale=1.0, scale_dtype=torch.float16,weight_min=None, weight_max=None, q_scale_thresh=0.0, **kwargs):"""Quantizes and dequantizes weight asymmetrically.Args:weight: Tensor containing the weight to be quantized- num_bits: Number of bits for quantization (e.g., 2, 3, 4, 8)+ bits: Number of bits for quantization (e.g., 2, 3, 4, 8)v: Rounding value perturbationmin_scale: Minimum scale coefficient for weightmax_scale: Maximum scale coefficient for weight@@ -34,7 +34,7 @@ def quant_tensor_asym(weight, num_bits=4, v=0, min_scale=1.0, max_scale=1.0, scaReturns:Quantized and dequantized weight, scale, zero-point"""- maxq = torch.tensor(2 ** num_bits - 1)+ maxq = torch.tensor(2 ** bits - 1)if isinstance(min_scale, torch.Tensor):if weight_min is None or weight_max is None:wmin_tmp = torch.clamp(weight.min(1)[0], max=0)@@ -68,14 +68,15 @@ def quant_tensor_asym(weight, num_bits=4, v=0, min_scale=1.0, max_scale=1.0, scaq = torch.clamp(int_w + zp, 0, maxq)return scale * (q - zp), scale, zp+@register_dtype("int_sym")-def quant_tensor_sym(weight, num_bits=4, v=0, min_scale=1.0, max_scale=1.0, scale_dtype=torch.float16, weight_min=None,+def quant_tensor_sym(weight, bits=4, v=0, min_scale=1.0, max_scale=1.0, scale_dtype=torch.float16, weight_min=None,weight_max=None, q_scale_thresh=0.0, **kargs):"""Quantizes and dequantizes weight symmetrically.Args:weight: Tensor containing the weight to be quantized- num_bits: Number of bits for quantization (e.g., 2, 3, 4, 8)+ bits: Number of bits for quantization (e.g., 2, 3, 4, 8)v: Rounding value perturbationmin_scale: Minimum scale coefficient for weightmax_scale: Maximum scale coefficient for weight@@ -85,7 +86,7 @@ def quant_tensor_sym(weight, num_bits=4, v=0, min_scale=1.0, max_scale=1.0, scalReturns:Quantized and dequantized weight, scale, zero-point"""- maxq = torch.tensor(2 ** num_bits - 1)+ maxq = torch.tensor(2 ** bits - 1)if isinstance(min_scale, torch.Tensor):if weight_min is None or weight_max is None:wmin_tmp = torch.clamp(weight.min(1)[0], max=0)@@ -125,3 +126,52 @@ def quant_tensor_sym(weight, num_bits=4, v=0, min_scale=1.0, max_scale=1.0, scalq = torch.clamp(int_w + zp, 0, maxq)return scale * (q - zp), scale, zp++def quant_tensor_asym_wo_round(weight, bits=4, v=0, min_scale=1.0, max_scale=1.0, scale_dtype=torch.float16,+ weight_min=None, weight_max=None, q_scale_thresh=0.0, **kwargs):+ """Quantizes and dequantizes weight asymmetrically without rounding, this is mainly for tuning bias, norm.++ Args:+ weight: Tensor containing the weight to be quantized+ bits: Number of bits for quantization (e.g., 2, 3, 4, 8)+ v: Rounding value perturbation+ min_scale: Minimum scale coefficient for weight+ max_scale: Maximum scale coefficient for weight+ weight_min (Tensor, optional): Minimum weight value for quantization. Defaults to None.+ weight_max (Tensor, optional): Maximum weight value for quantization. Defaults to None.++ Returns:+ Quantized and dequantized weight, scale, zero-point+ """+ maxq = torch.tensor(2 ** bits - 1)+ if isinstance(min_scale, torch.Tensor):+ if weight_min is None or weight_max is None:+ wmin_tmp = torch.clamp(weight.min(-1)[0], max=0)+ wmax_tmp = torch.clamp(weight.max(-1)[0], min=0)+ else:+ wmin_tmp = weight_min+ wmax_tmp = weight_max+ wmin_tmp = wmin_tmp * min_scale+ wmax_tmp = wmax_tmp * max_scale+ wmax = torch.maximum(wmax_tmp, wmin_tmp)+ wmin = torch.minimum(wmax_tmp, wmin_tmp)+ else:+ if weight_min is None or weight_max is None:+ wmin = torch.clamp(weight.min(-1)[0], max=0)+ wmax = torch.clamp(weight.max(-1)[0], min=0)+ else:+ wmin = weight_min+ wmax = weight_max++ tmp = (wmin == 0) & (wmax == 0)+ wmin[tmp] = -1+ wmax[tmp] = +1+ scale = ((wmax - wmin) / maxq).to(scale_dtype)+ scale = torch.clamp(scale, min=q_scale_thresh)+ zp = -wmin / scale # pylint: disable=E1130+ scale = scale.unsqueeze(dim=-1)+ zp = zp.unsqueeze(dim=-1)+ int_w = weight / scale + v+ q = torch.clamp(int_w + zp, 0, maxq)+ return scale * (q - zp), scale, zp+diff --git a/auto_round/export/export_to_itrex/export.py b/auto_round/export/export_to_itrex/export.pyindex 19833b217..f80d3076a 100644--- a/auto_round/export/export_to_itrex/export.py+++ b/auto_round/export/export_to_itrex/export.py@@ -205,7 +205,7 @@ def pack_model(continuelogger.info(f"Packing {k}")dtype = v["data_type"]- num_bits = v["bits"]+ bits = v["bits"]group_size = v["group_size"]sym = v["sym"]scale_dtype = v["scale_dtype"]@@ -236,7 +236,7 @@ def pack_model(new_module = WeightOnlyLinear(in_features,out_features,- num_bits,+ bits,group_size,dtype=dtype,scale_dtype=scale_dtype,diff --git a/auto_round/quantizer.py b/auto_round/quantizer.pyindex 34aa864a8..58a3858d0 100644--- a/auto_round/quantizer.py+++ b/auto_round/quantizer.py@@ -18,7 +18,8 @@ from auto_round.data_type import get_quant_funcfrom .utils import (check_to_quantized,get_scale_shape,- set_module+ set_module,+ logger)@@ -44,14 +45,14 @@ def reshape_tensor(v, group_size=-1):def quant_tensor(- quant_func, data, num_bits=4, group_size=-1, v=0, min_scale=1.0, max_scale=1.0, scale_dtype=torch.float16,+ quant_func, data, bits=4, group_size=-1, v=0, min_scale=1.0, max_scale=1.0, scale_dtype=torch.float16,weight_min=None, weight_max=None, q_scale_thresh=0.0, **kwargs,):"""Quantizes and dequantizes weight, handing the group size issue .Args:data: Tensor containing the weight to be quantized- num_bits: Number of bits for quantization (e.g., 2, 3, 4, 8)+ bits: Number of bits for quantization (e.g., 2, 3, 4, 8)group_size: The number of elements shares scale and zero pointsym: Sym or asymv: Rounding value perturbation@@ -67,7 +68,7 @@ def quant_tensor(if len(data.shape) > 2:data = data.reshape(-1, orig_shape[-1])if group_size == -1 or data.shape[1] < group_size:- data, scale, zp = quant_func(data, num_bits, v=v, min_scale=min_scale, max_scale=max_scale,+ data, scale, zp = quant_func(data, bits, v=v, min_scale=min_scale, max_scale=max_scale,scale_dtype=scale_dtype, weight_min=weight_min, weight_max=weight_max,q_scale_thresh=q_scale_thresh, **kwargs)data = data.reshape(orig_shape)@@ -75,7 +76,7 @@ def quant_tensor(if data.shape[1] % group_size == 0:data = data.reshape(-1, group_size)- data, scale, zp = quant_func(data, num_bits, v=v, min_scale=min_scale, max_scale=max_scale,+ data, scale, zp = quant_func(data, bits, v=v, min_scale=min_scale, max_scale=max_scale,scale_dtype=scale_dtype, weight_min=weight_min, weight_max=weight_max,q_scale_thresh=q_scale_thresh, **kwargs)data = data.reshape(orig_shape)@@ -86,7 +87,7 @@ def quant_tensor(pad_len = (data.shape[1] + group_size - 1) // group_size * group_size - data.shape[1]data_new = torch.nn.functional.pad(data, (0, pad_len))data_new = data_new.reshape(-1, group_size)- data_new, scale, zp = quant_func(data_new, num_bits, v=v, min_scale=min_scale,+ data_new, scale, zp = quant_func(data_new, bits, v=v, min_scale=min_scale,max_scale=max_scale, scale_dtype=scale_dtype, weight_min=weight_min,weight_max=weight_max, q_scale_thresh=q_scale_thresh, **kwargs)data_new = data_new.reshape(tmp_shape[0], -1)@@ -110,8 +111,101 @@ class WrapperWALayer(torch.nn.Module):return self.orig_layer.forward(x)+class WrapperLayerNorm(torch.nn.Module):+ """A wrapper for layer normalization with quantized weights.++ This class wraps a given layer normalization module and applies quantization without round+ to its weights. The quantization is parameterized by the number of bits and+ an optional group size.+ """++ def __init__(self, orig_layer, bit=4, group_size=-1, device="cpu"):+ super(WrapperLayerNorm, self).__init__()+ self.orig_layer = orig_layer+ self.bits = bit+ self.group_size = group_size+ self.device = device+ weight_dtype = torch.float32+ self.v = torch.nn.Parameter(+ reshape_tensor(+ torch.zeros(self.orig_layer.weight.shape, device=self.device, dtype=weight_dtype),+ self.group_size),+ requires_grad=True)+ self.params = {}+ self.params["v"] = self.v+ from auto_round.data_type.int import quant_tensor_asym_wo_round+ self.quant_func = quant_tensor_asym_wo_round++ def unwrapper(self, best_params):+ if best_params is None:+ return self.orig_layer+ v = best_params['v']+ weight_q, _, _ = quant_tensor(self.quant_func, self.orig_layer.weight, self.bits, self.group_size,+ v)+ self.orig_layer.weight.data.copy_(weight_q)+ return self.orig_layer++ def forward(self, input):+ weight_q, _, _ = quant_tensor(self.quant_func, self.orig_layer.weight, self.bits, self.group_size,+ self.v)+ import torch.nn.functional as F+ return F.layer_norm(+ input, self.orig_layer.normalized_shape, weight_q, self.orig_layer.bias, self.orig_layer.eps)+++class WrapperLlamaNorm(torch.nn.Module):+ """A wrapper for Llama normalization in HF with fake quantized weights without rounding.++ This class wraps a given layer normalization module and applies quantization without rounding+ to its weights. The quantization is parameterized by the number of bits and+ an optional group size.+ """++ def __init__(self, orig_layer, bit=4, group_size=-1, device="cpu"):+ super(WrapperLlamaNorm, self).__init__()+ self.orig_layer = orig_layer+ self.bits = bit+ self.group_size = group_size+ self.device = device+ weight_dtype = torch.float32+ self.v = torch.nn.Parameter(+ reshape_tensor(+ torch.zeros(self.orig_layer.weight.shape, device=self.device, dtype=weight_dtype),+ self.group_size),+ requires_grad=True)+ self.params = {"v": self.v}+ from auto_round.data_type.int import quant_tensor_asym_wo_round+ self.quant_func = quant_tensor_asym_wo_round++ def unwrapper(self, best_params):+ if best_params is None:+ return self.orig_layer+ v = best_params['v']+ weight_q, _, _ = quant_tensor(self.quant_func, self.orig_layer.weight, self.bits, self.group_size,+ v)+ self.orig_layer.weight.data.copy_(weight_q)+ return self.orig_layer++ def forward(self, hidden_states):+ weight_q, _, _ = quant_tensor(self.quant_func, self.orig_layer.weight, self.bits, self.group_size,+ self.v)+ input_dtype = hidden_states.dtype+ hidden_states = hidden_states.to(torch.float32)+ variance = hidden_states.pow(2).mean(-1, keepdim=True)+ hidden_states = hidden_states * torch.rsqrt(variance + self.orig_layer.variance_epsilon)+ return weight_q * hidden_states.to(input_dtype)+++norm_mapping = {}+norm_mapping["LayerNorm"] = WrapperLayerNorm+norm_mapping["LlamaRMSNorm"] = WrapperLlamaNorm+norm_mapping["Qwen2RMSNorm"] = WrapperLlamaNorm+norm_mapping["Phi3RMSNorm"] = WrapperLlamaNorm+norm_mapping["MistralRMSNorm"] = WrapperLlamaNorm++class WrapperLinear(torch.nn.Module):- def __init__(self, orig_layer, enable_minmax_tuning=True, device='cpu'):+ def __init__(self, orig_layer, enable_minmax_tuning=True, enable_norm_bias_tuning=False, device='cpu'):"""A wrapper module for linear layers that enables quantization and min-max tuning of weights.Args:@@ -120,7 +214,7 @@ class WrapperLinear(torch.nn.Module):Attributes:- orig_layer (torch.nn.Module): The original linear layer being wrapped.- - num_bits (int): The number of bits for quantization.+ - bits (int): The number of bits for quantization.- group_size (int): The size of the groups for quantization.- sym (bool): Whether the symmetric quantization is to be used.- value (torch.nn.Parameter): The learnable parameter for quantization.@@ -131,17 +225,19 @@ class WrapperLinear(torch.nn.Module):super(WrapperLinear, self).__init__()self.orig_layer = orig_layerself.device = device- self.num_bits = self.orig_layer.bits+ self.bits = self.orig_layer.bitsself.group_size = self.orig_layer.group_sizeself.scale_dtype = self.orig_layer.scale_dtypeself.sym = self.orig_layer.symself.data_type = self.orig_layer.data_type- self.weight_quant_func, self.data_type = get_quant_func(self.data_type, self.num_bits, self.sym)+ self.weight_quant_func, self.data_type = get_quant_func(self.data_type, self.bits, self.sym)self.act_bits = self.orig_layer.act_bitsself.act_group_size = self.orig_layer.act_group_sizeself.act_sym = self.orig_layer.act_symself.act_dynamic = self.orig_layer.act_dynamicself.act_quant = self.act_bits <= 8+ self.params = {}+if self.act_quant:self.act_quant_func, _ = get_quant_func(self.data_type, self.act_bits, self.act_sym)@@ -155,6 +251,7 @@ class WrapperLinear(torch.nn.Module):torch.zeros(self.orig_layer.weight.shape, device=self.device, dtype=weight_dtype),self.group_size),requires_grad=True)+ self.params["v"] = self.valueweight_reshape = reshape_tensor(orig_layer_weight.data, self.group_size)self.weight_min = torch.clamp(weight_reshape.min(1)[0], max=0)self.weight_max = torch.clamp(weight_reshape.max(1)[0], min=0)@@ -168,11 +265,26 @@ class WrapperLinear(torch.nn.Module):self.max_scale = torch.nn.Parameter(torch.ones(shape, device=self.device, dtype=weight_dtype), requires_grad=True)+ self.params["min_scale"] = self.min_scale+ self.params["max_scale"] = self.max_scaleelse:self.min_scale = torch.tensor(1.0, device=self.device, dtype=weight_dtype)self.max_scale = torch.tensor(1.0, device=self.device, dtype=weight_dtype)-- def unwrapper(self, v, min_scale, max_scale):+ self.enable_norm_bias_tuning = False+ if enable_norm_bias_tuning and self.orig_layer.bias is not None:+ self.enable_norm_bias_tuning = True+ self.bias_bits = 4 ## hard code+ self.bias_group_size = -1+ self.bias_v = torch.nn.Parameter(+ reshape_tensor(+ torch.zeros(self.orig_layer.bias.shape, device=self.device, dtype=weight_dtype),+ self.bias_group_size),+ requires_grad=True)+ from auto_round.data_type.int import quant_tensor_asym_wo_round+ self.bias_quant_func = quant_tensor_asym_wo_round+ self.params["bias_v"] = self.bias_v++ def unwrapper(self, best_params):"""Unwrapper the layer to the original layer.Args:@@ -183,31 +295,52 @@ class WrapperLinear(torch.nn.Module):Returns:- torch.nn.Module: The original linear layer with updated weights after quantization and dequantization."""+ min_scale = torch.tensor(1.0)+ max_scale = torch.tensor(1.0)+ v = torch.tensor(0.0)+ if best_params is not None:+ min_scale = best_params.get('min_scale', min_scale)+ max_scale = best_params.get('max_scale', max_scale)+ v = best_params.get('v', v)+min_scale.clamp_(0, 1.0)max_scale.clamp_(0, 1.0)+ v = v.to(self.device)+ min_scale = min_scale.to(self.device)+ max_scale = max_scale.to(self.device)if self.orig_layer.weight.device.type == 'meta':self.orig_layer.to(self.device)- qdq_weight, scale, zp = quant_tensor(self.weight_quant_func, self.orig_layer.weight, self.num_bits,+ qdq_weight, scale, zp = quant_tensor(self.weight_quant_func, self.orig_layer.weight, self.bits,self.group_size, v,min_scale, max_scale, self.scale_dtype, self.weight_min, self.weight_max,data_type=self.data_type)scale = scale.reshape(qdq_weight.shape[0], -1)if zp is not None:zp = zp.reshape(qdq_weight.shape[0], -1)+self.orig_layer.weight.data.copy_(qdq_weight)self.orig_layer.weight.grad = Noneself.orig_layer.scale = scale.to("cpu")self.orig_layer.zp = zp.to("cpu") if zp is not None else None+ if self.enable_norm_bias_tuning and "bias_v" in best_params.keys(): ##fake quant+ bias_v = best_params["bias_v"]+ bias, _, _ = quant_tensor(self.bias_quant_func, self.orig_layer.bias, self.bias_bits, self.bias_group_size,+ bias_v)+ self.orig_layer.bias.grad = None+ self.orig_layer.bias.data.copy_(bias)+self.orig_layer.q_scale_thresh = self.q_scale_threshself.orig_layer.data_type = self.data_type- if hasattr(self.orig_layer, 'update'):- self.orig_layer.update()- self.orig_layer.to('meta')if self.act_quant:self.orig_layer.act_quant_func = self.act_quant_funcwrapper_layer = WrapperWALayer(self.orig_layer)return wrapper_layer++ if hasattr(self.orig_layer, 'update'):+ self.orig_layer.update()+ self.orig_layer.to('meta')+return self.orig_layerdef forward(self, x):@@ -226,7 +359,7 @@ class WrapperLinear(torch.nn.Module):weight = self.orig_layer.get_weight().to(self.device)self.min_scale.data.copy_(torch.clamp(self.min_scale.data, 0, 1.0))self.max_scale.data.copy_(torch.clamp(self.max_scale.data, 0, 1.0))- weight_q, _, _ = quant_tensor(self.weight_quant_func, weight, self.num_bits, self.group_size, self.value,+ weight_q, _, _ = quant_tensor(self.weight_quant_func, weight, self.bits, self.group_size, self.value,self.min_scale,self.max_scale, self.scale_dtype, self.weight_min, self.weight_max,data_type=self.data_type)@@ -239,24 +372,27 @@ class WrapperLinear(torch.nn.Module):bias = self.orig_layer.biasif bias is not None and bias.device.type == 'meta':bias = self.orig_layer.get_bias().to(self.device)+ if self.enable_norm_bias_tuning:+ bias, _, _ = quant_tensor(self.bias_quant_func, bias, self.bias_bits, self.bias_group_size, self.bias_v)+return F.linear(x, weight_q, bias)class WrapperTransformerConv1d(torch.nn.Module):- def __init__(self, orig_layer, enable_minmax_tuning=True, device='cpu'):+ def __init__(self, orig_layer, enable_minmax_tuning=True, enable_norm_bias_tuning=False, device='cpu'):"""A wrapper module for transformers 1D convolutional layers used in transformers,enabling quantization and min-max tuning of weights.Args:- orig_layer (torch.nn.Module): The original 1D convolutional layer to be wrapped.- - num_bits (int): The number of bits for quantization.+ - bits (int): The number of bits for quantization.- group_size (int): The size of the groups for quantization.- sym (bool): Whether symmetric quantization is to be used.- enable_minmax_tuning (bool): Whether to enable min-max scaling tuning. Default is True.Attributes:- orig_layer (torch.nn.Module): The original 1D convolutional layer being wrapped.- - num_bits (int): The number of bits for quantization.+ - bits (int): The number of bits for quantization.- group_size (int): The size of the groups for quantization.- sym (bool): Whether symmetric quantization is to be used.- weight_t (torch.Tensor): Transposed weight tensor of the original layer.@@ -267,7 +403,7 @@ class WrapperTransformerConv1d(torch.nn.Module):"""super(WrapperTransformerConv1d, self).__init__()self.orig_layer = orig_layer- self.num_bits = self.orig_layer.bits+ self.bits = self.orig_layer.bitsself.group_size = self.orig_layer.group_sizeself.sym = self.orig_layer.symself.scale_dtype = self.orig_layer.scale_dtype@@ -277,13 +413,14 @@ class WrapperTransformerConv1d(torch.nn.Module):self.act_sym = self.orig_layer.act_symself.act_dynamic = self.orig_layer.act_dynamicself.act_quant = self.act_bits <= 8- self.weight_quant_func, self.data_type = get_quant_func(self.data_type, self.num_bits, self.sym)+ self.weight_quant_func, self.data_type = get_quant_func(self.data_type, self.bits, self.sym)if self.act_quant:self.act_quant_func, _ = get_quant_func(self.data_type, self.act_bits, self.act_sym)self.q_scale_thresh = 1e-5weight_dtype = torch.float32self.device = device+ self.params = {}if hasattr(self.orig_layer, 'get_weight'):self.weight_t = self.orig_layer.get_weight().t()else:@@ -294,6 +431,7 @@ class WrapperTransformerConv1d(torch.nn.Module):group_size=self.group_size),requires_grad=True)+ self.params["v"] = self.valueweight_reshape = reshape_tensor(self.weight_t, self.group_size)self.weight_min = torch.clamp(weight_reshape.min(1)[0], max=0)self.weight_max = torch.clamp(weight_reshape.max(1)[0], min=0)@@ -307,11 +445,28 @@ class WrapperTransformerConv1d(torch.nn.Module):self.max_scale = torch.nn.Parameter(torch.ones(shape, device=device, dtype=weight_dtype), requires_grad=True)+ self.params["min_scale"] = self.min_scale+ self.params["max_scale"] = self.max_scale+else:self.min_scale = torch.tensor(1.0, device=device, dtype=weight_dtype)self.max_scale = torch.tensor(1.0, device=device, dtype=weight_dtype)- def unwrapper(self, v=0, min_scale=1.0, max_scale=1.0):+ self.enable_norm_bias_tuning = False+ if enable_norm_bias_tuning and self.orig_layer.bias is not None:+ self.enable_norm_bias_tuning = True+ self.bias_bits = 4 ## hard code+ self.bias_group_size = -1+ self.bias_v = torch.nn.Parameter(+ reshape_tensor(+ torch.zeros(self.orig_layer.bias.shape, device=self.device, dtype=weight_dtype),+ self.bias_group_size),+ requires_grad=True)+ from auto_round.data_type.int import quant_tensor_asym_wo_round+ self.bias_quant_func = quant_tensor_asym_wo_round+ self.params["bias_v"] = self.bias_v++ def unwrapper(self, best_params):"""Unwrapper the layer to the original conv1d layer.Args:@@ -322,9 +477,21 @@ class WrapperTransformerConv1d(torch.nn.Module):Returns:- torch.nn.Module: The original 1D convolutional layer with updated weights after inverse quantization."""+ min_scale = torch.tensor(1.0)+ max_scale = torch.tensor(1.0)+ v = torch.tensor(0.0)+ if best_params is not None:+ min_scale = best_params.get('min_scale', min_scale)+ max_scale = best_params.get('max_scale', max_scale)+ v = best_params.get('v', v)+min_scale.clamp_(0, 1.0)max_scale.clamp_(0, 1.0)- qdq_weight, scale, zp = quant_tensor(self.weight_quant_func, self.weight_t, self.num_bits, self.group_size, v,+ v = v.to(self.device)+ min_scale = min_scale.to(self.device)+ max_scale = max_scale.to(self.device)++ qdq_weight, scale, zp = quant_tensor(self.weight_quant_func, self.weight_t, self.bits, self.group_size, v,min_scale,max_scale, self.scale_dtype, self.weight_min, self.weight_max,data_type=self.data_type)@@ -335,6 +502,14 @@ class WrapperTransformerConv1d(torch.nn.Module):self.orig_layer.weight.to(self.device)self.orig_layer.weight.data.copy_(qdq_weight.t())self.orig_layer.weight.grad = None++ if self.enable_norm_bias_tuning and "bias_v" in best_params.keys(): ##fake quant+ bias_v = best_params["bias_v"]+ bias, _, _ = quant_tensor(self.bias_quant_func, self.orig_layer.bias, self.bias_bits, self.bias_group_size,+ bias_v)+ self.orig_layer.bias.grad = None+ self.orig_layer.bias.data.copy_(bias)+self.orig_layer.scale = scale.to("cpu")self.orig_layer.zp = zp.to("cpu")self.orig_layer.q_scale_thresh = self.q_scale_thresh@@ -360,7 +535,7 @@ class WrapperTransformerConv1d(torch.nn.Module):with torch.no_grad():self.min_scale.clamp_(0, 1.0)self.max_scale.clamp_(0, 1.0)- weight_q, _, _ = quant_tensor(self.weight_quant_func, self.weight_t, self.num_bits, self.group_size, self.value,+ weight_q, _, _ = quant_tensor(self.weight_quant_func, self.weight_t, self.bits, self.group_size, self.value,self.min_scale, self.max_scale, self.scale_dtype, self.weight_min,self.weight_max, data_type=self.data_type)weight_q = weight_q.to(self.weight_t.dtype)@@ -369,7 +544,10 @@ class WrapperTransformerConv1d(torch.nn.Module):x, _, _ = quant_tensor(self.act_quant_func, x, self.act_bits, self.act_group_size,scale_dtype=self.scale_dtype, q_scale_thresh=self.q_scale_thresh,data_type=self.data_type)- x = torch.addmm(self.orig_layer.bias, x.view(-1, x.size(-1)), weight_q.t())+ bias = self.orig_layer.bias+ if self.enable_norm_bias_tuning:+ bias, _, _ = quant_tensor(self.bias_quant_func, bias, self.bias_bits, self.bias_group_size, self.bias_v)+ x = torch.addmm(bias, x.view(-1, x.size(-1)), weight_q.t())x = x.view(*size_out)return x@@ -395,7 +573,7 @@ class WrapperMultiblock(torch.nn.Module):return hidden_states-def wrapper_block(block, enable_minmax_tuning, device='cpu'):+def wrapper_block(block, enable_minmax_tuning, enable_norm_bias_tuning, device='cpu'):"""Wraps the layers in the given block with a custom Wrapper module.Args:@@ -412,7 +590,8 @@ def wrapper_block(block, enable_minmax_tuning, device='cpu'):if not check_to_quantized(m):unquantized_layers.append(n)continue- new_m = WrapperLinear(m, enable_minmax_tuning=enable_minmax_tuning, device=device)+ new_m = WrapperLinear(m, enable_minmax_tuning=enable_minmax_tuning,+ enable_norm_bias_tuning=enable_norm_bias_tuning, device=device)set_module(block, n, new_m)quantized_layers.append(n)@@ -424,11 +603,20 @@ def wrapper_block(block, enable_minmax_tuning, device='cpu'):set_module(block, n, new_m)quantized_layers.append(n)+ if enable_norm_bias_tuning:+ if "norm" in m.__class__.__name__.lower():+ if m.__class__.__name__ in norm_mapping.keys():+ wrapper_layer_class = norm_mapping[m.__class__.__name__]+ new_m = wrapper_layer_class(m, device=device)+ setattr(block, n, new_m)+ else:+ logger.warning_once(f"{m.__class__.__name__} is not supported")+return quantized_layers, unquantized_layers@torch.no_grad()-def unwrapper_layer(model, layer, layer_name, v=0, min_scale=0, max_scale=0):+def unwrapper_layer(model, layer, layer_name, best_params):"""Unwraps the WrapperLinear and WrapperTransformerConv1d modules in the given block.Args:@@ -439,21 +627,13 @@ def unwrapper_layer(model, layer, layer_name, v=0, min_scale=0, max_scale=0):"""if hasattr(layer, "orig_layer"):-- if isinstance(min_scale, torch.Tensor):- min_scale = torch.clamp(min_scale, 0, 1.0)- max_scale = torch.clamp(max_scale, 0, 1.0)-- else:- min_scale = torch.tensor(1.0)- max_scale = torch.tensor(1.0)- orig_layer = layer.unwrapper(v, min_scale, max_scale)+ orig_layer = layer.unwrapper(best_params)orig_layer = orig_layer.to("cpu")set_module(model, layer_name, orig_layer)@torch.no_grad()-def unwrapper_block(block, vs, min_scales, max_scales):+def unwrapper_block(block, best_params):"""Unwraps the WrapperLinear and WrapperTransformerConv1d modules in the given block.Args:@@ -464,16 +644,9 @@ def unwrapper_block(block, vs, min_scales, max_scales):"""for n, m in block.named_modules():if hasattr(m, "orig_layer"):- v = 0- min_scale = torch.tensor(1.0)- max_scale = torch.tensor(1.0)- if isinstance(vs, dict):- v = vs[n]- if isinstance(min_scales, dict):- min_scale = min_scales[n]- min_scale = torch.clamp(min_scale, 0, 1.0)- if isinstance(max_scales, dict):- max_scale = max_scales[n]- max_scale = torch.clamp(max_scale, 0, 1.0)- orig_layer = m.unwrapper(v, min_scale, max_scale)+ if n in best_params.keys():+ best_param = best_params[n]+ else:+ best_param = None+ orig_layer = m.unwrapper(best_param)set_module(block, n, orig_layer)diff --git a/auto_round/utils.py b/auto_round/utils.pyindex cbbe60eb6..bd54cdf45 100644--- a/auto_round/utils.py+++ b/auto_round/utils.py@@ -27,6 +27,7 @@ from torch.amp import autocastfrom functools import lru_cache+@lru_cache(None)def warning_once(self, msg: str):self.warning(msg)@@ -81,7 +82,6 @@ auto_gptq = LazyImport("auto_gptq")htcore = LazyImport("habana_frameworks.torch.core")-def is_optimum_habana_available():from transformers.utils.import_utils import is_optimum_available@@ -149,7 +149,7 @@ def unsupport_meta_device(model):if target_device is None:target_device = param.deviceif param.device != target_device:- if param.device.type == 'meta' or target_device.type == 'meta':+ if param.device.type == 'meta' or target_device.type == 'meta':return Trueif target_device.type == 'meta':if hasattr(model, "path"):@@ -201,7 +201,7 @@ def mv_module_from_gpu(module, low_cpu_mem_usage=False):The module on the specified device."""if hasattr(module, "device"):- target_device ="meta" if low_cpu_mem_usage else "cpu"+ target_device = "meta" if low_cpu_mem_usage else "cpu"if module.device.type == target_device:return moduleelse:@@ -257,36 +257,36 @@ def check_is_cpu(device):def validate_modules(module_names):- """- Test a list of modules' validity.-- Args:- modules (list of str): List of strings to be validated.-- Returns:- bool: True if all modules have equal length or not dependent, otherwise False.- """- if not bool(module_names): # pragma: no cover- raise ValueError(f"Empty modules")- if len(module_names) < 2:- return True- split_modules = [s.split('.') for s,_ in module_names]- lengths = [len(parts) for parts in split_modules]- if len(set(lengths)) == 1: # pragma: no cover- return True- max_length = max(lengths)- min_length = min(lengths)- longest_module = next(s for s in split_modules if len(s) == max_length)- shortest_module = next(s for s in split_modules if len(s) == min_length)- shortest_module = '.'.join(shortest_module)- longest_module = '.'.join(longest_module)- # Check if the shortest name is a substring of the longest name- if shortest_module in longest_module: # pragma: no cover- raise ValueError(f"Invalid modules, at least two modules detected"\- " as dependent, {shortest_module} and {longest_module}")+ """+ Test a list of modules' validity.++ Args:+ modules (list of str): List of strings to be validated.++ Returns:+ bool: True if all modules have equal length or not dependent, otherwise False.+ """+ if not bool(module_names): # pragma: no cover+ raise ValueError(f"Empty modules")+ if len(module_names) < 2:return True--+ split_modules = [s.split('.') for s, _ in module_names]+ lengths = [len(parts) for parts in split_modules]+ if len(set(lengths)) == 1: # pragma: no cover+ return True+ max_length = max(lengths)+ min_length = min(lengths)+ longest_module = next(s for s in split_modules if len(s) == max_length)+ shortest_module = next(s for s in split_modules if len(s) == min_length)+ shortest_module = '.'.join(shortest_module)+ longest_module = '.'.join(longest_module)+ # Check if the shortest name is a substring of the longest name+ if shortest_module in longest_module: # pragma: no cover+ raise ValueError(f"Invalid modules, at least two modules detected" \+ " as dependent, {shortest_module} and {longest_module}")+ return True++def get_block_names(model):"""Get the block names for transformers-like networks.@@ -300,9 +300,9 @@ def get_block_names(model):target_modules = []for n, m in model.named_modules():if hasattr(type(m), "__name__") and "ModuleList" in type(m).__name__:- target_modules.append((n, m))- break ## only find the first modulelist, may be not robust- for i,target_m in enumerate(target_modules):+ target_modules.append((n, m))+ break ## only find the first modulelist, may be not robust+ for i, target_m in enumerate(target_modules):block_names.append([])for n, m in target_m[1].named_children():block_names[i].append(target_m[0] + "." + n)@@ -326,47 +326,57 @@ def get_multimodal_block_names(model, quant_vision=False):if quant_vision or all(key not in n.lower() for key in (Vison_blocks_tuple)):target_modules.append((n, m))validate_modules(target_modules)- for i,target_m in enumerate(target_modules):+ for i, target_m in enumerate(target_modules):block_names.append([])for n, m in target_m[1].named_children():block_names[i].append(target_m[0] + "." + n)return block_names-def collect_round_v(block):- """Collects the round values for wrapped linear modules in the given block.-- Args:- block: The input block.-- Returns:- vs: A dictionary of round values for the wrapped linear modules.- """- vs = {}+def collect_best_params(block):+ params = {}for n, m in block.named_modules():if hasattr(m, "orig_layer"):- v = m.value.data- vs[n] = copy.deepcopy(v)- return vs---def collect_minmax_scale(block):- """Collects the min-max scaling values for wrapped linear modules in the given block.+ params[n] = {}+ for key in m.params.keys():+ params[n][key]=copy.deepcopy(m.params[key].data)+ return params- Args:- block: The input block.- Returns:- min_scales: A dictionary of minimum scaling values.- max_scales: A dictionary of maximum scaling values.- """- min_scales = {}- max_scales = {}- for n, m in block.named_modules():- if hasattr(m, "orig_layer"):- min_scales[n] = copy.deepcopy(torch.clamp(m.min_scale.data, 0, 1.0))- max_scales[n] = copy.deepcopy(torch.clamp(m.max_scale.data, 0, 1.0))- return min_scales, max_scales+# def collect_round_v(block):+# """Collects the round values for wrapped linear modules in the given block.+#+# Args:+# block: The input block.+#+# Returns:+# vs: A dictionary of round values for the wrapped linear modules.+# """+# vs = {}+# for n, m in block.named_modules():+# if hasattr(m, "orig_layer"):+# v = m.value.data+# vs[n] = copy.deepcopy(v)+# return vs+#+#+# def collect_minmax_scale(block):+# """Collects the min-max scaling values for wrapped linear modules in the given block.+#+# Args:+# block: The input block.+#+# Returns:+# min_scales: A dictionary of minimum scaling values.+# max_scales: A dictionary of maximum scaling values.+# """+# min_scales = {}+# max_scales = {}+# for n, m in block.named_modules():+# if hasattr(m, "orig_layer"):+# min_scales[n] = copy.deepcopy(torch.clamp(m.min_scale.data, 0, 1.0))+# max_scales[n] = copy.deepcopy(torch.clamp(m.max_scale.data, 0, 1.0))+# return min_scales, max_scales@torch.no_grad()@@ -468,7 +478,7 @@ def detect_device(device=None):if torch.cuda.is_available():device = torch.device("cuda")logger.info("Using GPU device")- elif is_optimum_habana_available(): # pragma: no cover+ elif is_optimum_habana_available(): # pragma: no coverdevice = torch.device("hpu")logger.info("Using HPU device")# Use CPU as a fallback@@ -665,7 +675,7 @@ def check_memory_availability(device, inputs, weight, org_seqlen, org_bs):total_memory = torch.cuda.get_device_properties(current_gpu_index).total_memoryused_memory = torch.cuda.memory_allocated(current_gpu_index)free_space = total_memory - used_memory- elif "hpu" in device: # pragma: no cover+ elif "hpu" in device: # pragma: no covercurrent_hpu_index = torch.hpu.current_device()free_space = torch.hpu.memory_reserved(current_hpu_index)else:@@ -800,7 +810,7 @@ def dynamic_import_inference_linear(backend, bits, group_size, sym):except Exception as e:raise ImportError("Please install auto-gptq via 'pip install auto-gptq' to support GPTQ backend ")return get_autogptq_infer_linear(backend, bits, group_size, sym)- else: # pragma: no cover+ else: # pragma: no covertry:import habana_frameworks.torch.hpu # noqa: F401 # pylint: disable=E0401except Exception as e:@@ -808,7 +818,7 @@ def dynamic_import_inference_linear(backend, bits, group_size, sym):else:from auto_round_extension.hpu.qlinear_hpu_gptq import QuantLinearreturn QuantLinear- if (bits == 4 and is_optimum_habana_available()) or "hpu" in backend: # pragma: no cover+ if (bits == 4 and is_optimum_habana_available()) or "hpu" in backend: # pragma: no covertry:import habana_frameworks.torch.hpu # noqa: F401 # pylint: disable=E0401except Exception as e:@@ -816,9 +826,9 @@ def dynamic_import_inference_linear(backend, bits, group_size, sym):else:from auto_round_extension.hpu.qlinear_hpu import QuantLinearreturn QuantLinear- if "awq" in backend: # pragma: no cover+ if "awq" in backend: # pragma: no covertry:- from awq.modules.linear import WQLinear_GEMM # pylint: disable=E0401+ from awq.modules.linear import WQLinear_GEMM # pylint: disable=E0401except:raise ImportError("autoawq is required. Please install it by 'pip install autoawq' to \support auto_awq format.")@@ -832,6 +842,3 @@ def dynamic_import_inference_linear(backend, bits, group_size, sym):else:from auto_round_extension.cuda.qliner_triton import QuantLinearreturn QuantLinear---diff --git a/docs/imgs/norm_bias_overview.png b/docs/imgs/norm_bias_overview.pngnew file mode 100644index 000000000..c431a917dBinary files /dev/null and b/docs/imgs/norm_bias_overview.png differdiff --git a/docs/tuning_norm_bias.md b/docs/tuning_norm_bias.mdnew file mode 100644index 000000000..4281134bd--- /dev/null+++ b/docs/tuning_norm_bias.md@@ -0,0 +1,60 @@+<div align="center">++## Fast tuning LayerNorm and Linear bias via fake quantization without rounding++**Personal view by Wenhua, welcome to discuss**++**Work in Progress**+<div align="left">+Recent studies have found that tuning LayerNorm and bias through optimizer like Adam can lead to better results, especially for low-bit quantization such as 2-bit. However, I personally do not favor the use of Adam for this purpose, as detailed in the following section, and introduce an alternative way, detailed in the last section.++### Why not using Adam++#### Reason1 hard to tune the learning rate and steps++Since Adam adaptively tunes the step size based on the gradient and its square, the learning rate often needs adjustment for different models, different quantization bits, or both, as observed in most papers. I hypothesize that this tuning requirement arises because most papers report results for only a limited range of model families, while many new models continually emerge. Despite my experience in this domain, I still find it challenging to tune the learning rate beyond using grid search. I believe many users encounter the same issue.++#### Reason2 Prone to overfitting++Since Adam adapts the step size with each iteration, it is difficult to control the changes in parameters, leading to significant deviations from the original model's weights in some scenarios. However, we only use hundreds or thousands of samples to fine-tune a low-bit model, whereas the original model is trained on a large corpus and specialized datasets (e.g., instruction datasets). Consequently, even if the low-bit tuned model performs well on some language-modeling tasks, it may lose other compatibility as the deviations increase.++++### Our way++**An overview of our method**+<div align="center">++++<div align="left">+++We limit the tuned parameters in a quantization space, expressed as:+$$+W' = s*clip(W/s+zp,N,M)+$$+where 𝑠 is the quantization scale, predefined by 𝑊 and hyperparameters such as bits.++To tune the W', following Signround, we add a trainable parameter V in the range [-0.5, 0.5], which can be easily tuned by SignSGD.++$$+W' = s*clip(W/s+zp+v,N,M)+$$+++An important note: We remove the rounding to reduce unnecessary rounding loss, as the final weights of LayerNorm and bias are typically kept at 16-bit precision in most cases.++++**Result at W2G32**++the tuning of layer normalization and Linear bias are fake quantized at W4G-1.++ Average accuracies of HellaSwag, WinoGrand, PIQA and LAMBADA, higher is better.++| | OPT125m | OPT1.3B | OPT2.7B | OPT6.7B | LLaMAV2-7b | LLaMAV3-8B-Instruct |+| --------- | ---------- | ---------- | ---------- | ---------- | ---------- | ------------------- |+| SignRound | 0.3978 | 0.5094 | 0.5267 | 0.3681 | 0.6267 | 0.5890 |+| Ours | **0.4077** | **0.5151** | **0.5596** | **0.3887** | **0.6315** | **0.5949** |+diff --git a/examples/language-modeling/main.py b/examples/language-modeling/main.pyindex 8605151b0..d476fd948 100644--- a/examples/language-modeling/main.py+++ b/examples/language-modeling/main.py@@ -106,7 +106,10 @@ if __name__ == '__main__':help="disable amp")parser.add_argument("--disable_minmax_tuning", action='store_true',- help="whether disable enable weight minmax tuning")+ help="whether disable enable weight minmax tuning")++ parser.add_argument("--enable_norm_bias_tuning", action='store_true',+ help="whether enable norm bias tuning")parser.add_argument("--disable_trust_remote_code", action='store_true',help="Whether to disable trust_remote_code")@@ -316,7 +319,7 @@ if __name__ == '__main__':seed=args.seed, gradient_accumulate_steps=args.gradient_accumulate_steps,scale_dtype=args.scale_dtype, layer_config=layer_config,enable_minmax_tuning=not args.disable_minmax_tuning, act_bits=args.act_bits,- low_cpu_mem_usage=low_cpu_mem_usage, data_type=args.data_type)+ low_cpu_mem_usage=low_cpu_mem_usage, data_type=args.data_type, enable_norm_bias_tuning=args.enable_norm_bias_tuning)model, _ = autoround.quantize()model_name = args.model_name.rstrip("/")if args.low_cpu_mem_mode == 1 or args.low_cpu_mem_mode == 2:diff --git a/test/test_autoround.py b/test/test_autoround.pyindex 6c11b3010..a1b871f5c 100644--- a/test/test_autoround.py+++ b/test/test_autoround.py@@ -149,7 +149,7 @@ class TestAutoRound(unittest.TestCase):)autoround.quantize()- def test_disable_enable_quanted_input(self):+ def test_disable_quanted_input(self):bits, group_size, sym = 4, -1, Trueautoround = AutoRound(self.model,@@ -164,6 +164,22 @@ class TestAutoRound(unittest.TestCase):)autoround.quantize()+ def test_enable_norm_bias_tuning(self):+ bits, group_size, sym = 4, -1, True+ autoround = AutoRound(+ self.model,+ self.tokenizer,+ bits=bits,+ group_size=group_size,+ sym=sym,+ iters=2,+ seqlen=10,+ enable_quanted_input=False,+ enable_norm_bias_tuning=True,+ dataset=self.llm_dataloader,+ )+ autoround.quantize()+def test_disable_minmax_tuning(self):bits, group_size, sym = 4, -1, Trueautoround = AutoRound(