Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

[Important update]set full range sym as the default (#278)

wenhuach2100122bc617ac
diff --git a/README.md b/README.md
index e9ee5d9a5..31ace1444 100644
--- a/README.md
+++ b/README.md
@@ -26,7 +26,13 @@ more accuracy data and recipes across various models.
<div align="left">
## What's New
-* [2024/09] AutoRound format supports several LVM models, check out the examples [Qwen2-Vl](./examples/multimodal-modeling/Qwen-VL),[Phi-3-vision](./examples/multimodal-modeling/Phi-3-vision), [Llava](./examples/multimodal-modeling/Llava)
+
+* [2024/10] Important update: We now support full-range symmetric quantization and have made it the default
+ configuration. This approach is typically better or comparable to asymmetric quantization and significantly
+ outperforms other symmetric variants, especially at low bit-widths like 2-bit. And,no need to compile from source to run
+ AutoRound format anymore.
+* [2024/09] AutoRound format supports several LVM models, check out the
+ examples [Qwen2-Vl](./examples/multimodal-modeling/Qwen-VL),[Phi-3-vision](./examples/multimodal-modeling/Phi-3-vision), [Llava](./examples/multimodal-modeling/Llava)
* [2024/08] AutoRound format supports Intel Gaudi2 devices. Please refer
to [Intel/Qwen2-7B-int4-inc](https://huggingface.co/Intel/Qwen2-7B-int4-inc).
* [2024/08] AutoRound introduces several experimental features, including fast tuning of norm/bias parameters (for 2-bit
@@ -61,14 +67,14 @@ tokenizer = AutoTokenizer.from_pretrained(model_name)
from auto_round import AutoRound
-bits, group_size, sym = 4, 128, False
-autoround = AutoRound(model, tokenizer, bits=bits, group_size=group_size, sym=sym)
+bits, group_size = 4, 128
+autoround = AutoRound(model, tokenizer, bits=bits, group_size=group_size)
-## best accuracy, 3X slower, low_gpu_mem_usage could save ~20G but ~30% slower
-# autoround = AutoRound(model, tokenizer, nsamples=512, iters=1000, low_gpu_mem_usage=True, bits=bits, group_size=group_size, sym=sym)
+## the best accuracy, 3X slower, low_gpu_mem_usage could save ~20G but ~30% slower
+# autoround = AutoRound(model, tokenizer, nsamples=512, iters=1000, low_gpu_mem_usage=True, bits=bits, group_size=group_size)
## fast and low memory, 2-3X speedup, slight accuracy drop at W4G128
-# autoround = AutoRound(model, tokenizer, nsamples=128, iters=200, seqlen=512, batch_size=4, bits=bits, group_size=group_size, sym=sym)
+# autoround = AutoRound(model, tokenizer, nsamples=128, iters=200, seqlen=512, batch_size=4, bits=bits, group_size=group_size)
autoround.quantize()
output_dir = "./tmp_autoround"
@@ -87,7 +93,7 @@ autoround.save_quantized(output_dir, format='auto_round', inplace=True)
- `group_size (int)`: Size of the quantization group (default is 128).
-- `sym (bool)`: Whether to use symmetric quantization (default is False).
+- `sym (bool)`: Whether to use symmetric quantization (default is True).
- `enable_quanted_input (bool)`: Whether to use the output of the previous quantized block as the input for the current
block for tuning (default is True).
@@ -173,10 +179,11 @@ We provide two recipes for best accuracy and fast running speed with low memory.
#### Formats
-**AutoRound Format**:This format is well-suited for CPU, HPU devices, 2 bits, as well as mixed-precision inference. [2,4]
-bits are supported. It
-resolves the asymmetric quantization kernel issues found in the AutoGPTQ format and supports both LM-head quantization
-and mixed precision. However, it has not yet gained widespread community adoption. For CUDA support, you will need to
+**AutoRound Format**:This format is well-suited for CPU, HPU devices, 2 bits, as well as mixed-precision
+inference. [2,4]
+bits are supported. It also benefits
+from the Marlin kernel, which can boost inference performance notably.However, it has not yet gained widespread
+community adoption. For CUDA support, you will need to
install from the source.
**AutoGPTQ Format**: This format is well-suited for symmetric quantization on CUDA devices and is widely adopted by the
@@ -187,8 +194,7 @@ models.
Additionally, symmetric quantization tends to perform poorly at 2-bit precision.
**AutoAWQ Format**: This format is well-suited for asymmetric 4-bit quantization on CUDA devices and is widely adopted
-within the community, only 4-bits quantization is supported. Asymmetric quantization typically improves
-accuracy but may reduce inference speed. It features
+within the community, only 4-bits quantization is supported. It features
specialized layer fusion tailored for Llama models.
## Model Inference
@@ -216,8 +222,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
**HPU**: docker image with Gaudi Software Stack is recommended. More details can be found
in [Gaudi Guide](https://docs.habana.ai/en/latest/).
-**CUDA**: git clone https://github.com/intel/auto-round.git && cd auto-round && pip install --no-build-isolation
--e .
+**CUDA**: pip install auto-gptq for sym quantization(tuning needs auto-round 0.30+), for asym quantization, need to install auto-round from source
#### CPU/HPU/CUDA on 0.3.0+
@@ -240,6 +245,8 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
#### CPU/HPU/CUDA on 0.3.0
+**CUDA**: need to install auto-round from source
+
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
from auto_round.auto_quantizer import AutoHfQuantizer ## must import
@@ -308,8 +315,8 @@ release most of the models ourselves.
| bigscience/bloom-3b | [accuracy](./docs/bloom-3B-acc.md), [recipe](./examples/language-modeling/scripts/bloom-3b.sh), [example](./examples/language-modeling/) |
| EleutherAI/gpt-j-6b | [accuracy](./docs/gpt-j-6B-acc.md), [recipe](./examples/language-modeling/scripts/gpt-j-6b.sh), [example](./examples/language-modeling/) |
-
## Integration
+
AutoRound has been integrated into multiple repositories.
[Intel Neural Compressor](https://github.com/intel/neural-compressor)
@@ -318,9 +325,6 @@ AutoRound has been integrated into multiple repositories.
[pytorch/ao](https://github.com/pytorch/ao)
-
-
-
## Reference
If you find AutoRound useful for your research, please cite our paper:
diff --git a/auto_round/__main__.py b/auto_round/__main__.py
index 3de56571b..8f8fcf417 100644
--- a/auto_round/__main__.py
+++ b/auto_round/__main__.py
@@ -56,8 +56,8 @@ def setup_parser():
help="The device to be used for tuning. The default is set to auto/None,"
"allowing for automatic detection. Currently, device settings support CPU, GPU, and HPU.")
- parser.add_argument("--sym", action='store_true',
- help=" sym quantization")
+ parser.add_argument("--asym", action='store_true',
+ help=" asym quantization")
parser.add_argument("--iters", default=200, type=int,
help=" iters")
@@ -92,7 +92,8 @@ def setup_parser():
parser.add_argument("--format", default=None, type=str,
help="The format in which to save the model. "
- "The options are 'auto_round', 'auto_gptq', 'auto_awq', 'itrex', 'itrex_xpu' and 'fake'."
+ "The options are 'auto_round', 'auto_round:gptq','auto_round:marlin',"
+ " 'auto_gptq', 'auto_awq', 'itrex', 'itrex_xpu' and 'fake'."
"default to 'auto_round."
)
@@ -161,6 +162,13 @@ def tune(args):
tasks = args.tasks
if args.format is None:
args.format = "auto_round"
+ if "auto_gptq" in args.format and args.asym is True:
+ print(
+ "warning: The auto_gptq kernel has issues with asymmetric quantization. "
+ "It is recommended to use sym quantization or --format='auto_round'")
+
+ if "marlin" in args.format and args.asym is True:
+ assert False, "marlin backend only supports sym quantization, please remove --asym"
model_name = args.model
if model_name[-1] == "/":
@@ -284,7 +292,7 @@ def tune(args):
raise EnvironmentError(error_message)
autoround = round(
- model, tokenizer, args.bits, args.group_size, sym=args.sym, batch_size=args.batch_size,
+ model, tokenizer, args.bits, args.group_size, sym=not args.asym, batch_size=args.batch_size,
dataset=args.dataset, seqlen=seqlen, nblocks=args.nblocks, iters=args.iters, lr=args.lr,
minmax_lr=args.minmax_lr, enable_quanted_input=not args.disable_quanted_input,
device=device_str, amp=not args.disable_amp, nsamples=args.nsamples, seed=args.seed,
diff --git a/auto_round/autoround.py b/auto_round/autoround.py
index ecd0ed0c6..3bde5b6f1 100644
--- a/auto_round/autoround.py
+++ b/auto_round/autoround.py
@@ -58,8 +58,7 @@ import accelerate
class AutoRound(object):
- """This is Signround+ which is an advanced version of Signround. For more information,
- please refer to Cheng, Wenhua, et al. "Optimize weight rounding via signed gradient descent
+ """For more information, please refer to Cheng, Wenhua, et al. "Optimize weight rounding via signed gradient descent
for the quantization of llms." arXiv preprint arXiv:2309.05516 (2023).
Args:
@@ -67,7 +66,7 @@ class AutoRound(object):
tokenizer: An optional tokenizer for processing input data. If none is provided, a dataloader must be supplied.
bits (int): Number of bits for quantization (default is 4).
group_size (int): Size of the quantization group (default is 128).
- sym (bool): Whether symmetric quantization is to be used (default is False).
+ sym (bool): Whether symmetric quantization is to be used (default is True).
layer_config (dict): Configuration for weight quantization (default is None).
layer_config={
'layer1':##layer_name
@@ -75,7 +74,7 @@ class AutoRound(object):
'data_type': 'int',
'bits': 4,
'group_size': 128,
- 'sym': False
+ 'sym': True
'act_data_type': None,
'act_bits': 32,
'act_group_size': None,
@@ -84,7 +83,6 @@ class AutoRound(object):
}
...
}
- enable_full_range (bool): Whether to enable full range quantization (default is False).
batch_size (int): Batch size for training (default is 8).
amp (bool): Whether to use automatic mixed precision (default is True).
device: The device to be used for tuning (default is "auto").
@@ -126,7 +124,6 @@ class AutoRound(object):
group_size: int = 128,
sym: bool = False,
layer_config: dict = None,
- enable_full_range: bool = False, ##for symmetric, TODO support later
batch_size: int = 8,
amp: bool = True,
device: str = None,
@@ -207,7 +204,6 @@ class AutoRound(object):
self.gradient_accumulate_steps = gradient_accumulate_steps
self.not_use_best_mse = not_use_best_mse
self.dynamic_max_gap = dynamic_max_gap
- self.enable_full_range = enable_full_range
self.lr_scheduler = lr_scheduler
self.optimizer = self.get_optimizer(None)
self.share_attention_mask_flag = None
@@ -243,7 +239,6 @@ class AutoRound(object):
assert self.seqlen > 0, "seqlen must be positive"
assert self.nblocks > 0, "nblocks must be positive"
assert self.gradient_accumulate_steps > 0, "gradient accumulate step must be positive"
- assert self.enable_full_range is False, "only support enable_full_range=False currently"
assert self.act_dynamic is True, "only support dynamic quantization for activation currently"
# assert self.tokenizer != None or self.dataloader != None
if self.act_bits <= 8:
@@ -1345,9 +1340,8 @@ class AutoOPTRound(AutoRound):
tokenizer: An optional tokenizer for processing input data.
bits (int): Number of bits for quantization (default is 4).
group_size (int): Size of the quantization group (default is 128).
- sym (bool): Whether sym to be used (default is False).
+ sym (bool): Whether sym to be used (default is True).
layer_config (dict): Configuration for weight quantization (default is None).
- enable_full_range (bool): Whether to enable full range quantization (default is False).
batch_size (int): Batch size for training (default is 8).
amp (bool): Whether to use automatic mixed precision (default is True).
device: The device to be used for training (default is "auto").
@@ -1388,9 +1382,8 @@ class AutoOPTRound(AutoRound):
tokenizer=None,
bits: int = 4,
group_size: int = 128,
- sym: bool = False,
+ sym: bool = True,
layer_config=None,
- enable_full_range: bool = False,
batch_size: int = 8,
amp: bool = True,
device=None,
@@ -1429,7 +1422,6 @@ class AutoOPTRound(AutoRound):
group_size=group_size,
sym=sym,
layer_config=layer_config,
- enable_full_range=enable_full_range,
batch_size=batch_size,
amp=amp,
device=device,
@@ -1514,9 +1506,8 @@ class AutoAdamRound(AutoOPTRound):
tokenizer: An optional tokenizer for processing input data.
bits (int): Number of bits for quantization (default is 4).
group_size (int): Size of the quantization group (default is 128).
- sym (str): Whether symmetric quantization to be used (default is False).
+ sym (str): Whether symmetric quantization to be used (default is True).
layer_config (dict): Configuration for weight quantization (default is None).
- enable_full_range (bool): Whether to enable full range quantization (default is False).
batch_size (int): Batch size for training (default is 8).
amp (bool): Whether to use automatic mixed precision (default is True).
device: The device to be used for training (default is "auto").
@@ -1557,9 +1548,8 @@ class AutoAdamRound(AutoOPTRound):
tokenizer=None,
bits: int = 4,
group_size: int = 128,
- sym: bool = False,
+ sym: bool = True,
layer_config=None,
- enable_full_range: bool = False,
batch_size: int = 8,
amp: bool = True,
device=None,
@@ -1598,7 +1588,6 @@ class AutoAdamRound(AutoOPTRound):
group_size=group_size,
sym=sym,
layer_config=layer_config,
- enable_full_range=enable_full_range,
batch_size=batch_size,
amp=amp,
device=device,
diff --git a/auto_round/data_type/int.py b/auto_round/data_type/int.py
index ef3508950..2ebd5ced3 100644
--- a/auto_round/data_type/int.py
+++ b/auto_round/data_type/int.py
@@ -17,6 +17,49 @@ from .utils import floor_ste, round_ste
from auto_round.data_type.register import register_dtype, QUANT_FUNC_WITH_DTYPE
+@register_dtype("int_sym")
+def quant_tensor_sym(weight, bits=4, v=0, min_scale=1.0, max_scale=1.0, scale_dtype=torch.float16, weight_min=None,
+ weight_max=None, q_scale_thresh=0.0, **kwargs):
+ """Quantizes and dequantizes weight asymmetrically. full range, credict goes to llamacpp community
+
+ Args:
+ weight: Tensor containing the weight to be quantized
+ bits: Number of bits for quantization (e.g., 2, 3, 4, 8)
+ v: Rounding value perturbation
+ min_scale: Minimum scale coefficient for weight
+ max_scale: Maximum scale coefficient for weight
+ weight_min (Tensor, optional): Minimum weight value for quantization. Defaults to None.
+ weight_max (Tensor, optional): Maximum weight value for quantization. Defaults to None.
+
+ Returns:
+ Quantized and dequantized weight, scale, zero-point
+ """
+ maxq = torch.tensor(2 ** (bits - 1))
+ if weight_min is None or weight_max is None:
+ wmin_tmp = torch.clamp(weight.min(-1)[0], max=0)
+ wmax_tmp = torch.clamp(weight.max(-1)[0], min=0)
+ else:
+ wmin_tmp = weight_min
+ wmax_tmp = weight_max
+ if isinstance(min_scale, torch.Tensor):
+ wmin = wmin_tmp * min_scale
+ wmax = wmax_tmp * max_scale
+ else:
+ wmin = wmin_tmp
+ wmax = wmax_tmp
+ max_v = (2 * (torch.abs(wmax) < torch.abs(wmin)).int() - 1) * torch.max(torch.abs(wmax), torch.abs(wmin))
+
+ scale = (max_v / maxq).to(scale_dtype)
+ scale = torch.where(scale < 0, torch.clamp(scale, max=-q_scale_thresh), torch.clamp(scale, min=q_scale_thresh))
+ zp = torch.full_like(scale, maxq) # pylint: disable=E1130
+ scale = scale.unsqueeze(dim=-1)
+ zp = zp.unsqueeze(dim=-1)
+ int_w = round_ste(weight / scale + v)
+ q = torch.clamp(int_w + zp, 0, 2 ** bits - 1)
+ qdq_result = (scale * (q - zp)).to(weight.dtype)
+ return qdq_result, scale, zp
+
+
@register_dtype("int_asym")
def quant_tensor_asym(weight, bits=4, v=0, min_scale=1.0, max_scale=1.0, scale_dtype=torch.float16,
weight_min=None, weight_max=None, q_scale_thresh=0.0, **kwargs):
@@ -49,7 +92,7 @@ def quant_tensor_asym(weight, bits=4, v=0, min_scale=1.0, max_scale=1.0, scale_d
wmax = wmax_tmp
scale = ((wmax - wmin) / maxq).to(scale_dtype)
scale = torch.clamp(scale, min=q_scale_thresh)
- zp = round_ste(-wmin / scale) # pylint: disable=E1130
+ zp = round_ste(-wmin / scale) # pylint: disable=E1130
scale = scale.unsqueeze(dim=-1)
zp = zp.unsqueeze(dim=-1)
int_w = round_ste(weight / scale + v)
@@ -58,8 +101,8 @@ def quant_tensor_asym(weight, bits=4, v=0, min_scale=1.0, max_scale=1.0, scale_d
return qdq_result, scale, zp
-@register_dtype("int_sym")
-def quant_tensor_sym(weight, bits=4, v=0, min_scale=1.0, max_scale=1.0, scale_dtype=torch.float16, weight_min=None,
+@register_dtype("int_sym_gptq")
+def quant_tensor_sym_gptq(weight, bits=4, v=0, min_scale=1.0, max_scale=1.0, scale_dtype=torch.float16, weight_min=None,
weight_max=None, q_scale_thresh=0.0, **kwargs):
"""Quantizes and dequantizes weight symmetrically.
@@ -106,6 +149,7 @@ def quant_tensor_sym(weight, bits=4, v=0, min_scale=1.0, max_scale=1.0, scale_dt
return qdq_result, scale, zp
+
def quant_tensor_asym_wo_round(weight, bits=4, v=0, min_scale=1.0, max_scale=1.0, scale_dtype=torch.float16,
weight_min=None, weight_max=None, q_scale_thresh=0.0, **kwargs):
"""Quantizes and dequantizes weight asymmetrically without rounding, this is mainly for tuning bias, norm.
diff --git a/auto_round/export/export_to_autoround/export.py b/auto_round/export/export_to_autoround/export.py
index e46b4800e..0ab2f505e 100644
--- a/auto_round/export/export_to_autoround/export.py
+++ b/auto_round/export/export_to_autoround/export.py
@@ -180,8 +180,12 @@ def save_quantized_as_autoround(output_dir, inplace=True, backend="auto_round:ex
backend = "auto_round:exllamav2"
backend = backend.replace("autoround", "auto_round")
backend = backend.replace("auto-round", "auto_round")
+ ##if using sym, we change to gptq sym kernel to avoid compiling from auto_round source
+ if (kwargs.get("sym") is None or kwargs.get("sym") == True) and ("gptq" not in backend and "awq" not in backend):
+ backend = backend.replace('auto_round','auto_round:gptq')
+
if not ("triton" in backend or "exllamav2" in backend or "awq" in backend or "gptq" in backend):
- logger.info(f"auto_round format does not support {backend}, try to pack each layer with autogptq")
+ logger.info(f"AutoRound format does not support {backend}, try to pack each layer with AutoGPTQ")
backend = backend.replace("auto_round", "auto_gptq")
model = kwargs["model"]
diff --git a/auto_round/utils.py b/auto_round/utils.py
index 05593c02a..10beff712 100644
--- a/auto_round/utils.py
+++ b/auto_round/utils.py
@@ -800,7 +800,7 @@ def get_autogptq_infer_linear(backend, bits=4, group_size=128, sym=False):
disable_marlin=disable_marlin
)
else:
- QuantLinear = dynamically_import_QuantLinear(
+ QuantLinear = dynamically_import_QuantLinear( # pylint: disable=E1123
use_triton=use_triton,
desc_act=False,
group_size=group_size,
@@ -960,7 +960,7 @@ def get_autogptq_packing_qlinear(backend, bits=4, group_size=128, sym=False):
disable_marlin=disable_marlin,
)
else:
- QuantLinear = dynamically_import_QuantLinear(
+ QuantLinear = dynamically_import_QuantLinear(# pylint: disable=E1123
use_triton=use_triton,
desc_act=False,
group_size=group_size,
diff --git a/docs/full_range_sym.md b/docs/full_range_sym.md
new file mode 100644
index 000000000..2165bb242
--- /dev/null
+++ b/docs/full_range_sym.md
@@ -0,0 +1,15 @@
+W2G32 nsamples 512,iter 200, average accuracy of 10 tasks
+
+| Models | gptq_sym | asym | full_range_sym |
+|----------------------------|----------|------------|----------------|
+| Meta-Llama-3.1-8B-Instruct | 0.4500 | 0.52802 | **0.5381** |
+| Qwen2-7B | 0.5229 | **0.5559** | 0.5486 |
+
+W4G128 nsamples 128,iter 200, average accuracy of 10 tasks
+
+| Models | asym | full_range_sym |
+|----------------------------|------------|----------------|
+| Meta-Llama-3.1-8B-Instruct | 0.6342 | **0.6370** |
+| Qwen2-7B | 0.6143 | **0.6167** |
+| Mistral-7B-Instruct-v0.2 | 0.6606 | **0.6635** |
+| Phi-3-mini-4k-instruct | **0.6475** | 0.6432 |
diff --git a/examples/language-modeling/main.py b/examples/language-modeling/main.py
index e40ed0699..b85c80f2b 100644
--- a/examples/language-modeling/main.py
+++ b/examples/language-modeling/main.py
@@ -50,8 +50,8 @@ if __name__ == '__main__':
help="The device to be used for tuning. The default is set to auto/None,"
"allowing for automatic detection. Currently, device settings support CPU, GPU, and HPU.")
- parser.add_argument("--sym", action='store_true',
- help=" sym quantization")
+ parser.add_argument("--asym", action='store_true',
+ help=" asym quantization")
parser.add_argument("--iters", default=200, type=int,
help=" iters")
@@ -184,12 +184,12 @@ if __name__ == '__main__':
if args.deployment_device:
warnings.warn("The deployment_device is deprecated and will be removed in future version."
"Please use format instead", DeprecationWarning)
- if "gpu" in args.deployment_device and args.sym is False:
+ if "gpu" in args.deployment_device and args.asym is True:
print(
- "warning: The auto_gptq kernel has issues with asymmetric quantization. It is recommended to use --format='auto_round'")
+ "warning: The auto_gptq kernel has issues with asymmetric quantization. It is recommended to use sym quantization or --format='auto_round'")
- if "marlin" in args.deployment_device and args.sym is False:
- assert False, "marlin backend only supports sym quantization, please set --sym"
+ if "marlin" in args.deployment_device and args.asym is True:
+ assert False, "marlin backend only supports sym quantization, please remove --asym"
model_name = args.model_name
if model_name[-1] == "/":
@@ -323,7 +323,7 @@ if __name__ == '__main__':
error_message = "Please upgrade transformers>=4.38.0 to support lm-head quantization."
raise EnvironmentError(error_message)
- autoround = round(model, tokenizer, args.bits, args.group_size, sym=args.sym, batch_size=args.train_bs,
+ autoround = round(model, tokenizer, args.bits, args.group_size, sym=not args.asym, batch_size=args.train_bs,
dataset=args.dataset, seqlen=seqlen, nblocks=args.nblocks, iters=args.iters, lr=args.lr,
minmax_lr=args.minmax_lr, enable_quanted_input=not args.disable_quanted_input, device=device_str,
amp=not args.disable_amp, nsamples=args.nsamples,
diff --git a/requirements.txt b/requirements.txt
index 0cc1327bb..5755cf0f4 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -9,4 +9,5 @@ numpy < 2.0
threadpoolctl
lm-eval==0.4.4
tqdm
-packaging
\ No newline at end of file
+packaging
+auto-gptq>=0.7.1
\ No newline at end of file
diff --git a/test/test_autoround_acc.py b/test/test_autoround_acc.py
index 53f64ae5a..9e741bd77 100644
--- a/test/test_autoround_acc.py
+++ b/test/test_autoround_acc.py
@@ -67,7 +67,7 @@ class TestAutoRound(unittest.TestCase):
out1 = model_tmp(inp)
assert out0[0].equal(out1[0])
- self.assertTrue(isclose(float(out0[0][0][0][0]), -0.02076812833547592, rel_tol=1e-04))
+ self.assertTrue(isclose(float(out0[0][0][0][0]), -0.021002087742090225, rel_tol=1e-04))
if __name__ == "__main__":
diff --git a/test/test_block_names.py b/test/test_block_names.py
index bee555a22..6e101f4a7 100644
--- a/test/test_block_names.py
+++ b/test/test_block_names.py
@@ -168,6 +168,8 @@ class TestQuantizationBlocks(unittest.TestCase):
import auto_gptq
except:
return
+ if not torch.cuda.is_available():
+ return
quantized_model_path = "./saved"
autoround.save_quantized(quantized_model_path, inplace=False, safe_serialization=False, format="auto_round")