PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
support xpu in triton backend (#563)
Wenhua Chengf683f99a9b06
diff --git a/README.md b/README.mdindex ba7e50a16..53ab09119 100644--- a/README.md+++ b/README.md@@ -14,11 +14,12 @@ AutoRound---<div align="left">-AutoRound is an advanced quantization algorithm that delivers strong accuracy, even at 2-bit precision.-It leverages sign gradient descent to fine-tune both rounding values and min-max clipping thresholds in just 200 steps.-Designed for broad compatibility, it seamlessly supports a wide range of LLMs and is actively expanding to cover more VLMs as well.-It also supports quantization and inference across multiple hardware platforms, including CPU, Intel GPU, and CUDA.-AutoRound also offers a variety of useful features, including mixed-bit tuning and inference, lm-head quantization,+AutoRound is an advanced quantization algorithm that delivers strong accuracy, even at 2-bit precision.+It leverages sign gradient descent to fine-tune both rounding values and min-max clipping thresholds in just 200 steps.+Designed for broad compatibility, it seamlessly supports a wide range of LLMs and is actively expanding to cover more+VLMs as well.+It also supports quantization and inference across multiple hardware platforms, including CPU, Intel GPU, and CUDA.+AutoRound also offers a variety of useful features, including mixed-bit tuning and inference, lm-head quantization,support for exporting to formats like GPTQ/AWQ/GGUF, and flexible tuning recipes. The belowimage presents an overview of AutoRound. Check out our paper on [arxiv](https://arxiv.org/pdf/2309.05516) for moredetails and quantized models in several Hugging Face Spaces,@@ -32,8 +33,12 @@ and [fbaldassarri](https://huggingface.co/fbaldassarri).<div align="left">## What's New-* [2025/04] AutoRound provides some recipes for Qwen3 series, please refer to [Qwen3-8B-sym-recipe](./docs/Qwen3-8B-sym-recipe.md) and [Qwen3-14B-sym-recipe](./docs/Qwen3-14B-sym-recipe.md) for more details.-* [2025/04] AutoRound has been integrated into Transformers. You can run models in the AutoRound format directly with Transformers versions later than 4.51.3.++* [2025/04] AutoRound provides some recipes for Qwen3 series, please refer+ to [Qwen3-8B-sym-recipe](./docs/Qwen3-8B-sym-recipe.md) and [Qwen3-14B-sym-recipe](./docs/Qwen3-14B-sym-recipe.md) for+ more details.+* [2025/04] AutoRound has been integrated into Transformers. You can run models in the AutoRound format directly with+ Transformers versions later than 4.51.3.* [2025/03] The INT2-mixed R1 model (~200GB) retains 97.9% accuracy. Checkout [OPEA/DeepSeek-R1-int2-mixed-sym-inc](https://huggingface.co/OPEA/DeepSeek-R1-int2-mixed-sym-inc).* [2025/01] We provide experimental support for GGUF q4_0 and q4_1 formats.@@ -83,7 +88,7 @@ auto-round \--output_dir ./tmp_autoround```-We offer another two configurations, `auto-round-best` and `auto-round-light`,+We offer another two configurations, `auto-round-best` and `auto-round-light`,designed for optimal accuracy and improved speed,respectively. Details are as follows.<details>@@ -313,7 +318,8 @@ in [Gaudi Guide](https://docs.habana.ai/en/latest/).#### HPU/CPU/XPU/CUDA-**Please avoid manually moving the quantized model to a different device** (e.g., model.to('cpu')) during inference, as this may cause unexpected exceptions.+**Please avoid manually moving the quantized model to a different device** (e.g., model.to('cpu')) during inference, as+this may cause unexpected exceptions.```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizer@@ -332,24 +338,25 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))AutoRound automatically selects the best available backend based on the installed libraries and prompts the user toinstall additional libraries when a better backend is found. On CUDA, the default priority is Marlin > ExLLaMAV2 >-Triton, but the final choice depends on factors such as bits, group_size, packing format compatibility, etc. And the backend may not always be the most suitable for certain devices. Please refer+Triton, but the final choice depends on factors such as bits, group_size, packing format compatibility, etc. And the+backend may not always be the most suitable for certain devices. Please referto the following table for the details and specify the backend you want.-| Name | Devices | Bits | Dtypes | Priority | Packing format | Requirements |-|--------------------------------------|---------|---------|-----------|----------|-----------------|-------------------------------|-| ipex | cpu/xpu | 4 | BF16/FP16 | 5 | gptq_zp+-1/awq | intel-extension-for-pytorch |-| itrex | cpu | 2,4,8 | BF16/FP16 | 0 | gptq_zp+-1/awq | intel-extension-for-transformers |-| marlin | cuda | 4,8 | BF16/FP16 | 6 | gptq/gptq_zp+-1 | gptqmodel |-| exllamav2 or<br/>gptqmodel:exllamav2 | cuda | 4 | BF16/FP16 | 5 | gptq | gptqmodel |-| exllamav2 or<br/>gptq:exllamav2 | cuda | 4 | FP16 | 5 | gptq_zp+-1 | auto-gptq |-| gptq:cuda | cuda | 2,3,4,8 | FP16 | 0 | gptq_zp+-1 | auto-gptq |-| triton | cuda | 2,4,8 | BF16/FP16 | 1 | gptq/gptq_zp+-1 | auto-round |-| awq | cuda | 4 | FP16 | 5 | awq | auto-awq |-| hpu | hpu | 4 | BF16 | 0 | gptq/gptq_zp+-1 | auto-round |+| Name | Devices | Bits | Dtypes | Priority | Packing format | Requirements |+|--------------------------------------|----------|---------|-----------|----------|-----------------|----------------------------------|+| ipex | cpu/xpu | 4 | BF16/FP16 | 5 | gptq_zp+-1/awq | intel-extension-for-pytorch |+| itrex | cpu | 2,4,8 | BF16/FP16 | 0 | gptq_zp+-1/awq | intel-extension-for-transformers |+| marlin | cuda | 4,8 | BF16/FP16 | 6 | gptq/gptq_zp+-1 | gptqmodel |+| exllamav2 or<br/>gptqmodel:exllamav2 | cuda | 4 | BF16/FP16 | 5 | gptq | gptqmodel |+| exllamav2 or<br/>gptq:exllamav2 | cuda | 4 | FP16 | 5 | gptq_zp+-1 | auto-gptq |+| gptq:cuda | cuda | 2,3,4,8 | FP16 | 0 | gptq_zp+-1 | auto-gptq |+| triton | cuda/xpu | 2,4,8 | BF16/FP16 | 1 | gptq/gptq_zp+-1 | auto-round |+| awq | cuda | 4 | FP16 | 5 | awq | auto-awq |+| hpu | hpu | 4 | BF16 | 0 | gptq/gptq_zp+-1 | auto-round |```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizer-from auto_round import AutoRoundConfig+from auto_round import AutoRoundConfigquantized_model_path = "./tmp_autoround"quantization_config = AutoRoundConfig(backend="auto")@@ -463,7 +470,7 @@ model architecture or kernel limitations.|--------------------------------|---------------------|----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|| allenai/Molmo | pile | X | [Molmo-7B-D-0924-int4-sym](https://huggingface.co/OPEA/Molmo-7B-D-0924-int4-sym-inc), [Molmo-72B-0924-int4-sym-gptq](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-gptq-inc), [Molmo-72B-0924-int4-sym](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-inc) || deepseek-ai/deepseek-vl2 | pile/llava | √ | [deepseek-vl2-int4-sym-gptq](https://huggingface.co/OPEA/deepseek-vl2-int4-sym-gptq-inc) |-| google/gemma-3 | pile/llava | √ | [gemma-3-12b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-12b-it-AutoRound-gguf-q4-0), [gemma-3-27b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-27b-it-AutoRound-gguf-q4-0), [gemma-3-12b-it-int4-AutoRound](https://huggingface.co/OPEA/gemma-3-12b-it-int4-AutoRound), [gemma-3-27b-it-int4-AutoRound](https://huggingface.co/OPEA/gemma-3-27b-it-int4-AutoRound) |+| google/gemma-3 | pile/llava | √ | [gemma-3-12b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-12b-it-AutoRound-gguf-q4-0), [gemma-3-27b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-27b-it-AutoRound-gguf-q4-0), [gemma-3-12b-it-int4-AutoRound](https://huggingface.co/OPEA/gemma-3-12b-it-int4-AutoRound), [gemma-3-27b-it-int4-AutoRound](https://huggingface.co/OPEA/gemma-3-27b-it-int4-AutoRound) || HuggingFaceTB/SmolVLM | pile/llava | √ | [SmolVLM-Instruct-int4-sym](https://huggingface.co/OPEA/SmolVLM-Instruct-int4-sym-inc) || ibm-granite/granite-vision-3.2 | pile/llava | - | || liuhaotian/Llava-v1.5 | pile/llava | X | [llava-v1.5-7b-int4-sym](https://huggingface.co/OPEA/llava-v1.5-7b-int4-sym-inc) |@@ -490,7 +497,6 @@ AutoRound has been integrated into multiple repositories.[pytorch/ao](https://github.com/pytorch/ao)-## ReferenceIf you find AutoRound useful for your research, please cite our paper:@@ -503,15 +509,3 @@ If you find AutoRound useful for your research, please cite our paper:year={2023}}```------------diff --git a/auto_round/export/export_to_autoround/export.py b/auto_round/export/export_to_autoround/export.pyindex 02ead0464..2d39eb48b 100644--- a/auto_round/export/export_to_autoround/export.py+++ b/auto_round/export/export_to_autoround/export.py@@ -23,7 +23,7 @@ import transformersimport auto_round.export.export_to_autoround.qlinear_triton_act-import auto_round_extension.cuda.qlinear_tritonv2+import auto_round_extension.triton.qlinear_tritonv2from auto_round.utils import get_module, logger, set_module, supported_layer_types, check_to_quantized, \filter_quantization_configimport threadpoolctl as tctl@@ -75,7 +75,7 @@ def dynamic_import_quant_linear_for_packing(backend, bits, group_size, sym, act_if act_bits <= 8: ##easily have bug for other configuration, need to refine code laterreturn auto_round.export.export_to_autoround.qlinear_triton_act.QuantLinear- from auto_round_extension.cuda.qlinear_tritonv2 import QuantLinear+ from auto_round_extension.triton.qlinear_tritonv2 import QuantLinearreturn QuantLinearelif "auto_round" in backend and "gptq" in backend and bits in (2, 4, 8):from auto_round.export.export_to_autoround.qlinear_triton import QuantLinear ##no g_idx@@ -84,7 +84,7 @@ def dynamic_import_quant_linear_for_packing(backend, bits, group_size, sym, act_from ..export_to_awq.utils import WQLinear_GEMMreturn WQLinear_GEMMelif "gptqmodel" in backend:- return auto_round_extension.cuda.qlinear_tritonv2.QuantLinear+ return auto_round_extension.triton.qlinear_tritonv2.QuantLinearelif "gptq" in backend and not "gptqmodel" in backend: ## have g_idxreturn get_autogptq_packing_qlinear(backend, bits, group_size, sym)else:@@ -194,7 +194,7 @@ def pack_layer(layer_name, model, backend):qlayer = new_layerimport auto_round.export.export_to_autoround.qlinear_tritonif sym and isinstance(QuantLinear, (auto_round.export.export_to_autoround.qlinear_triton.QuantLinear,- auto_round_extension.cuda.qlinear_tritonv2.QuantLinear)):+ auto_round_extension.triton.qlinear_tritonv2.QuantLinear)):zp = int(zp.flatten()[0])qlayer.to("cpu")diff --git a/auto_round/inference/backend.py b/auto_round/inference/backend.pyindex 53d99ada0..7f0ba35d5 100644--- a/auto_round/inference/backend.py+++ b/auto_round/inference/backend.py@@ -13,7 +13,6 @@# limitations under the License.import functools-import loggingfrom dataclasses import dataclass, fieldfrom typing import List, Any, Optional@@ -129,22 +128,23 @@ BackendInfos['auto_gptq:cuda'] = BackendInfo(device=["cuda"], sym=[True, False],requirements=["auto-gptq>=0.7.1"])-BackendInfos['auto_round:tritonv2'] = BackendInfo(device=["cuda"], sym=[True, False],+BackendInfos['auto_round:tritonv2'] = BackendInfo(device=["cuda","cpu","xpu"], sym=[True, False],packing_format="int32",dtype=["float16", "bfloat16"],bits=[2, 4, 8],priority=1, feature_checks=[feature_multiply_checker_32],alias=["auto_round", "tritonv2", "triton"],- requirements=["auto-round>=0.5.0", "triton>=2.0"]+ requirements=["triton>=2.0","auto-round>=0.5.0"])-BackendInfos['auto_round:tritonv2_zp'] = BackendInfo(device=["cuda"], sym=[True], ## asym has accuracy issue+BackendInfos['auto_round:tritonv2_zp'] = BackendInfo(device=["cuda","xpu"], sym=[True], ## asym has accuracy+ # issuepacking_format="int32_zp",dtype=["float16", "bfloat16"],bits=[2, 4, 8],priority=1, feature_checks=[feature_multiply_checker_32],alias=["tritonv2", "tritonv2_zp", "triton"],- requirements=["auto-round>=0.5.0", "triton>=2.0"]+ requirements=[ "triton>=2.0","auto-round>=0.5.0"])BackendInfos['gptqmodel:marlin'] = BackendInfo(device=["cuda"], sym=[True],@@ -406,11 +406,11 @@ def dynamic_import_inference_linear(backend, bits, group_size, sym):return WQLinear_GEMMif backend == "auto_round:tritonv2":- from auto_round_extension.cuda.qlinear_tritonv2 import QuantLinear+ from auto_round_extension.triton.qlinear_tritonv2 import QuantLinearreturn QuantLinearif backend == "auto_round:tritonv2_zp":- from auto_round_extension.cuda.qlinear_tritonv2_zp import QuantLinear+ from auto_round_extension.triton.qlinear_tritonv2_zp import QuantLinearreturn QuantLinearraise ValueError(f"unsupported backend {backend}, please set it to `auto` and retry")diff --git a/auto_round_extension/triton/__init__.py b/auto_round_extension/triton/__init__.pynew file mode 100644index 000000000..e69de29bbdiff --git a/auto_round_extension/cuda/qlinear_tritonv2.py b/auto_round_extension/triton/qlinear_tritonv2.pysimilarity index 95%rename from auto_round_extension/cuda/qlinear_tritonv2.pyrename to auto_round_extension/triton/qlinear_tritonv2.pyindex 655957e86..b9d8931b9 100644--- a/auto_round_extension/cuda/qlinear_tritonv2.py+++ b/auto_round_extension/triton/qlinear_tritonv2.py@@ -7,14 +7,17 @@ import torch.nn as nnimport transformers-from auto_round_extension.cuda.triton_utils.mixin import TritonModuleMixin+from auto_round_extension.triton.triton_utils.mixin import TritonModuleMixinlogger = getLogger(__name__)try:- from auto_round_extension.cuda.triton_utils.dequant import QuantLinearFunction, quant_matmul_248+ from auto_round_extension.triton.triton_utils.dequant import QuantLinearFunction, quant_matmul_248except ImportError as e:+ if torch.xpu.is_available():+ logger.error(f"please make sure your triton version is same with `pytorch-triton-xpu` library ")+ exit(-1)triton_import_exception = edef error_raiser_triton(*args, **kwargs):diff --git a/auto_round_extension/cuda/qlinear_tritonv2_zp.py b/auto_round_extension/triton/qlinear_tritonv2_zp.pysimilarity index 95%rename from auto_round_extension/cuda/qlinear_tritonv2_zp.pyrename to auto_round_extension/triton/qlinear_tritonv2_zp.pyindex 2dadfc0ae..530e14d8d 100644--- a/auto_round_extension/cuda/qlinear_tritonv2_zp.py+++ b/auto_round_extension/triton/qlinear_tritonv2_zp.py@@ -7,14 +7,17 @@ import torch.nn as nnimport transformers-from auto_round_extension.cuda.triton_utils_zp.mixin import TritonModuleMixin+from auto_round_extension.triton.triton_utils_zp.mixin import TritonModuleMixinlogger = getLogger(__name__)try:- from auto_round_extension.cuda.triton_utils_zp.dequant import QuantLinearFunction, quant_matmul_248+ from auto_round_extension.triton.triton_utils_zp.dequant import QuantLinearFunction, quant_matmul_248except ImportError as e:+ if torch.xpu.is_available():+ logger.error(f"please make sure your triton version is same with `pytorch-triton-xpu` library ")+ exit(-1)triton_import_exception = edef error_raiser_triton(*args, **kwargs):diff --git a/auto_round_extension/cuda/triton_utils/__init__.py b/auto_round_extension/triton/triton_utils/__init__.pysimilarity index 100%rename from auto_round_extension/cuda/triton_utils/__init__.pyrename to auto_round_extension/triton/triton_utils/__init__.pydiff --git a/auto_round_extension/cuda/triton_utils/custom_autotune.py b/auto_round_extension/triton/triton_utils/custom_autotune.pysimilarity index 100%rename from auto_round_extension/cuda/triton_utils/custom_autotune.pyrename to auto_round_extension/triton/triton_utils/custom_autotune.pydiff --git a/auto_round_extension/cuda/triton_utils/dequant.py b/auto_round_extension/triton/triton_utils/dequant.pysimilarity index 82%rename from auto_round_extension/cuda/triton_utils/dequant.pyrename to auto_round_extension/triton/triton_utils/dequant.pyindex 9e4b30744..43905629d 100644--- a/auto_round_extension/cuda/triton_utils/dequant.py+++ b/auto_round_extension/triton/triton_utils/dequant.py@@ -119,34 +119,44 @@ def dequant_kernel_248(tl.store(out_ptr + (x_index), weights, mask=xmask)-def dequant248(qweight, scales, qzeros, g_idx, bits, maxq=None, input_dtype=torch.float16):+def dequant248_core(qweight, scales, qzeros, g_idx, bits, maxq=None, input_dtype=torch.float16):"""Launcher for triton dequant kernel. Only valid for bits = 2, 4, 8"""- with torch.cuda.device(qweight.device):- num_groups = scales.shape[0]- outfeatures = scales.shape[1]- infeatures = g_idx.shape[0]-- out = torch.empty((infeatures, outfeatures), device=qweight.device, dtype=input_dtype)- numels = out.numel()- maxq = 2 ** bits - 1 if maxq is None else maxq- grid = lambda meta: (triton.cdiv(numels, meta["X_BLOCK"]),) # noqa: E731-- dequant_kernel_248[grid](- g_idx,- scales,- qweight,- qzeros,- out,- numels,- maxq=maxq,- bits=bits,- outfeatures=outfeatures,- num_groups=num_groups,- )- return out+ num_groups = scales.shape[0]+ outfeatures = scales.shape[1]+ infeatures = g_idx.shape[0]++ out = torch.empty((infeatures, outfeatures), device=qweight.device, dtype=input_dtype)+ numels = out.numel()+ maxq = 2 ** bits - 1 if maxq is None else maxq+ grid = lambda meta: (triton.cdiv(numels, meta["X_BLOCK"]),) # noqa: E731++ dequant_kernel_248[grid](+ g_idx,+ scales,+ qweight,+ qzeros,+ out,+ numels,+ maxq=maxq,+ bits=bits,+ outfeatures=outfeatures,+ num_groups=num_groups,+ )+ return out++def dequant248(qweight, scales, qzeros, g_idx, bits, maxq=None, input_dtype=torch.float16):+ """+ Launcher for triton dequant kernel. Only valid for bits = 2, 4, 8+ """+ device_type = qweight.device.type+ if device_type in {"cuda", "xpu"}:+ with getattr(torch, device_type).device(qweight.device):+ return dequant248_core(qweight, scales, qzeros, g_idx, bits, maxq=maxq, input_dtype=input_dtype)+ else:+ raise ValueError(f"Unsupported device type: {device_type}")def quant_matmul_248(input, qweight, scales, qzeros, g_idx, bits, maxq=None, transpose=Falsediff --git a/auto_round_extension/cuda/triton_utils/kernels.py b/auto_round_extension/triton/triton_utils/kernels.pysimilarity index 79%rename from auto_round_extension/cuda/triton_utils/kernels.pyrename to auto_round_extension/triton/triton_utils/kernels.pyindex 44981c7a3..eebd29cdb 100644--- a/auto_round_extension/cuda/triton_utils/kernels.py+++ b/auto_round_extension/triton/triton_utils/kernels.py@@ -380,65 +380,82 @@ def silu(x):return x * tl.sigmoid(x)+def quant_matmul_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq):+ output = torch.empty((input.shape[0], qweight.shape[1]), device=input.device, dtype=input.dtype)+ grid = lambda META: ( # noqa: E731+ triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(qweight.shape[1], META["BLOCK_SIZE_N"]),+ )+ quant_matmul_248_kernel[grid](+ input,+ qweight,+ output,+ scales.to(input.dtype),+ qzeros,+ g_idx,+ input.shape[0],+ qweight.shape[1],+ input.shape[1],+ bits,+ maxq,+ input.stride(0),+ input.stride(1),+ qweight.stride(0),+ qweight.stride(1),+ output.stride(0),+ output.stride(1),+ scales.stride(0),+ qzeros.stride(0),+ )+ return output++def quant_matmul_248(input, qweight, scales, qzeros, g_idx, bits, maxq):- with torch.cuda.device(input.device):- output = torch.empty((input.shape[0], qweight.shape[1]), device=input.device, dtype=input.dtype)- grid = lambda META: ( # noqa: E731- triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(qweight.shape[1], META["BLOCK_SIZE_N"]),- )- quant_matmul_248_kernel[grid](- input,- qweight,- output,- scales.to(input.dtype),- qzeros,- g_idx,- input.shape[0],- qweight.shape[1],- input.shape[1],- bits,- maxq,- input.stride(0),- input.stride(1),- qweight.stride(0),- qweight.stride(1),- output.stride(0),- output.stride(1),- scales.stride(0),- qzeros.stride(0),- )- return output+ device_type = input.device.type+ if device_type in {"cuda", "xpu"}:+ with getattr(torch, device_type).device(input.device):+ return quant_matmul_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq)+ else:+ raise ValueError(f"Unsupported device type: {device_type}")+++def transpose_quant_matmul_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq):+ output_dim = (qweight.shape[0] * 32) // bits+ output = torch.empty((input.shape[0], output_dim), device=input.device, dtype=input.dtype)+ grid = lambda META: ( # noqa: E731+ triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(output_dim, META["BLOCK_SIZE_K"]),+ )+ transpose_quant_matmul_248_kernel[grid](+ input,+ qweight,+ output,+ scales.to(input.dtype),+ qzeros,+ g_idx,+ input.shape[0],+ qweight.shape[1],+ output_dim,+ bits,+ maxq,+ input.stride(0),+ input.stride(1),+ qweight.stride(0),+ qweight.stride(1),+ output.stride(0),+ output.stride(1),+ scales.stride(0),+ qzeros.stride(0),+ )+ return outputdef transpose_quant_matmul_248(input, qweight, scales, qzeros, g_idx, bits, maxq):- with torch.cuda.device(input.device):- output_dim = (qweight.shape[0] * 32) // bits- output = torch.empty((input.shape[0], output_dim), device=input.device, dtype=input.dtype)- grid = lambda META: ( # noqa: E731- triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(output_dim, META["BLOCK_SIZE_K"]),- )- transpose_quant_matmul_248_kernel[grid](- input,- qweight,- output,- scales.to(input.dtype),- qzeros,- g_idx,- input.shape[0],- qweight.shape[1],- output_dim,- bits,- maxq,- input.stride(0),- input.stride(1),- qweight.stride(0),- qweight.stride(1),- output.stride(0),- output.stride(1),- scales.stride(0),- qzeros.stride(0),- )- return output+ device_type = input.device.type+ if device_type in {"cuda", "xpu"}:+ with getattr(torch, device_type).device(input.device):+ return transpose_quant_matmul_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq)+ else:+ raise ValueError(f"Unsupported device type: {device_type}")+class QuantLinearFunction(torch.autograd.Function):@@ -462,34 +479,43 @@ class QuantLinearFunction(torch.autograd.Function):return grad_input, None, None, None, None, None, None+def quant_matmul_inference_only_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq):+ output = torch.empty((input.shape[0], qweight.shape[1]), device=input.device, dtype=torch.float16)+ grid = lambda META: ( # noqa: E731+ triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(qweight.shape[1], META["BLOCK_SIZE_N"]),+ )+ quant_matmul_248_kernel[grid](+ input,+ qweight,+ output,+ scales,+ qzeros,+ g_idx,+ input.shape[0],+ qweight.shape[1],+ input.shape[1],+ bits,+ maxq,+ input.stride(0),+ input.stride(1),+ qweight.stride(0),+ qweight.stride(1),+ output.stride(0),+ output.stride(1),+ scales.stride(0),+ qzeros.stride(0),+ )+ return output++def quant_matmul_inference_only_248(input, qweight, scales, qzeros, g_idx, bits, maxq):- with torch.cuda.device(input.device):- output = torch.empty((input.shape[0], qweight.shape[1]), device=input.device, dtype=torch.float16)- grid = lambda META: ( # noqa: E731- triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(qweight.shape[1], META["BLOCK_SIZE_N"]),- )- quant_matmul_248_kernel[grid](- input,- qweight,- output,- scales,- qzeros,- g_idx,- input.shape[0],- qweight.shape[1],- input.shape[1],- bits,- maxq,- input.stride(0),- input.stride(1),- qweight.stride(0),- qweight.stride(1),- output.stride(0),- output.stride(1),- scales.stride(0),- qzeros.stride(0),- )- return output+ device_type = input.device.type+ if device_type in {"cuda", "xpu"}:+ with getattr(torch, device_type).device(input.device):+ return quant_matmul_inference_only_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq)+ else:+ raise ValueError(f"Unsupported device type: {device_type}")+class QuantLinearInferenceOnlyFunction(torch.autograd.Function):diff --git a/auto_round_extension/cuda/triton_utils/mixin.py b/auto_round_extension/triton/triton_utils/mixin.pysimilarity index 100%rename from auto_round_extension/cuda/triton_utils/mixin.pyrename to auto_round_extension/triton/triton_utils/mixin.pydiff --git a/auto_round_extension/cuda/triton_utils_zp/__init__.py b/auto_round_extension/triton/triton_utils_zp/__init__.pysimilarity index 100%rename from auto_round_extension/cuda/triton_utils_zp/__init__.pyrename to auto_round_extension/triton/triton_utils_zp/__init__.pydiff --git a/auto_round_extension/cuda/triton_utils_zp/custom_autotune.py b/auto_round_extension/triton/triton_utils_zp/custom_autotune.pysimilarity index 100%rename from auto_round_extension/cuda/triton_utils_zp/custom_autotune.pyrename to auto_round_extension/triton/triton_utils_zp/custom_autotune.pydiff --git a/auto_round_extension/cuda/triton_utils_zp/dequant.py b/auto_round_extension/triton/triton_utils_zp/dequant.pysimilarity index 82%rename from auto_round_extension/cuda/triton_utils_zp/dequant.pyrename to auto_round_extension/triton/triton_utils_zp/dequant.pyindex 8abf6f514..c6f47c1a1 100644--- a/auto_round_extension/cuda/triton_utils_zp/dequant.py+++ b/auto_round_extension/triton/triton_utils_zp/dequant.py@@ -38,7 +38,6 @@ import itertoolsimport torchimport tritonimport triton.language as tl-from torch.cuda.amp import custom_bwd, custom_fwddef make_dequant_configs(block_sizes, num_warps):@@ -119,33 +118,44 @@ def dequant_kernel_248(tl.store(out_ptr + (x_index), weights, mask=xmask)-def dequant248(qweight, scales, qzeros, g_idx, bits, maxq=None, input_dtype=torch.float16):+def dequant248_core(qweight, scales, qzeros, g_idx, bits, maxq=None, input_dtype=torch.float16):"""Launcher for triton dequant kernel. Only valid for bits = 2, 4, 8"""- with torch.cuda.device(qweight.device):- num_groups = scales.shape[0]- outfeatures = scales.shape[1]- infeatures = g_idx.shape[0]-- out = torch.empty((infeatures, outfeatures), device=qweight.device, dtype=input_dtype)- numels = out.numel()- maxq = 2 ** bits - 1 if maxq is None else maxq- grid = lambda meta: (triton.cdiv(numels, meta["X_BLOCK"]),) # noqa: E731-- dequant_kernel_248[grid](- g_idx,- scales,- qweight,- qzeros,- out,- numels,- maxq=maxq,- bits=bits,- outfeatures=outfeatures,- num_groups=num_groups,- )- return out+ num_groups = scales.shape[0]+ outfeatures = scales.shape[1]+ infeatures = g_idx.shape[0]++ out = torch.empty((infeatures, outfeatures), device=qweight.device, dtype=input_dtype)+ numels = out.numel()+ maxq = 2 ** bits - 1 if maxq is None else maxq+ grid = lambda meta: (triton.cdiv(numels, meta["X_BLOCK"]),) # noqa: E731++ dequant_kernel_248[grid](+ g_idx,+ scales,+ qweight,+ qzeros,+ out,+ numels,+ maxq=maxq,+ bits=bits,+ outfeatures=outfeatures,+ num_groups=num_groups,+ )+ return out+++def dequant248(qweight, scales, qzeros, g_idx, bits, maxq=None, input_dtype=torch.float16):+ """+ Launcher for triton dequant kernel. Only valid for bits = 2, 4, 8+ """+ device_type = qweight.device.type+ if device_type in {"cuda", "xpu"}:+ with getattr(torch, device_type).device(qweight.device):+ return dequant248_core(qweight, scales, qzeros, g_idx, bits, maxq=maxq, input_dtype=input_dtype)+ else:+ raise ValueError(f"Unsupported device type: {device_type}")def quant_matmul_248(diff --git a/auto_round_extension/cuda/triton_utils_zp/kernels.py b/auto_round_extension/triton/triton_utils_zp/kernels.pysimilarity index 79%rename from auto_round_extension/cuda/triton_utils_zp/kernels.pyrename to auto_round_extension/triton/triton_utils_zp/kernels.pyindex 44981c7a3..f361a3dde 100644--- a/auto_round_extension/cuda/triton_utils_zp/kernels.py+++ b/auto_round_extension/triton/triton_utils_zp/kernels.py@@ -39,7 +39,6 @@ from logging import getLoggerimport torchimport tritonimport triton.language as tl-from torch.cuda.amp import custom_bwd, custom_fwdfrom . import custom_autotune@@ -379,71 +378,85 @@ def transpose_quant_matmul_248_kernel(def silu(x):return x * tl.sigmoid(x)+def quant_matmul_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq):+ output = torch.empty((input.shape[0], qweight.shape[1]), device=input.device, dtype=input.dtype)+ grid = lambda META: ( # noqa: E731+ triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(qweight.shape[1], META["BLOCK_SIZE_N"]),+ )+ quant_matmul_248_kernel[grid](+ input,+ qweight,+ output,+ scales.to(input.dtype),+ qzeros,+ g_idx,+ input.shape[0],+ qweight.shape[1],+ input.shape[1],+ bits,+ maxq,+ input.stride(0),+ input.stride(1),+ qweight.stride(0),+ qweight.stride(1),+ output.stride(0),+ output.stride(1),+ scales.stride(0),+ qzeros.stride(0),+ )+ return output+def quant_matmul_248(input, qweight, scales, qzeros, g_idx, bits, maxq):- with torch.cuda.device(input.device):- output = torch.empty((input.shape[0], qweight.shape[1]), device=input.device, dtype=input.dtype)- grid = lambda META: ( # noqa: E731- triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(qweight.shape[1], META["BLOCK_SIZE_N"]),- )- quant_matmul_248_kernel[grid](- input,- qweight,- output,- scales.to(input.dtype),- qzeros,- g_idx,- input.shape[0],- qweight.shape[1],- input.shape[1],- bits,- maxq,- input.stride(0),- input.stride(1),- qweight.stride(0),- qweight.stride(1),- output.stride(0),- output.stride(1),- scales.stride(0),- qzeros.stride(0),- )- return output+ device_type = input.device.type+ if device_type in {"cuda", "xpu"}:+ with getattr(torch, device_type).device(input.device):+ return quant_matmul_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq)+ else:+ raise ValueError(f"Unsupported device type: {device_type}")+++def transpose_quant_matmul_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq):+ output_dim = (qweight.shape[0] * 32) // bits+ output = torch.empty((input.shape[0], output_dim), device=input.device, dtype=input.dtype)+ grid = lambda META: ( # noqa: E731+ triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(output_dim, META["BLOCK_SIZE_K"]),+ )+ transpose_quant_matmul_248_kernel[grid](+ input,+ qweight,+ output,+ scales.to(input.dtype),+ qzeros,+ g_idx,+ input.shape[0],+ qweight.shape[1],+ output_dim,+ bits,+ maxq,+ input.stride(0),+ input.stride(1),+ qweight.stride(0),+ qweight.stride(1),+ output.stride(0),+ output.stride(1),+ scales.stride(0),+ qzeros.stride(0),+ )+ return outputdef transpose_quant_matmul_248(input, qweight, scales, qzeros, g_idx, bits, maxq):- with torch.cuda.device(input.device):- output_dim = (qweight.shape[0] * 32) // bits- output = torch.empty((input.shape[0], output_dim), device=input.device, dtype=input.dtype)- grid = lambda META: ( # noqa: E731- triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(output_dim, META["BLOCK_SIZE_K"]),- )- transpose_quant_matmul_248_kernel[grid](- input,- qweight,- output,- scales.to(input.dtype),- qzeros,- g_idx,- input.shape[0],- qweight.shape[1],- output_dim,- bits,- maxq,- input.stride(0),- input.stride(1),- qweight.stride(0),- qweight.stride(1),- output.stride(0),- output.stride(1),- scales.stride(0),- qzeros.stride(0),- )- return output+ device_type = input.device.type+ if device_type in {"cuda", "xpu"}:+ with getattr(torch, device_type).device(input.device):+ return transpose_quant_matmul_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq)+ else:+ raise ValueError(f"Unsupported device type: {device_type}")class QuantLinearFunction(torch.autograd.Function):@staticmethod- @custom_fwddef forward(ctx, input, qweight, scales, qzeros, g_idx, bits, maxq):output = quant_matmul_248(input, qweight, scales, qzeros, g_idx, bits, maxq)ctx.save_for_backward(qweight, scales, qzeros, g_idx)@@ -451,7 +464,6 @@ class QuantLinearFunction(torch.autograd.Function):return output@staticmethod- @custom_bwddef backward(ctx, grad_output):qweight, scales, qzeros, g_idx = ctx.saved_tensorsbits, maxq = ctx.bits, ctx.maxq@@ -462,39 +474,46 @@ class QuantLinearFunction(torch.autograd.Function):return grad_input, None, None, None, None, None, None+def quant_matmul_inference_only_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq):+ output = torch.empty((input.shape[0], qweight.shape[1]), device=input.device, dtype=torch.float16)+ grid = lambda META: ( # noqa: E731+ triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(qweight.shape[1], META["BLOCK_SIZE_N"]),+ )+ quant_matmul_248_kernel[grid](+ input,+ qweight,+ output,+ scales,+ qzeros,+ g_idx,+ input.shape[0],+ qweight.shape[1],+ input.shape[1],+ bits,+ maxq,+ input.stride(0),+ input.stride(1),+ qweight.stride(0),+ qweight.stride(1),+ output.stride(0),+ output.stride(1),+ scales.stride(0),+ qzeros.stride(0),+ )+ return output++def quant_matmul_inference_only_248(input, qweight, scales, qzeros, g_idx, bits, maxq):- with torch.cuda.device(input.device):- output = torch.empty((input.shape[0], qweight.shape[1]), device=input.device, dtype=torch.float16)- grid = lambda META: ( # noqa: E731- triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(qweight.shape[1], META["BLOCK_SIZE_N"]),- )- quant_matmul_248_kernel[grid](- input,- qweight,- output,- scales,- qzeros,- g_idx,- input.shape[0],- qweight.shape[1],- input.shape[1],- bits,- maxq,- input.stride(0),- input.stride(1),- qweight.stride(0),- qweight.stride(1),- output.stride(0),- output.stride(1),- scales.stride(0),- qzeros.stride(0),- )- return output+ device_type = input.device.type+ if device_type in {"cuda", "xpu"}:+ with getattr(torch, device_type).device(input.device):+ return quant_matmul_inference_only_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq)+ else:+ raise ValueError(f"Unsupported device type: {device_type}")class QuantLinearInferenceOnlyFunction(torch.autograd.Function):@staticmethod- @custom_fwd(cast_inputs=torch.float16)def forward(ctx, input, qweight, scales, qzeros, g_idx, bits, maxq):output = quant_matmul_248(input, qweight, scales, qzeros, g_idx, bits, maxq)return outputdiff --git a/auto_round_extension/cuda/triton_utils_zp/mixin.py b/auto_round_extension/triton/triton_utils_zp/mixin.pysimilarity index 100%rename from auto_round_extension/cuda/triton_utils_zp/mixin.pyrename to auto_round_extension/triton/triton_utils_zp/mixin.pydiff --git a/docs/step_by_step.md b/docs/step_by_step.mdindex b7e6aab49..ee68d00c8 100644--- a/docs/step_by_step.md+++ b/docs/step_by_step.md@@ -480,17 +480,17 @@ text = "There is a girl who likes adventure,"inputs = tokenizer(text, return_tensors="pt").to(model.device)print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))```-| Name | Devices | Bits | Dtypes | Priority | Packing format | Requirements |-|--------------------------------------|---------|---------|-----------|----------|-----------------|-------------------------------|-| ipex | cpu/xpu | 4 | BF16/FP16 | 5 | gptq_zp+-1/awq | intel-extension-for-pytorch |-| itrex | cpu | 2,4,8 | BF16/FP16 | 0 | gptq_zp+-1/awq | intel-extension-for-transformers |-| marlin | cuda | 4,8 | BF16/FP16 | 6 | gptq/gptq_zp+-1 | gptqmodel |-| exllamav2 or<br/>gptqmodel:exllamav2 | cuda | 4 | BF16/FP16 | 5 | gptq | gptqmodel |-| exllamav2 or<br/>gptq:exllamav2 | cuda | 4 | FP16 | 5 | gptq_zp+-1 | auto-gptq |-| gptq:cuda | cuda | 2,3,4,8 | FP16 | 0 | gptq_zp+-1 | auto-gptq |-| triton | cuda | 2,4,8 | BF16/FP16 | 1 | gptq/gptq_zp+-1 | auto-round |-| awq | cuda | 4 | FP16 | 5 | awq | auto-awq |-| hpu | hpu | 4 | BF16 | 0 | gptq/gptq_zp+-1 | auto-round |+| Name | Devices | Bits | Dtypes | Priority | Packing format | Requirements |+|--------------------------------------|----------|---------|-----------|----------|-----------------|-------------------------------|+| ipex | cpu/xpu | 4 | BF16/FP16 | 5 | gptq_zp+-1/awq | intel-extension-for-pytorch |+| itrex | cpu | 2,4,8 | BF16/FP16 | 0 | gptq_zp+-1/awq | intel-extension-for-transformers |+| marlin | cuda | 4,8 | BF16/FP16 | 6 | gptq/gptq_zp+-1 | gptqmodel |+| exllamav2 or<br/>gptqmodel:exllamav2 | cuda | 4 | BF16/FP16 | 5 | gptq | gptqmodel |+| exllamav2 or<br/>gptq:exllamav2 | cuda | 4 | FP16 | 5 | gptq_zp+-1 | auto-gptq |+| gptq:cuda | cuda | 2,3,4,8 | FP16 | 0 | gptq_zp+-1 | auto-gptq |+| triton | cuda/xpu | 2,4,8 | BF16/FP16 | 1 | gptq/gptq_zp+-1 | auto-round |+| awq | cuda | 4 | FP16 | 5 | awq | auto-awq |+| hpu | hpu | 4 | BF16 | 0 | gptq/gptq_zp+-1 | auto-round |### Convert GPTQ/AWQ to AutoRound