Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

support xpu in triton backend (#563)

Wenhua Chengf683f99a9b06
diff --git a/README.md b/README.md
index ba7e50a16..53ab09119 100644
--- a/README.md
+++ b/README.md
@@ -14,11 +14,12 @@ AutoRound
---
<div align="left">
-AutoRound is an advanced quantization algorithm that delivers strong accuracy, even at 2-bit precision.
-It leverages sign gradient descent to fine-tune both rounding values and min-max clipping thresholds in just 200 steps.
-Designed for broad compatibility, it seamlessly supports a wide range of LLMs and is actively expanding to cover more VLMs as well.
-It also supports quantization and inference across multiple hardware platforms, including CPU, Intel GPU, and CUDA.
-AutoRound also offers a variety of useful features, including mixed-bit tuning and inference, lm-head quantization,
+AutoRound is an advanced quantization algorithm that delivers strong accuracy, even at 2-bit precision.
+It leverages sign gradient descent to fine-tune both rounding values and min-max clipping thresholds in just 200 steps.
+Designed for broad compatibility, it seamlessly supports a wide range of LLMs and is actively expanding to cover more
+VLMs as well.
+It also supports quantization and inference across multiple hardware platforms, including CPU, Intel GPU, and CUDA.
+AutoRound also offers a variety of useful features, including mixed-bit tuning and inference, lm-head quantization,
support for exporting to formats like GPTQ/AWQ/GGUF, and flexible tuning recipes. The below
image presents an overview of AutoRound. Check out our paper on [arxiv](https://arxiv.org/pdf/2309.05516) for more
details and quantized models in several Hugging Face Spaces,
@@ -32,8 +33,12 @@ and [fbaldassarri](https://huggingface.co/fbaldassarri).
<div align="left">
## What's New
-* [2025/04] AutoRound provides some recipes for Qwen3 series, please refer to [Qwen3-8B-sym-recipe](./docs/Qwen3-8B-sym-recipe.md) and [Qwen3-14B-sym-recipe](./docs/Qwen3-14B-sym-recipe.md) for more details.
-* [2025/04] AutoRound has been integrated into Transformers. You can run models in the AutoRound format directly with Transformers versions later than 4.51.3.
+
+* [2025/04] AutoRound provides some recipes for Qwen3 series, please refer
+ to [Qwen3-8B-sym-recipe](./docs/Qwen3-8B-sym-recipe.md) and [Qwen3-14B-sym-recipe](./docs/Qwen3-14B-sym-recipe.md) for
+ more details.
+* [2025/04] AutoRound has been integrated into Transformers. You can run models in the AutoRound format directly with
+ Transformers versions later than 4.51.3.
* [2025/03] The INT2-mixed R1 model (~200GB) retains 97.9% accuracy. Check
out [OPEA/DeepSeek-R1-int2-mixed-sym-inc](https://huggingface.co/OPEA/DeepSeek-R1-int2-mixed-sym-inc).
* [2025/01] We provide experimental support for GGUF q4_0 and q4_1 formats.
@@ -83,7 +88,7 @@ auto-round \
--output_dir ./tmp_autoround
```
-We offer another two configurations, `auto-round-best` and `auto-round-light`,
+We offer another two configurations, `auto-round-best` and `auto-round-light`,
designed for optimal accuracy and improved speed,
respectively. Details are as follows.
<details>
@@ -313,7 +318,8 @@ in [Gaudi Guide](https://docs.habana.ai/en/latest/).
#### HPU/CPU/XPU/CUDA
-**Please avoid manually moving the quantized model to a different device** (e.g., model.to('cpu')) during inference, as this may cause unexpected exceptions.
+**Please avoid manually moving the quantized model to a different device** (e.g., model.to('cpu')) during inference, as
+this may cause unexpected exceptions.
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
@@ -332,24 +338,25 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
AutoRound automatically selects the best available backend based on the installed libraries and prompts the user to
install additional libraries when a better backend is found. On CUDA, the default priority is Marlin > ExLLaMAV2 >
-Triton, but the final choice depends on factors such as bits, group_size, packing format compatibility, etc. And the backend may not always be the most suitable for certain devices. Please refer
+Triton, but the final choice depends on factors such as bits, group_size, packing format compatibility, etc. And the
+backend may not always be the most suitable for certain devices. Please refer
to the following table for the details and specify the backend you want.
-| Name | Devices | Bits | Dtypes | Priority | Packing format | Requirements |
-|--------------------------------------|---------|---------|-----------|----------|-----------------|-------------------------------|
-| ipex | cpu/xpu | 4 | BF16/FP16 | 5 | gptq_zp+-1/awq | intel-extension-for-pytorch |
-| itrex | cpu | 2,4,8 | BF16/FP16 | 0 | gptq_zp+-1/awq | intel-extension-for-transformers |
-| marlin | cuda | 4,8 | BF16/FP16 | 6 | gptq/gptq_zp+-1 | gptqmodel |
-| exllamav2 or<br/>gptqmodel:exllamav2 | cuda | 4 | BF16/FP16 | 5 | gptq | gptqmodel |
-| exllamav2 or<br/>gptq:exllamav2 | cuda | 4 | FP16 | 5 | gptq_zp+-1 | auto-gptq |
-| gptq:cuda | cuda | 2,3,4,8 | FP16 | 0 | gptq_zp+-1 | auto-gptq |
-| triton | cuda | 2,4,8 | BF16/FP16 | 1 | gptq/gptq_zp+-1 | auto-round |
-| awq | cuda | 4 | FP16 | 5 | awq | auto-awq |
-| hpu | hpu | 4 | BF16 | 0 | gptq/gptq_zp+-1 | auto-round |
+| Name | Devices | Bits | Dtypes | Priority | Packing format | Requirements |
+|--------------------------------------|----------|---------|-----------|----------|-----------------|----------------------------------|
+| ipex | cpu/xpu | 4 | BF16/FP16 | 5 | gptq_zp+-1/awq | intel-extension-for-pytorch |
+| itrex | cpu | 2,4,8 | BF16/FP16 | 0 | gptq_zp+-1/awq | intel-extension-for-transformers |
+| marlin | cuda | 4,8 | BF16/FP16 | 6 | gptq/gptq_zp+-1 | gptqmodel |
+| exllamav2 or<br/>gptqmodel:exllamav2 | cuda | 4 | BF16/FP16 | 5 | gptq | gptqmodel |
+| exllamav2 or<br/>gptq:exllamav2 | cuda | 4 | FP16 | 5 | gptq_zp+-1 | auto-gptq |
+| gptq:cuda | cuda | 2,3,4,8 | FP16 | 0 | gptq_zp+-1 | auto-gptq |
+| triton | cuda/xpu | 2,4,8 | BF16/FP16 | 1 | gptq/gptq_zp+-1 | auto-round |
+| awq | cuda | 4 | FP16 | 5 | awq | auto-awq |
+| hpu | hpu | 4 | BF16 | 0 | gptq/gptq_zp+-1 | auto-round |
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
-from auto_round import AutoRoundConfig
+from auto_round import AutoRoundConfig
quantized_model_path = "./tmp_autoround"
quantization_config = AutoRoundConfig(backend="auto")
@@ -463,7 +470,7 @@ model architecture or kernel limitations.
|--------------------------------|---------------------|----------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
| allenai/Molmo | pile | X | [Molmo-7B-D-0924-int4-sym](https://huggingface.co/OPEA/Molmo-7B-D-0924-int4-sym-inc), [Molmo-72B-0924-int4-sym-gptq](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-gptq-inc), [Molmo-72B-0924-int4-sym](https://huggingface.co/OPEA/Molmo-72B-0924-int4-sym-inc) |
| deepseek-ai/deepseek-vl2 | pile/llava | √ | [deepseek-vl2-int4-sym-gptq](https://huggingface.co/OPEA/deepseek-vl2-int4-sym-gptq-inc) |
-| google/gemma-3 | pile/llava | √ | [gemma-3-12b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-12b-it-AutoRound-gguf-q4-0), [gemma-3-27b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-27b-it-AutoRound-gguf-q4-0), [gemma-3-12b-it-int4-AutoRound](https://huggingface.co/OPEA/gemma-3-12b-it-int4-AutoRound), [gemma-3-27b-it-int4-AutoRound](https://huggingface.co/OPEA/gemma-3-27b-it-int4-AutoRound) |
+| google/gemma-3 | pile/llava | √ | [gemma-3-12b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-12b-it-AutoRound-gguf-q4-0), [gemma-3-27b-it-AutoRound-gguf-q4-0](https://huggingface.co/OPEA/gemma-3-27b-it-AutoRound-gguf-q4-0), [gemma-3-12b-it-int4-AutoRound](https://huggingface.co/OPEA/gemma-3-12b-it-int4-AutoRound), [gemma-3-27b-it-int4-AutoRound](https://huggingface.co/OPEA/gemma-3-27b-it-int4-AutoRound) |
| HuggingFaceTB/SmolVLM | pile/llava | √ | [SmolVLM-Instruct-int4-sym](https://huggingface.co/OPEA/SmolVLM-Instruct-int4-sym-inc) |
| ibm-granite/granite-vision-3.2 | pile/llava | - | |
| liuhaotian/Llava-v1.5 | pile/llava | X | [llava-v1.5-7b-int4-sym](https://huggingface.co/OPEA/llava-v1.5-7b-int4-sym-inc) |
@@ -490,7 +497,6 @@ AutoRound has been integrated into multiple repositories.
[pytorch/ao](https://github.com/pytorch/ao)
-
## Reference
If you find AutoRound useful for your research, please cite our paper:
@@ -503,15 +509,3 @@ If you find AutoRound useful for your research, please cite our paper:
year={2023}
}
```
-
-
-
-
-
-
-
-
-
-
-
-
diff --git a/auto_round/export/export_to_autoround/export.py b/auto_round/export/export_to_autoround/export.py
index 02ead0464..2d39eb48b 100644
--- a/auto_round/export/export_to_autoround/export.py
+++ b/auto_round/export/export_to_autoround/export.py
@@ -23,7 +23,7 @@ import transformers
import auto_round.export.export_to_autoround.qlinear_triton_act
-import auto_round_extension.cuda.qlinear_tritonv2
+import auto_round_extension.triton.qlinear_tritonv2
from auto_round.utils import get_module, logger, set_module, supported_layer_types, check_to_quantized, \
filter_quantization_config
import threadpoolctl as tctl
@@ -75,7 +75,7 @@ def dynamic_import_quant_linear_for_packing(backend, bits, group_size, sym, act_
if act_bits <= 8: ##easily have bug for other configuration, need to refine code later
return auto_round.export.export_to_autoround.qlinear_triton_act.QuantLinear
- from auto_round_extension.cuda.qlinear_tritonv2 import QuantLinear
+ from auto_round_extension.triton.qlinear_tritonv2 import QuantLinear
return QuantLinear
elif "auto_round" in backend and "gptq" in backend and bits in (2, 4, 8):
from auto_round.export.export_to_autoround.qlinear_triton import QuantLinear ##no g_idx
@@ -84,7 +84,7 @@ def dynamic_import_quant_linear_for_packing(backend, bits, group_size, sym, act_
from ..export_to_awq.utils import WQLinear_GEMM
return WQLinear_GEMM
elif "gptqmodel" in backend:
- return auto_round_extension.cuda.qlinear_tritonv2.QuantLinear
+ return auto_round_extension.triton.qlinear_tritonv2.QuantLinear
elif "gptq" in backend and not "gptqmodel" in backend: ## have g_idx
return get_autogptq_packing_qlinear(backend, bits, group_size, sym)
else:
@@ -194,7 +194,7 @@ def pack_layer(layer_name, model, backend):
qlayer = new_layer
import auto_round.export.export_to_autoround.qlinear_triton
if sym and isinstance(QuantLinear, (auto_round.export.export_to_autoround.qlinear_triton.QuantLinear,
- auto_round_extension.cuda.qlinear_tritonv2.QuantLinear)):
+ auto_round_extension.triton.qlinear_tritonv2.QuantLinear)):
zp = int(zp.flatten()[0])
qlayer.to("cpu")
diff --git a/auto_round/inference/backend.py b/auto_round/inference/backend.py
index 53d99ada0..7f0ba35d5 100644
--- a/auto_round/inference/backend.py
+++ b/auto_round/inference/backend.py
@@ -13,7 +13,6 @@
# limitations under the License.
import functools
-import logging
from dataclasses import dataclass, field
from typing import List, Any, Optional
@@ -129,22 +128,23 @@ BackendInfos['auto_gptq:cuda'] = BackendInfo(device=["cuda"], sym=[True, False],
requirements=["auto-gptq>=0.7.1"]
)
-BackendInfos['auto_round:tritonv2'] = BackendInfo(device=["cuda"], sym=[True, False],
+BackendInfos['auto_round:tritonv2'] = BackendInfo(device=["cuda","cpu","xpu"], sym=[True, False],
packing_format="int32",
dtype=["float16", "bfloat16"],
bits=[2, 4, 8],
priority=1, feature_checks=[feature_multiply_checker_32],
alias=["auto_round", "tritonv2", "triton"],
- requirements=["auto-round>=0.5.0", "triton>=2.0"]
+ requirements=["triton>=2.0","auto-round>=0.5.0"]
)
-BackendInfos['auto_round:tritonv2_zp'] = BackendInfo(device=["cuda"], sym=[True], ## asym has accuracy issue
+BackendInfos['auto_round:tritonv2_zp'] = BackendInfo(device=["cuda","xpu"], sym=[True], ## asym has accuracy
+ # issue
packing_format="int32_zp",
dtype=["float16", "bfloat16"],
bits=[2, 4, 8],
priority=1, feature_checks=[feature_multiply_checker_32],
alias=["tritonv2", "tritonv2_zp", "triton"],
- requirements=["auto-round>=0.5.0", "triton>=2.0"]
+ requirements=[ "triton>=2.0","auto-round>=0.5.0"]
)
BackendInfos['gptqmodel:marlin'] = BackendInfo(device=["cuda"], sym=[True],
@@ -406,11 +406,11 @@ def dynamic_import_inference_linear(backend, bits, group_size, sym):
return WQLinear_GEMM
if backend == "auto_round:tritonv2":
- from auto_round_extension.cuda.qlinear_tritonv2 import QuantLinear
+ from auto_round_extension.triton.qlinear_tritonv2 import QuantLinear
return QuantLinear
if backend == "auto_round:tritonv2_zp":
- from auto_round_extension.cuda.qlinear_tritonv2_zp import QuantLinear
+ from auto_round_extension.triton.qlinear_tritonv2_zp import QuantLinear
return QuantLinear
raise ValueError(f"unsupported backend {backend}, please set it to `auto` and retry")
diff --git a/auto_round_extension/triton/__init__.py b/auto_round_extension/triton/__init__.py
new file mode 100644
index 000000000..e69de29bb
diff --git a/auto_round_extension/cuda/qlinear_tritonv2.py b/auto_round_extension/triton/qlinear_tritonv2.py
similarity index 95%
rename from auto_round_extension/cuda/qlinear_tritonv2.py
rename to auto_round_extension/triton/qlinear_tritonv2.py
index 655957e86..b9d8931b9 100644
--- a/auto_round_extension/cuda/qlinear_tritonv2.py
+++ b/auto_round_extension/triton/qlinear_tritonv2.py
@@ -7,14 +7,17 @@ import torch.nn as nn
import transformers
-from auto_round_extension.cuda.triton_utils.mixin import TritonModuleMixin
+from auto_round_extension.triton.triton_utils.mixin import TritonModuleMixin
logger = getLogger(__name__)
try:
- from auto_round_extension.cuda.triton_utils.dequant import QuantLinearFunction, quant_matmul_248
+ from auto_round_extension.triton.triton_utils.dequant import QuantLinearFunction, quant_matmul_248
except ImportError as e:
+ if torch.xpu.is_available():
+ logger.error(f"please make sure your triton version is same with `pytorch-triton-xpu` library ")
+ exit(-1)
triton_import_exception = e
def error_raiser_triton(*args, **kwargs):
diff --git a/auto_round_extension/cuda/qlinear_tritonv2_zp.py b/auto_round_extension/triton/qlinear_tritonv2_zp.py
similarity index 95%
rename from auto_round_extension/cuda/qlinear_tritonv2_zp.py
rename to auto_round_extension/triton/qlinear_tritonv2_zp.py
index 2dadfc0ae..530e14d8d 100644
--- a/auto_round_extension/cuda/qlinear_tritonv2_zp.py
+++ b/auto_round_extension/triton/qlinear_tritonv2_zp.py
@@ -7,14 +7,17 @@ import torch.nn as nn
import transformers
-from auto_round_extension.cuda.triton_utils_zp.mixin import TritonModuleMixin
+from auto_round_extension.triton.triton_utils_zp.mixin import TritonModuleMixin
logger = getLogger(__name__)
try:
- from auto_round_extension.cuda.triton_utils_zp.dequant import QuantLinearFunction, quant_matmul_248
+ from auto_round_extension.triton.triton_utils_zp.dequant import QuantLinearFunction, quant_matmul_248
except ImportError as e:
+ if torch.xpu.is_available():
+ logger.error(f"please make sure your triton version is same with `pytorch-triton-xpu` library ")
+ exit(-1)
triton_import_exception = e
def error_raiser_triton(*args, **kwargs):
diff --git a/auto_round_extension/cuda/triton_utils/__init__.py b/auto_round_extension/triton/triton_utils/__init__.py
similarity index 100%
rename from auto_round_extension/cuda/triton_utils/__init__.py
rename to auto_round_extension/triton/triton_utils/__init__.py
diff --git a/auto_round_extension/cuda/triton_utils/custom_autotune.py b/auto_round_extension/triton/triton_utils/custom_autotune.py
similarity index 100%
rename from auto_round_extension/cuda/triton_utils/custom_autotune.py
rename to auto_round_extension/triton/triton_utils/custom_autotune.py
diff --git a/auto_round_extension/cuda/triton_utils/dequant.py b/auto_round_extension/triton/triton_utils/dequant.py
similarity index 82%
rename from auto_round_extension/cuda/triton_utils/dequant.py
rename to auto_round_extension/triton/triton_utils/dequant.py
index 9e4b30744..43905629d 100644
--- a/auto_round_extension/cuda/triton_utils/dequant.py
+++ b/auto_round_extension/triton/triton_utils/dequant.py
@@ -119,34 +119,44 @@ def dequant_kernel_248(
tl.store(out_ptr + (x_index), weights, mask=xmask)
-def dequant248(qweight, scales, qzeros, g_idx, bits, maxq=None, input_dtype=torch.float16):
+def dequant248_core(qweight, scales, qzeros, g_idx, bits, maxq=None, input_dtype=torch.float16):
"""
Launcher for triton dequant kernel. Only valid for bits = 2, 4, 8
"""
- with torch.cuda.device(qweight.device):
- num_groups = scales.shape[0]
- outfeatures = scales.shape[1]
- infeatures = g_idx.shape[0]
-
- out = torch.empty((infeatures, outfeatures), device=qweight.device, dtype=input_dtype)
- numels = out.numel()
- maxq = 2 ** bits - 1 if maxq is None else maxq
- grid = lambda meta: (triton.cdiv(numels, meta["X_BLOCK"]),) # noqa: E731
-
- dequant_kernel_248[grid](
- g_idx,
- scales,
- qweight,
- qzeros,
- out,
- numels,
- maxq=maxq,
- bits=bits,
- outfeatures=outfeatures,
- num_groups=num_groups,
- )
- return out
+ num_groups = scales.shape[0]
+ outfeatures = scales.shape[1]
+ infeatures = g_idx.shape[0]
+
+ out = torch.empty((infeatures, outfeatures), device=qweight.device, dtype=input_dtype)
+ numels = out.numel()
+ maxq = 2 ** bits - 1 if maxq is None else maxq
+ grid = lambda meta: (triton.cdiv(numels, meta["X_BLOCK"]),) # noqa: E731
+
+ dequant_kernel_248[grid](
+ g_idx,
+ scales,
+ qweight,
+ qzeros,
+ out,
+ numels,
+ maxq=maxq,
+ bits=bits,
+ outfeatures=outfeatures,
+ num_groups=num_groups,
+ )
+ return out
+
+def dequant248(qweight, scales, qzeros, g_idx, bits, maxq=None, input_dtype=torch.float16):
+ """
+ Launcher for triton dequant kernel. Only valid for bits = 2, 4, 8
+ """
+ device_type = qweight.device.type
+ if device_type in {"cuda", "xpu"}:
+ with getattr(torch, device_type).device(qweight.device):
+ return dequant248_core(qweight, scales, qzeros, g_idx, bits, maxq=maxq, input_dtype=input_dtype)
+ else:
+ raise ValueError(f"Unsupported device type: {device_type}")
def quant_matmul_248(
input, qweight, scales, qzeros, g_idx, bits, maxq=None, transpose=False
diff --git a/auto_round_extension/cuda/triton_utils/kernels.py b/auto_round_extension/triton/triton_utils/kernels.py
similarity index 79%
rename from auto_round_extension/cuda/triton_utils/kernels.py
rename to auto_round_extension/triton/triton_utils/kernels.py
index 44981c7a3..eebd29cdb 100644
--- a/auto_round_extension/cuda/triton_utils/kernels.py
+++ b/auto_round_extension/triton/triton_utils/kernels.py
@@ -380,65 +380,82 @@ def silu(x):
return x * tl.sigmoid(x)
+def quant_matmul_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq):
+ output = torch.empty((input.shape[0], qweight.shape[1]), device=input.device, dtype=input.dtype)
+ grid = lambda META: ( # noqa: E731
+ triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(qweight.shape[1], META["BLOCK_SIZE_N"]),
+ )
+ quant_matmul_248_kernel[grid](
+ input,
+ qweight,
+ output,
+ scales.to(input.dtype),
+ qzeros,
+ g_idx,
+ input.shape[0],
+ qweight.shape[1],
+ input.shape[1],
+ bits,
+ maxq,
+ input.stride(0),
+ input.stride(1),
+ qweight.stride(0),
+ qweight.stride(1),
+ output.stride(0),
+ output.stride(1),
+ scales.stride(0),
+ qzeros.stride(0),
+ )
+ return output
+
+
def quant_matmul_248(input, qweight, scales, qzeros, g_idx, bits, maxq):
- with torch.cuda.device(input.device):
- output = torch.empty((input.shape[0], qweight.shape[1]), device=input.device, dtype=input.dtype)
- grid = lambda META: ( # noqa: E731
- triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(qweight.shape[1], META["BLOCK_SIZE_N"]),
- )
- quant_matmul_248_kernel[grid](
- input,
- qweight,
- output,
- scales.to(input.dtype),
- qzeros,
- g_idx,
- input.shape[0],
- qweight.shape[1],
- input.shape[1],
- bits,
- maxq,
- input.stride(0),
- input.stride(1),
- qweight.stride(0),
- qweight.stride(1),
- output.stride(0),
- output.stride(1),
- scales.stride(0),
- qzeros.stride(0),
- )
- return output
+ device_type = input.device.type
+ if device_type in {"cuda", "xpu"}:
+ with getattr(torch, device_type).device(input.device):
+ return quant_matmul_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq)
+ else:
+ raise ValueError(f"Unsupported device type: {device_type}")
+
+
+def transpose_quant_matmul_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq):
+ output_dim = (qweight.shape[0] * 32) // bits
+ output = torch.empty((input.shape[0], output_dim), device=input.device, dtype=input.dtype)
+ grid = lambda META: ( # noqa: E731
+ triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(output_dim, META["BLOCK_SIZE_K"]),
+ )
+ transpose_quant_matmul_248_kernel[grid](
+ input,
+ qweight,
+ output,
+ scales.to(input.dtype),
+ qzeros,
+ g_idx,
+ input.shape[0],
+ qweight.shape[1],
+ output_dim,
+ bits,
+ maxq,
+ input.stride(0),
+ input.stride(1),
+ qweight.stride(0),
+ qweight.stride(1),
+ output.stride(0),
+ output.stride(1),
+ scales.stride(0),
+ qzeros.stride(0),
+ )
+ return output
def transpose_quant_matmul_248(input, qweight, scales, qzeros, g_idx, bits, maxq):
- with torch.cuda.device(input.device):
- output_dim = (qweight.shape[0] * 32) // bits
- output = torch.empty((input.shape[0], output_dim), device=input.device, dtype=input.dtype)
- grid = lambda META: ( # noqa: E731
- triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(output_dim, META["BLOCK_SIZE_K"]),
- )
- transpose_quant_matmul_248_kernel[grid](
- input,
- qweight,
- output,
- scales.to(input.dtype),
- qzeros,
- g_idx,
- input.shape[0],
- qweight.shape[1],
- output_dim,
- bits,
- maxq,
- input.stride(0),
- input.stride(1),
- qweight.stride(0),
- qweight.stride(1),
- output.stride(0),
- output.stride(1),
- scales.stride(0),
- qzeros.stride(0),
- )
- return output
+ device_type = input.device.type
+ if device_type in {"cuda", "xpu"}:
+ with getattr(torch, device_type).device(input.device):
+ return transpose_quant_matmul_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq)
+ else:
+ raise ValueError(f"Unsupported device type: {device_type}")
+
class QuantLinearFunction(torch.autograd.Function):
@@ -462,34 +479,43 @@ class QuantLinearFunction(torch.autograd.Function):
return grad_input, None, None, None, None, None, None
+def quant_matmul_inference_only_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq):
+ output = torch.empty((input.shape[0], qweight.shape[1]), device=input.device, dtype=torch.float16)
+ grid = lambda META: ( # noqa: E731
+ triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(qweight.shape[1], META["BLOCK_SIZE_N"]),
+ )
+ quant_matmul_248_kernel[grid](
+ input,
+ qweight,
+ output,
+ scales,
+ qzeros,
+ g_idx,
+ input.shape[0],
+ qweight.shape[1],
+ input.shape[1],
+ bits,
+ maxq,
+ input.stride(0),
+ input.stride(1),
+ qweight.stride(0),
+ qweight.stride(1),
+ output.stride(0),
+ output.stride(1),
+ scales.stride(0),
+ qzeros.stride(0),
+ )
+ return output
+
+
def quant_matmul_inference_only_248(input, qweight, scales, qzeros, g_idx, bits, maxq):
- with torch.cuda.device(input.device):
- output = torch.empty((input.shape[0], qweight.shape[1]), device=input.device, dtype=torch.float16)
- grid = lambda META: ( # noqa: E731
- triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(qweight.shape[1], META["BLOCK_SIZE_N"]),
- )
- quant_matmul_248_kernel[grid](
- input,
- qweight,
- output,
- scales,
- qzeros,
- g_idx,
- input.shape[0],
- qweight.shape[1],
- input.shape[1],
- bits,
- maxq,
- input.stride(0),
- input.stride(1),
- qweight.stride(0),
- qweight.stride(1),
- output.stride(0),
- output.stride(1),
- scales.stride(0),
- qzeros.stride(0),
- )
- return output
+ device_type = input.device.type
+ if device_type in {"cuda", "xpu"}:
+ with getattr(torch, device_type).device(input.device):
+ return quant_matmul_inference_only_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq)
+ else:
+ raise ValueError(f"Unsupported device type: {device_type}")
+
class QuantLinearInferenceOnlyFunction(torch.autograd.Function):
diff --git a/auto_round_extension/cuda/triton_utils/mixin.py b/auto_round_extension/triton/triton_utils/mixin.py
similarity index 100%
rename from auto_round_extension/cuda/triton_utils/mixin.py
rename to auto_round_extension/triton/triton_utils/mixin.py
diff --git a/auto_round_extension/cuda/triton_utils_zp/__init__.py b/auto_round_extension/triton/triton_utils_zp/__init__.py
similarity index 100%
rename from auto_round_extension/cuda/triton_utils_zp/__init__.py
rename to auto_round_extension/triton/triton_utils_zp/__init__.py
diff --git a/auto_round_extension/cuda/triton_utils_zp/custom_autotune.py b/auto_round_extension/triton/triton_utils_zp/custom_autotune.py
similarity index 100%
rename from auto_round_extension/cuda/triton_utils_zp/custom_autotune.py
rename to auto_round_extension/triton/triton_utils_zp/custom_autotune.py
diff --git a/auto_round_extension/cuda/triton_utils_zp/dequant.py b/auto_round_extension/triton/triton_utils_zp/dequant.py
similarity index 82%
rename from auto_round_extension/cuda/triton_utils_zp/dequant.py
rename to auto_round_extension/triton/triton_utils_zp/dequant.py
index 8abf6f514..c6f47c1a1 100644
--- a/auto_round_extension/cuda/triton_utils_zp/dequant.py
+++ b/auto_round_extension/triton/triton_utils_zp/dequant.py
@@ -38,7 +38,6 @@ import itertools
import torch
import triton
import triton.language as tl
-from torch.cuda.amp import custom_bwd, custom_fwd
def make_dequant_configs(block_sizes, num_warps):
@@ -119,33 +118,44 @@ def dequant_kernel_248(
tl.store(out_ptr + (x_index), weights, mask=xmask)
-def dequant248(qweight, scales, qzeros, g_idx, bits, maxq=None, input_dtype=torch.float16):
+def dequant248_core(qweight, scales, qzeros, g_idx, bits, maxq=None, input_dtype=torch.float16):
"""
Launcher for triton dequant kernel. Only valid for bits = 2, 4, 8
"""
- with torch.cuda.device(qweight.device):
- num_groups = scales.shape[0]
- outfeatures = scales.shape[1]
- infeatures = g_idx.shape[0]
-
- out = torch.empty((infeatures, outfeatures), device=qweight.device, dtype=input_dtype)
- numels = out.numel()
- maxq = 2 ** bits - 1 if maxq is None else maxq
- grid = lambda meta: (triton.cdiv(numels, meta["X_BLOCK"]),) # noqa: E731
-
- dequant_kernel_248[grid](
- g_idx,
- scales,
- qweight,
- qzeros,
- out,
- numels,
- maxq=maxq,
- bits=bits,
- outfeatures=outfeatures,
- num_groups=num_groups,
- )
- return out
+ num_groups = scales.shape[0]
+ outfeatures = scales.shape[1]
+ infeatures = g_idx.shape[0]
+
+ out = torch.empty((infeatures, outfeatures), device=qweight.device, dtype=input_dtype)
+ numels = out.numel()
+ maxq = 2 ** bits - 1 if maxq is None else maxq
+ grid = lambda meta: (triton.cdiv(numels, meta["X_BLOCK"]),) # noqa: E731
+
+ dequant_kernel_248[grid](
+ g_idx,
+ scales,
+ qweight,
+ qzeros,
+ out,
+ numels,
+ maxq=maxq,
+ bits=bits,
+ outfeatures=outfeatures,
+ num_groups=num_groups,
+ )
+ return out
+
+
+def dequant248(qweight, scales, qzeros, g_idx, bits, maxq=None, input_dtype=torch.float16):
+ """
+ Launcher for triton dequant kernel. Only valid for bits = 2, 4, 8
+ """
+ device_type = qweight.device.type
+ if device_type in {"cuda", "xpu"}:
+ with getattr(torch, device_type).device(qweight.device):
+ return dequant248_core(qweight, scales, qzeros, g_idx, bits, maxq=maxq, input_dtype=input_dtype)
+ else:
+ raise ValueError(f"Unsupported device type: {device_type}")
def quant_matmul_248(
diff --git a/auto_round_extension/cuda/triton_utils_zp/kernels.py b/auto_round_extension/triton/triton_utils_zp/kernels.py
similarity index 79%
rename from auto_round_extension/cuda/triton_utils_zp/kernels.py
rename to auto_round_extension/triton/triton_utils_zp/kernels.py
index 44981c7a3..f361a3dde 100644
--- a/auto_round_extension/cuda/triton_utils_zp/kernels.py
+++ b/auto_round_extension/triton/triton_utils_zp/kernels.py
@@ -39,7 +39,6 @@ from logging import getLogger
import torch
import triton
import triton.language as tl
-from torch.cuda.amp import custom_bwd, custom_fwd
from . import custom_autotune
@@ -379,71 +378,85 @@ def transpose_quant_matmul_248_kernel(
def silu(x):
return x * tl.sigmoid(x)
+def quant_matmul_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq):
+ output = torch.empty((input.shape[0], qweight.shape[1]), device=input.device, dtype=input.dtype)
+ grid = lambda META: ( # noqa: E731
+ triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(qweight.shape[1], META["BLOCK_SIZE_N"]),
+ )
+ quant_matmul_248_kernel[grid](
+ input,
+ qweight,
+ output,
+ scales.to(input.dtype),
+ qzeros,
+ g_idx,
+ input.shape[0],
+ qweight.shape[1],
+ input.shape[1],
+ bits,
+ maxq,
+ input.stride(0),
+ input.stride(1),
+ qweight.stride(0),
+ qweight.stride(1),
+ output.stride(0),
+ output.stride(1),
+ scales.stride(0),
+ qzeros.stride(0),
+ )
+ return output
+
def quant_matmul_248(input, qweight, scales, qzeros, g_idx, bits, maxq):
- with torch.cuda.device(input.device):
- output = torch.empty((input.shape[0], qweight.shape[1]), device=input.device, dtype=input.dtype)
- grid = lambda META: ( # noqa: E731
- triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(qweight.shape[1], META["BLOCK_SIZE_N"]),
- )
- quant_matmul_248_kernel[grid](
- input,
- qweight,
- output,
- scales.to(input.dtype),
- qzeros,
- g_idx,
- input.shape[0],
- qweight.shape[1],
- input.shape[1],
- bits,
- maxq,
- input.stride(0),
- input.stride(1),
- qweight.stride(0),
- qweight.stride(1),
- output.stride(0),
- output.stride(1),
- scales.stride(0),
- qzeros.stride(0),
- )
- return output
+ device_type = input.device.type
+ if device_type in {"cuda", "xpu"}:
+ with getattr(torch, device_type).device(input.device):
+ return quant_matmul_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq)
+ else:
+ raise ValueError(f"Unsupported device type: {device_type}")
+
+
+def transpose_quant_matmul_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq):
+ output_dim = (qweight.shape[0] * 32) // bits
+ output = torch.empty((input.shape[0], output_dim), device=input.device, dtype=input.dtype)
+ grid = lambda META: ( # noqa: E731
+ triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(output_dim, META["BLOCK_SIZE_K"]),
+ )
+ transpose_quant_matmul_248_kernel[grid](
+ input,
+ qweight,
+ output,
+ scales.to(input.dtype),
+ qzeros,
+ g_idx,
+ input.shape[0],
+ qweight.shape[1],
+ output_dim,
+ bits,
+ maxq,
+ input.stride(0),
+ input.stride(1),
+ qweight.stride(0),
+ qweight.stride(1),
+ output.stride(0),
+ output.stride(1),
+ scales.stride(0),
+ qzeros.stride(0),
+ )
+ return output
def transpose_quant_matmul_248(input, qweight, scales, qzeros, g_idx, bits, maxq):
- with torch.cuda.device(input.device):
- output_dim = (qweight.shape[0] * 32) // bits
- output = torch.empty((input.shape[0], output_dim), device=input.device, dtype=input.dtype)
- grid = lambda META: ( # noqa: E731
- triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(output_dim, META["BLOCK_SIZE_K"]),
- )
- transpose_quant_matmul_248_kernel[grid](
- input,
- qweight,
- output,
- scales.to(input.dtype),
- qzeros,
- g_idx,
- input.shape[0],
- qweight.shape[1],
- output_dim,
- bits,
- maxq,
- input.stride(0),
- input.stride(1),
- qweight.stride(0),
- qweight.stride(1),
- output.stride(0),
- output.stride(1),
- scales.stride(0),
- qzeros.stride(0),
- )
- return output
+ device_type = input.device.type
+ if device_type in {"cuda", "xpu"}:
+ with getattr(torch, device_type).device(input.device):
+ return transpose_quant_matmul_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq)
+ else:
+ raise ValueError(f"Unsupported device type: {device_type}")
class QuantLinearFunction(torch.autograd.Function):
@staticmethod
- @custom_fwd
def forward(ctx, input, qweight, scales, qzeros, g_idx, bits, maxq):
output = quant_matmul_248(input, qweight, scales, qzeros, g_idx, bits, maxq)
ctx.save_for_backward(qweight, scales, qzeros, g_idx)
@@ -451,7 +464,6 @@ class QuantLinearFunction(torch.autograd.Function):
return output
@staticmethod
- @custom_bwd
def backward(ctx, grad_output):
qweight, scales, qzeros, g_idx = ctx.saved_tensors
bits, maxq = ctx.bits, ctx.maxq
@@ -462,39 +474,46 @@ class QuantLinearFunction(torch.autograd.Function):
return grad_input, None, None, None, None, None, None
+def quant_matmul_inference_only_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq):
+ output = torch.empty((input.shape[0], qweight.shape[1]), device=input.device, dtype=torch.float16)
+ grid = lambda META: ( # noqa: E731
+ triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(qweight.shape[1], META["BLOCK_SIZE_N"]),
+ )
+ quant_matmul_248_kernel[grid](
+ input,
+ qweight,
+ output,
+ scales,
+ qzeros,
+ g_idx,
+ input.shape[0],
+ qweight.shape[1],
+ input.shape[1],
+ bits,
+ maxq,
+ input.stride(0),
+ input.stride(1),
+ qweight.stride(0),
+ qweight.stride(1),
+ output.stride(0),
+ output.stride(1),
+ scales.stride(0),
+ qzeros.stride(0),
+ )
+ return output
+
+
def quant_matmul_inference_only_248(input, qweight, scales, qzeros, g_idx, bits, maxq):
- with torch.cuda.device(input.device):
- output = torch.empty((input.shape[0], qweight.shape[1]), device=input.device, dtype=torch.float16)
- grid = lambda META: ( # noqa: E731
- triton.cdiv(input.shape[0], META["BLOCK_SIZE_M"]) * triton.cdiv(qweight.shape[1], META["BLOCK_SIZE_N"]),
- )
- quant_matmul_248_kernel[grid](
- input,
- qweight,
- output,
- scales,
- qzeros,
- g_idx,
- input.shape[0],
- qweight.shape[1],
- input.shape[1],
- bits,
- maxq,
- input.stride(0),
- input.stride(1),
- qweight.stride(0),
- qweight.stride(1),
- output.stride(0),
- output.stride(1),
- scales.stride(0),
- qzeros.stride(0),
- )
- return output
+ device_type = input.device.type
+ if device_type in {"cuda", "xpu"}:
+ with getattr(torch, device_type).device(input.device):
+ return quant_matmul_inference_only_248_core(input, qweight, scales, qzeros, g_idx, bits, maxq)
+ else:
+ raise ValueError(f"Unsupported device type: {device_type}")
class QuantLinearInferenceOnlyFunction(torch.autograd.Function):
@staticmethod
- @custom_fwd(cast_inputs=torch.float16)
def forward(ctx, input, qweight, scales, qzeros, g_idx, bits, maxq):
output = quant_matmul_248(input, qweight, scales, qzeros, g_idx, bits, maxq)
return output
diff --git a/auto_round_extension/cuda/triton_utils_zp/mixin.py b/auto_round_extension/triton/triton_utils_zp/mixin.py
similarity index 100%
rename from auto_round_extension/cuda/triton_utils_zp/mixin.py
rename to auto_round_extension/triton/triton_utils_zp/mixin.py
diff --git a/docs/step_by_step.md b/docs/step_by_step.md
index b7e6aab49..ee68d00c8 100644
--- a/docs/step_by_step.md
+++ b/docs/step_by_step.md
@@ -480,17 +480,17 @@ text = "There is a girl who likes adventure,"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))
```
-| Name | Devices | Bits | Dtypes | Priority | Packing format | Requirements |
-|--------------------------------------|---------|---------|-----------|----------|-----------------|-------------------------------|
-| ipex | cpu/xpu | 4 | BF16/FP16 | 5 | gptq_zp+-1/awq | intel-extension-for-pytorch |
-| itrex | cpu | 2,4,8 | BF16/FP16 | 0 | gptq_zp+-1/awq | intel-extension-for-transformers |
-| marlin | cuda | 4,8 | BF16/FP16 | 6 | gptq/gptq_zp+-1 | gptqmodel |
-| exllamav2 or<br/>gptqmodel:exllamav2 | cuda | 4 | BF16/FP16 | 5 | gptq | gptqmodel |
-| exllamav2 or<br/>gptq:exllamav2 | cuda | 4 | FP16 | 5 | gptq_zp+-1 | auto-gptq |
-| gptq:cuda | cuda | 2,3,4,8 | FP16 | 0 | gptq_zp+-1 | auto-gptq |
-| triton | cuda | 2,4,8 | BF16/FP16 | 1 | gptq/gptq_zp+-1 | auto-round |
-| awq | cuda | 4 | FP16 | 5 | awq | auto-awq |
-| hpu | hpu | 4 | BF16 | 0 | gptq/gptq_zp+-1 | auto-round |
+| Name | Devices | Bits | Dtypes | Priority | Packing format | Requirements |
+|--------------------------------------|----------|---------|-----------|----------|-----------------|-------------------------------|
+| ipex | cpu/xpu | 4 | BF16/FP16 | 5 | gptq_zp+-1/awq | intel-extension-for-pytorch |
+| itrex | cpu | 2,4,8 | BF16/FP16 | 0 | gptq_zp+-1/awq | intel-extension-for-transformers |
+| marlin | cuda | 4,8 | BF16/FP16 | 6 | gptq/gptq_zp+-1 | gptqmodel |
+| exllamav2 or<br/>gptqmodel:exllamav2 | cuda | 4 | BF16/FP16 | 5 | gptq | gptqmodel |
+| exllamav2 or<br/>gptq:exllamav2 | cuda | 4 | FP16 | 5 | gptq_zp+-1 | auto-gptq |
+| gptq:cuda | cuda | 2,3,4,8 | FP16 | 0 | gptq_zp+-1 | auto-gptq |
+| triton | cuda/xpu | 2,4,8 | BF16/FP16 | 1 | gptq/gptq_zp+-1 | auto-round |
+| awq | cuda | 4 | FP16 | 5 | awq | auto-awq |
+| hpu | hpu | 4 | BF16 | 0 | gptq/gptq_zp+-1 | auto-round |
### Convert GPTQ/AWQ to AutoRound