Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

[research feature] Introduce INT4 support at the algorithm level (#1641)

Wenhua Cheng11e91869415e
diff --git a/README.md b/README.md
index d710f0a4e..dc7acfc8e 100644
--- a/README.md
+++ b/README.md
@@ -36,9 +36,11 @@ See our papers [SignRoundV1](https://arxiv.org/pdf/2309.05516) and [SignRoundV2]
## 🆕 What's New
+* [2026/04] We demonstrate that AutoRound achieves SOTA or near SOTA performance under INT4 (W4A4) quantization [accuracy data](./docs/int_acc.md). This capability is currently a **research-only** feature, with no production model export.
+
* [2026/03] **Block-wise FP8** quantization is available via `--scheme FP8_BLOCK --iters 0 --disable_opt_rtn`.
-* [2026/03] MTP layer quantization has been supported in [PR](https://github.com/intel/auto-round/pull/1526)
+* [2026/03] **MTP layer quantization** has been supported in this [PR](https://github.com/intel/auto-round/pull/1526)
* [2025/12] The **SignRoundV2** paper is available. Turn on `enable_alg_ext` and use the **AutoScheme** API for mixed-precision quantization to reproduce the results: [*Paper*](http://arxiv.org/abs/2512.04746), [*Notes for evaluating LLaMA models*](./docs/alg_202508.md).
@@ -226,7 +228,7 @@ ar.quantize_and_save(output_dir="./qmodel", format="auto_round")
- **`iters` (int)**: Number of tuning iterations (default is `200`). Common values: 0 (RTN mode), 50 (with lr=5e-3 recommended), 1000. Higher values increase accuracy but slow down tuning.
- **`lr` (float)**: The learning rate for rounding value (default is `None`). When None, it will be set to `1.0/iters` automatically.
- **`batch_size` (int)**: Batch size for training (default is `8`). 4 is also commonly used.
-- ** `enable_deterministic_algorithms` (bool)**: Whether to enable deterministic algorithms for reproducibility (default is `False`).
+- **`enable_deterministic_algorithms` (bool)**: Whether to enable deterministic algorithms for reproducibility (default is `False`).
##### Calibration Dataset
- **`dataset` (str|list|tuple|torch.utils.data.DataLoader)**: The dataset for tuning (default is `"NeelNanda/pile-10k"`). Supports local JSON files and dataset combinations, e.g. `"./tmp.json,NeelNanda/pile-10k:train,mbpp:train+validation+test"`.
diff --git a/README_CN.md b/README_CN.md
index 9f3a5667f..72e99df18 100644
--- a/README_CN.md
+++ b/README_CN.md
@@ -35,6 +35,12 @@ AutoRound 是专为大语言模型(LLMs)和视觉-语言模型(VLMs)设
## 🆕 最新进展
+* [2026/04] AutoRound可以在INT4上取得非常好的精度[accuracy data](./docs/int_acc.md). 不过目前这个特性还停留在研究性质,我们正在努力将其优化到生产环境中。
+
+* [2026/03] 支持**Block-wise FP8**,只需要将scheme改成 `--scheme FP8_BLOCK`.为了更快的量化,我们推荐使用RTN模式,即·--iters 0 --disable_opt_rtn·
+
+* [2026/03] 支持**MTP层量化**
+*
* [2025/12] 发布 **SignRoundV2** 论文。如果要复现论文成果,可启用 `enable_alg_ext`,并使用 **AutoScheme** API 对模型进行混合精度量化。相关链接:[*论文*](http://arxiv.org/abs/2512.04746),[*LLaMA 模型评估说明*](./docs/alg_202508.md)。
* [2025/11] **LLM-Compressor** 已支持 AutoRound 算法。相关链接:[*使用方法*](https://github.com/vllm-project/llm-compressor/tree/main/examples/autoround/README.md),[*vLLM 博客*](https://blog.vllm.ai/2025/12/09/intel-autoround-llmc.html),[*RedHat 博客*](https://developers.redhat.com/articles/2025/12/09/advancing-low-bit-quantization-llms-autoround-x-llm-compressor),[*X 推文*](https://x.com/vllm_project/status/1998710451312771532),[*Intel 博客*](https://community.intel.com/t5/Blogs/Products-and-Solutions/HPC/Advancing-Low-Bit-Quantization-for-LLMs-AutoRound-x-LLM/post/1729336),[*LinkedIn*](https://www.linkedin.com/posts/vllm-project_advancing-lowbit-quantization-for-llms-activity-7404478053768441856-ru8f/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAapNW8BLnAdCAr57GOwSCJXjf76ZvOEOAg),[*微信*](https://mp.weixin.qq.com/s/l5WA-1_4ipffQN6GOH2Iqg),[*知乎*](https://zhuanlan.zhihu.com/p/1982167638315664412)。
diff --git a/auto_round/__main__.py b/auto_round/__main__.py
index 627d4ebef..d7fde248a 100644
--- a/auto_round/__main__.py
+++ b/auto_round/__main__.py
@@ -286,6 +286,7 @@ class BasicArgumentParser(argparse.ArgumentParser):
help="Disable optimization for RTN (Round-To-Nearest) mode when iters=0. "
"RTN is fast but less accurate; keeping optimization enabled is recommended.",
)
+
group_opt_rtn.add_argument(
"--enable_opt_rtn",
action="store_const",
@@ -303,6 +304,9 @@ class BasicArgumentParser(argparse.ArgumentParser):
help="Group size for weight quantization.",
)
scheme.add_argument("--asym", action="store_true", help="Use asymmetric quantization instead of symmetric.")
+ scheme.add_argument(
+ "--act_asym", action="store_true", help="Use asymmetric quantization for activation instead of symmetric."
+ )
scheme.add_argument(
"--data_type",
"--dtype",
@@ -373,6 +377,13 @@ class BasicArgumentParser(argparse.ArgumentParser):
choices=["fp8", "float8_e4m3fn"],
help="Data type for static quantize attention. ",
)
+ scheme.add_argument(
+ "--rotation_type",
+ default=None,
+ type=str,
+ choices=["hadamard", "random_hadamard"],
+ help="Research feature: applies a rotation (e.g., Hadamard) to reduce activation/weight outliers",
+ )
gguf = self.add_argument_group("Double Quant Arguments")
gguf.add_argument(
"--super_group_size", default=None, type=int, help="Super group size for double quantization."
@@ -659,6 +670,7 @@ def tune(args):
act_group_size=args.act_group_size,
act_data_type=args.act_data_type,
act_dynamic=act_dynamic,
+ act_sym=None if not args.asym else False,
super_bits=args.super_bits,
super_group_size=args.super_group_size,
quant_lm_head=args.quant_lm_head,
@@ -704,6 +716,11 @@ def tune(args):
low_gpu_mem_usage=True, # force it to be True as it uses much smaller vram but similar time cost
low_cpu_mem_usage=low_cpu_mem_usage,
)
+ rot_config = None
+ if args.rotation_type:
+ from auto_round.experimental.transform.rotation_config import RotationConfig
+
+ rot_config = RotationConfig(hadamard_type=args.rotation_type)
autoround: BaseCompressor = AutoRound(
model=model_name,
@@ -727,6 +744,7 @@ def tune(args):
model_dtype=args.model_dtype,
momentum=args.momentum,
trust_remote_code=not args.disable_trust_remote_code,
+ rotation_config=rot_config,
)
model_name = args.model.rstrip("/")
diff --git a/auto_round/alg_ext.py b/auto_round/alg_ext.py
index 4d74f8bd9..cedd0fdc0 100644
--- a/auto_round/alg_ext.py
+++ b/auto_round/alg_ext.py
@@ -24,7 +24,7 @@ from torch import autocast
from torch.functional import F
from auto_round import AutoRound
-from auto_round.compressors.utils import check_need_act_calibration, is_nv_fp
+from auto_round.compressors.utils import check_need_act_calibration, is_nv_fp, is_wint4aint4
from auto_round.data_type.int import search_scales
from auto_round.data_type.mxfp import MXFP_FORMAT_CACHE, quant_element
from auto_round.data_type.nvfp import FLOAT4_E2M1_MAX, FLOAT8_E4M3_MAX, ref_nvfp4_quant
@@ -42,16 +42,12 @@ def wrapper_autoround(cls: AutoRound):
cls.sym
and cls.enable_alg_ext
and cls.super_group_size is None
- and (
- (cls.data_type.startswith("int") and cls.act_bits >= 8)
- or cls.data_type.startswith("mx")
- or cls.data_type.startswith("nv")
- )
+ and ((cls.data_type.startswith("int")) or cls.data_type.startswith("mx") or cls.data_type.startswith("nv"))
):
if cls.bits > 2 and (not cls.data_type.startswith("mx") or not cls.data_type.startswith("nv")):
logger.warning_once(
"algorithm extension has only undergone limited validation on "
- "INT2,mxfp4 and nvfp4; use with caution."
+ "W2A16,INT4, MXFP4 and NVFP4; use with caution."
)
cls._get_loss = types.MethodType(_get_loss_ext, cls)
setattr(cls, "wrapper_block", wrapper_block_v2)
@@ -323,8 +319,6 @@ class WrapperLinearV2(WrapperLinear):
disable_opt_rtn=disable_opt_rtn,
**kwargs,
)
- if self.enable_act_quant and self.orig_layer.data_type.startswith("int"):
- raise ValueError("does not support act quantization in WrapperLinearV2, please use WrapperLinear instead")
def _init_tuning_params_and_quant_func(self):
"""Initializes tuning parameters and quantization functions.
@@ -342,21 +336,23 @@ class WrapperLinearV2(WrapperLinear):
imatrix = imatrix.expand(weight_reshape.numel() // imatrix.numel(), -1)
imatrix = imatrix.reshape(weight_reshape.shape)
imatrix = imatrix.to(orig_weight.device)
- if self.orig_layer.data_type.startswith("int"):
- self.init_scale = search_scales(weight_reshape, self.orig_layer.bits, imatrix)
- self.init_scale = torch.where(
- self.init_scale < 0,
- torch.clamp(self.init_scale, max=-self.q_scale_thresh),
- torch.clamp(self.init_scale, min=self.q_scale_thresh),
- )
- elif self.orig_layer.data_type.startswith("mx"):
- self.init_scale = mx_init(weight_reshape, self.orig_layer.bits, imatrix)
- elif self.orig_layer.data_type.startswith("nv"):
- self.init_scale = nv_init(weight_reshape, self.orig_layer.bits, imatrix)
- self.orig_layer.imatrix = None
- delattr(self.orig_layer, "imatrix")
+ else:
+ imatrix = 1.0
+ if self.orig_layer.data_type.startswith("int"):
+ self.init_scale = search_scales(weight_reshape, self.orig_layer.bits, imatrix)
+ self.init_scale = torch.where(
+ self.init_scale < 0,
+ torch.clamp(self.init_scale, max=-self.q_scale_thresh),
+ torch.clamp(self.init_scale, min=self.q_scale_thresh),
+ )
+ elif self.orig_layer.data_type.startswith("mx"):
+ self.init_scale = mx_init(weight_reshape, self.orig_layer.bits, imatrix)
+ elif self.orig_layer.data_type.startswith("nv"):
+ self.init_scale = nv_init(weight_reshape, self.orig_layer.bits, imatrix)
else:
self.init_scale = 1.0
+ self.orig_layer.imatrix = None
+ delattr(self.orig_layer, "imatrix")
# self.weight_quant_func, self.data_type = get_quant_func(orig_layer.data_type, orig_layer.bits, orig_layer.sym)
if self.orig_layer.data_type.startswith("int"):
@@ -496,8 +492,9 @@ def _register_act_max_hook_ext(self, model):
for n, m in model.named_modules():
if isinstance(m, self.supported_types) and check_to_quantized(m):
- hook = m.register_forward_hook(get_imatrix_hook)
- hook_handles.append(hook)
+ if not is_wint4aint4(self): # INT4 no imatrix is much better
+ hook = m.register_forward_hook(get_imatrix_hook)
+ hook_handles.append(hook)
if (
hasattr(m, "act_dynamic")
diff --git a/auto_round/auto_scheme/delta_loss.py b/auto_round/auto_scheme/delta_loss.py
index 7610e94af..3b88c0e60 100644
--- a/auto_round/auto_scheme/delta_loss.py
+++ b/auto_round/auto_scheme/delta_loss.py
@@ -98,11 +98,11 @@ class AutoSchemeWrapperLinear(WrapperLinear):
if self.need_weight_grad:
self.orig_layer.weight.requires_grad = True
- def _qdq_act(self, x, act_max_scale, act_max=None):
+ def _qdq_act(self, x, act_min_scale=1.0, act_max_scale=1.0, act_max=None):
if hasattr(self.orig_layer, "act_bits") and self.orig_layer.act_bits > 8:
return x, 1.0, None
- qdq_x, scale, zp = self.act_qdq_func(x, act_max_scale, act_max)
+ qdq_x, scale, zp = self.act_qdq_func(x, act_min_scale, act_max_scale, act_max)
if self.grad_mode:
with torch.no_grad():
self.max_act_value = torch.abs(x).max()
diff --git a/auto_round/compressors/base.py b/auto_round/compressors/base.py
index 18a3f1d89..bbeb85d0d 100644
--- a/auto_round/compressors/base.py
+++ b/auto_round/compressors/base.py
@@ -51,12 +51,14 @@ from auto_round.compressors.utils import (
is_nv_fp,
is_static_wfp8afp8,
is_wfp8afp8,
+ is_wint4aint4,
reset_params,
set_layer_config,
)
from auto_round.data_type import QUANT_FUNC_WITH_DTYPE
from auto_round.data_type.utils import reshape_pad_tensor_by_group_size, update_block_global_scale_if_needed
-from auto_round.experimental.transform.hadamard_config import HadamardConfig
+from auto_round.experimental.transform.rotation_config import RotationConfig
+from auto_round.experimental.utils import dump_group_size_to_rotation_config, normalize_rotation_config
from auto_round.export.export_to_gguf.config import GGUF_INNER_CONFIG
from auto_round.formats import OutputFormat, get_formats
from auto_round.logger import logger
@@ -153,7 +155,7 @@ SERIALIZATION_KEYS = (
"super_bits",
"super_group_size",
"to_quant_block_names",
- "hadamard_config",
+ "rotation_config",
)
@@ -204,7 +206,7 @@ class BaseCompressor(object):
disable_opt_rtn: bool | None = None,
seed: int = 42,
low_cpu_mem_usage: bool = True,
- hadamard_config: str | dict | HadamardConfig | None = None,
+ rotation_config: str | dict | RotationConfig | None = None,
**kwargs,
):
"""Initialize AutoRound with quantization and tuning configuration.
@@ -275,7 +277,6 @@ class BaseCompressor(object):
value = kwargs.pop(k)
if value is not None:
self.user_scheme_overrides[k] = value
-
# Model related
model_dtype = kwargs.pop("model_dtype", None)
self.mllm = kwargs.pop("mllm") if "mllm" in kwargs else False
@@ -559,14 +560,34 @@ class BaseCompressor(object):
except (ImportError, ModuleNotFoundError):
logger.error("algorithm extension import error, fallback to default mode")
- # apply hadamard transform
- if hadamard_config:
- logger.info("Applying Hadamard transform to the model.")
- from auto_round.experimental.transform.apply import apply_hadamard_transform
- from auto_round.experimental.utils import normalize_hadamard_config
-
- self.hadamard_config = normalize_hadamard_config(hadamard_config, self.data_type)
- self.model = apply_hadamard_transform(self.model, self.hadamard_config, data_type=self.data_type)
+ # Hadamard rotation / transform – unified entry that auto-dispatches
+ # between the "inplace" (QuaRot-style residual rotation; supports any
+ # dtype incl. INT4/INT8/FPx; can fuse online Had into weights) and
+ # "transform" (per-Linear weight+activation Had with fused triton
+ # kernel; only MXFP4 / NVFP4; cannot fuse) backends.
+ # See :class:`RotationConfig` for routing rules.
+ if rotation_config is not None:
+ from auto_round.experimental.apply_rotation_transform import apply_hadamard_rotation
+
+ default_block_size = self.group_size
+ if self.act_bits < 16:
+ default_block_size = self.act_group_size
+ rotation_config = dump_group_size_to_rotation_config(rotation_config, default_block_size)
+
+ self.model, _ = apply_hadamard_rotation(
+ self.model,
+ rotation_config,
+ data_type=self.data_type,
+ compute_device=self.device,
+ )
+ stored = getattr(self.model, "rotation_config", rotation_config)
+ # Normalize to a plain dict so it round-trips through JSON
+ # (HF saves quantization_config to config.json on save_pretrained).
+ if hasattr(stored, "model_dump"):
+ stored = stored.model_dump()
+ self.rotation_config = stored
+ # Mirror the JSON-safe form back onto the model.
+ setattr(self.model, "rotation_config", stored)
self.blocks_requiring_input_ids = []
self.has_variable_block_shape = False
@@ -1081,7 +1102,9 @@ class BaseCompressor(object):
dtype = f"{dtype}_{'sym' if config['sym'] else 'asym'}"
# Optionally use optimized rounding (RTN) variant
- if not self.disable_opt_rtn and f"rtn_{dtype}" in QUANT_FUNC_WITH_DTYPE:
+ if not self.disable_opt_rtn and f"opt_rtn_{dtype}" in QUANT_FUNC_WITH_DTYPE:
+ dtype = f"opt_rtn_{dtype}"
+ elif f"rtn_{dtype}" in QUANT_FUNC_WITH_DTYPE:
dtype = f"rtn_{dtype}"
quant_func = QUANT_FUNC_WITH_DTYPE[dtype]
@@ -1194,7 +1217,10 @@ class BaseCompressor(object):
hook_handles.append(hook)
return hook_handles
- hooks = register_act_hook(model)
+ if not is_wint4aint4(self): # INT4 no imatrix is much better
+ hooks = register_act_hook(model)
+ else:
+ hooks = []
try:
if hasattr(model, "hf_device_map") and len(model.hf_device_map) > 1:
@@ -1284,6 +1310,7 @@ class BaseCompressor(object):
enable_round_tuning=False,
enable_torch_compile=self.enable_torch_compile,
disable_opt_rtn=disable_opt_rtn,
+ iters=self.iters,
)
m = m.unwrapper({})
except torch.OutOfMemoryError:
@@ -1299,6 +1326,7 @@ class BaseCompressor(object):
enable_norm_bias_tuning=False,
enable_round_tuning=False,
enable_torch_compile=self.enable_torch_compile,
+ iters=self.iters,
)
m = m.unwrapper({})
except Exception as e:
@@ -1985,6 +2013,7 @@ class BaseCompressor(object):
enable_torch_compile=self.enable_torch_compile,
device=self.device,
disable_opt_rtn=self.disable_opt_rtn,
+ iters=self.iters,
)
new_layer = wrapper_layer.unwrapper({})
set_module(self.model, layer_name, new_layer)
@@ -2792,6 +2821,7 @@ class BaseCompressor(object):
enable_minmax_tuning=self.enable_minmax_tuning,
enable_torch_compile=self.enable_torch_compile,
device=device,
+ iters=self.iters,
).to(device)
round_params = []
minmax_params = []
@@ -3107,6 +3137,7 @@ class BaseCompressor(object):
self.enable_norm_bias_tuning,
enable_torch_compile=self.enable_torch_compile,
device=device,
+ iters=self.iters,
)
# Call this before quantization and after applying the block wrapper.
if is_nv_fp(self.data_type): # enable qkv and moe structure global_scale fuse.
diff --git a/auto_round/compressors/utils.py b/auto_round/compressors/utils.py
index 8240c8304..7b6f04545 100644
--- a/auto_round/compressors/utils.py
+++ b/auto_round/compressors/utils.py
@@ -107,7 +107,7 @@ def is_wint_a16(ar: Callable) -> bool:
def is_dynamic_wint8aint8(ar_or_format: Union[str, Callable]) -> bool:
if isinstance(ar_or_format, str):
- return "int8_w8a8" in ar_or_format.lower()
+ return "int8_w8a8" in ar_or_format.lower() or "int8" in ar_or_format.lower()
if not ar_or_format.act_dynamic:
return False
if is_wint8aint8(ar_or_format):
@@ -115,6 +115,16 @@ def is_dynamic_wint8aint8(ar_or_format: Union[str, Callable]) -> bool:
return False
+def is_wint4aint4(ar_or_scheme: Union[str, Callable]):
+ if isinstance(ar_or_scheme, str):
+ return "int4" in ar_or_scheme.lower()
+ elif (
+ "int4" in ar_or_scheme.act_data_type or ("int" in ar_or_scheme.act_data_type and ar_or_scheme.act_bits == 4)
+ ) and ("int4" in ar_or_scheme.data_type or ("int" in ar_or_scheme.data_type and ar_or_scheme.bits == 4)):
+ return True
+ return False
+
+
def is_dynamic_afp8(ar_or_format: Callable) -> bool:
return ar_or_format.act_dynamic and ar_or_format.act_data_type.startswith("fp") and ar_or_format.act_bits == 8
diff --git a/auto_round/data_type/gguf.py b/auto_round/data_type/gguf.py
index a63c3df90..1a419d880 100644
--- a/auto_round/data_type/gguf.py
+++ b/auto_round/data_type/gguf.py
@@ -40,7 +40,7 @@ def quant_tensor_sym_dq(
super_bits=6,
**kwargs,
):
- """Quantize and de-quantize tensor symmetrically. full range, credict goes to llamacpp community
+ """Quantize and de-quantize tensor symmetrically. full range, credit goes to llamacpp community
Args:
tensor: Tensor containing the tensor to be quantized
diff --git a/auto_round/data_type/int.py b/auto_round/data_type/int.py
index 960a7fc08..73645dd60 100644
--- a/auto_round/data_type/int.py
+++ b/auto_round/data_type/int.py
@@ -15,9 +15,9 @@ from typing import Union
import torch
+from auto_round import envs
from auto_round.data_type.register import register_dtype
from auto_round.data_type.utils import reshape_pad_tensor_by_group_size, revert_tensor_by_pad, round_ste
-from auto_round.logger import logger
from auto_round.utils import get_reciprocal
@@ -41,20 +41,27 @@ def search_scales(data: torch.Tensor, bits: int, qw: Union[None, torch.Tensor, f
# Set default weight if None
if qw is None:
qw = 1.0
-
# Compute initial best loss
best_loss = ((scales * L - data).to(torch.float32)) ** 2
if isinstance(qw, torch.Tensor):
best_loss.mul_(qw) # inplace multiply by weight
best_loss = torch.sum(best_loss, dim=-1)
-
+ if bits == 2:
+ search_min = 18 * 5
+ step = 0.01
+ else:
+ grid = 200
+ search_ratio = envs.AR_SEARCH_SCALE_RATIO or 0.75 # default 0.5 -> nmax/2
+ search_min = nmax * search_ratio
+ step = search_min / grid * 2 # 0.08
+ search_min = int(search_min / step)
# Iterative search over small adjustments
- for _is in range(-18 * 5, 18 * 5 + 1):
+ for _is in range(-search_min, search_min + 1):
if _is == 0:
continue
# Update iscales in-place
- iscales_tmp = -(nmax - 0.01 * _is) * get_reciprocal(group_max)
+ iscales_tmp = -(nmax - step * _is) * get_reciprocal(group_max)
# Compute temporary quantized values (in-place round + clamp)
tmp_L = torch.empty_like(data)
@@ -79,9 +86,9 @@ def search_scales(data: torch.Tensor, bits: int, qw: Union[None, torch.Tensor, f
return scales
-@register_dtype("rtn_int_sym")
-def quant_tensor_rtn_sym(tensor, bits=4, group_size=-1, v=0, q_scale_thresh=1e-5, imatrix=None, **kwargs):
- """Quantize and de-quantize tensor asymmetrically. full range, credict goes to llamacpp community
+@register_dtype("opt_rtn_int_sym")
+def quant_tensor_opt_rtn_sym(tensor, bits=4, group_size=-1, v=0, q_scale_thresh=1e-5, imatrix=None, **kwargs):
+ """Quantize and de-quantize tensor asymmetrically. full range, credit goes to llamacpp community
Args:
tensor: Tensor containing the tensor to be quantized
@@ -115,6 +122,46 @@ def quant_tensor_rtn_sym(tensor, bits=4, group_size=-1, v=0, q_scale_thresh=1e-5
return qdq_result, scale, maxq
+@register_dtype("rtn_int_sym")
+def quant_tensor_rtn_sym(
+ tensor,
+ bits=4,
+ group_size=-1,
+ q_scale_thresh=1e-5,
+ min_scale=1.0,
+ max_scale=1.0,
+ scale_dtype=torch.float16,
+ **kwargs
+):
+ """Quantize and de-quantize tensor asymmetrically. full range, credit goes to llamacpp community
+
+ Args:
+ tensor: Tensor containing the tensor to be quantized
+ bits: Number of bits for quantization (e.g., 2, 3, 4, 8)
+ group_size: Number of elements to share scale for quantization
+ v: Rounding value perturbation
+ q_scale_thresh: clip the quantized scale's magnitude to this value to improve the numerical stability
+
+ Returns:
+ Quantized and de-quantized tensor, scale, zero-point
+ """
+ tensor, orig_shape, pad_len = reshape_pad_tensor_by_group_size(tensor, group_size)
+ maxq = 2 ** (bits - 1)
+
+ wmin_tmp = torch.clamp(tensor.min(-1)[0], max=0)
+ wmax_tmp = torch.clamp(tensor.max(-1)[0], min=0)
+ wmin_abs = -(wmin_tmp * min_scale) # pylint: disable=E1130
+ wmax_abs = wmax_tmp * max_scale
+ max_v = (2 * (wmax_abs < wmin_abs).int() - 1) * torch.max(wmax_abs, wmin_abs)
+ scale = (max_v / maxq).to(scale_dtype)
+ scale = torch.where(scale < 0, torch.clamp(scale, max=-q_scale_thresh), torch.clamp(scale, min=q_scale_thresh))
+ scale = scale.unsqueeze(dim=-1)
+ int_w = tensor.div(scale).round_().clamp_(-maxq, maxq - 1)
+ qdq_result = (int_w.mul_(scale)).to(tensor.dtype)
+ qdq_result = revert_tensor_by_pad(qdq_result, orig_shape=orig_shape, pad_len=pad_len)
+ return qdq_result, scale, maxq
+
+
@register_dtype("int_sym")
def quant_tensor_sym(
tensor,
@@ -129,7 +176,7 @@ def quant_tensor_sym(
q_scale_thresh=1e-5,
**kwargs
):
- """Quantize and de-quantize tensor asymmetrically. full range, credict goes to llamacpp community
+ """Quantize and de-quantize tensor asymmetrically. full range, credit goes to llamacpp community
Args:
tensor: Tensor containing the tensor to be quantized
diff --git a/auto_round/data_type/utils.py b/auto_round/data_type/utils.py
index a8e3eb218..bd4d74eaa 100644
--- a/auto_round/data_type/utils.py
+++ b/auto_round/data_type/utils.py
@@ -101,7 +101,9 @@ def revert_tensor_by_pad(data: torch.Tensor, orig_shape: tuple, pad_len: Union[i
return data_new
-def get_quant_func(dtype: str, bits: int, sym: bool, disable_opt_rtn=False, group_size=None) -> tuple[callable, str]:
+def get_quant_func(
+ dtype: str, bits: int, sym: bool, disable_opt_rtn=False, group_size=None, iters=200
+) -> tuple[callable, str]:
"""Retrieve the quantization function based on data type, bit width, and symmetry.
This function returns the appropriate quantization function from the QUANT_FUNC_WITH_DTYPE
@@ -130,7 +132,15 @@ def get_quant_func(dtype: str, bits: int, sym: bool, disable_opt_rtn=False, grou
def pad_bits(data_type):
return data_type + str(bits)
- if not disable_opt_rtn:
+ if not disable_opt_rtn and iters == 0:
+ rtn_data_type = "opt_rtn_" + dtype
+ data_types = [rtn_data_type, pad_bits(rtn_data_type), pad_sym(rtn_data_type), pad_sym(pad_bits(rtn_data_type))]
+ for data_type in data_types:
+ from auto_round.data_type import QUANT_FUNC_WITH_DTYPE
+
+ if data_type in QUANT_FUNC_WITH_DTYPE:
+ return QUANT_FUNC_WITH_DTYPE[data_type], data_type
+ if iters == 0:
rtn_data_type = "rtn_" + dtype
data_types = [rtn_data_type, pad_bits(rtn_data_type), pad_sym(rtn_data_type), pad_sym(pad_bits(rtn_data_type))]
for data_type in data_types:
diff --git a/auto_round/envs.py b/auto_round/envs.py
index 8a731ad10..a1a0e4f0d 100644
--- a/auto_round/envs.py
+++ b/auto_round/envs.py
@@ -34,6 +34,16 @@ environment_variables: dict[str, Callable[[], Any]] = {
"AR_DISABLE_DATASET_SUBPROCESS": lambda: os.getenv("AR_DISABLE_DATASET_SUBPROCESS", "0").lower() in ("1", "true"),
"AR_DISABLE_COPY_MTP_WEIGHTS": lambda: os.getenv("AR_DISABLE_COPY_MTP_WEIGHTS", "0").lower()
in ("1", "true", "yes"),
+ "AR_ACT_SCALE": lambda: float(os.getenv("AR_ACT_SCALE", "1.0")),
+ "AR_ENABLE_ACT_MINMAX_TUNING": lambda: os.getenv("AR_ENABLE_ACT_MINMAX_TUNING", "0").lower()
+ in ("1", "true", "yes"),
+ "AR_FUSE_ONLINE_ROTATION": lambda: os.getenv("AR_FUSE_ONLINE_ROTATION", "0").lower() in ("1", "true", "yes"),
+ # Controls the search range ratio for symmetric int scale search in
+ # `auto_round.data_type.int.search_scales`. The search bound is
+ # `nmax * AR_SEARCH_SCALE_RATIO` (default None).
+ "AR_SEARCH_SCALE_RATIO": lambda: (
+ float(os.getenv("AR_SEARCH_SCALE_RATIO")) if os.getenv("AR_SEARCH_SCALE_RATIO") is not None else None
+ ),
}
diff --git a/auto_round/eval/evaluation.py b/auto_round/eval/evaluation.py
index d4c4ec917..b3136c32f 100644
--- a/auto_round/eval/evaluation.py
+++ b/auto_round/eval/evaluation.py
@@ -428,7 +428,7 @@ def run_model_evaluation(model, tokenizer, autoround, folders, formats, device_s
eval_gguf_model = any(file.endswith("gguf") for file in os.listdir(eval_folder))
# Determine if model instance evaluation is needed
- need_model_instance = (autoround.act_bits <= 8 and formats[-1] == "fake") or eval_gguf_model
+ need_model_instance = formats[-1] == "fake" or eval_gguf_model
if need_model_instance:
# Load or prepare model instance
diff --git a/auto_round/experimental/apply_rotation_transform.py b/auto_round/experimental/apply_rotation_transform.py
new file mode 100644
index 000000000..fc7dbf297
--- /dev/null
+++ b/auto_round/experimental/apply_rotation_transform.py
@@ -0,0 +1,154 @@
+# Copyright (c) 2026 Intel Corporation
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+# http://www.apache.org/licenses/LICENSE-2.0
+
+"""Unified entry point for Hadamard rotation/transform.
+
+Two backend implementations exist:
+
+* ``inplace`` – :mod:`auto_round.experimental.rotation_inplace`
+ QuaRot-style residual-stream rotation. Works for any weight/activation
+ dtype. Optionally fuses the online Hadamard into weights
+ (``fuse_online_to_weight=True``).
+* ``transform`` – :mod:`auto_round.experimental.transform`
+ Per-Linear weight + activation Hadamard with a fused triton kernel.
+ Only supports MXFP4 / NVFP4 and **cannot** fuse online to weight.
+
+Routing is controlled by :class:`RotationConfig.backend`:
+
+ "inplace" -> always inplace
+ "transform" -> always transform (validates dtype + no-fuse)
+ "auto" -> if user asked to fuse -> inplace
+ elif data_type is mx_fp / nv_fp -> transform
+ else -> inplace
+"""
+
+from __future__ import annotations
+
+from typing import Any, Union
+
+import torch
+
+import auto_round.envs as envs
+from auto_round.compressors.utils import is_mx_fp, is_nv_fp
+from auto_round.experimental.transform.rotation_config import RotationConfig
+from auto_round.experimental.utils import normalize_rotation_config
+from auto_round.utils import logger
+
+__all__ = ["apply_hadamard_rotation", "resolve_hadamard_backend"]
+
+
+def _to_config(
+ rotation_config: Union[str, dict, RotationConfig, None],
+ data_type: str,
+) -> RotationConfig:
+ """Normalise *rotation_config* and return a :class:`RotationConfig` instance."""
+ cfg_dict = normalize_rotation_config(rotation_config, data_type)
+ if isinstance(cfg_dict, RotationConfig):
+ return cfg_dict
+ return RotationConfig.model_validate(cfg_dict or {})
+
+
+def resolve_hadamard_backend(config: RotationConfig, data_type: str) -> str:
+ """Resolve the actual backend (``"inplace"`` / ``"transform"``) from config."""
+ requested = config.backend
+ fuse_requested = bool(config.fuse_online_to_weight)
+ allow_online_rotation: bool = config.allow_online_rotation
+
+ if requested == "inplace":
+ return "inplace"
+
+ transform_backend_name = "transform"
+ if requested == "transform":
+ if fuse_requested:
+ raise ValueError(
+ f"backend='{transform_backend_name}' does not support fuse_online_to_weight=True. "
+ "Use backend='inplace' (or backend='auto' with fuse_online_to_weight=True) instead."
+ )
+ if not (is_mx_fp(data_type) or is_nv_fp(data_type)):
+ raise ValueError(
+ f"backend='{transform_backend_name}' only supports MXFP4 / NVFP4 (got data_type={data_type!r}). "
+ "Use backend='inplace' or backend='auto' for other dtypes."
+ )
+ if not allow_online_rotation:
+ raise ValueError(f"backend='{transform_backend_name}' only supports `allow_online_rotation`=True")
+
+ return "transform"
+
+ # backend == "auto"
+ if fuse_requested:
+ return "inplace"
+ if is_mx_fp(data_type) or is_nv_fp(data_type):
+ return "transform"
+ return "inplace"
+
+
+def apply_hadamard_rotation(
+ model: torch.nn.Module,
+ rotation_config: Union[str, dict, RotationConfig, None],
+ data_type: str,
+ compute_device: torch.device | str = None,
+) -> (torch.nn.Module, Any):
+ """Apply Hadamard rotation/transform to *model*, dispatching by backend.
+
+ Args:
+ model: Target model.
+ rotation_config: ``str`` / ``dict`` / :class:`RotationConfig` / ``None``.
+ See :class:`RotationConfig` for fields.
+ data_type: Quantization data type (e.g. ``"mx_fp"``, ``"nv_fp"``,
+ ``"int"``, ``"fp"``).
+ compute_device: Device for inplace-backend computation. Ignored by
+ the transform backend.
+
+ Returns:
+ The same model (for chaining); also stored on ``model.rotation_config``.
+ """
+ config = _to_config(rotation_config, data_type)
+ backend = resolve_hadamard_backend(config, data_type)
+
+ # Resolve fuse flag: explicit > env var > default(True)
+ fuse_online_to_weight = config.fuse_online_to_weight
+ if config.fuse_online_to_weight is not None:
+ fuse_online_to_weight = bool(config.fuse_online_to_weight)
+ elif envs.AR_FUSE_ONLINE_ROTATION:
+ fuse_online_to_weight = bool(envs.AR_FUSE_ONLINE_ROTATION)
+
+ logger.info(
+ f"Applying Hadamard (backend={backend}, "
+ f"data_type={data_type}, fuse_online_to_weight={fuse_online_to_weight if backend == 'inplace' else False})."
+ )
+
+ if backend == "inplace":
+ logger.warning("this backend does not support real exporting, please export the model to fake format")
+ from auto_round.experimental.rotation_inplace import apply_rotation_transform
+
+ # block_size -> group_size (None / -1 / 0 means full-dimension)
+ bs = config.block_size
+ group_size = bs if (bs is not None and bs > 0) else None
+
+ model, hooks = apply_rotation_transform(
+ model,
+ group_size=group_size,
+ allow_online_rotation=config.allow_online_rotation,
+ rotation_matrix=config.hadamard_type,
+ fuse_online_to_weight=fuse_online_to_weight,
+ compute_device=compute_device,
+ )
+ # Stash for downstream (export / serialization). Plain dict so JSON
+ # serialization (HF save_pretrained -> config.json) round-trips.
+ setattr(model, "rotation_config", config.model_dump() if hasattr(config, "model_dump") else config)
+ return model, hooks
+
+ elif backend == "transform":
+ supported_hadamard_types = ("hadamard", "random_hadamard")
+ if config.hadamard_type not in supported_hadamard_types:
+ raise ValueError("this backend only supports hadamard or random_hadamard")
+ from auto_round.experimental.transform.apply import apply_rotation_transform
+
+ return apply_rotation_transform(model, config, data_type=data_type)
+ else:
+ raise ValueError(f"Unsupported Hadamard backend {backend!r}")
diff --git a/auto_round/experimental/qmodules/mx.py b/auto_round/experimental/qmodules/mx.py
index 23d05872c..dea099760 100644
--- a/auto_round/experimental/qmodules/mx.py
+++ b/auto_round/experimental/qmodules/mx.py
@@ -96,9 +96,9 @@ class MXQuantLinearBase(QModuleBase):
)
self.register_buffer("weight_scale", init_weight_scale)
- hadamard_config = getattr(config, "hadamard_config", None)
- # TODO: remove the limit: hadamard_config["hadamard_type"] == "random_hadamard"
- if hadamard_config is not None and hadamard_config["hadamard_type"] == "random_hadamard":
+ rotation_config = getattr(config, "rotation_config", None)
+ # TODO: remove the limit: rotation_config["hadamard_type"] == "random_hadamard"
+ if rotation_config is not None and rotation_config["hadamard_type"] == "random_hadamard":
self.enable_transform = True
self.register_buffer(
"hadamard_matrix",
diff --git a/auto_round/experimental/qmodules/nvfp4.py b/auto_round/experimental/qmodules/nvfp4.py
index 4350f0567..dafeb2567 100644
--- a/auto_round/experimental/qmodules/nvfp4.py
+++ b/auto_round/experimental/qmodules/nvfp4.py
@@ -119,9 +119,9 @@ class NVFP4QuantLinear(QModuleBase):
),
)
- hadamard_config = getattr(config, "hadamard_config", None)
- # TODO: remove the limit: hadamard_config["hadamard_type"] == "random_hadamard"
- if hadamard_config is not None and hadamard_config["hadamard_type"] == "random_hadamard":
+ rotation_config = getattr(config, "rotation_config", None)
+ # TODO: remove the limit: rotation_config["hadamard_type"] == "random_hadamard"
+ if rotation_config is not None and rotation_config["hadamard_type"] == "random_hadamard":
self.enable_transform = True
self.register_buffer(
"hadamard_matrix",
diff --git a/auto_round/experimental/rotation_inplace/__init__.py b/auto_round/experimental/rotation_inplace/__init__.py
new file mode 100644
index 000000000..8cdef31b0
--- /dev/null
+++ b/auto_round/experimental/rotation_inplace/__init__.py
@@ -0,0 +1,5 @@
+# # Copyright (C) 2026 Intel Corporation
+# # SPDX-License-Identifier: Apache-2.0
+
+from auto_round.experimental.rotation_inplace.apply_rotation_transform import apply_rotation_transform
+from auto_round.experimental.rotation_inplace.utils import clear_random_hadamard_cache
diff --git a/auto_round/experimental/rotation_inplace/apply_rotation_transform.py b/auto_round/experimental/rotation_inplace/apply_rotation_transform.py
new file mode 100644
index 000000000..1052b036f
--- /dev/null
+++ b/auto_round/experimental/rotation_inplace/apply_rotation_transform.py
@@ -0,0 +1,882 @@
+# # Copyright (C) 2026 Intel Corporation
+# # SPDX-License-Identifier: Apache-2.0
+
+"""Hadamard inplace rotation — public API and rotation primitives.
+
+Supports LLaMA-2, LLaMA-3, Qwen-3 (and any model with the same layout).
+The entry point is :func:`apply_hadamard_rotation`.
+"""
+
+import gc
+import typing
+from typing import Dict, Union
+
+import torch
+import tqdm
+
+from auto_round.experimental.rotation_inplace.model_config import (
+ MAPPING_REGISTRY,
+ RotationMapping,
+ _resolve,
+ infer_mapping_from_model,
+)
+from auto_round.experimental.rotation_inplace.utils import (
+ CrossHeadOnlineHadamardHook,
+ FullOnlineHadamardHook,
+ GroupOnlineHadamardHook,
+ _get_custom_had,
+ _normalize_rotation_matrix,
+ _resolve_compute_device,
+ _rotate_embedding_grouped,
+ _rotate_linear_grouped,
+ apply_cross_head_had_to_linear,
+ apply_exact_had_to_linear,
+ deterministic_hadamard_matrix,
+ get_hadK,
+ get_or_create_random_hadamard,
+)
+
+# ---------------------------------------------------------------------------
+# Low-level primitives (model-agnostic via RotationMapping)
+# ---------------------------------------------------------------------------
+
+
+def _fuse_ln_linear(
+ layernorm: torch.nn.Module,
+ linear_layers: typing.Iterable[torch.nn.Linear],
+) -> None:
+ """Fuse the linear operations in LayerNorm into adjacent linear blocks."""
+ for linear in linear_layers:
+ linear_dtype = linear.weight.dtype
+ dev = linear.weight.device
+
+ W_ = linear.weight.data.double()
+ ln_weight = layernorm.weight.double().to(dev)
+ linear.weight.data = (W_ * ln_weight).to(linear_dtype)
+
+ if hasattr(layernorm, "bias") and layernorm.bias is not None:
+ if linear.bias is None:
+ linear.bias = torch.nn.Parameter(torch.zeros(linear.out_features, dtype=torch.float64, device=dev))
+ ln_bias = layernorm.bias.double().to(dev)
+ linear.bias.data = linear.bias.data.double() + torch.matmul(W_, ln_bias)
+ linear.bias.data = linear.bias.data.to(linear_dtype)
+
+
+def _reset_ln_params(layernorm: torch.nn.Module) -> None:
+ """Reset LayerNorm to identity: weight=1, bias=0."""
+ layernorm.weight.data.fill_(1.0)
+ if hasattr(layernorm, "bias") and layernorm.bias is not None:
+ layernorm.bias.data.fill_(0.0)
+
+
+def _rotate_linear_by_Q(module: torch.nn.Linear, Q: torch.Tensor, side: str, compute_device=None) -> None:
+ """Apply rotation *Q* to a Linear layer's weight (and bias if present).
+
+ Args:
+ side: ``'input'`` → W = W @ Q (rotate input side)
+ ``'output'`` → W = Q^T @ W (rotate output side)
+ compute_device: Device to run computation on. If None, auto-detects GPU.
+ """
+ dtype = module.weight.data.dtype
+ dev = module.weight.data.device
+ cdev = _resolve_compute_device(compute_device)
+ W_ = module.weight.data.to(device=cdev, dtype=torch.float64)
+ Q_ = Q.to(device=cdev)
+ if side == "input":
+ new_W = torch.matmul(W_, Q_).to(device=dev, dtype=dtype)
+ else:
+ new_W = torch.matmul(Q_.T, W_).to(device=dev, dtype=dtype)
+ # Release fp64 copy before assigning back so peak memory ≈ 1× weight + 1× rotated.
+ del W_
+ module.weight.data = new_W
+ if side == "output" and module.bias is not None:
+ b = module.bias.data.to(device=cdev, dtype=torch.float64)
+ new_b = torch.matmul(Q_.T, b).to(device=dev, dtype=dtype)
+ del b
+ module.bias.data = new_b
+ del Q_
+
+
+def _untie_word_embeddings(model, mapping: RotationMapping) -> None:
+ """Break tied weights between lm_head and embedding if they share the same tensor."""
+ embedding = _resolve(model, mapping.embedding)
+ lm_head = _resolve(model, mapping.lm_head)
+
+ if lm_head.weight.data_ptr() != embedding.weight.data_ptr():
+ return
+
+ lm_head.weight = torch.nn.Parameter(lm_head.weight.data.clone())
+ if hasattr(model.config, "tie_word_embeddings"):
+ model.config.tie_word_embeddings = False
+
+
+def _uses_layernorm_with_mean(model, mapping: RotationMapping) -> bool:
+ """Check whether the model uses standard LayerNorm (which subtracts mean)."""
+ layers = _resolve(model, mapping.layers_attr)
+ first_ln = _resolve(layers[0], mapping.attn_input_ln)
+ return isinstance(first_ln, torch.nn.LayerNorm)
+
+
+def _bake_mean_into_linear(linear: torch.nn.Linear) -> None:
+ """Subtract column-wise mean from a Linear layer's weight (and mean from bias)."""
+ linear_dtype = linear.weight.dtype
+ W_ = linear.weight.data.double()
+ linear.weight.data = (W_ - W_.mean(dim=-2, keepdim=True)).to(linear_dtype)
+ if linear.bias is not None:
+ b_ = linear.bias.data.double()
+ linear.bias.data = (b_ - b_.mean()).to(linear_dtype)
+
+
+def _subtract_embedding_mean(model, mapping: RotationMapping) -> None:
+ """Subtract per-row mean from the embedding weight matrix."""
+ W = _resolve(model, mapping.embedding)
+ dtype = W.weight.data.dtype
+ W_ = W.weight.data.to(dtype=torch.float64)
+ W.weight.data = (W_ - W_.mean(dim=-1, keepdim=True)).to(dtype=dtype)
+
+ if mapping.positional_embedding is not None:
+ P = _resolve(model, mapping.positional_embedding)
+ p_dtype = P.weight.data.dtype
+ P_ = P.weight.data.to(dtype=torch.float64)
+ P.weight.data = (P_ - P_.mean(dim=-1, keepdim=True)).to(dtype=p_dtype)
+
+
+class _RMSNorm(torch.nn.Module):
+ """RMS Normalization (no mean subtraction)."""
+
+ def __init__(self, dim: int, eps: float = 1e-5):
+ super().__init__()
+ self.eps = eps
+ self.register_buffer("weight", torch.ones(dim))
+
+ def forward(self, x: torch.Tensor) -> torch.Tensor:
+ rms = torch.sqrt(x.pow(2).mean(dim=-1, keepdim=True) + self.eps)
+ return x / rms * self.weight
+
+
+def _replace_layernorms_with_rmsnorm(model) -> None:
+ """Replace all ``nn.LayerNorm`` modules with ``_RMSNorm``."""
+ replacements = []
+ for name, module in model.named_modules():
+ if isinstance(module, torch.nn.LayerNorm):
+ replacements.append((name, module))
+
+ for name, module in replacements:
+ parts = name.rsplit(".", 1)
+ if len(parts) == 2:
+ parent = _resolve(model, parts[0])
+ attr = parts[1]
+ else:
+ parent = model
+ attr = parts[0]
+ rms = _RMSNorm(module.normalized_shape[0], eps=module.eps)
+ rms = rms.to(device=module.weight.device, dtype=module.weight.dtype)
+ setattr(parent, attr, rms)
+
+
+# ---------------------------------------------------------------------------
+# High-level steps driven by RotationMapping
+# ---------------------------------------------------------------------------
+
+
+def _fuse_layer_norms(model, mapping: RotationMapping) -> None:
+ """Fuse all LayerNorm parameters into adjacent Linear layers."""
+ layers = _resolve(model, mapping.layers_attr)
+
+ for layer in layers:
+ mlp_ln = _resolve(layer, mapping.mlp_input_ln)
+ mlp_linears = [_resolve(layer, p) for p in mapping.mlp_in]
+ _fuse_ln_linear(mlp_ln, mlp_linears)
+ _reset_ln_params(mlp_ln)
+
+ attn_ln = _resolve(layer, mapping.attn_input_ln)
+ attn_linears = [
+ _resolve(layer, mapping.attn_q),
+ _resolve(layer, mapping.attn_k),
+ _resolve(layer, mapping.attn_v),
+ ]
+ _fuse_ln_linear(attn_ln, attn_linears)
+ _reset_ln_params(attn_ln)
+
+ pre_head_ln = _resolve(model, mapping.pre_head_ln)
+ lm_head = _resolve(model, mapping.lm_head)
+ _fuse_ln_linear(pre_head_ln, [lm_head])
+ _reset_ln_params(pre_head_ln)
+
+
+# ---------------------------------------------------------------------------
+# Unified weight rotation (full or grouped)
+# ---------------------------------------------------------------------------
+
+
+def _rotate_weights(
+ model,
+ mapping: RotationMapping,
+ use_fast_had: bool = True,
+ group_size: int = None,
+ compute_device: torch.device = None,
+ had_dict: dict = None,
+ preset: str = None,
+ fuse_online_to_weight: bool = True,
+) -> None:
+ """Apply Hadamard rotation to all weights.
+
+ Args:
+ group_size: ``None`` → full Hadamard rotation.
+ ``int`` → block-diagonal rotation with this block size.
+ compute_device: Device to run Hadamard computation on (e.g. ``"cuda:0"``).
+ Weights are moved there temporarily and moved back afterwards.
+ If ``None``, auto-detects GPU availability.
+ allow_online_rotation: If ``True`` (default), apply extra input-side
+ Hadamard rotations on ``down_proj`` and the OV pair (``v_proj``
+ output + ``o_proj`` input) that require compensating online hooks
+ at inference time. If ``False``, skip those extra rotations so
+ that **no** online hooks are needed.
+ had_dict: Normalized ``dict[int, Tensor]`` of custom Hadamard matrices
+ (keyed by dimension). Only used in grouped mode.
+ preset: Rotation preset name (``"quarot_hadamard"``, ``"hadamard"``,
+ ``"random_hadamard"``, or ``None``).
+
+ * ``"quarot_hadamard"``: fusable (residual-stream) rotations use
+ ``fast_hadamard_transform`` / random Hadamard; non-fusable
+ (online-paired) rotations and their weight-side counterparts use
+ deterministic ``get_hadK``/``matmul_hadU`` so that the online
+ hook at inference produces the exact same transform.
+ * ``"hadamard"``: all rotations use deterministic ``get_hadK`` /
+ ``matmul_hadU``. Full-mode Q is a deterministic Hadamard matrix.
+ * ``"random_hadamard"``: all rotations use random Hadamard matrices
+ from the global cache (``get_or_create_random_hadamard``).
+ Same dimension → same matrix everywhere.
+ * ``None``: same behaviour as ``"hadamard"`` (built-in butterfly).
+ """
+ compute_device = _resolve_compute_device(compute_device)
+ config = model.config
+ hidden_size = getattr(config, mapping.hidden_size_attr)
+ intermediate_size = getattr(config, mapping.intermediate_size_attr)
+ num_heads = getattr(config, mapping.num_heads_attr)
+ head_dim = mapping.attn_head_dim or (hidden_size // num_heads)
+
+ is_grouped = group_size is not None and group_size > 0
+ desc = f"Rotating (group_size={group_size})" if is_grouped else "Rotating"
+
+ # ----- Resolve per-operation Hadamard sources -----
+ fused_fast = use_fast_had
+ online_fast = False
+ if preset == "random_hadamard":
+ fused_fast = False
+
+ # -- Matrix resolution --
+ had_matrix, _found = _get_custom_had(had_dict, group_size) if is_grouped else (None, False)
+
+ online_had_matrix = had_matrix
+ if preset == "random_hadamard" and had_matrix is None:
+ had_matrix = get_or_create_random_hadamard(group_size if is_grouped else hidden_size, compute_device)
+ online_had_matrix = had_matrix
+ if preset == "quarot_hadamard" and is_grouped:
+ online_had_matrix = None # force deterministic for online-paired
+
+ # -- Helper: look up cached random matrix for online-paired ops --
+ def _online_had(dim):
+ """Return cached random matrix for *dim* under random_hadamard, else None."""
+ if preset == "random_hadamard":
+ return get_or_create_random_hadamard(dim, compute_device)
+ return None
+
+ if is_grouped:
+ assert hidden_size % group_size == 0, f"group_size={group_size} must divide hidden_size={hidden_size}"
+ assert (
+ intermediate_size % group_size == 0
+ ), f"group_size={group_size} must divide intermediate_size={intermediate_size}"
+
+ # --- Full mode: build Hadamard matrix Q ---
+ Q = None
+ if not is_grouped:
+ if preset == "hadamard":
+ Q = deterministic_hadamard_matrix(hidden_size, compute_device)
+ else:
+ # "random_hadamard", "quarot_hadamard", None — same shape → same matrix
+ Q = get_or_create_random_hadamard(hidden_size, compute_device)
+
+ # ---- Top-level: embedding / lm_head ----
+ # When fuse_online_to_weight=False, skip embedding and lm_head rotation:
+ # each layer is self-contained (weight rotation + online hook cancel out).
+ if fuse_online_to_weight:
+ embedding = _resolve(model, mapping.embedding)
+ if is_grouped:
+ _rotate_embedding_grouped(
+ embedding, group_size, use_fast_had=fused_fast, compute_device=compute_device, had_matrix=had_matrix
+ )
+ else:
+ dtype = embedding.weight.data.dtype
+ dev = embedding.weight.data.device
+ cdev = compute_device
+ W_ = embedding.weight.data.to(device=cdev, dtype=torch.float64)
+ new_W = torch.matmul(W_, Q.to(cdev)).to(device=dev, dtype=dtype)
+ del W_
+ embedding.weight.data = new_W
+
+ if mapping.positional_embedding is not None:
+ pos_emb = _resolve(model, mapping.positional_embedding)
+ if is_grouped:
+ _rotate_embedding_grouped(
+ pos_emb, group_size, use_fast_had=fused_fast, compute_device=compute_device, had_matrix=had_matrix
+ )
+ else:
+ pos_dtype = pos_emb.weight.data.dtype
+ pos_dev = pos_emb.weight.data.device
+ cdev = compute_device
+ P_ = pos_emb.weight.data.to(device=cdev, dtype=torch.float64)
+ new_P = torch.matmul(P_, Q.to(cdev)).to(device=pos_dev, dtype=pos_dtype)
+ del P_
+ pos_emb.weight.data = new_P
+
+ # ---- Top-level: lm_head ----
+ lm_head = _resolve(model, mapping.lm_head)
+ if is_grouped:
+ _rotate_linear_grouped(
+ lm_head,
+ group_size,
+ side="input",
+ use_fast_had=fused_fast,
+ compute_device=compute_device,
+ had_matrix=had_matrix,
+ )
+ else:
+ _rotate_linear_by_Q(lm_head, Q, side="input", compute_device=compute_device)
+
+ gc.collect()
+ if torch.cuda.is_available():
+ torch.cuda.empty_cache()
+
+ # ---- Per-layer rotation ----
+ layers = _resolve(model, mapping.layers_attr)
+ for layer in tqdm.tqdm(layers, unit="layer", desc=desc):
+ if fuse_online_to_weight:
+ # ---- fuse mode: QuaRot-style residual stream rotation ----
+ # Q/K/V: only residual Q on input (no online Had stacking, no hook).
+ # When Q == online Had (e.g. preset="hadamard"), Q @ Q = I cancels
+ # the rotation entirely, destroying quantization benefit.
+ # gate/up: only residual Q on input (no online Had stacking, no hook).
+ # down_proj: residual Q^T on output + online Had on input (+ hook).
+ # v_proj/o_proj: per-head/cross-head Had below (+ hook on o_proj).
+ for attr in (mapping.attn_q, mapping.attn_k, mapping.attn_v):
+ mod = _resolve(layer, attr)
+ if is_grouped:
+ _rotate_linear_grouped(
+ mod,
+ group_size,
+ side="input",
+ use_fast_had=fused_fast,
+ compute_device=compute_device,
+ had_matrix=had_matrix,
+ )
+ else:
+ _rotate_linear_by_Q(mod, Q, side="input", compute_device=compute_device)
+
+ # o_proj: residual stream output rotation
+ if is_grouped:
+ _rotate_linear_grouped(
+ _resolve(layer, mapping.attn_o),
+ group_size,
+ side="output",
+ use_fast_had=fused_fast,
+ compute_device=compute_device,
+ had_matrix=had_matrix,
+ )
+ else:
+ _rotate_linear_by_Q(_resolve(layer, mapping.attn_o), Q, side="output", compute_device=compute_device)
+
+ # gate/up: only residual Q on input
+ for attr in mapping.mlp_in:
+ mod = _resolve(layer, attr)
+ if is_grouped:
+ _rotate_linear_grouped(
+ mod,
+ group_size,
+ side="input",
+ use_fast_had=fused_fast,
+ compute_device=compute_device,
+ had_matrix=had_matrix,
+ )
+ else:
+ _rotate_linear_by_Q(mod, Q, side="input", compute_device=compute_device)
+
+ # down_proj: residual output + online input Had
+ down_proj = _resolve(layer, mapping.mlp_out)
+ if is_grouped:
+ _rotate_linear_grouped(
+ down_proj,
+ group_size,
+ side="output",
+ use_fast_had=fused_fast,
+ compute_device=compute_device,
+ had_matrix=had_matrix,
+ )
+ _rotate_linear_grouped(
+ down_proj,
+ group_size,
+ side="input",
+ use_fast_had=online_fast,
+ compute_device=compute_device,
+ had_matrix=online_had_matrix,
+ )
+ else:
+ _rotate_linear_by_Q(down_proj, Q, side="output", compute_device=compute_device)
+ apply_exact_had_to_linear(
+ down_proj,
+ had_dim=-1,
+ output=False,
+ use_fast_had=online_fast,
+ compute_device=compute_device,
+ had_matrix=_online_had(intermediate_size),
+ )
+
+ # OV projection: v_proj per-head output + o_proj full/cross-head input
+ v_proj = _resolve(layer, mapping.attn_v)
+ o_proj = _resolve(layer, mapping.attn_o)
+ if is_grouped:
+ pass
+ else:
+ online_head_had = _online_had(head_dim)
+ apply_exact_had_to_linear(
+ v_proj,
+ had_dim=head_dim,
+ output=True,
+ use_fast_had=online_fast,
+ compute_device=compute_device,
+ had_matrix=online_head_had,
+ )
+ if preset == "random_hadamard":
+ apply_exact_had_to_linear(
+ o_proj,
+ had_dim=head_dim,
+ output=False,
+ use_fast_had=online_fast,
+ compute_device=compute_device,
+ had_matrix=online_head_had,
+ )
+ apply_cross_head_had_to_linear(
+ o_proj,
+ num_heads,
+ head_dim,
+ use_fast_had=online_fast,
+ compute_device=compute_device,
+ had_matrix=_online_had(num_heads),
+ )
+ else:
+ apply_exact_had_to_linear(
+ o_proj,
+ had_dim=-1,
+ output=False,
+ use_fast_had=online_fast,
+ compute_device=compute_device,
+ )
+
+ else:
+ # ---- unfused mode: no residual rotation, only input-side Had ----
+ # Each layer gets Had fused on input side + compensating hook → equivalent.
+ # No embedding/lm_head rotation. No self-cancelling pair.
+ # v_proj treated same as Q/K (input Had only, no per-head/cross-head).
+
+ # Q/K/V: input-side Had on hidden_size
+ for attr in (mapping.attn_q, mapping.attn_k, mapping.attn_v):
+ mod = _resolve(layer, attr)
+ if is_grouped:
+ _rotate_linear_grouped(
+ mod,
+ group_size,
+ side="input",
+ use_fast_had=online_fast,
+ compute_device=compute_device,
+ had_matrix=online_had_matrix,
+ )
+ else:
+ apply_exact_had_to_linear(
+ mod,
+ had_dim=-1,
+ output=False,
+ use_fast_had=online_fast,
+ compute_device=compute_device,
+ had_matrix=_online_had(hidden_size),
+ )
+
+ # o_proj: input-side Had on hidden_size (full Had, not cross-head)
+ o_proj = _resolve(layer, mapping.attn_o)
+ if is_grouped:
+ _rotate_linear_grouped(
+ o_proj,
+ group_size,
+ side="input",
+ use_fast_had=online_fast,
+ compute_device=compute_device,
+ had_matrix=online_had_matrix,
+ )
+ else:
+ apply_exact_had_to_linear(
+ o_proj,
+ had_dim=-1,
+ output=False,
+ use_fast_had=online_fast,
+ compute_device=compute_device,
+ had_matrix=_online_had(hidden_size),
+ )
+
+ # gate/up: input-side Had on hidden_size
+ for attr in mapping.mlp_in:
+ mod = _resolve(layer, attr)
+ if is_grouped:
+ _rotate_linear_grouped(
+ mod,
+ group_size,
+ side="input",
+ use_fast_had=online_fast,
+ compute_device=compute_device,
+ had_matrix=online_had_matrix,
+ )
+ else:
+ apply_exact_had_to_linear(
+ mod,
+ had_dim=-1,
+ output=False,
+ use_fast_had=online_fast,
+ compute_device=compute_device,
+ had_matrix=_online_had(hidden_size),
+ )
+
+ # down_proj: input-side Had on intermediate_size
+ down_proj = _resolve(layer, mapping.mlp_out)
+ if is_grouped:
+ _rotate_linear_grouped(
+ down_proj,
+ group_size,
+ side="input",
+ use_fast_had=online_fast,
+ compute_device=compute_device,
+ had_matrix=online_had_matrix,
+ )
+ else:
+ apply_exact_had_to_linear(
+ down_proj,
+ had_dim=-1,
+ output=False,
+ use_fast_had=online_fast,
+ compute_device=compute_device,
+ had_matrix=_online_had(intermediate_size),
+ )
+
+ # Per-layer cleanup: drop fp64 temporaries and CUDA caching allocator
+ # blocks so peak memory stays at ~1 layer's worth instead of accumulating
+ # across all 32+ decoder layers (was the main cause of 33 GB RAM on 8B).
+ gc.collect()
+ if torch.cuda.is_available():
+ torch.cuda.empty_cache()
+
+
+# ---------------------------------------------------------------------------
+# Unified online hook registration
+# ---------------------------------------------------------------------------
+
+
+def _register_online_hooks(
+ model,
+ mapping: RotationMapping,
+ fp32_had: bool = False,
+ use_fast_had: bool = True,
+ group_size: int = None,
+ had_dict: dict = None,
+ preset: str = None,
+ fuse_online_to_weight: bool = True,
+):
+ """Register online Hadamard pre-forward hooks on ``down_proj`` and ``o_proj``.
+
+ Online hooks must use the **same** Hadamard matrix that was applied to the
+ weight-side counterpart during ``_rotate_weights``. For ``quarot_hadamard``
+ this is always the deterministic ``get_hadK``/``matmul_hadU`` path
+ (``use_fast_had=False``). For ``"random_hadamard"`` it is the random matrix that
+ was generated once and stored in ``had_dict``.
+
+ Args:
+ group_size: ``None`` → full Hadamard hooks (original QuaRot).
+ ``int`` → per-group Hadamard hooks.
+ had_dict: Normalized ``dict[int, Tensor]`` of custom Hadamard matrices.
+ preset: Rotation preset name.
+ Returns:
+ list of hook handles.
+ """
+ config = model.config
+ num_heads = getattr(config, mapping.num_heads_attr)
+ hidden_size = getattr(config, mapping.hidden_size_attr)
+ intermediate_size = getattr(config, mapping.intermediate_size_attr)
+ head_dim = mapping.attn_head_dim or (hidden_size // num_heads)
+
+ is_grouped = group_size is not None and group_size > 0
+
+ # Online hooks always use deterministic (fixed) Hadamard — never fast_had
+ # for quarot_hadamard; for "random_hadamard" they use the same random matrix
+ # that was cached in had_dict by _rotate_weights.
+ online_fast = False
+
+ # -- Matrix resolution (must match the *online-paired* matrix used by
+ # _rotate_weights for down_proj input / OV pair). Variable name kept in
+ # sync with _rotate_weights to make any future drift obvious.
+ online_had_matrix, _ = _get_custom_had(had_dict, group_size) if is_grouped else (None, False)
+ if preset == "random_hadamard" and online_had_matrix is None:
+ online_had_matrix = get_or_create_random_hadamard(group_size if is_grouped else hidden_size)
+ if preset == "quarot_hadamard" and is_grouped:
+ online_had_matrix = None
+
+ # -- Helper: look up cached random matrix for online-paired hooks --
+ def _online_had(dim):
+ if preset == "random_hadamard":
+ return get_or_create_random_hadamard(dim)
+ return None
+
+ mlp_out_suffix = mapping.mlp_out.split(".")[-1]
+ attn_o_suffix = mapping.attn_o.split(".")[-1]
+
+ # Suffixes for Q/K/V and gate/up (for online input Had hooks)
+ attn_qkv_suffixes = set(attr.split(".")[-1] for attr in (mapping.attn_q, mapping.attn_k, mapping.attn_v))
+ mlp_in_suffixes = set(attr.split(".")[-1] for attr in mapping.mlp_in)
+
+ # --- Build hook factories ---
+ def _make_down_proj_hook():
+ if is_grouped:
+ return GroupOnlineHadamardHook(
+ group_size=group_size, fp32_had=fp32_had, use_fast_had=online_fast, had_matrix=online_had_matrix
+ )
+ online_mat = _online_had(intermediate_size)
+ if online_mat is not None:
+ return FullOnlineHadamardHook(
+ had_K=None, K=None, fp32_had=fp32_had, use_fast_had=online_fast, had_matrix=online_mat
+ )
+ had_K, K = get_hadK(intermediate_size)
+ return FullOnlineHadamardHook(had_K=had_K, K=K, fp32_had=fp32_had, use_fast_had=online_fast)
+
+ def _make_hidden_had_hook():
+ """Full Had hook on hidden_size (for Q/K/V and gate/up input)."""
+ if is_grouped:
+ return GroupOnlineHadamardHook(
+ group_size=group_size, fp32_had=fp32_had, use_fast_had=online_fast, had_matrix=online_had_matrix
+ )
+ online_mat = _online_had(hidden_size)
+ if online_mat is not None:
+ return FullOnlineHadamardHook(
+ had_K=None, K=None, fp32_had=fp32_had, use_fast_had=online_fast, had_matrix=online_mat
+ )
+ had_K, K = get_hadK(hidden_size)
+ return FullOnlineHadamardHook(had_K=had_K, K=K, fp32_had=fp32_had, use_fast_had=online_fast)
+
+ def _make_o_proj_hook():
+ online_mat = _online_had(num_heads)
+ if online_mat is not None:
+ return CrossHeadOnlineHadamardHook(
+ had_K=None,
+ K=None,
+ head_dim=head_dim,
+ fp32_had=fp32_had,
+ use_fast_had=online_fast,
+ had_matrix=online_mat,
+ )
+ had_K, K = get_hadK(num_heads)
+ return CrossHeadOnlineHadamardHook(
+ had_K=had_K,
+ K=K,
+ head_dim=head_dim,
+ fp32_had=fp32_had,
+ use_fast_had=online_fast,
+ )
+
+ # --- Register ---
+ handles = []
+
+ for name, module in model.named_modules():
+ if not isinstance(module, torch.nn.Linear):
+ continue
+ suffix = name.split(".")[-1]
+
+ if name.endswith(mlp_out_suffix):
+ # down_proj: full Had on intermediate_size input
+ h = module.register_forward_pre_hook(_make_down_proj_hook())
+ handles.append(h)
+ elif name.endswith(attn_o_suffix):
+ if fuse_online_to_weight and not is_grouped:
+ # o_proj: cross-head Had on input (fused mode, full only)
+ h = module.register_forward_pre_hook(_make_o_proj_hook())
+ handles.append(h)
+ elif not fuse_online_to_weight:
+ # o_proj: full Had on hidden_size input (unfused mode, matches weight rotation)
+ h = module.register_forward_pre_hook(_make_hidden_had_hook())
+ handles.append(h)
+ elif suffix in attn_qkv_suffixes:
+ if not fuse_online_to_weight:
+ # Q/K/V: full Had on hidden_size input (unfused mode only).
+ # In fused mode Q/K/V only have residual Q on weight (no online Had),
+ # and activations come pre-rotated from residual stream → no hook needed.
+ h = module.register_forward_pre_hook(_make_hidden_had_hook())
+ handles.append(h)
+ elif suffix in mlp_in_suffixes:
+ if not fuse_online_to_weight:
+ # gate/up: full Had on hidden_size input (unfused mode only).
+ # Same reasoning as Q/K/V above.
+ h = module.register_forward_pre_hook(_make_hidden_had_hook())
+ handles.append(h)
+
+ return handles
+
+
+# ---------------------------------------------------------------------------
+# Public API
+# ---------------------------------------------------------------------------
+
+
+def apply_rotation_transform(
+ model,
+ group_size: int = None,
+ allow_online_rotation: bool = True,
+ rotation_matrix: Union[str, torch.Tensor, Dict[int, torch.Tensor], None] = None,
+ compute_device: torch.device | str = None,
+ fp32_had: bool = False,
+ fuse_online_to_weight: bool = None,
+):
+ """Fuse layer norms, rotate weights, and register online Hadamard hooks.
+
+ This is the single entry point for applying Hadamard inplace rotation.
+ The model architecture is auto-detected via ``model.config.model_type``.
+
+ Args:
+ model: A HuggingFace CausalLM model (LLaMA-2/3, Qwen-3, etc.).
+ fp32_had: Whether to compute the online Hadamard transform in fp32.
+ group_size: If ``None`` (default), use full-dimension Hadamard rotation.
+ compute_device: Device to run Hadamard computation on.
+ allow_online_rotation: If ``True`` (default), apply online Hadamard
+ rotations on ``down_proj`` input and the OV pair.
+ rotation_matrix: Rotation matrix selection (``"hadamard"``,
+ ``"random_hadamard"``, ``"quarot_hadamard"``, Tensor, dict, or None).
+ fuse_online_to_weight: If ``True`` (default), fuse online Hadamard
+ rotation into weights (down_proj input, v_proj output, o_proj input)
+ and register compensating online hooks. If ``False``, skip
+ embedding/lm_head rotation; each linear layer is self-contained
+ with input-side Had on weight + compensating online hook on
+ activation. No v_proj cross-head or inner-head rotation.
+
+ Returns:
+ list of hook handles."""
+ if fuse_online_to_weight is None:
+ if model.config.model_type in MAPPING_REGISTRY or model.__class__.__name__ in MAPPING_REGISTRY:
+ fuse_online_to_weight = True
+ else:
+ fuse_online_to_weight = False
+ had_dict, use_fast_had, preset = _normalize_rotation_matrix(rotation_matrix, group_size)
+ compute_device = _resolve_compute_device(compute_device)
+
+ if use_fast_had:
+ from auto_round.utils import logger
+
+ try:
+ import fast_hadamard_transform # noqa: F401
+
+ if group_size is None:
+ logger.warning(
+ "fast_hadamard_transform uses a different Hadamard matrix than the "
+ "default implementation. Please ensure consistency between training "
+ "and inference. This will be refined later."
+ )
+ except ImportError:
+ logger.warning("Importing fast_hadamard_transform failed, falling back to default implementation.")
+ use_fast_had = False
+
+ mapping = infer_mapping_from_model(model)
+
+ _untie_word_embeddings(model, mapping)
+
+ if _uses_layernorm_with_mean(model, mapping):
+ _subtract_embedding_mean(model, mapping)
+
+ _fuse_layer_norms(model, mapping)
+
+ if _uses_layernorm_with_mean(model, mapping):
+ layers = _resolve(model, mapping.layers_attr)
+ for layer in layers:
+ _bake_mean_into_linear(_resolve(layer, mapping.attn_o))
+ _bake_mean_into_linear(_resolve(layer, mapping.mlp_out))
+ _replace_layernorms_with_rmsnorm(model)
+
+ _rotate_weights(
+ model,
+ mapping,
+ use_fast_had=use_fast_had,
+ group_size=group_size,
+ compute_device=compute_device,
+ had_dict=had_dict,
+ preset=preset,
+ fuse_online_to_weight=fuse_online_to_weight,
+ )
+
+ handles = []
+ if fuse_online_to_weight or allow_online_rotation:
+ handles = _register_online_hooks(
+ model,
+ mapping,
+ fp32_had=fp32_had,
+ use_fast_had=use_fast_had,
+ group_size=group_size,
+ had_dict=had_dict,
+ preset=preset,
+ fuse_online_to_weight=fuse_online_to_weight,
+ )
+
+ return model, handles
+
+
+# ---------------------------------------------------------------------------
+# Quick smoke test
+# ---------------------------------------------------------------------------
+
+if __name__ == "__main__":
+ from transformers import AutoModelForCausalLM, AutoTokenizer
+
+ model_name = "/models/opt-125m"
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
+ model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto")
+ model.to("cuda")
+ text = "There is a girl who likes adventure,"
+ inputs = tokenizer(text, return_tensors="pt").to(model.device)
+ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
+
+ apply_rotation_transform(
+ model, group_size=-1, allow_online_rotation=True, rotation_matrix="random_hadamard", fuse_online_to_weight=False
+ )
+ model.to("cuda")
+ text = "There is a girl who likes adventure,"
+ inputs = tokenizer(text, return_tensors="pt").to(model.device)
+ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
+
+ model_name = "/models/Qwen3-8B"
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
+ model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto")
+ apply_rotation_transform(model, group_size=-1, allow_online_rotation=True, fuse_online_to_weight=True)
+ model.to("cuda")
+ text = "There is a girl who likes adventure,"
+ inputs = tokenizer(text, return_tensors="pt").to(model.device)
+ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
+
+ from transformers import AutoModelForCausalLM, AutoTokenizer
+
+ model_name = "/models/Meta-Llama-3.1-8B-Instruct"
+ tokenizer = AutoTokenizer.from_pretrained(model_name)
+ model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto")
+ apply_rotation_transform(model, fuse_online_to_weight=True, group_size=32)
+ model.to("cuda")
+ text = "There is a girl who likes adventure,"
+ inputs = tokenizer(text, return_tensors="pt").to(model.device)
+ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
+ #
+ # model_name = "/models/Llama-2-7b-chat-hf"
+ # tokenizer = AutoTokenizer.from_pretrained(model_name)
+ # model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto")
+ # apply_hadamard_rotation(model)
+ # model.to("cuda")
+ # text = "There is a girl who likes adventure,"
+ # inputs = tokenizer(text, return_tensors="pt").to(model.device)
+ # print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
diff --git a/auto_round/experimental/rotation_inplace/model_config.py b/auto_round/experimental/rotation_inplace/model_config.py
new file mode 100644
index 000000000..3ecbf9b69
--- /dev/null
+++ b/auto_round/experimental/rotation_inplace/model_config.py
@@ -0,0 +1,169 @@
+# # Copyright (C) 2026 Intel Corporation
+# # SPDX-License-Identifier: Apache-2.0
+
+"""Model architecture mapping for Hadamard rotation.
+
+Each :class:`RotationMapping` describes *where* the rotation-relevant modules
+live inside a model. Currently supports LLaMA-2, LLaMA-3, and Qwen-3 (dense).
+
+New architectures can be supported by calling :func:`register_mapping`.
+"""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field
+from typing import Dict, List, Optional
+
+from auto_round.utils import logger
+
+__all__ = [
+ "RotationMapping",
+ "register_mapping",
+ "get_mapping",
+ "infer_mapping_from_model",
+ "MAPPING_REGISTRY",
+]
+
+
+# ---------------------------------------------------------------------------
+# Mapping dataclass
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class RotationMapping:
+ """Declarative description of a transformer architecture for Hadamard rotation.
+
+ Attribute names follow the dot-path convention relative to the model or
+ each decoder layer.
+
+ Config attribute names (read from ``model.config``):
+ num_heads_attr, hidden_size_attr, intermediate_size_attr
+ head_dim_override – explicit head dim (skip hidden_size // num_heads)
+ """
+
+ # -- top-level modules (dot-path from model root) --
+ embedding: str = "model.embed_tokens"
+ lm_head: str = "lm_head"
+ positional_embedding: Optional[str] = None # e.g. "model.decoder.embed_positions" for OPT
+
+ # -- layers container (dot-path from model root) --
+ layers_attr: str = "model.layers"
+
+ # -- per-layer: attention (dot-path from each layer) --
+ attn_input_ln: str = "input_layernorm"
+ attn_q: str = "self_attn.q_proj"
+ attn_k: str = "self_attn.k_proj"
+ attn_v: str = "self_attn.v_proj"
+ attn_o: str = "self_attn.o_proj"
+
+ # -- per-layer: MLP (dot-path from each layer) --
+ mlp_input_ln: str = "post_attention_layernorm"
+ mlp_in: List[str] = field(default_factory=lambda: ["mlp.up_proj", "mlp.gate_proj"])
+ mlp_out: str = "mlp.down_proj"
+
+ # -- final norm (dot-path from model root) --
+ pre_head_ln: str = "model.norm"
+
+ # -- head dim override (None = hidden_size // num_heads) --
+ attn_head_dim: Optional[int] = None
+
+ # -- config attr names --
+ num_heads_attr: str = "num_attention_heads"
+ hidden_size_attr: str = "hidden_size"
+ intermediate_size_attr: str = "intermediate_size"
+
+
+# ---------------------------------------------------------------------------
+# Helper: resolve a dot-path attribute on a module
+# ---------------------------------------------------------------------------
+
+
+def _resolve(root, dot_path: str):
+ """Resolve ``'a.b.c'`` to ``root.a.b.c``."""
+ obj = root
+ for attr in dot_path.split("."):
+ obj = getattr(obj, attr)
+ return obj
+
+
+# ---------------------------------------------------------------------------
+# Registry
+# ---------------------------------------------------------------------------
+
+MAPPING_REGISTRY: Dict[str, RotationMapping] = {}
+
+
+def register_mapping(key: str, mapping: RotationMapping) -> RotationMapping:
+ """Register a :class:`RotationMapping` under *key* (model_type or architecture)."""
+ MAPPING_REGISTRY[key] = mapping
+ return mapping
+
+
+def get_mapping(key: str) -> RotationMapping:
+ """Look up a mapping by *key*; fall back to default if not found."""
+ if key in MAPPING_REGISTRY:
+ return MAPPING_REGISTRY[key]
+ logger.warning(f"No rotation mapping registered for '{key}', " "falling back to default (LLaMA-like) mapping.")
+ return RotationMapping()
+
+
+def infer_mapping_from_model(model) -> RotationMapping:
+ """Return the best :class:`RotationMapping` for *model*.
+
+ Tries ``model.config.model_type`` first, then ``model.__class__.__name__``.
+ """
+ model_type = getattr(getattr(model, "config", None), "model_type", "")
+ if model_type in MAPPING_REGISTRY:
+ return MAPPING_REGISTRY[model_type]
+
+ arch = model.__class__.__name__
+ if arch in MAPPING_REGISTRY:
+ return MAPPING_REGISTRY[arch]
+
+ logger.warning(
+ f"Unrecognised architecture '{arch}' (model_type='{model_type}'). "
+ "Falling back to default (LLaMA-like) mapping."
+ )
+ return RotationMapping()
+
+
+# ===================================================================
+# Built-in mappings
+# ===================================================================
+
+# LLaMA-2 / LLaMA-3 / Mistral / Yi — all share the same layout
+_default = RotationMapping()
+
+register_mapping("llama", _default)
+register_mapping("LlamaForCausalLM", _default)
+
+# Qwen-3 dense — identical layout to LLaMA
+register_mapping("qwen3", _default)
+register_mapping("Qwen3ForCausalLM", _default)
+
+# Qwen-2 / Qwen-2.5 dense — identical layout to LLaMA
+register_mapping("qwen2", _default)
+register_mapping("Qwen2ForCausalLM", _default)
+
+# ---- OPT ----
+# OPT uses standard LayerNorm (with bias, subtracts mean),
+# different module names, and tied lm_head ↔ embedding weights.
+_opt = RotationMapping(
+ embedding="model.decoder.embed_tokens",
+ lm_head="lm_head",
+ positional_embedding="model.decoder.embed_positions",
+ layers_attr="model.decoder.layers",
+ attn_input_ln="self_attn_layer_norm",
+ attn_q="self_attn.q_proj",
+ attn_k="self_attn.k_proj",
+ attn_v="self_attn.v_proj",
+ attn_o="self_attn.out_proj",
+ mlp_input_ln="final_layer_norm",
+ mlp_in=["fc1"],
+ mlp_out="fc2",
+ pre_head_ln="model.decoder.final_layer_norm",
+ intermediate_size_attr="ffn_dim",
+)
+register_mapping("opt", _opt)
+register_mapping("OPTForCausalLM", _opt)
diff --git a/auto_round/experimental/rotation_inplace/utils.py b/auto_round/experimental/rotation_inplace/utils.py
new file mode 100644
index 000000000..04bb18981
--- /dev/null
+++ b/auto_round/experimental/rotation_inplace/utils.py
@@ -0,0 +1,786 @@
+# # Copyright (C) 2026 Intel Corporation
+# # SPDX-License-Identifier: Apache-2.0
+
+"""Online Hadamard transform hooks.
+
+After weight rotation, down_proj and o_proj require an online Hadamard
+transform on their *input activations* at inference time. This module
+provides the hooks and a helper to register them on the model.
+"""
+
+import math
+
+import torch
+import torch.nn as nn
+
+try:
+ import fast_hadamard_transform
+except ImportError:
+ fast_hadamard_transform = None
+
+
+def _resolve_compute_device(compute_device) -> torch.device:
+ """Return *compute_device* if explicitly given, otherwise auto-detect GPU.
+
+ When ``compute_device`` is ``None`` the function checks for CUDA / XPU
+ availability and returns the first accelerator it finds so that heavy
+ matrix operations are offloaded to GPU even when the model weights live
+ on CPU. Falls back to ``torch.device("cpu")`` when no accelerator is
+ present.
+ """
+ if compute_device is not None:
+ return torch.device(compute_device) if not isinstance(compute_device, torch.device) else compute_device
+ if torch.cuda.is_available():
+ return torch.device("cuda:0")
+ if hasattr(torch, "xpu") and torch.xpu.is_available():
+ return torch.device("xpu:0")
+ return torch.device("cpu")
+
+
+BUILTIN_ROTATION_PRESETS = {"quarot_hadamard", "hadamard", "random_hadamard"}
+
+# Global cache for random Hadamard matrices keyed by dimension.
+# Ensures the same shape always returns the exact same random matrix within
+# a process, across all calls to ``_rotate_weights`` / ``_register_online_hooks``.
+_RANDOM_HADAMARD_CACHE: dict = {}
+
+
+def get_or_create_random_hadamard(dim: int, device=None) -> torch.Tensor:
+ """Return a random Hadamard matrix for *dim*, creating and caching it if needed.
+
+ The matrix is cached globally in ``_RANDOM_HADAMARD_CACHE`` so that every
+ caller that requests the same *dim* receives the identical matrix.
+ """
+ if dim in _RANDOM_HADAMARD_CACHE:
+ mat = _RANDOM_HADAMARD_CACHE[dim]
+ if device is not None:
+ mat = mat.to(device)
+ return mat
+ mat = random_hadamard_matrix(dim, device or torch.device("cpu"))
+ _RANDOM_HADAMARD_CACHE[dim] = mat
+ return mat
+
+
+def clear_random_hadamard_cache():
+ """Clear the global random Hadamard matrix cache.
+
+ Call this when you want subsequent ``random_hadamard`` preset runs to
+ generate fresh random matrices (e.g. between independent experiments).
+ """
+ _RANDOM_HADAMARD_CACHE.clear()
+
+
+def _normalize_rotation_matrix(rotation_matrix, group_size):
+ """Normalize ``rotation_matrix`` into a ``(had_dict, use_fast_had, preset)`` tuple.
+
+ Accepted inputs:
+ * ``None`` → ``(None, False, None)`` — use built-in butterfly ``matmul_hadU``.
+ * ``"quarot_hadamard"`` → ``(None, True, "quarot_hadamard")`` — fusable
+ rotations use ``fast_hadamard_transform`` (random); non-fusable
+ (online-paired) rotations use deterministic ``get_hadK``/``matmul_hadU``.
+ * ``"hadamard"`` → ``(None, False, "hadamard")`` — all rotations use
+ deterministic ``get_hadK``/``matmul_hadU``.
+ * ``"random_hadamard"`` → ``(None, False, "random_hadamard")`` — all rotations use
+ ``random_hadamard_matrix``.
+ * A ``torch.Tensor`` of shape ``(n, n)`` → ``({n: tensor}, False, None)``.
+ * A ``dict[int, Tensor]`` → ``(dict, False, None)`` — returned as-is.
+
+ Returns:
+ ``(had_dict, use_fast_had, preset)``
+
+ Raises:
+ ValueError: if a non-``str`` *rotation_matrix* is given but
+ *group_size* is not a positive integer, or an unknown preset.
+ """
+ if rotation_matrix is None:
+ return None, False, None
+
+ if isinstance(rotation_matrix, str):
+ if rotation_matrix not in BUILTIN_ROTATION_PRESETS:
+ raise ValueError(
+ f"Unknown rotation_matrix preset '{rotation_matrix}'. "
+ f"Supported presets: {BUILTIN_ROTATION_PRESETS}."
+ )
+ if rotation_matrix == "quarot_hadamard":
+ return None, True, "quarot_hadamard"
+ elif rotation_matrix == "hadamard":
+ return None, False, "hadamard"
+ else: # "random_hadamard"
+ return None, False, "random_hadamard"
+
+ is_grouped = group_size is not None and group_size > 0
+ if not is_grouped and not isinstance(rotation_matrix, dict):
+ raise ValueError(
+ "rotation_matrix (Tensor/dict) can only be used with a positive group_size. "
+ f"Got group_size={group_size}."
+ )
+
+ if isinstance(rotation_matrix, torch.Tensor):
+ assert (
+ rotation_matrix.ndim == 2 and rotation_matrix.shape[0] == rotation_matrix.shape[1]
+ ), f"rotation_matrix must be square, got shape {rotation_matrix.shape}"
+ return {rotation_matrix.shape[0]: rotation_matrix}, False, None
+
+ if isinstance(rotation_matrix, dict):
+ for k, t in rotation_matrix.items():
+ assert (
+ isinstance(t, torch.Tensor) and t.ndim == 2 and t.shape[0] == t.shape[1]
+ ), f"rotation_matrix[{k}] must be a square tensor, got shape {t.shape}"
+ return rotation_matrix, False, None
+
+ raise TypeError(
+ f"rotation_matrix must be a Tensor, dict[int, Tensor], str, or None. " f"Got {type(rotation_matrix)}."
+ )
+
+
+def _get_custom_had(had_dict, size):
+ """Look up a custom Hadamard matrix for *size* from the normalized dict.
+
+ Returns ``(had_tensor, True)`` if found, ``(None, False)`` otherwise.
+ """
+ if had_dict is None:
+ return None, False
+ if size in had_dict:
+ return had_dict[size], True
+ return None, False
+
+
+# ---------------------------------------------------------------------------
+# Hook implementations
+# ---------------------------------------------------------------------------
+
+
+class FullOnlineHadamardHook(nn.Module):
+ """Pre-forward hook: full Hadamard on the entire last dimension (for ``down_proj``)."""
+
+ def __init__(self, had_K, K, fp32_had=False, use_fast_had=True, had_matrix=None):
+ super().__init__()
+ self.custom_had = had_matrix is not None
+ if had_matrix is not None:
+ self.register_buffer("had_matrix", had_matrix)
+ self.had_K = None
+ self.K = None
+ else:
+ if had_K is not None:
+ self.register_buffer("had_K", had_K)
+ else:
+ self.had_K = None
+ self.K = K
+ self.fp32_had = fp32_had
+ self.use_fast_had = use_fast_had
+
+ def __call__(self, module: nn.Module, args):
+ x = args[0] if isinstance(args, tuple) else args
+ x_dtype = x.dtype
+
+ if self.custom_had:
+ H = self.had_matrix.to(device=x.device, dtype=x.dtype)
+ if self.fp32_had:
+ H = self.had_matrix.to(device=x.device).float()
+ x = (x.float() @ H.T).to(x_dtype)
+ else:
+ x = x @ H.T
+ elif self.fp32_had:
+ x = matmul_hadU_cuda(x.float(), self.had_K, self.K, use_fast_had=self.use_fast_had).to(x_dtype)
+ else:
+ x = matmul_hadU_cuda(x, self.had_K, self.K, use_fast_had=self.use_fast_had)
+
+ if isinstance(args, tuple):
+ return (x,) + args[1:]
+ return x
+
+
+class CrossHeadOnlineHadamardHook(nn.Module):
+ """Pre-forward hook: **cross-head** Hadamard on the ``num_heads`` dimension
+ (for ``o_proj``).
+
+ After offline rotation:
+ - ``v_proj`` absorbed a per-head (within-head) Hadamard on ``head_dim``.
+ - ``o_proj`` absorbed a full Hadamard on ``hidden_size``.
+
+ Since ``H_full = H_cross ⊗ H_within`` (Kronecker decomposition) and the
+ within-head part is already cancelled by ``v_proj`` through the attention
+ path (``H_within² = I``), the online hook only needs to apply the residual
+ **cross-head** Hadamard (``H_cross ⊗ I``):
+
+ * reshape ``(*, hidden_size)`` → ``(*, num_heads, head_dim)``
+ * transpose → ``(*, head_dim, num_heads)``
+ * Hadamard on the **num_heads** axis (last dim)
+ * transpose back and reshape
+ """
+
+ def __init__(self, had_K, K, head_dim, fp32_had=False, use_fast_had=True, had_matrix=None):
+ """
+ Args:
+ had_K: Hadamard sub-matrix from ``get_hadK(num_heads)``.
+ K: Block size from ``get_hadK(num_heads)``.
+ head_dim: ``hidden_size // num_attention_heads``.
+ fp32_had: Compute in fp32.
+ use_fast_had: If True use fast_hadamard_transform; if False use matmul_hadU.
+ had_matrix: Optional custom rotation matrix of shape ``(num_heads, num_heads)``.
+ """
+ super().__init__()
+ self.custom_had = had_matrix is not None
+ if had_matrix is not None:
+ self.register_buffer("had_matrix", had_matrix)
+ self.had_K = None
+ self.K = None
+ else:
+ if had_K is not None:
+ self.register_buffer("had_K", had_K)
+ else:
+ self.had_K = None
+ self.K = K
+ self.had_dim = head_dim
+ self.fp32_had = fp32_had
+ self.use_fast_had = use_fast_had
+
+ def __call__(self, module: nn.Module, args):
+ x = args[0] if isinstance(args, tuple) else args
+ x_dtype = x.dtype
+
+ if self.fp32_had:
+ x = x.float()
+
+ init_shape = x.shape
+ num_heads = init_shape[-1] // self.had_dim
+
+ if self.custom_had:
+ H = self.had_matrix.to(device=x.device, dtype=x.dtype)
+ # reshape (*, hidden) → (*, num_heads, head_dim), transpose → (*, head_dim, num_heads)
+ x = x.reshape(-1, num_heads, self.had_dim).transpose(1, 2)
+ # apply H on last dim (num_heads): x @ H.T
+ x = (x @ H.T).transpose(1, 2)
+ elif self.use_fast_had and fast_hadamard_transform is not None and self.K == 1:
+ x = fast_hadamard_transform.hadamard_transform(
+ x.reshape(-1, num_heads, self.had_dim).transpose(1, 2),
+ scale=1 / math.sqrt(num_heads),
+ ).transpose(1, 2)
+ else:
+ # Fallback: use matmul_hadU (pure butterfly + had_K, no fast_hadamard_transform)
+ x = x.reshape(-1, num_heads, self.had_dim).transpose(1, 2)
+ x = matmul_hadU(x.contiguous())
+ x = x.transpose(1, 2)
+
+ if self.fp32_had:
+ x = x.to(x_dtype)
+ x = x.reshape(init_shape)
+
+ if isinstance(args, tuple):
+ return (x,) + args[1:]
+ return x
+
+
+# ---------------------------------------------------------------------------
+# Registration helper
+# ---------------------------------------------------------------------------
+
+
+def register_online_had_hooks(model, mapping=None, fp32_had=False, use_fast_had=True):
+ """Register online Hadamard pre-forward hooks on ``down_proj`` and ``o_proj``.
+
+ * **down_proj** (``online_full_had``): full Hadamard on ``intermediate_size``.
+ Compensates ``apply_exact_had_to_linear(down_proj, had_dim=-1, output=False)``.
+
+ * **o_proj** (``online cross-head had``): cross-head Hadamard on ``num_heads``.
+ Compensates the residual after v_proj's within-head Hadamard cancels.
+
+ Args:
+ model: A HuggingFace model whose weights have already been rotated.
+ mapping: A :class:`RotationMapping` (auto-inferred if ``None``).
+ fp32_had: Whether to compute the Hadamard transform in fp32.
+ use_fast_had: If True use fast_hadamard_transform; if False use matmul_hadU.
+
+ Returns:
+ list of hook handles (call ``handle.remove()`` to detach).
+ """
+ if mapping is None:
+ from auto_round.experimental.rotation_inplace.model_config import infer_mapping_from_model
+
+ mapping = infer_mapping_from_model(model)
+
+ config = model.config
+ num_heads = getattr(config, mapping.num_heads_attr)
+ hidden_size = getattr(config, mapping.hidden_size_attr)
+ intermediate_size = getattr(config, mapping.intermediate_size_attr)
+ head_dim = mapping.attn_head_dim or (hidden_size // num_heads)
+
+ # down_proj: full Hadamard on intermediate_size
+ had_K_full, K_full = get_hadK(intermediate_size)
+
+ # o_proj: cross-head Hadamard on num_heads
+ had_K_head, K_head = get_hadK(num_heads)
+
+ # Identify target module suffixes from mapping
+ mlp_out_suffix = mapping.mlp_out.split(".")[-1] # e.g. "down_proj"
+ attn_o_suffix = mapping.attn_o.split(".")[-1] # e.g. "o_proj"
+
+ handles = []
+ for name, module in model.named_modules():
+ if name.endswith(mlp_out_suffix) and isinstance(module, nn.Linear):
+ hook = FullOnlineHadamardHook(
+ had_K=had_K_full,
+ K=K_full,
+ fp32_had=fp32_had,
+ use_fast_had=use_fast_had,
+ )
+ h = module.register_forward_pre_hook(hook)
+ handles.append(h)
+ elif name.endswith(attn_o_suffix) and isinstance(module, nn.Linear):
+ hook = CrossHeadOnlineHadamardHook(
+ had_K=had_K_head,
+ K=K_head,
+ head_dim=head_dim,
+ fp32_had=fp32_had,
+ use_fast_had=use_fast_had,
+ )
+ h = module.register_forward_pre_hook(hook)
+ handles.append(h)
+
+ return handles
+
+
+def is_pow2(n):
+ return (n & (n - 1) == 0) and (n > 0)
+
+
+# Adapted from https://github.com/Cornell-RelaxML/quip-sharp/blob/main/lib/utils/matmul_had.py
+def get_hadK(n: int, transpose=False) -> (torch.Tensor, int):
+ hadK, K = None, None
+
+ if is_pow2(n):
+ K = 1
+ return hadK, K
+ else:
+ from auto_round.experimental.transform.utils.hadamard import _fetch_hadamard_divisor
+
+ hadK = _fetch_hadamard_divisor(n, torch.float, torch.device("cpu"))
+ if transpose:
+ hadK = hadK.T
+ if hadK is not None:
+ return hadK, 1 if is_pow2(hadK.shape[0]) else hadK.shape[0]
+ assert is_pow2(n)
+
+
+def matmul_hadU(X, transpose=False):
+ n = X.shape[-1]
+ hadK, K = get_hadK(n, transpose)
+ input = X.clone().view(-1, n, 1)
+ output = input.clone()
+ while input.shape[1] > K:
+ input = input.view(input.shape[0], input.shape[1] // 2, 2, input.shape[2])
+ output = output.view(input.shape)
+ output[:, :, 0, :] = input[:, :, 0, :] + input[:, :, 1, :]
+ output[:, :, 1, :] = input[:, :, 0, :] - input[:, :, 1, :]
+ output = output.view(input.shape[0], input.shape[1], -1)
+ input, output = (output, input)
+ del output
+
+ if K > 1:
+ # Do not explicitly repeat - OOM
+ # input = torch.bmm(
+ # hadK.repeat(len(input), 1, 1).to(input.device).to(input.dtype), input)
+ # Use bcast instead
+ input = hadK.view(1, K, K).to(input) @ input
+
+ return input.view(X.shape) / torch.tensor(n).sqrt()
+
+
+def matmul_hadUt(X):
+ return matmul_hadU(X, transpose=True)
+
+
+def random_hadamard_matrix(size, device):
+ # See https://cornell-relaxml.github.io/quip-sharp/ , Section "Randomized Hadamard Transformation"
+ Q = torch.randint(low=0, high=2, size=(size,)).to(torch.float64)
+ Q = Q * 2 - 1
+ Q = torch.diag(Q)
+ return matmul_hadU(Q).to(device)
+
+
+def deterministic_hadamard_matrix(size, device):
+ """Build a deterministic Hadamard matrix of the given *size*.
+
+ Applies the butterfly ``matmul_hadU`` to an identity matrix so that the
+ result is purely determined by ``get_hadK`` (no random sign flips).
+ """
+ Q = torch.eye(size, dtype=torch.float64)
+ return matmul_hadU(Q).to(device)
+
+
+def matmul_hadU_cuda(X, hadK, K, use_fast_had=True):
+ n = X.shape[-1]
+ if not use_fast_had or fast_hadamard_transform is None:
+ return matmul_hadU(X)
+ if K == 1:
+ return fast_hadamard_transform.hadamard_transform(X.contiguous(), 1.0 / torch.tensor(n).sqrt())
+ # if transpose:
+ # hadK = hadK.T.contiguous()
+ input = X.view(*X.shape[:-1], K, n // K)
+ input = fast_hadamard_transform.hadamard_transform(input.contiguous(), 1.0 / torch.tensor(n).sqrt())
+ input = hadK.to(input.device).to(input.dtype) @ input
+ return input.reshape(X.shape)
+
+
+def matmul_hadUt_cuda(X, hadK, K, use_fast_had=True):
+ return matmul_hadU_cuda(X, hadK, K, use_fast_had=use_fast_had)
+
+
+def apply_exact_had_to_linear(
+ module, had_dim=-1, output=False, use_fast_had=True, compute_device=None, had_matrix=None
+):
+ """Apply Hadamard rotation to a Linear layer's weight in-place.
+
+ Args:
+ module: ``nn.Linear`` layer.
+ had_dim: Dimension of each Hadamard block (``-1`` for full dimension).
+ output: If ``True`` rotate the output (row) side; otherwise input (col).
+ use_fast_had: Use ``fast_hadamard_transform`` when available.
+ compute_device: Device to run computation on.
+ had_matrix: Optional custom rotation matrix. When ``had_dim == -1``
+ this should be a square tensor whose size equals
+ ``out_features`` (output) or ``in_features`` (input). When
+ ``had_dim > 0`` the size should equal ``had_dim``.
+ """
+ assert isinstance(module, torch.nn.Linear)
+ in_features, out_features = module.in_features, module.out_features
+
+ if had_dim != -1 and had_matrix is None:
+ assert is_pow2(had_dim), "Hadamard dimension must be a power of 2!"
+
+ W_ = module.weight.data
+ dtype = W_.dtype
+ dev = W_.device
+ init_shape = W_.shape
+ compute_dev = _resolve_compute_device(compute_device)
+ W_ = W_.double().to(compute_dev)
+
+ if had_matrix is not None:
+ H = had_matrix.to(device=compute_dev, dtype=torch.float64)
+ if had_dim == -1:
+ # Full-dimension custom matrix
+ if output:
+ # W.T = H @ W.T → W = (H @ W.T).T = W @ H.T
+ W_ = W_ @ H.T
+ else:
+ # W = H @ W (rotate input columns: W_new[i,:] = sum H[i,j]*W[j,:])
+ # Actually for input side: W_new = W @ H (each row is rotated)
+ W_ = W_ @ H.T
+ else:
+ # Per-block custom matrix
+ if output:
+ W_ = W_.t()
+ transposed_shape = W_.shape
+ flat = W_.reshape(-1, had_dim)
+ W_ = (flat @ H.T).reshape(transposed_shape).t()
+ else:
+ flat = W_.reshape(-1, had_dim)
+ W_ = (flat @ H.T).reshape(init_shape)
+ elif had_dim == -1:
+ if output:
+ had_K, K = get_hadK(out_features)
+ W_ = matmul_hadU_cuda(W_.t(), had_K, K, use_fast_had=use_fast_had).t()
+ if not output:
+ had_K, K = get_hadK(in_features)
+ W_ = matmul_hadU_cuda(W_, had_K, K, use_fast_had=use_fast_had)
+ else:
+ # Apply Hadamard to the last had_dim chunks of the weights
+ if output:
+ W_ = W_.t()
+ transposed_shape = W_.shape
+ if use_fast_had and fast_hadamard_transform is not None:
+ W_ = (
+ fast_hadamard_transform.hadamard_transform(
+ W_.reshape(-1, transposed_shape[-1] // had_dim, had_dim), scale=1 / math.sqrt(had_dim)
+ )
+ .reshape(transposed_shape)
+ .t()
+ )
+ else:
+ W_ = matmul_hadU(W_.reshape(-1, had_dim)).reshape(transposed_shape).t()
+ else:
+ if use_fast_had and fast_hadamard_transform is not None:
+ n = W_.shape[1]
+ W_ = fast_hadamard_transform.hadamard_transform(
+ W_.reshape(-1, n // had_dim, had_dim), scale=1 / math.sqrt(had_dim)
+ ).reshape(init_shape)
+ else:
+ W_ = matmul_hadU(W_.reshape(-1, had_dim)).reshape(init_shape)
+ module.weight.data = W_.to(device=dev, dtype=dtype)
+
+
+def apply_cross_head_had_to_linear(
+ module, num_heads, head_dim, use_fast_had=True, compute_device=None, had_matrix=None
+):
+ """Apply a cross-head Hadamard rotation to a Linear layer's input side.
+
+ The operation is equivalent to ``(H_cross ⊗ I_head_dim)`` applied to the
+ input columns:
+
+ * Reshape columns ``(hidden_size,)`` → ``(num_heads, head_dim)``
+ * Transpose → ``(head_dim, num_heads)``
+ * Hadamard on the ``num_heads`` axis
+ * Transpose back and reshape
+
+ This mirrors what :class:`CrossHeadOnlineHadamardHook` does at runtime.
+
+ Args:
+ module: ``nn.Linear`` layer whose ``in_features == num_heads * head_dim``.
+ num_heads: Number of attention heads.
+ head_dim: Per-head dimension.
+ use_fast_had: Use ``fast_hadamard_transform`` when available.
+ compute_device: Device to run computation on.
+ had_matrix: Optional custom rotation matrix of shape ``(num_heads, num_heads)``.
+ """
+ assert isinstance(module, torch.nn.Linear)
+ W_ = module.weight.data
+ dtype = W_.dtype
+ dev = W_.device
+ compute_dev = _resolve_compute_device(compute_device)
+ W_ = W_.double().to(compute_dev)
+
+ out_f = W_.shape[0]
+ # W shape: (out_features, hidden_size) where hidden_size = num_heads * head_dim
+ # Reshape columns: (out_f, num_heads, head_dim)
+ W_ = W_.reshape(out_f, num_heads, head_dim)
+ # Transpose last two dims: (out_f, head_dim, num_heads)
+ W_ = W_.transpose(1, 2).contiguous()
+
+ if had_matrix is not None:
+ H = had_matrix.to(device=compute_dev, dtype=torch.float64)
+ # Apply H on last dim (num_heads): flat @ H.T
+ flat = W_.reshape(-1, num_heads)
+ W_ = (flat @ H.T).reshape(out_f, head_dim, num_heads)
+ elif use_fast_had and fast_hadamard_transform is not None and is_pow2(num_heads):
+ W_ = fast_hadamard_transform.hadamard_transform(W_, scale=1.0 / math.sqrt(num_heads))
+ else:
+ W_ = matmul_hadU(W_.reshape(-1, num_heads)).reshape(out_f, head_dim, num_heads)
+
+ # Transpose back: (out_f, num_heads, head_dim) → (out_f, hidden_size)
+ W_ = W_.transpose(1, 2).contiguous().reshape(out_f, num_heads * head_dim)
+ module.weight.data = W_.to(device=dev, dtype=dtype)
+
+
+# ---------------------------------------------------------------------------
+# Grouped (block-diagonal) Hadamard utilities
+# ---------------------------------------------------------------------------
+
+
+class OnlineHadamardPostHook(nn.Module):
+ """Forward hook (post-hook) adapter: wraps a pre-hook-style Hadamard
+ transform to apply it on the layer's **output** instead of input.
+
+ Used for v_proj per-head Hadamard on the output side when online
+ rotation is not fused into weights.
+ """
+
+ def __init__(self, pre_hook):
+ super().__init__()
+ self.pre_hook = pre_hook
+
+ def __call__(self, module, input, output):
+ result = self.pre_hook(module, (output,))
+ if isinstance(result, tuple):
+ return result[0]
+ return result
+
+
+class GroupOnlineHadamardHook(nn.Module):
+ """Pre-forward hook: block-diagonal Hadamard with fixed ``group_size`` on last dim.
+
+ Reshapes ``(*, D)`` → ``(*, D // group_size, group_size)``, applies Hadamard
+ per group, then reshapes back. Much cheaper than a full-dimension Hadamard.
+ """
+
+ def __init__(self, group_size, fp32_had=False, use_fast_had=True, had_matrix=None):
+ super().__init__()
+ self.group_size = group_size
+ self.fp32_had = fp32_had
+ self.use_fast_had = use_fast_had
+ self.custom_had = had_matrix is not None
+
+ if had_matrix is not None:
+ self.register_buffer("had_matrix", had_matrix)
+ self.had_K = None
+ self.K = None
+ elif not is_pow2(group_size):
+ had_K, K = get_hadK(group_size)
+ if had_K is not None:
+ self.register_buffer("had_K", had_K)
+ else:
+ self.had_K = None
+ self.K = K
+ else:
+ self.had_K = None
+ self.K = 1
+
+ def __call__(self, module: nn.Module, args):
+ x = args[0] if isinstance(args, tuple) else args
+ x_dtype = x.dtype
+ init_shape = x.shape
+ gs = self.group_size
+
+ if self.fp32_had:
+ x = x.float()
+
+ # Reshape: (*, D) → (*, D//gs, gs)
+ x = x.reshape(*init_shape[:-1], init_shape[-1] // gs, gs)
+
+ if self.custom_had:
+ H = self.had_matrix.to(device=x.device, dtype=x.dtype)
+ flat = x.reshape(-1, gs)
+ x = (flat @ H.T).reshape(*init_shape[:-1], init_shape[-1] // gs, gs)
+ elif self.use_fast_had and fast_hadamard_transform is not None and self.K == 1:
+ x = fast_hadamard_transform.hadamard_transform(x, scale=1.0 / math.sqrt(gs))
+ else:
+ x = x.reshape(-1, gs)
+ x = matmul_hadU(x)
+ x = x.reshape(*init_shape[:-1], init_shape[-1] // gs, gs)
+
+ x = x.reshape(init_shape)
+
+ if self.fp32_had:
+ x = x.to(x_dtype)
+
+ if isinstance(args, tuple):
+ return (x,) + args[1:]
+ return x
+
+
+def _apply_grouped_had_to_weight(W, group_size, side="input", use_fast_had=True, had_matrix=None):
+ """Apply block-diagonal Hadamard to a weight matrix.
+
+ Args:
+ W: Weight tensor, shape (out_features, in_features).
+ group_size: Block size for the Hadamard rotation.
+ side: ``'input'`` rotates columns (in_features dim),
+ ``'output'`` rotates rows (out_features dim).
+ use_fast_had: Use fast_hadamard_transform if available.
+ had_matrix: Optional custom Hadamard matrix of shape ``(gs, gs)``
+ to use instead of the built-in Hadamard.
+
+ Returns:
+ Rotated weight tensor.
+ """
+ gs = group_size
+ dtype = W.dtype
+ W = W.double()
+
+ def _had_on_last_dim(X):
+ """Apply Hadamard on the last dimension (size gs) of X shaped (..., gs)."""
+ if had_matrix is not None:
+ H = had_matrix.to(device=X.device, dtype=X.dtype)
+ # X: (..., gs) → batch matmul with H^T → X @ H^T
+ flat = X.reshape(-1, gs)
+ return (flat @ H.T).reshape(X.shape)
+ if use_fast_had and fast_hadamard_transform is not None and is_pow2(gs):
+ return fast_hadamard_transform.hadamard_transform(X, scale=1.0 / math.sqrt(gs))
+ orig_shape = X.shape
+ return matmul_hadU(X.reshape(-1, gs)).reshape(orig_shape)
+
+ if side == "input":
+ out_f, in_f = W.shape
+ W = W.reshape(out_f, in_f // gs, gs)
+ W = _had_on_last_dim(W)
+ W = W.reshape(out_f, in_f)
+ else:
+ out_f, in_f = W.shape
+ Wt = W.t().contiguous()
+ Wt = Wt.reshape(in_f, out_f // gs, gs)
+ Wt = _had_on_last_dim(Wt)
+ W = Wt.reshape(in_f, out_f).t().contiguous()
+
+ return W.to(dtype)
+
+
+def _rotate_linear_grouped(module, group_size, side="input", use_fast_had=True, compute_device=None, had_matrix=None):
+ """Apply block-diagonal Hadamard rotation to a Linear layer's weight.
+
+ Args:
+ module: ``nn.Linear`` layer.
+ group_size: Block size.
+ side: ``'input'`` or ``'output'``.
+ use_fast_had: Use fast_hadamard_transform.
+ compute_device: Device to run computation on. If None, auto-detects GPU.
+ had_matrix: Optional custom Hadamard matrix of shape ``(gs, gs)``.
+ """
+ dtype = module.weight.data.dtype
+ dev = module.weight.data.device
+ compute_dev = _resolve_compute_device(compute_device)
+ W = module.weight.data.to(device=compute_dev, dtype=torch.float64)
+ W = _apply_grouped_had_to_weight(W, group_size, side=side, use_fast_had=use_fast_had, had_matrix=had_matrix)
+ module.weight.data = W.to(device=dev, dtype=dtype)
+
+ if side == "output" and module.bias is not None:
+ bias = module.bias.data.to(device=compute_dev, dtype=torch.float64)
+ gs = group_size
+ bias = bias.reshape(-1, gs)
+ if had_matrix is not None:
+ H = had_matrix.to(device=compute_dev, dtype=torch.float64)
+ bias = (bias @ H.T).reshape(-1)
+ elif use_fast_had and fast_hadamard_transform is not None and is_pow2(gs):
+ bias = (
+ fast_hadamard_transform.hadamard_transform(bias.unsqueeze(0), scale=1.0 / math.sqrt(gs))
+ .squeeze(0)
+ .reshape(-1)
+ )
+ else:
+ bias = matmul_hadU(bias).reshape(-1)
+ module.bias.data = bias.to(device=dev, dtype=dtype)
+
+
+def _rotate_embedding_grouped(embedding, group_size, use_fast_had=True, compute_device=None, had_matrix=None):
+ """Apply block-diagonal Hadamard rotation to an Embedding layer.
+
+ Embedding weight: (vocab, hidden_size) → rotate on hidden_size (columns).
+ """
+ dtype = embedding.weight.data.dtype
+ dev = embedding.weight.data.device
+ compute_dev = _resolve_compute_device(compute_device)
+ W = embedding.weight.data.to(device=compute_dev, dtype=torch.float64)
+ W = _apply_grouped_had_to_weight(W, group_size, side="input", use_fast_had=use_fast_had, had_matrix=had_matrix)
+ new_W = W.to(device=dev, dtype=dtype)
+ del W
+ embedding.weight.data = new_W
+
+
+def register_online_had_hooks_grouped(model, mapping, group_size, fp32_had=False, use_fast_had=True):
+ """Register per-group online Hadamard hooks on ``down_proj`` and ``o_proj``.
+
+ In grouped mode:
+ - **down_proj**: block-diagonal Hadamard on ``intermediate_size`` with ``group_size``.
+ - **o_proj**: block-diagonal Hadamard on ``hidden_size`` with ``group_size``.
+
+ Args:
+ model: HuggingFace model with rotated weights.
+ mapping: RotationMapping.
+ group_size: Block size for block-diagonal Hadamard.
+ fp32_had: Compute in fp32.
+ use_fast_had: Use fast_hadamard_transform.
+
+ Returns:
+ list of hook handles.
+ """
+ mlp_out_suffix = mapping.mlp_out.split(".")[-1]
+ attn_o_suffix = mapping.attn_o.split(".")[-1]
+
+ handles = []
+ for name, module in model.named_modules():
+ if name.endswith(mlp_out_suffix) and isinstance(module, nn.Linear):
+ hook = GroupOnlineHadamardHook(
+ group_size=group_size,
+ fp32_had=fp32_had,
+ use_fast_had=use_fast_had,
+ )
+ h = module.register_forward_pre_hook(hook)
+ handles.append(h)
+ elif name.endswith(attn_o_suffix) and isinstance(module, nn.Linear):
+ hook = GroupOnlineHadamardHook(
+ group_size=group_size,
+ fp32_had=fp32_had,
+ use_fast_had=use_fast_had,
+ )
+ h = module.register_forward_pre_hook(hook)
+ handles.append(h)
+
+ return handles
diff --git a/auto_round/experimental/transform/apply.py b/auto_round/experimental/transform/apply.py
index 4ae74f5fb..aafd4c1b0 100644
--- a/auto_round/experimental/transform/apply.py
+++ b/auto_round/experimental/transform/apply.py
@@ -5,16 +5,16 @@ import torch
import tqdm
from auto_round.experimental.qmodules.base import QModuleBase
-from auto_round.experimental.transform.hadamard_config import HadamardConfig
from auto_round.experimental.transform.hadamards import build_hadamard_transform
-from auto_round.experimental.utils import is_triton_kernel_available, normalize_hadamard_config
+from auto_round.experimental.transform.rotation_config import RotationConfig
+from auto_round.experimental.utils import is_triton_kernel_available, normalize_rotation_config
-__all__ = ["apply_hadamard_transform"]
+__all__ = ["apply_rotation_transform"]
-def apply_hadamard_transform(
+def apply_rotation_transform(
model: torch.nn.Module,
- config: str | dict | HadamardConfig | None,
+ config: str | dict | RotationConfig | None,
location: str = "weight",
use_tqdm=True,
desc=None,
@@ -31,13 +31,13 @@ def apply_hadamard_transform(
* ``str``: A named/preset transform configuration. In this case,
resolved to a concrete quantization/transform configuration.
* ``dict``: A raw configuration mapping that will be normalized
- (via :func:`normalize_hadamard_config`) and then passed to
+ (via :func:`normalize_rotation_config`) and then passed to
:class:`TransformConfig`.
* :class:`TransformConfig`: An existing configuration instance.
This will be used to construct the final configuration after
normalization.
* ``None``: Uses the default behavior of
- :func:`_normalize_hadamard_config` (for example, inferring a
+ :func:`_normalize_rotation_config` (for example, inferring a
configuration from ``data_type`` or other project defaults), if
supported.
:param data_type: quantization data type.
@@ -48,9 +48,9 @@ def apply_hadamard_transform(
``config.transform_type``.
"""
- config = normalize_hadamard_config(config, data_type)
- if not isinstance(config, HadamardConfig):
- config = HadamardConfig(**config)
+ config = normalize_rotation_config(config, data_type)
+ if not isinstance(config, RotationConfig):
+ config = RotationConfig(**config)
modules_config = [
(name, module, config)
@@ -64,16 +64,19 @@ def apply_hadamard_transform(
continue
_apply_to_module(model, module, config, location, data_type)
- # attach config to model for compression/serialization
- setattr(model, "hadamard_config", config)
+ # attach config to model for compression/serialization. Use a plain dict so
+ # that downstream HF `save_pretrained` -> JSON works (RotationConfig is a
+ # pydantic model and is not directly JSON serializable).
+ setattr(model, "rotation_config", config.model_dump() if hasattr(config, "model_dump") else config)
+ hooks = None
- return model
+ return model, hooks
def _apply_to_module(
model: torch.nn.Module,
module: torch.nn.Module,
- config: HadamardConfig,
+ config: RotationConfig,
location: str = "weight",
data_type: str = "mx_fp",
):
diff --git a/auto_round/experimental/transform/hadamard_config.py b/auto_round/experimental/transform/hadamard_config.py
deleted file mode 100644
index b58cce9dc..000000000
--- a/auto_round/experimental/transform/hadamard_config.py
+++ /dev/null
@@ -1,28 +0,0 @@
-# # Copyright (C) 2026 Intel Corporation
-# # SPDX-License-Identifier: Apache-2.0
-
-from pydantic import BaseModel, Field, field_validator
-
-__all__ = ["HadamardConfig"]
-
-
-class HadamardConfig(BaseModel):
- """
- Configuration of transforms to be applied to a model. This config is to be
- serialized within a model's `config.json` file
- """
-
- block_size: int = Field(default=32)
-
- hadamard_type: str = Field(default="hadamard")
-
- # for random hadamard transform
- random_seed: bool = Field(default=False, exclude=True)
-
- @field_validator("hadamard_type")
- @classmethod
- def validate_hadamard_type(cls, v: str) -> str:
- allowed = {"hadamard", "random_hadamard"}
- if v not in allowed:
- raise ValueError(f"Unsupported hadamard_type: {v}. Supported values: {sorted(allowed)}")
- return v
diff --git a/auto_round/experimental/transform/patch_modules.py b/auto_round/experimental/transform/patch_modules.py
index 34cb2f1ce..e0f9adbae 100644
--- a/auto_round/experimental/transform/patch_modules.py
+++ b/auto_round/experimental/transform/patch_modules.py
@@ -51,11 +51,11 @@ def patch_wrapperlinear_to_apply_transform(w_transform, inp_transform):
orig_qdq_act = WrapperLinear._qdq_act
- def _qdq_act_patched(self, x, act_max_scale, act_max=None):
+ def _qdq_act_patched(self, x, act_min_scale, act_max_scale, act_max=None):
x = inp_transform(x)
- return orig_qdq_act(self, x, act_max_scale, act_max)
+ return orig_qdq_act(self, x, act_min_scale, act_max_scale, act_max)
WrapperLinear._qdq_weight = _qdq_weight_patched
WrapperLinear._qdq_act = _qdq_act_patched
diff --git a/auto_round/experimental/transform/rotation_config.py b/auto_round/experimental/transform/rotation_config.py
new file mode 100644
index 000000000..dfcfb5d45
--- /dev/null
+++ b/auto_round/experimental/transform/rotation_config.py
@@ -0,0 +1,61 @@
+# # Copyright (C) 2026 Intel Corporation
+# # SPDX-License-Identifier: Apache-2.0
+
+from typing import Optional
+
+from pydantic import BaseModel, Field, field_validator
+
+__all__ = ["RotationConfig"]
+
+
+class RotationConfig(BaseModel):
+ """
+ Unified configuration for Hadamard rotation/transform applied to a model.
+
+ Two implementation paths are supported:
+
+ * ``backend="inplace"`` -> ``auto_round.experimental.rotation_inplace``
+ QuaRot-style residual-stream / per-layer rotation. Supports any
+ weight/activation dtype (incl. INT4/INT8/FPx). Can optionally fuse
+ the online Hadamard into weights (``fuse_online_to_weight=True``).
+ * ``backend="transform"`` -> ``auto_round.experimental.transform``
+ Per-Linear weight + activation Hadamard with a fused triton kernel.
+ **Only supports MXFP4 / NVFP4** and **cannot fuse online to weight.**
+ * ``backend="auto"`` (default)
+ - If ``fuse_online_to_weight=True`` -> inplace (fused).
+ - Else if ``data_type`` is MX-FP / NV-FP -> transform.
+ - Otherwise -> inplace (unfused).
+
+ Notes:
+ * ``block_size`` is the group/block size for grouped Hadamard.
+ For ``backend="inplace"`` it is forwarded as ``group_size`` (``None``
+ / ``-1`` means full-dimension Hadamard).
+ """
+
+ # ---- shared ----
+ backend: str = Field(default="auto")
+ block_size: Optional[int] = Field(default=None)
+ hadamard_type: str = Field(default="hadamard")
+
+ # ---- inplace-only ----
+ fuse_online_to_weight: Optional[bool] = Field(default=None)
+ allow_online_rotation: bool = Field(default=True)
+
+ # for random hadamard transform (transform path)
+ random_seed: bool = Field(default=False, exclude=True)
+
+ @field_validator("backend")
+ @classmethod
+ def validate_backend(cls, v: str) -> str:
+ allowed = {"auto", "inplace", "transform"}
+ if v not in allowed:
+ raise ValueError(f"Unsupported backend: {v}. Supported values: {sorted(allowed)}")
+ return v
+
+ @field_validator("hadamard_type")
+ @classmethod
+ def validate_hadamard_type(cls, v: str) -> str:
+ allowed = {"hadamard", "random_hadamard", "quarot_hadamard"}
+ if v not in allowed:
+ raise ValueError(f"Unsupported hadamard_type: {v}. Supported values: {sorted(allowed)}")
+ return v
diff --git a/auto_round/experimental/utils.py b/auto_round/experimental/utils.py
index 8527dbe00..ea1dade78 100644
--- a/auto_round/experimental/utils.py
+++ b/auto_round/experimental/utils.py
@@ -17,8 +17,8 @@ from typing import Any
import torch
from auto_round.compressors.utils import is_mx_fp, is_nv_fp
-from auto_round.experimental.transform.hadamard_config import HadamardConfig
from auto_round.experimental.transform.hadamards import HADAMARDS
+from auto_round.experimental.transform.rotation_config import RotationConfig
from auto_round.utils import logger
SUPPORTED_QUANTIZATION_SCHEMES = ["MXFP8", "MXFP4", "NVFP4"]
@@ -137,14 +137,38 @@ def is_triton_kernel_available(data_type: str) -> bool:
return True
-def normalize_hadamard_config(hadamard_config: str | dict | HadamardConfig | None, data_type: str) -> dict[str, Any]:
+def dump_group_size_to_rotation_config(rotation_config: str | dict | RotationConfig, group_size: int):
+ rotation_dict = to_dict_rotation_config(rotation_config)
+ if rotation_dict.get("block_size", None) is None:
+ rotation_dict["block_size"] = group_size
+ return rotation_dict
+
+
+def to_dict_rotation_config(rotation_config: str | dict | RotationConfig):
+ if isinstance(rotation_config, str):
+ key = rotation_config.strip()
+ if not key:
+ return {}
+
+ if key == "default":
+ cfg_dict = {"hadamard_type": "hadamard"}
+ else:
+ cfg_dict = {"hadamard_type": key}
+ elif isinstance(rotation_config, RotationConfig):
+ cfg_dict = rotation_config.model_dump()
+ else:
+ cfg_dict = dict(rotation_config)
+ return cfg_dict
+
+
+def normalize_rotation_config(rotation_config: str | dict | RotationConfig | None, data_type: str) -> dict[str, Any]:
"""
- Normalize and validate `hadamard_config`.
+ Normalize and validate `rotation_config`.
Supported input types:
- None -> {}
- - dict -> validated via HadamardConfig
- - HadamardConfig -> validated & converted to dict
+ - dict -> validated via RotationConfig
+ - RotationConfig -> validated & converted to dict
- str -> shorthand for `hadamard_type` in HADAMARDS keys
Additional behavior:
@@ -171,7 +195,7 @@ def normalize_hadamard_config(hadamard_config: str | dict | HadamardConfig | Non
else:
logger.warning(
f"block_size is not set and cannot be inferred for data_type {data_type!r}; "
- "please set block_size explicitly in hadamard_config if needed."
+ "please set block_size explicitly in rotation_config if needed."
)
else:
if is_mx_fp(data_type) and block_size != 32:
@@ -182,62 +206,16 @@ def normalize_hadamard_config(hadamard_config: str | dict | HadamardConfig | Non
return cfg_dict
# 1) None -> {}
- if hadamard_config is None:
+ if rotation_config is None:
return {}
- # 2) HadamardConfig instance
- if isinstance(hadamard_config, HadamardConfig):
- raw_cfg_dict = hadamard_config.model_dump(exclude_unset=True)
- block_size_explicitly_set = "block_size" in raw_cfg_dict
-
- cfg_dict = dict(raw_cfg_dict)
- cfg_dict = _apply_data_type_block_size(cfg_dict, block_size_explicitly_set)
-
- try:
- return HadamardConfig.model_validate(cfg_dict).model_dump()
- except Exception as e:
- raise ValueError(f"Invalid HadamardConfig: {e}") from e
-
- # 3) dict
- if isinstance(hadamard_config, dict):
- block_size_explicitly_set = "block_size" in hadamard_config
-
- cfg_dict = dict(hadamard_config)
- cfg_dict = _apply_data_type_block_size(cfg_dict, block_size_explicitly_set)
-
- try:
- return HadamardConfig.model_validate(cfg_dict).model_dump()
- except Exception as e:
- raise ValueError(f"Invalid hadamard_config dict: {e}") from e
-
- # 4) str -> shorthand for hadamard_type
- if isinstance(hadamard_config, str):
- key = hadamard_config.strip()
- if not key:
- return {}
-
- if key == "default":
- cfg_dict = {}
- cfg_dict = _apply_data_type_block_size(cfg_dict, block_size_explicitly_set=False)
- try:
- return HadamardConfig.model_validate(cfg_dict).model_dump()
- except Exception as e:
- raise ValueError(f"Invalid default hadamard_config after data_type adjustment: {e}") from e
-
- if key not in HADAMARDS:
- raise ValueError(f"Invalid hadamard_config string: {key!r}. Expected one of {sorted(HADAMARDS.keys())}.")
-
- cfg_dict = {"hadamard_type": key}
- cfg_dict = _apply_data_type_block_size(cfg_dict, block_size_explicitly_set=False)
-
- try:
- return HadamardConfig.model_validate(cfg_dict).model_dump()
- except Exception as e:
- raise ValueError(f"hadamard_config built from string {key!r} is invalid for HadamardConfig: {e}") from e
-
- raise TypeError(
- "hadamard_config must be one of: None, dict, HadamardConfig, or str " f"(got {type(hadamard_config).__name__})"
- )
+ rotation_dict = to_dict_rotation_config(rotation_config)
+ block_size_explicitly_set = "block_size" in rotation_dict
+ cfg_dict = _apply_data_type_block_size(rotation_dict, block_size_explicitly_set)
+ try:
+ return RotationConfig.model_validate(cfg_dict).model_dump()
+ except Exception as e:
+ raise ValueError(f"Invalid RotationConfig: {e}") from e
def check_supported_schemes(scheme: str):
diff --git a/auto_round/formats.py b/auto_round/formats.py
index 40f984cca..68d392d48 100644
--- a/auto_round/formats.py
+++ b/auto_round/formats.py
@@ -269,7 +269,7 @@ class OutputFormat(ABC):
if ar.act_bits <= 8 and (not is_standard_fp(ar.act_data_type) or ar.act_dynamic) and not is_block_dynamic_fp8:
logger.warning(
- f"{self.format_name} format not support for current activation quantization configuration,"
+ f"{self.format_name} format does not support the current activation quantization configuration,"
" reset to fake format and save."
)
return "fake"
diff --git a/auto_round/inference/convert_model.py b/auto_round/inference/convert_model.py
index e769e1eca..e8a2779ec 100644
--- a/auto_round/inference/convert_model.py
+++ b/auto_round/inference/convert_model.py
@@ -214,7 +214,7 @@ def get_layer_config(model, quantization_config):
act_data_type = getattr(quantization_config, "act_data_type", None)
act_dynamic = getattr(quantization_config, "act_dynamic", False)
- hadamard_config = getattr(quantization_config, "hadamard_config", None)
+ rotation_config = getattr(quantization_config, "rotation_config", None)
default_quant_scheme = QuantizationScheme(
bits=bits,
@@ -226,7 +226,7 @@ def get_layer_config(model, quantization_config):
act_sym=act_sym,
act_data_type=act_data_type,
act_dynamic=act_dynamic,
- hadamard_config=hadamard_config,
+ rotation_config=rotation_config,
)
# Determine the quantization block list
@@ -680,19 +680,19 @@ def convert_hf_model(model: nn.Module, target_device: str = "cpu") -> tuple[nn.M
layer_configs = get_layer_config(model, quantization_config)
used_backends = _replace_by_quant_layers(model, layer_configs, backend, target_device, packing_format)
- hadamard_config = getattr(quantization_config, "hadamard_config", None)
- if hadamard_config is not None and hadamard_config:
- from auto_round.experimental.transform.apply import apply_hadamard_transform
- from auto_round.experimental.transform.hadamard_config import HadamardConfig
+ rotation_config = getattr(quantization_config, "rotation_config", None)
+ if rotation_config is not None and rotation_config:
+ from auto_round.experimental.transform.apply import apply_rotation_transform
+ from auto_round.experimental.transform.rotation_config import RotationConfig
# apply forward hook
- act_hadamard_config = HadamardConfig(
- block_size=hadamard_config["block_size"],
- hadamard_type=hadamard_config["hadamard_type"],
+ act_rotation_config = RotationConfig(
+ block_size=rotation_config["block_size"],
+ hadamard_type=rotation_config["hadamard_type"],
) # apply to activation
- model = apply_hadamard_transform(
+ model, _ = apply_rotation_transform(
model,
- act_hadamard_config,
+ act_rotation_config,
location="input",
desc="Register pre forward hook for hadamard transform",
data_type=quantization_config.data_type,
diff --git a/auto_round/schemes.py b/auto_round/schemes.py
index 0cfd757bb..0e9b922ba 100644
--- a/auto_round/schemes.py
+++ b/auto_round/schemes.py
@@ -36,7 +36,7 @@ class QuantizationScheme:
act_dynamic: Optional[bool] = None
super_bits: Optional[int] = None
super_group_size: Optional[int] = None
- hadamard_config: Optional[dict] = None
+ rotation_config: Optional[dict] = None
@classmethod
def from_dict(cls, config: dict):
@@ -88,7 +88,7 @@ class QuantizationScheme:
continue
self_val = getattr(self, field)
other_val = getattr(other, field)
- # Treat None and empty dict as equivalent for dict fields like hadamard_config
+ # Treat None and empty dict as equivalent for dict fields like rotation_config
if self_val != other_val:
if isinstance(self_val, dict) and not self_val and other_val is None:
continue
@@ -206,6 +206,19 @@ MXFP4 = QuantizationScheme.from_dict(
}
)
+MXFP6 = QuantizationScheme.from_dict(
+ {
+ "bits": 6,
+ "group_size": 32,
+ "data_type": "mx_fp",
+ "act_bits": 6,
+ "act_data_type": "mx_fp",
+ "act_group_size": 32,
+ "act_sym": True,
+ "act_dynamic": True,
+ }
+)
+
MXFP4_RCEIL = QuantizationScheme.from_dict(
{
"bits": 4,
@@ -330,6 +343,20 @@ INT8 = QuantizationScheme.from_dict(
}
)
+INT4 = QuantizationScheme.from_dict(
+ {
+ "bits": 4,
+ "group_size": -1,
+ "data_type": "int",
+ "sym": True,
+ "act_bits": 4,
+ "act_group_size": -1,
+ "act_data_type": "int",
+ "act_dynamic": True,
+ "act_sym": True,
+ }
+)
+
# For AutoScheme 16 bits options
BF16 = QuantizationScheme.from_dict(
@@ -348,6 +375,7 @@ PRESET_SCHEMES = {
"W3A16": W3A16,
"W8A16": W8A16,
"MXFP4": MXFP4,
+ "MXFP6": MXFP6,
"MXFP4_RCEIL": MXFP4_RCEIL,
"MXFP8": MXFP8,
"MXFP8_RCEIL": MXFP8_RCEIL,
@@ -358,6 +386,7 @@ PRESET_SCHEMES = {
"FP8_STATIC": FP8_STATIC,
"BF16": BF16,
"W4A16_MIXED": W4A16,
+ "INT4": INT4,
"INT8": INT8,
"INT8_W8A8": INT8,
"FP8_BLOCK": FP8_BLOCK,
diff --git a/auto_round/wrapper.py b/auto_round/wrapper.py
index 782786a13..a48a654be 100644
--- a/auto_round/wrapper.py
+++ b/auto_round/wrapper.py
@@ -12,12 +12,14 @@
# See the License for the specific language governing permissions and
# limitations under the License.
+import math
from math import ceil
import torch
import transformers
from torch.functional import F
+import auto_round.envs as envs
from auto_round.compressors.utils import is_nv_fp
from auto_round.data_type import get_quant_func, reshape_pad_tensor_by_group_size
from auto_round.logger import logger
@@ -92,6 +94,7 @@ class WrapperLinear(torch.nn.Module):
"""
super(WrapperLinear, self).__init__()
self.orig_layer = orig_layer
+ self.orig_layer.iters = kwargs.pop("iters", 200)
self.disable_opt_rtn = disable_opt_rtn
self.output_device = device
self.device = self.orig_layer.tuning_device if hasattr(self.orig_layer, "tuning_device") else device
@@ -169,18 +172,30 @@ class WrapperLinear(torch.nn.Module):
self._init_params("max_scale", p_dtype, shape, 1.0, (self.enable_minmax_tuning and self.orig_layer.bits < 16))
self.weight_quant_func, self.data_type = get_quant_func(
- orig_layer.data_type, orig_layer.bits, orig_layer.sym, self.disable_opt_rtn, orig_layer.group_size
+ orig_layer.data_type,
+ orig_layer.bits,
+ orig_layer.sym,
+ self.disable_opt_rtn,
+ orig_layer.group_size,
+ iters=orig_layer.iters,
)
if self.enable_torch_compile:
self.weight_quant_func = compile_func(self.weight_quant_func, self.device)
if self.enable_act_quant:
self.act_quant_func, self.act_data_type = get_quant_func(
- orig_layer.act_data_type, orig_layer.act_bits, orig_layer.act_sym, self.disable_opt_rtn
+ orig_layer.act_data_type,
+ orig_layer.act_bits,
+ orig_layer.act_sym,
+ disable_opt_rtn=True,
+ iters=orig_layer.iters,
)
if self.enable_torch_compile:
self.act_quant_func = compile_func(self.act_quant_func, self.device)
- self._init_params("act_max_scale", p_dtype, (1), 1.0, not orig_layer.act_dynamic)
+ self._init_params(
+ "act_max_scale", p_dtype, (1), 1.0, envs.AR_ENABLE_ACT_MINMAX_TUNING or (not orig_layer.act_dynamic)
+ )
+ self._init_params("act_min_scale", p_dtype, (1), 1.0, envs.AR_ENABLE_ACT_MINMAX_TUNING)
# Bias tuning
if self.enable_norm_bias_tuning:
@@ -221,8 +236,8 @@ class WrapperLinear(torch.nn.Module):
"""
if self.orig_layer.bits >= 16:
return self.orig_layer.weight, None, None
- min_scale.data.clamp_(0, 1.0)
- max_scale.data.clamp_(0, 1.0)
+ min_scale.data.clamp_(0.0, 1.0)
+ max_scale.data.clamp_(0.0, 1.0)
weight = self.orig_layer.weight
if weight.device.type == "meta":
weight = self.orig_layer.get_weight().to(self.device)
@@ -255,7 +270,7 @@ class WrapperLinear(torch.nn.Module):
weight_q = weight_q.t()
return weight_q, scale, zp
- def _qdq_act(self, x, act_max_scale, act_max=None):
+ def _qdq_act(self, x, act_min_scale=torch.tensor(1.0), act_max_scale=torch.tensor(1.0), act_max=None):
"""Quantizes and dequantizes activations.
Args:
@@ -267,6 +282,8 @@ class WrapperLinear(torch.nn.Module):
tuple: Quantized activation, scale, and zero point.
"""
act_max_scale.data.clamp_(0, 1.0)
+ act_min_scale.data.clamp_(0, 1.0)
+ env_act_scale = envs.AR_ACT_SCALE # fixed activation ratio,prioritize to use this one if set
x, scale, zp = self.act_quant_func(
x,
bits=self.orig_layer.act_bits,
@@ -274,8 +291,9 @@ class WrapperLinear(torch.nn.Module):
scale_dtype=self.orig_layer.scale_dtype,
q_scale_thresh=self.q_scale_thresh,
data_type=self.act_data_type,
- max_scale=act_max_scale,
- tensor_max=act_max,
+ tensor_max=act_max, # for static
+ max_scale=act_max_scale if math.isclose(env_act_scale, 1.0, rel_tol=1e-6) else env_act_scale,
+ min_scale=act_min_scale if math.isclose(env_act_scale, 1.0, rel_tol=1e-6) else env_act_scale,
global_scale=getattr(self, "input_global_scale", None),
)
return x, scale, zp
@@ -394,7 +412,10 @@ class WrapperLinear(torch.nn.Module):
elif self.orig_layer.act_group_size == -1:
tmp_shape = (act_max.shape[0], 1)
_, act_scale, _ = self._qdq_act(
- torch.zeros(tmp_shape).to(self.device), act_max_scale=self.act_max_scale, act_max=act_max
+ torch.zeros(tmp_shape).to(self.device),
+ act_min_scale=self.act_min_scale,
+ act_max_scale=self.act_max_scale,
+ act_max=act_max,
)
self.orig_layer.act_max = self.orig_layer.act_max * act_max_scale.item()
self.orig_layer.act_max = self.orig_layer.act_max.to("cpu")
@@ -404,6 +425,8 @@ class WrapperLinear(torch.nn.Module):
self.orig_layer.q_scale_thresh = self.q_scale_thresh
self.orig_layer.data_type = self.data_type
+ self.orig_layer.act_min_scale = self.act_min_scale
+ self.orig_layer.act_max_scale = self.act_max_scale
self.orig_layer.act_data_type = self.act_data_type
self.orig_layer.act_quant_func = self.act_quant_func
@@ -477,8 +500,23 @@ class WrapperLinear(torch.nn.Module):
weight_q, *_ = self._qdq_weight(self.value, self.min_scale, self.max_scale)
if self.enable_act_quant:
+ # Run orig_layer's forward_pre_hooks (e.g., online Hadamard transform)
+ # BEFORE activation quantization, to match inference behavior in WrapperWALayer.
+ for hook in self.orig_layer._forward_pre_hooks.values():
+ result = hook(self.orig_layer, (x,))
+ if result is not None:
+ x = result[0] if isinstance(result, tuple) else result
act_max = self.orig_layer.act_max if hasattr(self.orig_layer, "act_max") else None
- x, _, _ = self._qdq_act(x, act_max_scale=self.act_max_scale, act_max=act_max)
+ x, _, _ = self._qdq_act(
+ x, act_max_scale=self.act_max_scale, act_min_scale=self.act_min_scale, act_max=act_max
+ )
+ elif len(self.orig_layer._forward_pre_hooks) > 0:
+ # Even without act_quant, run pre-hooks for online Hadamard correctness
+ # (when fuse_online_to_weight=False, pre-hooks transform activations).
+ for hook in self.orig_layer._forward_pre_hooks.values():
+ result = hook(self.orig_layer, (x,))
+ if result is not None:
+ x = result[0] if isinstance(result, tuple) else result
# pylint: disable=not-callable
bias = self.orig_layer.bias
@@ -488,6 +526,14 @@ class WrapperLinear(torch.nn.Module):
bias, _, _ = self._qdq_bias(bias, self.bias_v)
output = self.orig_forward(x, weight_q, bias).to(self.output_device)
+
+ # Execute post-hooks from orig_layer (e.g., v_proj per-head Hadamard
+ # when online rotation is not fused into weights).
+ for hook in self.orig_layer._forward_hooks.values():
+ hook_result = hook(self.orig_layer, (x,), output)
+ if hook_result is not None:
+ output = hook_result
+
return output
@@ -504,6 +550,12 @@ class WrapperWALayer(torch.nn.Module):
self.act_quant_func = compile_func(self.act_quant_func, self.device)
self.extra_repr_org = orig_layer.extra_repr
+ # Steal forward_pre_hooks from orig_layer (e.g., Hadamard transform hooks)
+ # and remove them from orig_layer so they won't fire again inside orig_layer.forward().
+ # We will run them explicitly in our forward() BEFORE activation quantization.
+ self._stolen_pre_hooks = list(orig_layer._forward_pre_hooks.values())
+ orig_layer._forward_pre_hooks.clear()
+
@property
def weight(self):
"""Exposes the weight of the wrapped layer for external access."""
@@ -515,16 +567,42 @@ class WrapperWALayer(torch.nn.Module):
return self.orig_layer.bias
def forward(self, x):
+ # 1) Run stolen pre_hooks first (e.g., online Hadamard) → smooths activation
+ for hook in self._stolen_pre_hooks:
+ result = hook(self.orig_layer, (x,))
+ if result is not None:
+ x = result[0] if isinstance(result, tuple) else result
+
+ # 2) Activation quantization on the smoothed activation
+ import auto_round.envs as envs
+
+ act_scale = envs.AR_ACT_SCALE
act_max = self.orig_layer.act_max if hasattr(self.orig_layer, "act_max") else None
- x, _, _ = self.orig_layer.act_quant_func(
- x,
- bits=self.orig_layer.act_bits,
- group_size=self.orig_layer.act_group_size,
- scale_dtype=self.orig_layer.scale_dtype,
- q_scale_thresh=self.orig_layer.q_scale_thresh,
- data_type=self.orig_layer.act_data_type,
- tensor_max=act_max,
- )
+
+ max_scale = self.orig_layer.act_max_scale if math.isclose(act_scale, 1.0, rel_tol=1e-6) else act_scale
+ min_scale = self.orig_layer.act_min_scale if math.isclose(act_scale, 1.0, rel_tol=1e-6) else act_scale
+ if act_max is None:
+ x, _, _ = self.orig_layer.act_quant_func(
+ x,
+ bits=self.orig_layer.act_bits,
+ group_size=self.orig_layer.act_group_size,
+ scale_dtype=self.orig_layer.scale_dtype,
+ q_scale_thresh=self.orig_layer.q_scale_thresh,
+ data_type=self.orig_layer.act_data_type,
+ min_scale=min_scale,
+ max_scale=max_scale,
+ )
+ else:
+ x, _, _ = self.orig_layer.act_quant_func(
+ x,
+ bits=self.orig_layer.act_bits,
+ group_size=self.orig_layer.act_group_size,
+ scale_dtype=self.orig_layer.scale_dtype,
+ q_scale_thresh=self.orig_layer.q_scale_thresh,
+ data_type=self.orig_layer.act_data_type,
+ act_max=act_max,
+ )
+ # 3) Linear computation via orig_layer (pre_hooks already removed, no double execution)
return self.orig_layer.forward(x)
def extra_repr(self):
@@ -640,6 +718,7 @@ NORM_MAPPING["Qwen2RMSNorm"] = WrapperLlamaNorm
NORM_MAPPING["Phi3RMSNorm"] = WrapperLlamaNorm
NORM_MAPPING["MistralRMSNorm"] = WrapperLlamaNorm
NORM_MAPPING["Qwen3RMSNorm"] = WrapperLlamaNorm
+NORM_MAPPING["Qwen3_5MoeRMSNorm"] = WrapperLlamaNorm
class WrapperMultiblock(torch.nn.Module):
diff --git a/docs/environments.md b/docs/environments.md
index 242d660d7..ae1052976 100644
--- a/docs/environments.md
+++ b/docs/environments.md
@@ -60,7 +60,7 @@ export AR_DISABLE_OFFLOAD=1
```
### AR_DISABLE_DATASET_SUBPROCESS
-- **Description**: Disables the use of a subprocess for dataset preprocessing. By default, AutoRound uses a subprocess to ensure all temporary memory is reclaimed by the OS.
+- **Description**: Only for research. Disables the use of a subprocess for dataset preprocessing. By default, AutoRound uses a subprocess to ensure all temporary memory is reclaimed by the OS.
- **Default**: `False`
- **Valid Values**: `"1"`, `"true"` (case-insensitive) for disabling; any other value for enabling
- **Usage**: Set this to run dataset preprocessing in the main process
@@ -69,6 +69,36 @@ export AR_DISABLE_OFFLOAD=1
export AR_DISABLE_DATASET_SUBPROCESS=true
```
+### AR_ACT_SCALE
+- **Description**: Only for research. Controls the scaling factor applied to activation min/max values during activation quantization. A value less than 1.0 shrinks the clipping range, which can reduce outlier impact.
+- **Default**: `1.0`
+- **Valid Values**: float>=0.0, e.g. `0.8`, `0.9`, `1.0`
+- **Usage**: Set this to adjust the activation clipping range
+
+```bash
+export AR_ACT_SCALE=0.9
+```
+
+### AR_ENABLE_ACT_MINMAX_TUNING
+- **Description**: Enables tuning of activation min/max scale parameters (`act_min_scale`, `act_max_scale`) during quantization optimization. When enabled, these scales become tunable instead of remaining fixed at `1.0`.
+- **Default**: `False` (equivalent to `"0"`)
+- **Valid Values**: `"1"`, `"true"`, `"yes"` (case-insensitive) for enabling tuning; any other value keeps tuning disabled
+- **Usage**: Set this to enable activation min-max scale tuning
+
+```bash
+export AR_ENABLE_ACT_MINMAX_TUNING=1
+```
+
+### AR_SEARCH_SCALE_RATIO
+- **Description**: Controls the search range ratio used by the symmetric INT scale search in `auto_round.data_type.int.search_scales`. The search bound is `nmax * AR_SEARCH_SCALE_RATIO`, where `nmax = 2^(bits-1)`. Smaller values restrict the search to a tighter neighborhood around the initial scale (faster, less thorough); larger values broaden the search (slower, may improve accuracy on outlier-heavy weights).
+- **Default**: unset → falls back to the built-in default (`0.5`, i.e. `nmax/2`).
+- **Valid Values**: positive float, e.g. `0.25`, `0.5`, `0.75`, `1.0`
+- **Usage**: Set this to override the default scale-search range
+
+```bash
+export AR_SEARCH_SCALE_RATIO=0.75
+```
+
## Usage Examples
### Setting Environment Variables
diff --git a/docs/environments_CN.md b/docs/environments_CN.md
index 7407354cf..c360f5b99 100644
--- a/docs/environments_CN.md
+++ b/docs/environments_CN.md
@@ -69,6 +69,36 @@ export AR_DISABLE_OFFLOAD=1
export AR_DISABLE_DATASET_SUBPROCESS=true
```
+### AR_ACT_SCALE
+- **描述**:只用于研究性质,控制激活量化时对激活值最小/最大值的缩放系数。小于 1.0 的值会缩小裁剪范围,有助于减小离群值的影响。
+- **默认值**:`1.0`
+- **有效值**:任意浮点数,如 `0.8`、`0.9`、`1.0`
+- **用途**:调整激活裁剪范围
+
+```bash
+export AR_ACT_SCALE=0.9
+```
+
+### AR_ENABLE_ACT_MINMAX_TUNING
+- **描述**:只用于研究性质,使用激活量化中最小/最大缩放参数(`act_min_scale`、`act_max_scale`)的调优。启用后,这些缩放参数将固定为 1.0。
+- **默认值**:`False`(等同于 `"0"`)
+- **有效值**:`"1"`、`"true"`、`"yes"`(不区分大小写)表示禁用调优;其他值表示保持调优
+- **用途**:禁用激活最小-最大缩放参数的调优
+
+```bash
+export AR_ENABLE_ACT_MINMAX_TUNING=1
+```
+
+### AR_SEARCH_SCALE_RATIO
+- **描述**:控制 `auto_round.data_type.int.search_scales` 中对称 INT 量化 scale 搜索的范围比例。搜索上界为 `nmax * AR_SEARCH_SCALE_RATIO`,其中 `nmax = 2^(bits-1)`。值越小搜索范围越窄(更快,但可能漏掉较优解);值越大搜索范围越广(更慢,对离群权重可能更准)。
+- **默认值**:未设置 → 走内置默认值(`0.5`,即 `nmax/2`)。
+- **有效值**:正浮点数,如 `0.25`、`0.5`、`0.75`、`1.0`
+- **用途**:覆盖默认的 scale 搜索范围
+
+```bash
+export AR_SEARCH_SCALE_RATIO=0.75
+```
+
## 使用示例
### 设置环境变量
diff --git a/docs/int_acc.md b/docs/int_acc.md
new file mode 100644
index 000000000..e3f09a39c
--- /dev/null
+++ b/docs/int_acc.md
@@ -0,0 +1,59 @@
+
+### Algorithm Details
+
+| Alg | Rotation Type | Activation Clip Ratio | weight quantizer | w/a symmetric |
+|-----------|---------------|-----------------------|------------------|--------------|
+| Quarot | fixed | 0.9 | GPTQ | yes |
+| FlatQuant | learnable | learnable | GPTQ | yes |
+| Ours | fixed | 0.9 | SignRound | yes |
+
+### Eval Details.
+
+lm-eval 0.4.11 for all the data evaluation.
+
+We report the average of 10 tasks (AVG10) as well as the individual performance on each task. The 10 tasks are: ARC-Challenge (arc_c), ARC-Easy (arc_e), BoolQ (boolq), HellaSwag (hellaswag), Lambada (lambada), MMLU (mmlu), OpenBookQA (obqa), PiQA (piqa), TriviaQA (tfq), and Winogrande (winog).
+
+Following Quarot and FlatQuant, we report `acc_norm` instead of accuracy for tasks where acc_norm is available.
+
+All results below `exclude` KV-cache quantization, as it is not yet supported in our implementation.
+
+### How to Reproduce
+~~~ bash
+AR_ACT_SCALE=0.9 auto-round Qwen/Qwen3-8B --scheme INT4 --rotation_type hadamard --format fake --tasks mmlu
+~~~
+### Qwen3-8B
+
+| Method | AVG10 | arc_c | arc_e | boolq | hellaswag | lambada | mmlu | obqa | piqa | tfq | winog |
+|-----------|------------|------------|------------|------------|------------|------------|------------|-----------|------------|------------|------------|
+| 16 bit | 0.6598 | 0.5640 | 0.8068 | 0.8670 | 0.7491 | 0.6420 | 0.7288 | 0.420 | 0.7764 | 0.3647 | 0.6788 |
+| QuaRot | 0.6123 | 0.5060 | 0.7428 | 0.8220 | 0.6836 | 0.5762 | 0.6545 | 0.412 | 0.7383 | 0.3464 | 0.6409 |
+| FlatQuant | 0.6511 | 0.5657 | 0.8072 | 0.8609 | 0.7256 | 0.6369 | 0.7032 | 0.410 | **0.7682** | 0.3525 | 0.6803 |
+| Ours | **0.6533** | **0.5614** | **0.8035** | **0.8657** | **0.7336** | **0.6336** | **0.7188** | **0.408** | 0.7704 | **0.3562** | **0.6819** |
+
+### Llama-3-8B
+
+| Method | AVG10 | arc_c | arc_e | boolq | hellaswag | lambada | mmlu | obqa | piqa | tfq | winog |
+|-----------|------------|------------|------------|------------|------------|------------|------------|-----------|------------|------------|------------|
+| 16 bit | 0.6569 | 0.5392 | 0.7769 | 0.8190 | 0.7919 | 0.7545 | 0.6216 | 0.452 | 0.8069 | 0.2717 | 0.7356 |
+| QuaRot | 0.5924 | 0.4326 | 0.7155 | 0.7526 | 0.7342 | 0.6810 | 0.5007 | 0.426 | 0.7492 | 0.2546 | 0.6772 |
+| FlatQuant | 0.6442 | 0.5213 | 0.7706 | 0.8168 | 0.7724 | 0.7363 | 0.5875 | 0.438 | **0.7987** | **0.2754** | 0.7253 |
+| Ours | **0.6521** | **0.5452** | **0.7862** | **0.8119** | **0.7832** | **0.7460** | **0.6107** | **0.436** | 0.8079 | 0.2668 | **0.7269** |
+
+### Llama-2-7B
+
+| Method | AVG10 | arc_c | arc_e | boolq | hellaswag | lambada | mmlu | obqa | piqa | tfq | winog |
+|-----------|------------|------------|------------|------------|------------|------------|------------|-----------|------------|------------|------------|
+| 16 bit | 0.6071 | 0.4505 | 0.7382 | 0.7939 | 0.7615 | 0.7367 | 0.4177 | 0.442 | 0.7878 | 0.2485 | 0.6946 |
+| QuaRot | 0.5742 | 0.4352 | 0.6907 | 0.7388 | 0.7275 | 0.7093 | 0.3527 | 0.402 | 0.7715 | 0.2509 | 0.6630 |
+| FlatQuant | 0.5954 | 0.4454 | 0.7239 | 0.7743 | 0.7452 | 0.7233 | **0.3920** | 0.430 | **0.7840** | 0.2448 | **0.6914** |
+| Ours | **0.5991** | **0.4403** | **0.7281** | **0.7902** | **0.7495** | **0.7256** | 0.3915 | **0.450** | 0.7802 | **0.2497** | 0.6859 |
+
+### Llama-2-13B
+
+| Method | AVG10 | arc_c | arc_e | boolq | hellaswag | lambada | mmlu | obqa | piqa | tfq | winog |
+|-----------|------------|------------|------------|------------|------------|------------|------------|-----------|------------|------------|------------|
+| 16 bit | 0.6446 | 0.4881 | 0.7652 | 0.8217 | 0.7965 | 0.7652 | 0.5239 | 0.452 | 0.8030 | 0.2681 | 0.7621 |
+| QuaRot | 0.6114 | 0.4573 | 0.7348 | 0.7872 | 0.7644 | 0.7444 | 0.4584 | 0.430 | 0.7818 | 0.2546 | 0.7009 |
+| FlatQuant | 0.6308 | 0.4795 | 0.7534 | 0.8180 | 0.7817 | 0.7588 | 0.4985 | 0.446 | **0.7982** | **0.2583** | 0.7151 |
+| Ours | **0.6369** | **0.4804** | **0.7626** | **0.8245** | **0.7886** | **0.7720** | **0.5106** | **0.450** | 0.8014 | 0.2497 | **0.7293** |
+
diff --git a/docs/step_by_step.md b/docs/step_by_step.md
index 0327f4dc1..4ca147926 100644
--- a/docs/step_by_step.md
+++ b/docs/step_by_step.md
@@ -659,8 +659,8 @@ from auto_round import AutoRound
model_name_or_path = "meta-llama/Llama-3.1-8B-Instruct"
output_dir = "./Llama-3.1-8B-Instruct-mxfp4-ht"
-# hadamard_config="default": block_size=32, hadamard_type="hadamard"
-ar = AutoRound(model_name_or_path, scheme="MXFP4", hadamard_config="default")
+# rotation_config="default": block_size=32, hadamard_type="hadamard"
+ar = AutoRound(model_name_or_path, scheme="MXFP4", rotation_config="default")
ar.quantize_and_save(output_dir=output_dir, format="auto_round")
```
diff --git a/docs/step_by_step_CN.md b/docs/step_by_step_CN.md
index d535718a6..0e2c6bec1 100644
--- a/docs/step_by_step_CN.md
+++ b/docs/step_by_step_CN.md
@@ -619,8 +619,8 @@ from auto_round import AutoRound
model_name_or_path = "meta-llama/Llama-3.1-8B-Instruct"
output_dir = "./Llama-3.1-8B-Instruct-mxfp4-ht"
-# hadamard_config="default": block_size=32, hadamard_type="hadamard"
-ar = AutoRound(model_name_or_path, scheme="MXFP4", hadamard_config="default")
+# rotation_config="default": block_size=32, hadamard_type="hadamard"
+ar = AutoRound(model_name_or_path, scheme="MXFP4", rotation_config="default")
ar.quantize_and_save(output_dir=output_dir, format="auto_round")
```
diff --git a/test/test_cuda/transform/test_mxfp4_transform.py b/test/test_cuda/transform/test_mxfp4_transform.py
index fba84e678..16511edad 100644
--- a/test/test_cuda/transform/test_mxfp4_transform.py
+++ b/test/test_cuda/transform/test_mxfp4_transform.py
@@ -36,7 +36,7 @@ class TestAutoRound:
iters=0,
seqlen=2,
scheme=scheme,
- hadamard_config="default",
+ rotation_config="default",
)
compressed_model, _ = ar.quantize_and_save(output_dir=self.save_dir, format="auto_round")
@@ -55,7 +55,7 @@ class TestAutoRound:
iters=2,
seqlen=2,
scheme=scheme,
- hadamard_config="default",
+ rotation_config="default",
)
compressed_model, _ = ar.quantize_and_save(output_dir=self.save_dir, format="auto_round")
@@ -74,7 +74,7 @@ class TestAutoRound:
iters=0,
seqlen=2,
scheme=scheme,
- hadamard_config="random_hadamard",
+ rotation_config="random_hadamard",
)
compressed_model, _ = ar.quantize_and_save(output_dir=self.save_dir, format="auto_round")