PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
[research feature] Introduce INT4 support at the algorithm level (#1641)
Wenhua Cheng11e91869415e
diff --git a/README.md b/README.mdindex d710f0a4e..dc7acfc8e 100644--- a/README.md+++ b/README.md@@ -36,9 +36,11 @@ See our papers [SignRoundV1](https://arxiv.org/pdf/2309.05516) and [SignRoundV2]## 🆕 What's New+* [2026/04] We demonstrate that AutoRound achieves SOTA or near SOTA performance under INT4 (W4A4) quantization [accuracy data](./docs/int_acc.md). This capability is currently a **research-only** feature, with no production model export.+* [2026/03] **Block-wise FP8** quantization is available via `--scheme FP8_BLOCK --iters 0 --disable_opt_rtn`.-* [2026/03] MTP layer quantization has been supported in [PR](https://github.com/intel/auto-round/pull/1526)+* [2026/03] **MTP layer quantization** has been supported in this [PR](https://github.com/intel/auto-round/pull/1526)* [2025/12] The **SignRoundV2** paper is available. Turn on `enable_alg_ext` and use the **AutoScheme** API for mixed-precision quantization to reproduce the results: [*Paper*](http://arxiv.org/abs/2512.04746), [*Notes for evaluating LLaMA models*](./docs/alg_202508.md).@@ -226,7 +228,7 @@ ar.quantize_and_save(output_dir="./qmodel", format="auto_round")- **`iters` (int)**: Number of tuning iterations (default is `200`). Common values: 0 (RTN mode), 50 (with lr=5e-3 recommended), 1000. Higher values increase accuracy but slow down tuning.- **`lr` (float)**: The learning rate for rounding value (default is `None`). When None, it will be set to `1.0/iters` automatically.- **`batch_size` (int)**: Batch size for training (default is `8`). 4 is also commonly used.-- ** `enable_deterministic_algorithms` (bool)**: Whether to enable deterministic algorithms for reproducibility (default is `False`).+- **`enable_deterministic_algorithms` (bool)**: Whether to enable deterministic algorithms for reproducibility (default is `False`).##### Calibration Dataset- **`dataset` (str|list|tuple|torch.utils.data.DataLoader)**: The dataset for tuning (default is `"NeelNanda/pile-10k"`). Supports local JSON files and dataset combinations, e.g. `"./tmp.json,NeelNanda/pile-10k:train,mbpp:train+validation+test"`.diff --git a/README_CN.md b/README_CN.mdindex 9f3a5667f..72e99df18 100644--- a/README_CN.md+++ b/README_CN.md@@ -35,6 +35,12 @@ AutoRound 是专为大语言模型(LLMs)和视觉-语言模型(VLMs)设## 🆕 最新进展+* [2026/04] AutoRound可以在INT4上取得非常好的精度[accuracy data](./docs/int_acc.md). 不过目前这个特性还停留在研究性质,我们正在努力将其优化到生产环境中。++* [2026/03] 支持**Block-wise FP8**,只需要将scheme改成 `--scheme FP8_BLOCK`.为了更快的量化,我们推荐使用RTN模式,即·--iters 0 --disable_opt_rtn·++* [2026/03] 支持**MTP层量化**+** [2025/12] 发布 **SignRoundV2** 论文。如果要复现论文成果,可启用 `enable_alg_ext`,并使用 **AutoScheme** API 对模型进行混合精度量化。相关链接:[*论文*](http://arxiv.org/abs/2512.04746),[*LLaMA 模型评估说明*](./docs/alg_202508.md)。* [2025/11] **LLM-Compressor** 已支持 AutoRound 算法。相关链接:[*使用方法*](https://github.com/vllm-project/llm-compressor/tree/main/examples/autoround/README.md),[*vLLM 博客*](https://blog.vllm.ai/2025/12/09/intel-autoround-llmc.html),[*RedHat 博客*](https://developers.redhat.com/articles/2025/12/09/advancing-low-bit-quantization-llms-autoround-x-llm-compressor),[*X 推文*](https://x.com/vllm_project/status/1998710451312771532),[*Intel 博客*](https://community.intel.com/t5/Blogs/Products-and-Solutions/HPC/Advancing-Low-Bit-Quantization-for-LLMs-AutoRound-x-LLM/post/1729336),[*LinkedIn*](https://www.linkedin.com/posts/vllm-project_advancing-lowbit-quantization-for-llms-activity-7404478053768441856-ru8f/?utm_source=share&utm_medium=member_desktop&rcm=ACoAAAapNW8BLnAdCAr57GOwSCJXjf76ZvOEOAg),[*微信*](https://mp.weixin.qq.com/s/l5WA-1_4ipffQN6GOH2Iqg),[*知乎*](https://zhuanlan.zhihu.com/p/1982167638315664412)。diff --git a/auto_round/__main__.py b/auto_round/__main__.pyindex 627d4ebef..d7fde248a 100644--- a/auto_round/__main__.py+++ b/auto_round/__main__.py@@ -286,6 +286,7 @@ class BasicArgumentParser(argparse.ArgumentParser):help="Disable optimization for RTN (Round-To-Nearest) mode when iters=0. ""RTN is fast but less accurate; keeping optimization enabled is recommended.",)+group_opt_rtn.add_argument("--enable_opt_rtn",action="store_const",@@ -303,6 +304,9 @@ class BasicArgumentParser(argparse.ArgumentParser):help="Group size for weight quantization.",)scheme.add_argument("--asym", action="store_true", help="Use asymmetric quantization instead of symmetric.")+ scheme.add_argument(+ "--act_asym", action="store_true", help="Use asymmetric quantization for activation instead of symmetric."+ )scheme.add_argument("--data_type","--dtype",@@ -373,6 +377,13 @@ class BasicArgumentParser(argparse.ArgumentParser):choices=["fp8", "float8_e4m3fn"],help="Data type for static quantize attention. ",)+ scheme.add_argument(+ "--rotation_type",+ default=None,+ type=str,+ choices=["hadamard", "random_hadamard"],+ help="Research feature: applies a rotation (e.g., Hadamard) to reduce activation/weight outliers",+ )gguf = self.add_argument_group("Double Quant Arguments")gguf.add_argument("--super_group_size", default=None, type=int, help="Super group size for double quantization."@@ -659,6 +670,7 @@ def tune(args):act_group_size=args.act_group_size,act_data_type=args.act_data_type,act_dynamic=act_dynamic,+ act_sym=None if not args.asym else False,super_bits=args.super_bits,super_group_size=args.super_group_size,quant_lm_head=args.quant_lm_head,@@ -704,6 +716,11 @@ def tune(args):low_gpu_mem_usage=True, # force it to be True as it uses much smaller vram but similar time costlow_cpu_mem_usage=low_cpu_mem_usage,)+ rot_config = None+ if args.rotation_type:+ from auto_round.experimental.transform.rotation_config import RotationConfig++ rot_config = RotationConfig(hadamard_type=args.rotation_type)autoround: BaseCompressor = AutoRound(model=model_name,@@ -727,6 +744,7 @@ def tune(args):model_dtype=args.model_dtype,momentum=args.momentum,trust_remote_code=not args.disable_trust_remote_code,+ rotation_config=rot_config,)model_name = args.model.rstrip("/")diff --git a/auto_round/alg_ext.py b/auto_round/alg_ext.pyindex 4d74f8bd9..cedd0fdc0 100644--- a/auto_round/alg_ext.py+++ b/auto_round/alg_ext.py@@ -24,7 +24,7 @@ from torch import autocastfrom torch.functional import Ffrom auto_round import AutoRound-from auto_round.compressors.utils import check_need_act_calibration, is_nv_fp+from auto_round.compressors.utils import check_need_act_calibration, is_nv_fp, is_wint4aint4from auto_round.data_type.int import search_scalesfrom auto_round.data_type.mxfp import MXFP_FORMAT_CACHE, quant_elementfrom auto_round.data_type.nvfp import FLOAT4_E2M1_MAX, FLOAT8_E4M3_MAX, ref_nvfp4_quant@@ -42,16 +42,12 @@ def wrapper_autoround(cls: AutoRound):cls.symand cls.enable_alg_extand cls.super_group_size is None- and (- (cls.data_type.startswith("int") and cls.act_bits >= 8)- or cls.data_type.startswith("mx")- or cls.data_type.startswith("nv")- )+ and ((cls.data_type.startswith("int")) or cls.data_type.startswith("mx") or cls.data_type.startswith("nv"))):if cls.bits > 2 and (not cls.data_type.startswith("mx") or not cls.data_type.startswith("nv")):logger.warning_once("algorithm extension has only undergone limited validation on "- "INT2,mxfp4 and nvfp4; use with caution."+ "W2A16,INT4, MXFP4 and NVFP4; use with caution.")cls._get_loss = types.MethodType(_get_loss_ext, cls)setattr(cls, "wrapper_block", wrapper_block_v2)@@ -323,8 +319,6 @@ class WrapperLinearV2(WrapperLinear):disable_opt_rtn=disable_opt_rtn,**kwargs,)- if self.enable_act_quant and self.orig_layer.data_type.startswith("int"):- raise ValueError("does not support act quantization in WrapperLinearV2, please use WrapperLinear instead")def _init_tuning_params_and_quant_func(self):"""Initializes tuning parameters and quantization functions.@@ -342,21 +336,23 @@ class WrapperLinearV2(WrapperLinear):imatrix = imatrix.expand(weight_reshape.numel() // imatrix.numel(), -1)imatrix = imatrix.reshape(weight_reshape.shape)imatrix = imatrix.to(orig_weight.device)- if self.orig_layer.data_type.startswith("int"):- self.init_scale = search_scales(weight_reshape, self.orig_layer.bits, imatrix)- self.init_scale = torch.where(- self.init_scale < 0,- torch.clamp(self.init_scale, max=-self.q_scale_thresh),- torch.clamp(self.init_scale, min=self.q_scale_thresh),- )- elif self.orig_layer.data_type.startswith("mx"):- self.init_scale = mx_init(weight_reshape, self.orig_layer.bits, imatrix)- elif self.orig_layer.data_type.startswith("nv"):- self.init_scale = nv_init(weight_reshape, self.orig_layer.bits, imatrix)- self.orig_layer.imatrix = None- delattr(self.orig_layer, "imatrix")+ else:+ imatrix = 1.0+ if self.orig_layer.data_type.startswith("int"):+ self.init_scale = search_scales(weight_reshape, self.orig_layer.bits, imatrix)+ self.init_scale = torch.where(+ self.init_scale < 0,+ torch.clamp(self.init_scale, max=-self.q_scale_thresh),+ torch.clamp(self.init_scale, min=self.q_scale_thresh),+ )+ elif self.orig_layer.data_type.startswith("mx"):+ self.init_scale = mx_init(weight_reshape, self.orig_layer.bits, imatrix)+ elif self.orig_layer.data_type.startswith("nv"):+ self.init_scale = nv_init(weight_reshape, self.orig_layer.bits, imatrix)else:self.init_scale = 1.0+ self.orig_layer.imatrix = None+ delattr(self.orig_layer, "imatrix")# self.weight_quant_func, self.data_type = get_quant_func(orig_layer.data_type, orig_layer.bits, orig_layer.sym)if self.orig_layer.data_type.startswith("int"):@@ -496,8 +492,9 @@ def _register_act_max_hook_ext(self, model):for n, m in model.named_modules():if isinstance(m, self.supported_types) and check_to_quantized(m):- hook = m.register_forward_hook(get_imatrix_hook)- hook_handles.append(hook)+ if not is_wint4aint4(self): # INT4 no imatrix is much better+ hook = m.register_forward_hook(get_imatrix_hook)+ hook_handles.append(hook)if (hasattr(m, "act_dynamic")diff --git a/auto_round/auto_scheme/delta_loss.py b/auto_round/auto_scheme/delta_loss.pyindex 7610e94af..3b88c0e60 100644--- a/auto_round/auto_scheme/delta_loss.py+++ b/auto_round/auto_scheme/delta_loss.py@@ -98,11 +98,11 @@ class AutoSchemeWrapperLinear(WrapperLinear):if self.need_weight_grad:self.orig_layer.weight.requires_grad = True- def _qdq_act(self, x, act_max_scale, act_max=None):+ def _qdq_act(self, x, act_min_scale=1.0, act_max_scale=1.0, act_max=None):if hasattr(self.orig_layer, "act_bits") and self.orig_layer.act_bits > 8:return x, 1.0, None- qdq_x, scale, zp = self.act_qdq_func(x, act_max_scale, act_max)+ qdq_x, scale, zp = self.act_qdq_func(x, act_min_scale, act_max_scale, act_max)if self.grad_mode:with torch.no_grad():self.max_act_value = torch.abs(x).max()diff --git a/auto_round/compressors/base.py b/auto_round/compressors/base.pyindex 18a3f1d89..bbeb85d0d 100644--- a/auto_round/compressors/base.py+++ b/auto_round/compressors/base.py@@ -51,12 +51,14 @@ from auto_round.compressors.utils import (is_nv_fp,is_static_wfp8afp8,is_wfp8afp8,+ is_wint4aint4,reset_params,set_layer_config,)from auto_round.data_type import QUANT_FUNC_WITH_DTYPEfrom auto_round.data_type.utils import reshape_pad_tensor_by_group_size, update_block_global_scale_if_needed-from auto_round.experimental.transform.hadamard_config import HadamardConfig+from auto_round.experimental.transform.rotation_config import RotationConfig+from auto_round.experimental.utils import dump_group_size_to_rotation_config, normalize_rotation_configfrom auto_round.export.export_to_gguf.config import GGUF_INNER_CONFIGfrom auto_round.formats import OutputFormat, get_formatsfrom auto_round.logger import logger@@ -153,7 +155,7 @@ SERIALIZATION_KEYS = ("super_bits","super_group_size","to_quant_block_names",- "hadamard_config",+ "rotation_config",)@@ -204,7 +206,7 @@ class BaseCompressor(object):disable_opt_rtn: bool | None = None,seed: int = 42,low_cpu_mem_usage: bool = True,- hadamard_config: str | dict | HadamardConfig | None = None,+ rotation_config: str | dict | RotationConfig | None = None,**kwargs,):"""Initialize AutoRound with quantization and tuning configuration.@@ -275,7 +277,6 @@ class BaseCompressor(object):value = kwargs.pop(k)if value is not None:self.user_scheme_overrides[k] = value-# Model relatedmodel_dtype = kwargs.pop("model_dtype", None)self.mllm = kwargs.pop("mllm") if "mllm" in kwargs else False@@ -559,14 +560,34 @@ class BaseCompressor(object):except (ImportError, ModuleNotFoundError):logger.error("algorithm extension import error, fallback to default mode")- # apply hadamard transform- if hadamard_config:- logger.info("Applying Hadamard transform to the model.")- from auto_round.experimental.transform.apply import apply_hadamard_transform- from auto_round.experimental.utils import normalize_hadamard_config-- self.hadamard_config = normalize_hadamard_config(hadamard_config, self.data_type)- self.model = apply_hadamard_transform(self.model, self.hadamard_config, data_type=self.data_type)+ # Hadamard rotation / transform – unified entry that auto-dispatches+ # between the "inplace" (QuaRot-style residual rotation; supports any+ # dtype incl. INT4/INT8/FPx; can fuse online Had into weights) and+ # "transform" (per-Linear weight+activation Had with fused triton+ # kernel; only MXFP4 / NVFP4; cannot fuse) backends.+ # See :class:`RotationConfig` for routing rules.+ if rotation_config is not None:+ from auto_round.experimental.apply_rotation_transform import apply_hadamard_rotation++ default_block_size = self.group_size+ if self.act_bits < 16:+ default_block_size = self.act_group_size+ rotation_config = dump_group_size_to_rotation_config(rotation_config, default_block_size)++ self.model, _ = apply_hadamard_rotation(+ self.model,+ rotation_config,+ data_type=self.data_type,+ compute_device=self.device,+ )+ stored = getattr(self.model, "rotation_config", rotation_config)+ # Normalize to a plain dict so it round-trips through JSON+ # (HF saves quantization_config to config.json on save_pretrained).+ if hasattr(stored, "model_dump"):+ stored = stored.model_dump()+ self.rotation_config = stored+ # Mirror the JSON-safe form back onto the model.+ setattr(self.model, "rotation_config", stored)self.blocks_requiring_input_ids = []self.has_variable_block_shape = False@@ -1081,7 +1102,9 @@ class BaseCompressor(object):dtype = f"{dtype}_{'sym' if config['sym'] else 'asym'}"# Optionally use optimized rounding (RTN) variant- if not self.disable_opt_rtn and f"rtn_{dtype}" in QUANT_FUNC_WITH_DTYPE:+ if not self.disable_opt_rtn and f"opt_rtn_{dtype}" in QUANT_FUNC_WITH_DTYPE:+ dtype = f"opt_rtn_{dtype}"+ elif f"rtn_{dtype}" in QUANT_FUNC_WITH_DTYPE:dtype = f"rtn_{dtype}"quant_func = QUANT_FUNC_WITH_DTYPE[dtype]@@ -1194,7 +1217,10 @@ class BaseCompressor(object):hook_handles.append(hook)return hook_handles- hooks = register_act_hook(model)+ if not is_wint4aint4(self): # INT4 no imatrix is much better+ hooks = register_act_hook(model)+ else:+ hooks = []try:if hasattr(model, "hf_device_map") and len(model.hf_device_map) > 1:@@ -1284,6 +1310,7 @@ class BaseCompressor(object):enable_round_tuning=False,enable_torch_compile=self.enable_torch_compile,disable_opt_rtn=disable_opt_rtn,+ iters=self.iters,)m = m.unwrapper({})except torch.OutOfMemoryError:@@ -1299,6 +1326,7 @@ class BaseCompressor(object):enable_norm_bias_tuning=False,enable_round_tuning=False,enable_torch_compile=self.enable_torch_compile,+ iters=self.iters,)m = m.unwrapper({})except Exception as e:@@ -1985,6 +2013,7 @@ class BaseCompressor(object):enable_torch_compile=self.enable_torch_compile,device=self.device,disable_opt_rtn=self.disable_opt_rtn,+ iters=self.iters,)new_layer = wrapper_layer.unwrapper({})set_module(self.model, layer_name, new_layer)@@ -2792,6 +2821,7 @@ class BaseCompressor(object):enable_minmax_tuning=self.enable_minmax_tuning,enable_torch_compile=self.enable_torch_compile,device=device,+ iters=self.iters,).to(device)round_params = []minmax_params = []@@ -3107,6 +3137,7 @@ class BaseCompressor(object):self.enable_norm_bias_tuning,enable_torch_compile=self.enable_torch_compile,device=device,+ iters=self.iters,)# Call this before quantization and after applying the block wrapper.if is_nv_fp(self.data_type): # enable qkv and moe structure global_scale fuse.diff --git a/auto_round/compressors/utils.py b/auto_round/compressors/utils.pyindex 8240c8304..7b6f04545 100644--- a/auto_round/compressors/utils.py+++ b/auto_round/compressors/utils.py@@ -107,7 +107,7 @@ def is_wint_a16(ar: Callable) -> bool:def is_dynamic_wint8aint8(ar_or_format: Union[str, Callable]) -> bool:if isinstance(ar_or_format, str):- return "int8_w8a8" in ar_or_format.lower()+ return "int8_w8a8" in ar_or_format.lower() or "int8" in ar_or_format.lower()if not ar_or_format.act_dynamic:return Falseif is_wint8aint8(ar_or_format):@@ -115,6 +115,16 @@ def is_dynamic_wint8aint8(ar_or_format: Union[str, Callable]) -> bool:return False+def is_wint4aint4(ar_or_scheme: Union[str, Callable]):+ if isinstance(ar_or_scheme, str):+ return "int4" in ar_or_scheme.lower()+ elif (+ "int4" in ar_or_scheme.act_data_type or ("int" in ar_or_scheme.act_data_type and ar_or_scheme.act_bits == 4)+ ) and ("int4" in ar_or_scheme.data_type or ("int" in ar_or_scheme.data_type and ar_or_scheme.bits == 4)):+ return True+ return False++def is_dynamic_afp8(ar_or_format: Callable) -> bool:return ar_or_format.act_dynamic and ar_or_format.act_data_type.startswith("fp") and ar_or_format.act_bits == 8diff --git a/auto_round/data_type/gguf.py b/auto_round/data_type/gguf.pyindex a63c3df90..1a419d880 100644--- a/auto_round/data_type/gguf.py+++ b/auto_round/data_type/gguf.py@@ -40,7 +40,7 @@ def quant_tensor_sym_dq(super_bits=6,**kwargs,):- """Quantize and de-quantize tensor symmetrically. full range, credict goes to llamacpp community+ """Quantize and de-quantize tensor symmetrically. full range, credit goes to llamacpp communityArgs:tensor: Tensor containing the tensor to be quantizeddiff --git a/auto_round/data_type/int.py b/auto_round/data_type/int.pyindex 960a7fc08..73645dd60 100644--- a/auto_round/data_type/int.py+++ b/auto_round/data_type/int.py@@ -15,9 +15,9 @@ from typing import Unionimport torch+from auto_round import envsfrom auto_round.data_type.register import register_dtypefrom auto_round.data_type.utils import reshape_pad_tensor_by_group_size, revert_tensor_by_pad, round_ste-from auto_round.logger import loggerfrom auto_round.utils import get_reciprocal@@ -41,20 +41,27 @@ def search_scales(data: torch.Tensor, bits: int, qw: Union[None, torch.Tensor, f# Set default weight if Noneif qw is None:qw = 1.0-# Compute initial best lossbest_loss = ((scales * L - data).to(torch.float32)) ** 2if isinstance(qw, torch.Tensor):best_loss.mul_(qw) # inplace multiply by weightbest_loss = torch.sum(best_loss, dim=-1)-+ if bits == 2:+ search_min = 18 * 5+ step = 0.01+ else:+ grid = 200+ search_ratio = envs.AR_SEARCH_SCALE_RATIO or 0.75 # default 0.5 -> nmax/2+ search_min = nmax * search_ratio+ step = search_min / grid * 2 # 0.08+ search_min = int(search_min / step)# Iterative search over small adjustments- for _is in range(-18 * 5, 18 * 5 + 1):+ for _is in range(-search_min, search_min + 1):if _is == 0:continue# Update iscales in-place- iscales_tmp = -(nmax - 0.01 * _is) * get_reciprocal(group_max)+ iscales_tmp = -(nmax - step * _is) * get_reciprocal(group_max)# Compute temporary quantized values (in-place round + clamp)tmp_L = torch.empty_like(data)@@ -79,9 +86,9 @@ def search_scales(data: torch.Tensor, bits: int, qw: Union[None, torch.Tensor, freturn scales-@register_dtype("rtn_int_sym")-def quant_tensor_rtn_sym(tensor, bits=4, group_size=-1, v=0, q_scale_thresh=1e-5, imatrix=None, **kwargs):- """Quantize and de-quantize tensor asymmetrically. full range, credict goes to llamacpp community+@register_dtype("opt_rtn_int_sym")+def quant_tensor_opt_rtn_sym(tensor, bits=4, group_size=-1, v=0, q_scale_thresh=1e-5, imatrix=None, **kwargs):+ """Quantize and de-quantize tensor asymmetrically. full range, credit goes to llamacpp communityArgs:tensor: Tensor containing the tensor to be quantized@@ -115,6 +122,46 @@ def quant_tensor_rtn_sym(tensor, bits=4, group_size=-1, v=0, q_scale_thresh=1e-5return qdq_result, scale, maxq+@register_dtype("rtn_int_sym")+def quant_tensor_rtn_sym(+ tensor,+ bits=4,+ group_size=-1,+ q_scale_thresh=1e-5,+ min_scale=1.0,+ max_scale=1.0,+ scale_dtype=torch.float16,+ **kwargs+):+ """Quantize and de-quantize tensor asymmetrically. full range, credit goes to llamacpp community++ Args:+ tensor: Tensor containing the tensor to be quantized+ bits: Number of bits for quantization (e.g., 2, 3, 4, 8)+ group_size: Number of elements to share scale for quantization+ v: Rounding value perturbation+ q_scale_thresh: clip the quantized scale's magnitude to this value to improve the numerical stability++ Returns:+ Quantized and de-quantized tensor, scale, zero-point+ """+ tensor, orig_shape, pad_len = reshape_pad_tensor_by_group_size(tensor, group_size)+ maxq = 2 ** (bits - 1)++ wmin_tmp = torch.clamp(tensor.min(-1)[0], max=0)+ wmax_tmp = torch.clamp(tensor.max(-1)[0], min=0)+ wmin_abs = -(wmin_tmp * min_scale) # pylint: disable=E1130+ wmax_abs = wmax_tmp * max_scale+ max_v = (2 * (wmax_abs < wmin_abs).int() - 1) * torch.max(wmax_abs, wmin_abs)+ scale = (max_v / maxq).to(scale_dtype)+ scale = torch.where(scale < 0, torch.clamp(scale, max=-q_scale_thresh), torch.clamp(scale, min=q_scale_thresh))+ scale = scale.unsqueeze(dim=-1)+ int_w = tensor.div(scale).round_().clamp_(-maxq, maxq - 1)+ qdq_result = (int_w.mul_(scale)).to(tensor.dtype)+ qdq_result = revert_tensor_by_pad(qdq_result, orig_shape=orig_shape, pad_len=pad_len)+ return qdq_result, scale, maxq++@register_dtype("int_sym")def quant_tensor_sym(tensor,@@ -129,7 +176,7 @@ def quant_tensor_sym(q_scale_thresh=1e-5,**kwargs):- """Quantize and de-quantize tensor asymmetrically. full range, credict goes to llamacpp community+ """Quantize and de-quantize tensor asymmetrically. full range, credit goes to llamacpp communityArgs:tensor: Tensor containing the tensor to be quantizeddiff --git a/auto_round/data_type/utils.py b/auto_round/data_type/utils.pyindex a8e3eb218..bd4d74eaa 100644--- a/auto_round/data_type/utils.py+++ b/auto_round/data_type/utils.py@@ -101,7 +101,9 @@ def revert_tensor_by_pad(data: torch.Tensor, orig_shape: tuple, pad_len: Union[ireturn data_new-def get_quant_func(dtype: str, bits: int, sym: bool, disable_opt_rtn=False, group_size=None) -> tuple[callable, str]:+def get_quant_func(+ dtype: str, bits: int, sym: bool, disable_opt_rtn=False, group_size=None, iters=200+) -> tuple[callable, str]:"""Retrieve the quantization function based on data type, bit width, and symmetry.This function returns the appropriate quantization function from the QUANT_FUNC_WITH_DTYPE@@ -130,7 +132,15 @@ def get_quant_func(dtype: str, bits: int, sym: bool, disable_opt_rtn=False, groudef pad_bits(data_type):return data_type + str(bits)- if not disable_opt_rtn:+ if not disable_opt_rtn and iters == 0:+ rtn_data_type = "opt_rtn_" + dtype+ data_types = [rtn_data_type, pad_bits(rtn_data_type), pad_sym(rtn_data_type), pad_sym(pad_bits(rtn_data_type))]+ for data_type in data_types:+ from auto_round.data_type import QUANT_FUNC_WITH_DTYPE++ if data_type in QUANT_FUNC_WITH_DTYPE:+ return QUANT_FUNC_WITH_DTYPE[data_type], data_type+ if iters == 0:rtn_data_type = "rtn_" + dtypedata_types = [rtn_data_type, pad_bits(rtn_data_type), pad_sym(rtn_data_type), pad_sym(pad_bits(rtn_data_type))]for data_type in data_types:diff --git a/auto_round/envs.py b/auto_round/envs.pyindex 8a731ad10..a1a0e4f0d 100644--- a/auto_round/envs.py+++ b/auto_round/envs.py@@ -34,6 +34,16 @@ environment_variables: dict[str, Callable[[], Any]] = {"AR_DISABLE_DATASET_SUBPROCESS": lambda: os.getenv("AR_DISABLE_DATASET_SUBPROCESS", "0").lower() in ("1", "true"),"AR_DISABLE_COPY_MTP_WEIGHTS": lambda: os.getenv("AR_DISABLE_COPY_MTP_WEIGHTS", "0").lower()in ("1", "true", "yes"),+ "AR_ACT_SCALE": lambda: float(os.getenv("AR_ACT_SCALE", "1.0")),+ "AR_ENABLE_ACT_MINMAX_TUNING": lambda: os.getenv("AR_ENABLE_ACT_MINMAX_TUNING", "0").lower()+ in ("1", "true", "yes"),+ "AR_FUSE_ONLINE_ROTATION": lambda: os.getenv("AR_FUSE_ONLINE_ROTATION", "0").lower() in ("1", "true", "yes"),+ # Controls the search range ratio for symmetric int scale search in+ # `auto_round.data_type.int.search_scales`. The search bound is+ # `nmax * AR_SEARCH_SCALE_RATIO` (default None).+ "AR_SEARCH_SCALE_RATIO": lambda: (+ float(os.getenv("AR_SEARCH_SCALE_RATIO")) if os.getenv("AR_SEARCH_SCALE_RATIO") is not None else None+ ),}diff --git a/auto_round/eval/evaluation.py b/auto_round/eval/evaluation.pyindex d4c4ec917..b3136c32f 100644--- a/auto_round/eval/evaluation.py+++ b/auto_round/eval/evaluation.py@@ -428,7 +428,7 @@ def run_model_evaluation(model, tokenizer, autoround, folders, formats, device_seval_gguf_model = any(file.endswith("gguf") for file in os.listdir(eval_folder))# Determine if model instance evaluation is needed- need_model_instance = (autoround.act_bits <= 8 and formats[-1] == "fake") or eval_gguf_model+ need_model_instance = formats[-1] == "fake" or eval_gguf_modelif need_model_instance:# Load or prepare model instancediff --git a/auto_round/experimental/apply_rotation_transform.py b/auto_round/experimental/apply_rotation_transform.pynew file mode 100644index 000000000..fc7dbf297--- /dev/null+++ b/auto_round/experimental/apply_rotation_transform.py@@ -0,0 +1,154 @@+# Copyright (c) 2026 Intel Corporation+#+# Licensed under the Apache License, Version 2.0 (the "License");+# you may not use this file except in compliance with the License.+# You may obtain a copy of the License at+#+# http://www.apache.org/licenses/LICENSE-2.0++"""Unified entry point for Hadamard rotation/transform.++Two backend implementations exist:++* ``inplace`` – :mod:`auto_round.experimental.rotation_inplace`+ QuaRot-style residual-stream rotation. Works for any weight/activation+ dtype. Optionally fuses the online Hadamard into weights+ (``fuse_online_to_weight=True``).+* ``transform`` – :mod:`auto_round.experimental.transform`+ Per-Linear weight + activation Hadamard with a fused triton kernel.+ Only supports MXFP4 / NVFP4 and **cannot** fuse online to weight.++Routing is controlled by :class:`RotationConfig.backend`:++ "inplace" -> always inplace+ "transform" -> always transform (validates dtype + no-fuse)+ "auto" -> if user asked to fuse -> inplace+ elif data_type is mx_fp / nv_fp -> transform+ else -> inplace+"""++from __future__ import annotations++from typing import Any, Union++import torch++import auto_round.envs as envs+from auto_round.compressors.utils import is_mx_fp, is_nv_fp+from auto_round.experimental.transform.rotation_config import RotationConfig+from auto_round.experimental.utils import normalize_rotation_config+from auto_round.utils import logger++__all__ = ["apply_hadamard_rotation", "resolve_hadamard_backend"]+++def _to_config(+ rotation_config: Union[str, dict, RotationConfig, None],+ data_type: str,+) -> RotationConfig:+ """Normalise *rotation_config* and return a :class:`RotationConfig` instance."""+ cfg_dict = normalize_rotation_config(rotation_config, data_type)+ if isinstance(cfg_dict, RotationConfig):+ return cfg_dict+ return RotationConfig.model_validate(cfg_dict or {})+++def resolve_hadamard_backend(config: RotationConfig, data_type: str) -> str:+ """Resolve the actual backend (``"inplace"`` / ``"transform"``) from config."""+ requested = config.backend+ fuse_requested = bool(config.fuse_online_to_weight)+ allow_online_rotation: bool = config.allow_online_rotation++ if requested == "inplace":+ return "inplace"++ transform_backend_name = "transform"+ if requested == "transform":+ if fuse_requested:+ raise ValueError(+ f"backend='{transform_backend_name}' does not support fuse_online_to_weight=True. "+ "Use backend='inplace' (or backend='auto' with fuse_online_to_weight=True) instead."+ )+ if not (is_mx_fp(data_type) or is_nv_fp(data_type)):+ raise ValueError(+ f"backend='{transform_backend_name}' only supports MXFP4 / NVFP4 (got data_type={data_type!r}). "+ "Use backend='inplace' or backend='auto' for other dtypes."+ )+ if not allow_online_rotation:+ raise ValueError(f"backend='{transform_backend_name}' only supports `allow_online_rotation`=True")++ return "transform"++ # backend == "auto"+ if fuse_requested:+ return "inplace"+ if is_mx_fp(data_type) or is_nv_fp(data_type):+ return "transform"+ return "inplace"+++def apply_hadamard_rotation(+ model: torch.nn.Module,+ rotation_config: Union[str, dict, RotationConfig, None],+ data_type: str,+ compute_device: torch.device | str = None,+) -> (torch.nn.Module, Any):+ """Apply Hadamard rotation/transform to *model*, dispatching by backend.++ Args:+ model: Target model.+ rotation_config: ``str`` / ``dict`` / :class:`RotationConfig` / ``None``.+ See :class:`RotationConfig` for fields.+ data_type: Quantization data type (e.g. ``"mx_fp"``, ``"nv_fp"``,+ ``"int"``, ``"fp"``).+ compute_device: Device for inplace-backend computation. Ignored by+ the transform backend.++ Returns:+ The same model (for chaining); also stored on ``model.rotation_config``.+ """+ config = _to_config(rotation_config, data_type)+ backend = resolve_hadamard_backend(config, data_type)++ # Resolve fuse flag: explicit > env var > default(True)+ fuse_online_to_weight = config.fuse_online_to_weight+ if config.fuse_online_to_weight is not None:+ fuse_online_to_weight = bool(config.fuse_online_to_weight)+ elif envs.AR_FUSE_ONLINE_ROTATION:+ fuse_online_to_weight = bool(envs.AR_FUSE_ONLINE_ROTATION)++ logger.info(+ f"Applying Hadamard (backend={backend}, "+ f"data_type={data_type}, fuse_online_to_weight={fuse_online_to_weight if backend == 'inplace' else False})."+ )++ if backend == "inplace":+ logger.warning("this backend does not support real exporting, please export the model to fake format")+ from auto_round.experimental.rotation_inplace import apply_rotation_transform++ # block_size -> group_size (None / -1 / 0 means full-dimension)+ bs = config.block_size+ group_size = bs if (bs is not None and bs > 0) else None++ model, hooks = apply_rotation_transform(+ model,+ group_size=group_size,+ allow_online_rotation=config.allow_online_rotation,+ rotation_matrix=config.hadamard_type,+ fuse_online_to_weight=fuse_online_to_weight,+ compute_device=compute_device,+ )+ # Stash for downstream (export / serialization). Plain dict so JSON+ # serialization (HF save_pretrained -> config.json) round-trips.+ setattr(model, "rotation_config", config.model_dump() if hasattr(config, "model_dump") else config)+ return model, hooks++ elif backend == "transform":+ supported_hadamard_types = ("hadamard", "random_hadamard")+ if config.hadamard_type not in supported_hadamard_types:+ raise ValueError("this backend only supports hadamard or random_hadamard")+ from auto_round.experimental.transform.apply import apply_rotation_transform++ return apply_rotation_transform(model, config, data_type=data_type)+ else:+ raise ValueError(f"Unsupported Hadamard backend {backend!r}")diff --git a/auto_round/experimental/qmodules/mx.py b/auto_round/experimental/qmodules/mx.pyindex 23d05872c..dea099760 100644--- a/auto_round/experimental/qmodules/mx.py+++ b/auto_round/experimental/qmodules/mx.py@@ -96,9 +96,9 @@ class MXQuantLinearBase(QModuleBase):)self.register_buffer("weight_scale", init_weight_scale)- hadamard_config = getattr(config, "hadamard_config", None)- # TODO: remove the limit: hadamard_config["hadamard_type"] == "random_hadamard"- if hadamard_config is not None and hadamard_config["hadamard_type"] == "random_hadamard":+ rotation_config = getattr(config, "rotation_config", None)+ # TODO: remove the limit: rotation_config["hadamard_type"] == "random_hadamard"+ if rotation_config is not None and rotation_config["hadamard_type"] == "random_hadamard":self.enable_transform = Trueself.register_buffer("hadamard_matrix",diff --git a/auto_round/experimental/qmodules/nvfp4.py b/auto_round/experimental/qmodules/nvfp4.pyindex 4350f0567..dafeb2567 100644--- a/auto_round/experimental/qmodules/nvfp4.py+++ b/auto_round/experimental/qmodules/nvfp4.py@@ -119,9 +119,9 @@ class NVFP4QuantLinear(QModuleBase):),)- hadamard_config = getattr(config, "hadamard_config", None)- # TODO: remove the limit: hadamard_config["hadamard_type"] == "random_hadamard"- if hadamard_config is not None and hadamard_config["hadamard_type"] == "random_hadamard":+ rotation_config = getattr(config, "rotation_config", None)+ # TODO: remove the limit: rotation_config["hadamard_type"] == "random_hadamard"+ if rotation_config is not None and rotation_config["hadamard_type"] == "random_hadamard":self.enable_transform = Trueself.register_buffer("hadamard_matrix",diff --git a/auto_round/experimental/rotation_inplace/__init__.py b/auto_round/experimental/rotation_inplace/__init__.pynew file mode 100644index 000000000..8cdef31b0--- /dev/null+++ b/auto_round/experimental/rotation_inplace/__init__.py@@ -0,0 +1,5 @@+# # Copyright (C) 2026 Intel Corporation+# # SPDX-License-Identifier: Apache-2.0++from auto_round.experimental.rotation_inplace.apply_rotation_transform import apply_rotation_transform+from auto_round.experimental.rotation_inplace.utils import clear_random_hadamard_cachediff --git a/auto_round/experimental/rotation_inplace/apply_rotation_transform.py b/auto_round/experimental/rotation_inplace/apply_rotation_transform.pynew file mode 100644index 000000000..1052b036f--- /dev/null+++ b/auto_round/experimental/rotation_inplace/apply_rotation_transform.py@@ -0,0 +1,882 @@+# # Copyright (C) 2026 Intel Corporation+# # SPDX-License-Identifier: Apache-2.0++"""Hadamard inplace rotation — public API and rotation primitives.++Supports LLaMA-2, LLaMA-3, Qwen-3 (and any model with the same layout).+The entry point is :func:`apply_hadamard_rotation`.+"""++import gc+import typing+from typing import Dict, Union++import torch+import tqdm++from auto_round.experimental.rotation_inplace.model_config import (+ MAPPING_REGISTRY,+ RotationMapping,+ _resolve,+ infer_mapping_from_model,+)+from auto_round.experimental.rotation_inplace.utils import (+ CrossHeadOnlineHadamardHook,+ FullOnlineHadamardHook,+ GroupOnlineHadamardHook,+ _get_custom_had,+ _normalize_rotation_matrix,+ _resolve_compute_device,+ _rotate_embedding_grouped,+ _rotate_linear_grouped,+ apply_cross_head_had_to_linear,+ apply_exact_had_to_linear,+ deterministic_hadamard_matrix,+ get_hadK,+ get_or_create_random_hadamard,+)++# ---------------------------------------------------------------------------+# Low-level primitives (model-agnostic via RotationMapping)+# ---------------------------------------------------------------------------+++def _fuse_ln_linear(+ layernorm: torch.nn.Module,+ linear_layers: typing.Iterable[torch.nn.Linear],+) -> None:+ """Fuse the linear operations in LayerNorm into adjacent linear blocks."""+ for linear in linear_layers:+ linear_dtype = linear.weight.dtype+ dev = linear.weight.device++ W_ = linear.weight.data.double()+ ln_weight = layernorm.weight.double().to(dev)+ linear.weight.data = (W_ * ln_weight).to(linear_dtype)++ if hasattr(layernorm, "bias") and layernorm.bias is not None:+ if linear.bias is None:+ linear.bias = torch.nn.Parameter(torch.zeros(linear.out_features, dtype=torch.float64, device=dev))+ ln_bias = layernorm.bias.double().to(dev)+ linear.bias.data = linear.bias.data.double() + torch.matmul(W_, ln_bias)+ linear.bias.data = linear.bias.data.to(linear_dtype)+++def _reset_ln_params(layernorm: torch.nn.Module) -> None:+ """Reset LayerNorm to identity: weight=1, bias=0."""+ layernorm.weight.data.fill_(1.0)+ if hasattr(layernorm, "bias") and layernorm.bias is not None:+ layernorm.bias.data.fill_(0.0)+++def _rotate_linear_by_Q(module: torch.nn.Linear, Q: torch.Tensor, side: str, compute_device=None) -> None:+ """Apply rotation *Q* to a Linear layer's weight (and bias if present).++ Args:+ side: ``'input'`` → W = W @ Q (rotate input side)+ ``'output'`` → W = Q^T @ W (rotate output side)+ compute_device: Device to run computation on. If None, auto-detects GPU.+ """+ dtype = module.weight.data.dtype+ dev = module.weight.data.device+ cdev = _resolve_compute_device(compute_device)+ W_ = module.weight.data.to(device=cdev, dtype=torch.float64)+ Q_ = Q.to(device=cdev)+ if side == "input":+ new_W = torch.matmul(W_, Q_).to(device=dev, dtype=dtype)+ else:+ new_W = torch.matmul(Q_.T, W_).to(device=dev, dtype=dtype)+ # Release fp64 copy before assigning back so peak memory ≈ 1× weight + 1× rotated.+ del W_+ module.weight.data = new_W+ if side == "output" and module.bias is not None:+ b = module.bias.data.to(device=cdev, dtype=torch.float64)+ new_b = torch.matmul(Q_.T, b).to(device=dev, dtype=dtype)+ del b+ module.bias.data = new_b+ del Q_+++def _untie_word_embeddings(model, mapping: RotationMapping) -> None:+ """Break tied weights between lm_head and embedding if they share the same tensor."""+ embedding = _resolve(model, mapping.embedding)+ lm_head = _resolve(model, mapping.lm_head)++ if lm_head.weight.data_ptr() != embedding.weight.data_ptr():+ return++ lm_head.weight = torch.nn.Parameter(lm_head.weight.data.clone())+ if hasattr(model.config, "tie_word_embeddings"):+ model.config.tie_word_embeddings = False+++def _uses_layernorm_with_mean(model, mapping: RotationMapping) -> bool:+ """Check whether the model uses standard LayerNorm (which subtracts mean)."""+ layers = _resolve(model, mapping.layers_attr)+ first_ln = _resolve(layers[0], mapping.attn_input_ln)+ return isinstance(first_ln, torch.nn.LayerNorm)+++def _bake_mean_into_linear(linear: torch.nn.Linear) -> None:+ """Subtract column-wise mean from a Linear layer's weight (and mean from bias)."""+ linear_dtype = linear.weight.dtype+ W_ = linear.weight.data.double()+ linear.weight.data = (W_ - W_.mean(dim=-2, keepdim=True)).to(linear_dtype)+ if linear.bias is not None:+ b_ = linear.bias.data.double()+ linear.bias.data = (b_ - b_.mean()).to(linear_dtype)+++def _subtract_embedding_mean(model, mapping: RotationMapping) -> None:+ """Subtract per-row mean from the embedding weight matrix."""+ W = _resolve(model, mapping.embedding)+ dtype = W.weight.data.dtype+ W_ = W.weight.data.to(dtype=torch.float64)+ W.weight.data = (W_ - W_.mean(dim=-1, keepdim=True)).to(dtype=dtype)++ if mapping.positional_embedding is not None:+ P = _resolve(model, mapping.positional_embedding)+ p_dtype = P.weight.data.dtype+ P_ = P.weight.data.to(dtype=torch.float64)+ P.weight.data = (P_ - P_.mean(dim=-1, keepdim=True)).to(dtype=p_dtype)+++class _RMSNorm(torch.nn.Module):+ """RMS Normalization (no mean subtraction)."""++ def __init__(self, dim: int, eps: float = 1e-5):+ super().__init__()+ self.eps = eps+ self.register_buffer("weight", torch.ones(dim))++ def forward(self, x: torch.Tensor) -> torch.Tensor:+ rms = torch.sqrt(x.pow(2).mean(dim=-1, keepdim=True) + self.eps)+ return x / rms * self.weight+++def _replace_layernorms_with_rmsnorm(model) -> None:+ """Replace all ``nn.LayerNorm`` modules with ``_RMSNorm``."""+ replacements = []+ for name, module in model.named_modules():+ if isinstance(module, torch.nn.LayerNorm):+ replacements.append((name, module))++ for name, module in replacements:+ parts = name.rsplit(".", 1)+ if len(parts) == 2:+ parent = _resolve(model, parts[0])+ attr = parts[1]+ else:+ parent = model+ attr = parts[0]+ rms = _RMSNorm(module.normalized_shape[0], eps=module.eps)+ rms = rms.to(device=module.weight.device, dtype=module.weight.dtype)+ setattr(parent, attr, rms)+++# ---------------------------------------------------------------------------+# High-level steps driven by RotationMapping+# ---------------------------------------------------------------------------+++def _fuse_layer_norms(model, mapping: RotationMapping) -> None:+ """Fuse all LayerNorm parameters into adjacent Linear layers."""+ layers = _resolve(model, mapping.layers_attr)++ for layer in layers:+ mlp_ln = _resolve(layer, mapping.mlp_input_ln)+ mlp_linears = [_resolve(layer, p) for p in mapping.mlp_in]+ _fuse_ln_linear(mlp_ln, mlp_linears)+ _reset_ln_params(mlp_ln)++ attn_ln = _resolve(layer, mapping.attn_input_ln)+ attn_linears = [+ _resolve(layer, mapping.attn_q),+ _resolve(layer, mapping.attn_k),+ _resolve(layer, mapping.attn_v),+ ]+ _fuse_ln_linear(attn_ln, attn_linears)+ _reset_ln_params(attn_ln)++ pre_head_ln = _resolve(model, mapping.pre_head_ln)+ lm_head = _resolve(model, mapping.lm_head)+ _fuse_ln_linear(pre_head_ln, [lm_head])+ _reset_ln_params(pre_head_ln)+++# ---------------------------------------------------------------------------+# Unified weight rotation (full or grouped)+# ---------------------------------------------------------------------------++[email protected]_mode()+def _rotate_weights(+ model,+ mapping: RotationMapping,+ use_fast_had: bool = True,+ group_size: int = None,+ compute_device: torch.device = None,+ had_dict: dict = None,+ preset: str = None,+ fuse_online_to_weight: bool = True,+) -> None:+ """Apply Hadamard rotation to all weights.++ Args:+ group_size: ``None`` → full Hadamard rotation.+ ``int`` → block-diagonal rotation with this block size.+ compute_device: Device to run Hadamard computation on (e.g. ``"cuda:0"``).+ Weights are moved there temporarily and moved back afterwards.+ If ``None``, auto-detects GPU availability.+ allow_online_rotation: If ``True`` (default), apply extra input-side+ Hadamard rotations on ``down_proj`` and the OV pair (``v_proj``+ output + ``o_proj`` input) that require compensating online hooks+ at inference time. If ``False``, skip those extra rotations so+ that **no** online hooks are needed.+ had_dict: Normalized ``dict[int, Tensor]`` of custom Hadamard matrices+ (keyed by dimension). Only used in grouped mode.+ preset: Rotation preset name (``"quarot_hadamard"``, ``"hadamard"``,+ ``"random_hadamard"``, or ``None``).++ * ``"quarot_hadamard"``: fusable (residual-stream) rotations use+ ``fast_hadamard_transform`` / random Hadamard; non-fusable+ (online-paired) rotations and their weight-side counterparts use+ deterministic ``get_hadK``/``matmul_hadU`` so that the online+ hook at inference produces the exact same transform.+ * ``"hadamard"``: all rotations use deterministic ``get_hadK`` /+ ``matmul_hadU``. Full-mode Q is a deterministic Hadamard matrix.+ * ``"random_hadamard"``: all rotations use random Hadamard matrices+ from the global cache (``get_or_create_random_hadamard``).+ Same dimension → same matrix everywhere.+ * ``None``: same behaviour as ``"hadamard"`` (built-in butterfly).+ """+ compute_device = _resolve_compute_device(compute_device)+ config = model.config+ hidden_size = getattr(config, mapping.hidden_size_attr)+ intermediate_size = getattr(config, mapping.intermediate_size_attr)+ num_heads = getattr(config, mapping.num_heads_attr)+ head_dim = mapping.attn_head_dim or (hidden_size // num_heads)++ is_grouped = group_size is not None and group_size > 0+ desc = f"Rotating (group_size={group_size})" if is_grouped else "Rotating"++ # ----- Resolve per-operation Hadamard sources -----+ fused_fast = use_fast_had+ online_fast = False+ if preset == "random_hadamard":+ fused_fast = False++ # -- Matrix resolution --+ had_matrix, _found = _get_custom_had(had_dict, group_size) if is_grouped else (None, False)++ online_had_matrix = had_matrix+ if preset == "random_hadamard" and had_matrix is None:+ had_matrix = get_or_create_random_hadamard(group_size if is_grouped else hidden_size, compute_device)+ online_had_matrix = had_matrix+ if preset == "quarot_hadamard" and is_grouped:+ online_had_matrix = None # force deterministic for online-paired++ # -- Helper: look up cached random matrix for online-paired ops --+ def _online_had(dim):+ """Return cached random matrix for *dim* under random_hadamard, else None."""+ if preset == "random_hadamard":+ return get_or_create_random_hadamard(dim, compute_device)+ return None++ if is_grouped:+ assert hidden_size % group_size == 0, f"group_size={group_size} must divide hidden_size={hidden_size}"+ assert (+ intermediate_size % group_size == 0+ ), f"group_size={group_size} must divide intermediate_size={intermediate_size}"++ # --- Full mode: build Hadamard matrix Q ---+ Q = None+ if not is_grouped:+ if preset == "hadamard":+ Q = deterministic_hadamard_matrix(hidden_size, compute_device)+ else:+ # "random_hadamard", "quarot_hadamard", None — same shape → same matrix+ Q = get_or_create_random_hadamard(hidden_size, compute_device)++ # ---- Top-level: embedding / lm_head ----+ # When fuse_online_to_weight=False, skip embedding and lm_head rotation:+ # each layer is self-contained (weight rotation + online hook cancel out).+ if fuse_online_to_weight:+ embedding = _resolve(model, mapping.embedding)+ if is_grouped:+ _rotate_embedding_grouped(+ embedding, group_size, use_fast_had=fused_fast, compute_device=compute_device, had_matrix=had_matrix+ )+ else:+ dtype = embedding.weight.data.dtype+ dev = embedding.weight.data.device+ cdev = compute_device+ W_ = embedding.weight.data.to(device=cdev, dtype=torch.float64)+ new_W = torch.matmul(W_, Q.to(cdev)).to(device=dev, dtype=dtype)+ del W_+ embedding.weight.data = new_W++ if mapping.positional_embedding is not None:+ pos_emb = _resolve(model, mapping.positional_embedding)+ if is_grouped:+ _rotate_embedding_grouped(+ pos_emb, group_size, use_fast_had=fused_fast, compute_device=compute_device, had_matrix=had_matrix+ )+ else:+ pos_dtype = pos_emb.weight.data.dtype+ pos_dev = pos_emb.weight.data.device+ cdev = compute_device+ P_ = pos_emb.weight.data.to(device=cdev, dtype=torch.float64)+ new_P = torch.matmul(P_, Q.to(cdev)).to(device=pos_dev, dtype=pos_dtype)+ del P_+ pos_emb.weight.data = new_P++ # ---- Top-level: lm_head ----+ lm_head = _resolve(model, mapping.lm_head)+ if is_grouped:+ _rotate_linear_grouped(+ lm_head,+ group_size,+ side="input",+ use_fast_had=fused_fast,+ compute_device=compute_device,+ had_matrix=had_matrix,+ )+ else:+ _rotate_linear_by_Q(lm_head, Q, side="input", compute_device=compute_device)++ gc.collect()+ if torch.cuda.is_available():+ torch.cuda.empty_cache()++ # ---- Per-layer rotation ----+ layers = _resolve(model, mapping.layers_attr)+ for layer in tqdm.tqdm(layers, unit="layer", desc=desc):+ if fuse_online_to_weight:+ # ---- fuse mode: QuaRot-style residual stream rotation ----+ # Q/K/V: only residual Q on input (no online Had stacking, no hook).+ # When Q == online Had (e.g. preset="hadamard"), Q @ Q = I cancels+ # the rotation entirely, destroying quantization benefit.+ # gate/up: only residual Q on input (no online Had stacking, no hook).+ # down_proj: residual Q^T on output + online Had on input (+ hook).+ # v_proj/o_proj: per-head/cross-head Had below (+ hook on o_proj).+ for attr in (mapping.attn_q, mapping.attn_k, mapping.attn_v):+ mod = _resolve(layer, attr)+ if is_grouped:+ _rotate_linear_grouped(+ mod,+ group_size,+ side="input",+ use_fast_had=fused_fast,+ compute_device=compute_device,+ had_matrix=had_matrix,+ )+ else:+ _rotate_linear_by_Q(mod, Q, side="input", compute_device=compute_device)++ # o_proj: residual stream output rotation+ if is_grouped:+ _rotate_linear_grouped(+ _resolve(layer, mapping.attn_o),+ group_size,+ side="output",+ use_fast_had=fused_fast,+ compute_device=compute_device,+ had_matrix=had_matrix,+ )+ else:+ _rotate_linear_by_Q(_resolve(layer, mapping.attn_o), Q, side="output", compute_device=compute_device)++ # gate/up: only residual Q on input+ for attr in mapping.mlp_in:+ mod = _resolve(layer, attr)+ if is_grouped:+ _rotate_linear_grouped(+ mod,+ group_size,+ side="input",+ use_fast_had=fused_fast,+ compute_device=compute_device,+ had_matrix=had_matrix,+ )+ else:+ _rotate_linear_by_Q(mod, Q, side="input", compute_device=compute_device)++ # down_proj: residual output + online input Had+ down_proj = _resolve(layer, mapping.mlp_out)+ if is_grouped:+ _rotate_linear_grouped(+ down_proj,+ group_size,+ side="output",+ use_fast_had=fused_fast,+ compute_device=compute_device,+ had_matrix=had_matrix,+ )+ _rotate_linear_grouped(+ down_proj,+ group_size,+ side="input",+ use_fast_had=online_fast,+ compute_device=compute_device,+ had_matrix=online_had_matrix,+ )+ else:+ _rotate_linear_by_Q(down_proj, Q, side="output", compute_device=compute_device)+ apply_exact_had_to_linear(+ down_proj,+ had_dim=-1,+ output=False,+ use_fast_had=online_fast,+ compute_device=compute_device,+ had_matrix=_online_had(intermediate_size),+ )++ # OV projection: v_proj per-head output + o_proj full/cross-head input+ v_proj = _resolve(layer, mapping.attn_v)+ o_proj = _resolve(layer, mapping.attn_o)+ if is_grouped:+ pass+ else:+ online_head_had = _online_had(head_dim)+ apply_exact_had_to_linear(+ v_proj,+ had_dim=head_dim,+ output=True,+ use_fast_had=online_fast,+ compute_device=compute_device,+ had_matrix=online_head_had,+ )+ if preset == "random_hadamard":+ apply_exact_had_to_linear(+ o_proj,+ had_dim=head_dim,+ output=False,+ use_fast_had=online_fast,+ compute_device=compute_device,+ had_matrix=online_head_had,+ )+ apply_cross_head_had_to_linear(+ o_proj,+ num_heads,+ head_dim,+ use_fast_had=online_fast,+ compute_device=compute_device,+ had_matrix=_online_had(num_heads),+ )+ else:+ apply_exact_had_to_linear(+ o_proj,+ had_dim=-1,+ output=False,+ use_fast_had=online_fast,+ compute_device=compute_device,+ )++ else:+ # ---- unfused mode: no residual rotation, only input-side Had ----+ # Each layer gets Had fused on input side + compensating hook → equivalent.+ # No embedding/lm_head rotation. No self-cancelling pair.+ # v_proj treated same as Q/K (input Had only, no per-head/cross-head).++ # Q/K/V: input-side Had on hidden_size+ for attr in (mapping.attn_q, mapping.attn_k, mapping.attn_v):+ mod = _resolve(layer, attr)+ if is_grouped:+ _rotate_linear_grouped(+ mod,+ group_size,+ side="input",+ use_fast_had=online_fast,+ compute_device=compute_device,+ had_matrix=online_had_matrix,+ )+ else:+ apply_exact_had_to_linear(+ mod,+ had_dim=-1,+ output=False,+ use_fast_had=online_fast,+ compute_device=compute_device,+ had_matrix=_online_had(hidden_size),+ )++ # o_proj: input-side Had on hidden_size (full Had, not cross-head)+ o_proj = _resolve(layer, mapping.attn_o)+ if is_grouped:+ _rotate_linear_grouped(+ o_proj,+ group_size,+ side="input",+ use_fast_had=online_fast,+ compute_device=compute_device,+ had_matrix=online_had_matrix,+ )+ else:+ apply_exact_had_to_linear(+ o_proj,+ had_dim=-1,+ output=False,+ use_fast_had=online_fast,+ compute_device=compute_device,+ had_matrix=_online_had(hidden_size),+ )++ # gate/up: input-side Had on hidden_size+ for attr in mapping.mlp_in:+ mod = _resolve(layer, attr)+ if is_grouped:+ _rotate_linear_grouped(+ mod,+ group_size,+ side="input",+ use_fast_had=online_fast,+ compute_device=compute_device,+ had_matrix=online_had_matrix,+ )+ else:+ apply_exact_had_to_linear(+ mod,+ had_dim=-1,+ output=False,+ use_fast_had=online_fast,+ compute_device=compute_device,+ had_matrix=_online_had(hidden_size),+ )++ # down_proj: input-side Had on intermediate_size+ down_proj = _resolve(layer, mapping.mlp_out)+ if is_grouped:+ _rotate_linear_grouped(+ down_proj,+ group_size,+ side="input",+ use_fast_had=online_fast,+ compute_device=compute_device,+ had_matrix=online_had_matrix,+ )+ else:+ apply_exact_had_to_linear(+ down_proj,+ had_dim=-1,+ output=False,+ use_fast_had=online_fast,+ compute_device=compute_device,+ had_matrix=_online_had(intermediate_size),+ )++ # Per-layer cleanup: drop fp64 temporaries and CUDA caching allocator+ # blocks so peak memory stays at ~1 layer's worth instead of accumulating+ # across all 32+ decoder layers (was the main cause of 33 GB RAM on 8B).+ gc.collect()+ if torch.cuda.is_available():+ torch.cuda.empty_cache()+++# ---------------------------------------------------------------------------+# Unified online hook registration+# ---------------------------------------------------------------------------+++def _register_online_hooks(+ model,+ mapping: RotationMapping,+ fp32_had: bool = False,+ use_fast_had: bool = True,+ group_size: int = None,+ had_dict: dict = None,+ preset: str = None,+ fuse_online_to_weight: bool = True,+):+ """Register online Hadamard pre-forward hooks on ``down_proj`` and ``o_proj``.++ Online hooks must use the **same** Hadamard matrix that was applied to the+ weight-side counterpart during ``_rotate_weights``. For ``quarot_hadamard``+ this is always the deterministic ``get_hadK``/``matmul_hadU`` path+ (``use_fast_had=False``). For ``"random_hadamard"`` it is the random matrix that+ was generated once and stored in ``had_dict``.++ Args:+ group_size: ``None`` → full Hadamard hooks (original QuaRot).+ ``int`` → per-group Hadamard hooks.+ had_dict: Normalized ``dict[int, Tensor]`` of custom Hadamard matrices.+ preset: Rotation preset name.+ Returns:+ list of hook handles.+ """+ config = model.config+ num_heads = getattr(config, mapping.num_heads_attr)+ hidden_size = getattr(config, mapping.hidden_size_attr)+ intermediate_size = getattr(config, mapping.intermediate_size_attr)+ head_dim = mapping.attn_head_dim or (hidden_size // num_heads)++ is_grouped = group_size is not None and group_size > 0++ # Online hooks always use deterministic (fixed) Hadamard — never fast_had+ # for quarot_hadamard; for "random_hadamard" they use the same random matrix+ # that was cached in had_dict by _rotate_weights.+ online_fast = False++ # -- Matrix resolution (must match the *online-paired* matrix used by+ # _rotate_weights for down_proj input / OV pair). Variable name kept in+ # sync with _rotate_weights to make any future drift obvious.+ online_had_matrix, _ = _get_custom_had(had_dict, group_size) if is_grouped else (None, False)+ if preset == "random_hadamard" and online_had_matrix is None:+ online_had_matrix = get_or_create_random_hadamard(group_size if is_grouped else hidden_size)+ if preset == "quarot_hadamard" and is_grouped:+ online_had_matrix = None++ # -- Helper: look up cached random matrix for online-paired hooks --+ def _online_had(dim):+ if preset == "random_hadamard":+ return get_or_create_random_hadamard(dim)+ return None++ mlp_out_suffix = mapping.mlp_out.split(".")[-1]+ attn_o_suffix = mapping.attn_o.split(".")[-1]++ # Suffixes for Q/K/V and gate/up (for online input Had hooks)+ attn_qkv_suffixes = set(attr.split(".")[-1] for attr in (mapping.attn_q, mapping.attn_k, mapping.attn_v))+ mlp_in_suffixes = set(attr.split(".")[-1] for attr in mapping.mlp_in)++ # --- Build hook factories ---+ def _make_down_proj_hook():+ if is_grouped:+ return GroupOnlineHadamardHook(+ group_size=group_size, fp32_had=fp32_had, use_fast_had=online_fast, had_matrix=online_had_matrix+ )+ online_mat = _online_had(intermediate_size)+ if online_mat is not None:+ return FullOnlineHadamardHook(+ had_K=None, K=None, fp32_had=fp32_had, use_fast_had=online_fast, had_matrix=online_mat+ )+ had_K, K = get_hadK(intermediate_size)+ return FullOnlineHadamardHook(had_K=had_K, K=K, fp32_had=fp32_had, use_fast_had=online_fast)++ def _make_hidden_had_hook():+ """Full Had hook on hidden_size (for Q/K/V and gate/up input)."""+ if is_grouped:+ return GroupOnlineHadamardHook(+ group_size=group_size, fp32_had=fp32_had, use_fast_had=online_fast, had_matrix=online_had_matrix+ )+ online_mat = _online_had(hidden_size)+ if online_mat is not None:+ return FullOnlineHadamardHook(+ had_K=None, K=None, fp32_had=fp32_had, use_fast_had=online_fast, had_matrix=online_mat+ )+ had_K, K = get_hadK(hidden_size)+ return FullOnlineHadamardHook(had_K=had_K, K=K, fp32_had=fp32_had, use_fast_had=online_fast)++ def _make_o_proj_hook():+ online_mat = _online_had(num_heads)+ if online_mat is not None:+ return CrossHeadOnlineHadamardHook(+ had_K=None,+ K=None,+ head_dim=head_dim,+ fp32_had=fp32_had,+ use_fast_had=online_fast,+ had_matrix=online_mat,+ )+ had_K, K = get_hadK(num_heads)+ return CrossHeadOnlineHadamardHook(+ had_K=had_K,+ K=K,+ head_dim=head_dim,+ fp32_had=fp32_had,+ use_fast_had=online_fast,+ )++ # --- Register ---+ handles = []++ for name, module in model.named_modules():+ if not isinstance(module, torch.nn.Linear):+ continue+ suffix = name.split(".")[-1]++ if name.endswith(mlp_out_suffix):+ # down_proj: full Had on intermediate_size input+ h = module.register_forward_pre_hook(_make_down_proj_hook())+ handles.append(h)+ elif name.endswith(attn_o_suffix):+ if fuse_online_to_weight and not is_grouped:+ # o_proj: cross-head Had on input (fused mode, full only)+ h = module.register_forward_pre_hook(_make_o_proj_hook())+ handles.append(h)+ elif not fuse_online_to_weight:+ # o_proj: full Had on hidden_size input (unfused mode, matches weight rotation)+ h = module.register_forward_pre_hook(_make_hidden_had_hook())+ handles.append(h)+ elif suffix in attn_qkv_suffixes:+ if not fuse_online_to_weight:+ # Q/K/V: full Had on hidden_size input (unfused mode only).+ # In fused mode Q/K/V only have residual Q on weight (no online Had),+ # and activations come pre-rotated from residual stream → no hook needed.+ h = module.register_forward_pre_hook(_make_hidden_had_hook())+ handles.append(h)+ elif suffix in mlp_in_suffixes:+ if not fuse_online_to_weight:+ # gate/up: full Had on hidden_size input (unfused mode only).+ # Same reasoning as Q/K/V above.+ h = module.register_forward_pre_hook(_make_hidden_had_hook())+ handles.append(h)++ return handles+++# ---------------------------------------------------------------------------+# Public API+# ---------------------------------------------------------------------------+++def apply_rotation_transform(+ model,+ group_size: int = None,+ allow_online_rotation: bool = True,+ rotation_matrix: Union[str, torch.Tensor, Dict[int, torch.Tensor], None] = None,+ compute_device: torch.device | str = None,+ fp32_had: bool = False,+ fuse_online_to_weight: bool = None,+):+ """Fuse layer norms, rotate weights, and register online Hadamard hooks.++ This is the single entry point for applying Hadamard inplace rotation.+ The model architecture is auto-detected via ``model.config.model_type``.++ Args:+ model: A HuggingFace CausalLM model (LLaMA-2/3, Qwen-3, etc.).+ fp32_had: Whether to compute the online Hadamard transform in fp32.+ group_size: If ``None`` (default), use full-dimension Hadamard rotation.+ compute_device: Device to run Hadamard computation on.+ allow_online_rotation: If ``True`` (default), apply online Hadamard+ rotations on ``down_proj`` input and the OV pair.+ rotation_matrix: Rotation matrix selection (``"hadamard"``,+ ``"random_hadamard"``, ``"quarot_hadamard"``, Tensor, dict, or None).+ fuse_online_to_weight: If ``True`` (default), fuse online Hadamard+ rotation into weights (down_proj input, v_proj output, o_proj input)+ and register compensating online hooks. If ``False``, skip+ embedding/lm_head rotation; each linear layer is self-contained+ with input-side Had on weight + compensating online hook on+ activation. No v_proj cross-head or inner-head rotation.++ Returns:+ list of hook handles."""+ if fuse_online_to_weight is None:+ if model.config.model_type in MAPPING_REGISTRY or model.__class__.__name__ in MAPPING_REGISTRY:+ fuse_online_to_weight = True+ else:+ fuse_online_to_weight = False+ had_dict, use_fast_had, preset = _normalize_rotation_matrix(rotation_matrix, group_size)+ compute_device = _resolve_compute_device(compute_device)++ if use_fast_had:+ from auto_round.utils import logger++ try:+ import fast_hadamard_transform # noqa: F401++ if group_size is None:+ logger.warning(+ "fast_hadamard_transform uses a different Hadamard matrix than the "+ "default implementation. Please ensure consistency between training "+ "and inference. This will be refined later."+ )+ except ImportError:+ logger.warning("Importing fast_hadamard_transform failed, falling back to default implementation.")+ use_fast_had = False++ mapping = infer_mapping_from_model(model)++ _untie_word_embeddings(model, mapping)++ if _uses_layernorm_with_mean(model, mapping):+ _subtract_embedding_mean(model, mapping)++ _fuse_layer_norms(model, mapping)++ if _uses_layernorm_with_mean(model, mapping):+ layers = _resolve(model, mapping.layers_attr)+ for layer in layers:+ _bake_mean_into_linear(_resolve(layer, mapping.attn_o))+ _bake_mean_into_linear(_resolve(layer, mapping.mlp_out))+ _replace_layernorms_with_rmsnorm(model)++ _rotate_weights(+ model,+ mapping,+ use_fast_had=use_fast_had,+ group_size=group_size,+ compute_device=compute_device,+ had_dict=had_dict,+ preset=preset,+ fuse_online_to_weight=fuse_online_to_weight,+ )++ handles = []+ if fuse_online_to_weight or allow_online_rotation:+ handles = _register_online_hooks(+ model,+ mapping,+ fp32_had=fp32_had,+ use_fast_had=use_fast_had,+ group_size=group_size,+ had_dict=had_dict,+ preset=preset,+ fuse_online_to_weight=fuse_online_to_weight,+ )++ return model, handles+++# ---------------------------------------------------------------------------+# Quick smoke test+# ---------------------------------------------------------------------------++if __name__ == "__main__":+ from transformers import AutoModelForCausalLM, AutoTokenizer++ model_name = "/models/opt-125m"+ tokenizer = AutoTokenizer.from_pretrained(model_name)+ model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto")+ model.to("cuda")+ text = "There is a girl who likes adventure,"+ inputs = tokenizer(text, return_tensors="pt").to(model.device)+ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))++ apply_rotation_transform(+ model, group_size=-1, allow_online_rotation=True, rotation_matrix="random_hadamard", fuse_online_to_weight=False+ )+ model.to("cuda")+ text = "There is a girl who likes adventure,"+ inputs = tokenizer(text, return_tensors="pt").to(model.device)+ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))++ model_name = "/models/Qwen3-8B"+ tokenizer = AutoTokenizer.from_pretrained(model_name)+ model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto")+ apply_rotation_transform(model, group_size=-1, allow_online_rotation=True, fuse_online_to_weight=True)+ model.to("cuda")+ text = "There is a girl who likes adventure,"+ inputs = tokenizer(text, return_tensors="pt").to(model.device)+ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))++ from transformers import AutoModelForCausalLM, AutoTokenizer++ model_name = "/models/Meta-Llama-3.1-8B-Instruct"+ tokenizer = AutoTokenizer.from_pretrained(model_name)+ model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto")+ apply_rotation_transform(model, fuse_online_to_weight=True, group_size=32)+ model.to("cuda")+ text = "There is a girl who likes adventure,"+ inputs = tokenizer(text, return_tensors="pt").to(model.device)+ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))+ #+ # model_name = "/models/Llama-2-7b-chat-hf"+ # tokenizer = AutoTokenizer.from_pretrained(model_name)+ # model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=torch.float16, device_map="auto")+ # apply_hadamard_rotation(model)+ # model.to("cuda")+ # text = "There is a girl who likes adventure,"+ # inputs = tokenizer(text, return_tensors="pt").to(model.device)+ # print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))diff --git a/auto_round/experimental/rotation_inplace/model_config.py b/auto_round/experimental/rotation_inplace/model_config.pynew file mode 100644index 000000000..3ecbf9b69--- /dev/null+++ b/auto_round/experimental/rotation_inplace/model_config.py@@ -0,0 +1,169 @@+# # Copyright (C) 2026 Intel Corporation+# # SPDX-License-Identifier: Apache-2.0++"""Model architecture mapping for Hadamard rotation.++Each :class:`RotationMapping` describes *where* the rotation-relevant modules+live inside a model. Currently supports LLaMA-2, LLaMA-3, and Qwen-3 (dense).++New architectures can be supported by calling :func:`register_mapping`.+"""++from __future__ import annotations++from dataclasses import dataclass, field+from typing import Dict, List, Optional++from auto_round.utils import logger++__all__ = [+ "RotationMapping",+ "register_mapping",+ "get_mapping",+ "infer_mapping_from_model",+ "MAPPING_REGISTRY",+]+++# ---------------------------------------------------------------------------+# Mapping dataclass+# ---------------------------------------------------------------------------+++@dataclass+class RotationMapping:+ """Declarative description of a transformer architecture for Hadamard rotation.++ Attribute names follow the dot-path convention relative to the model or+ each decoder layer.++ Config attribute names (read from ``model.config``):+ num_heads_attr, hidden_size_attr, intermediate_size_attr+ head_dim_override – explicit head dim (skip hidden_size // num_heads)+ """++ # -- top-level modules (dot-path from model root) --+ embedding: str = "model.embed_tokens"+ lm_head: str = "lm_head"+ positional_embedding: Optional[str] = None # e.g. "model.decoder.embed_positions" for OPT++ # -- layers container (dot-path from model root) --+ layers_attr: str = "model.layers"++ # -- per-layer: attention (dot-path from each layer) --+ attn_input_ln: str = "input_layernorm"+ attn_q: str = "self_attn.q_proj"+ attn_k: str = "self_attn.k_proj"+ attn_v: str = "self_attn.v_proj"+ attn_o: str = "self_attn.o_proj"++ # -- per-layer: MLP (dot-path from each layer) --+ mlp_input_ln: str = "post_attention_layernorm"+ mlp_in: List[str] = field(default_factory=lambda: ["mlp.up_proj", "mlp.gate_proj"])+ mlp_out: str = "mlp.down_proj"++ # -- final norm (dot-path from model root) --+ pre_head_ln: str = "model.norm"++ # -- head dim override (None = hidden_size // num_heads) --+ attn_head_dim: Optional[int] = None++ # -- config attr names --+ num_heads_attr: str = "num_attention_heads"+ hidden_size_attr: str = "hidden_size"+ intermediate_size_attr: str = "intermediate_size"+++# ---------------------------------------------------------------------------+# Helper: resolve a dot-path attribute on a module+# ---------------------------------------------------------------------------+++def _resolve(root, dot_path: str):+ """Resolve ``'a.b.c'`` to ``root.a.b.c``."""+ obj = root+ for attr in dot_path.split("."):+ obj = getattr(obj, attr)+ return obj+++# ---------------------------------------------------------------------------+# Registry+# ---------------------------------------------------------------------------++MAPPING_REGISTRY: Dict[str, RotationMapping] = {}+++def register_mapping(key: str, mapping: RotationMapping) -> RotationMapping:+ """Register a :class:`RotationMapping` under *key* (model_type or architecture)."""+ MAPPING_REGISTRY[key] = mapping+ return mapping+++def get_mapping(key: str) -> RotationMapping:+ """Look up a mapping by *key*; fall back to default if not found."""+ if key in MAPPING_REGISTRY:+ return MAPPING_REGISTRY[key]+ logger.warning(f"No rotation mapping registered for '{key}', " "falling back to default (LLaMA-like) mapping.")+ return RotationMapping()+++def infer_mapping_from_model(model) -> RotationMapping:+ """Return the best :class:`RotationMapping` for *model*.++ Tries ``model.config.model_type`` first, then ``model.__class__.__name__``.+ """+ model_type = getattr(getattr(model, "config", None), "model_type", "")+ if model_type in MAPPING_REGISTRY:+ return MAPPING_REGISTRY[model_type]++ arch = model.__class__.__name__+ if arch in MAPPING_REGISTRY:+ return MAPPING_REGISTRY[arch]++ logger.warning(+ f"Unrecognised architecture '{arch}' (model_type='{model_type}'). "+ "Falling back to default (LLaMA-like) mapping."+ )+ return RotationMapping()+++# ===================================================================+# Built-in mappings+# ===================================================================++# LLaMA-2 / LLaMA-3 / Mistral / Yi — all share the same layout+_default = RotationMapping()++register_mapping("llama", _default)+register_mapping("LlamaForCausalLM", _default)++# Qwen-3 dense — identical layout to LLaMA+register_mapping("qwen3", _default)+register_mapping("Qwen3ForCausalLM", _default)++# Qwen-2 / Qwen-2.5 dense — identical layout to LLaMA+register_mapping("qwen2", _default)+register_mapping("Qwen2ForCausalLM", _default)++# ---- OPT ----+# OPT uses standard LayerNorm (with bias, subtracts mean),+# different module names, and tied lm_head ↔ embedding weights.+_opt = RotationMapping(+ embedding="model.decoder.embed_tokens",+ lm_head="lm_head",+ positional_embedding="model.decoder.embed_positions",+ layers_attr="model.decoder.layers",+ attn_input_ln="self_attn_layer_norm",+ attn_q="self_attn.q_proj",+ attn_k="self_attn.k_proj",+ attn_v="self_attn.v_proj",+ attn_o="self_attn.out_proj",+ mlp_input_ln="final_layer_norm",+ mlp_in=["fc1"],+ mlp_out="fc2",+ pre_head_ln="model.decoder.final_layer_norm",+ intermediate_size_attr="ffn_dim",+)+register_mapping("opt", _opt)+register_mapping("OPTForCausalLM", _opt)diff --git a/auto_round/experimental/rotation_inplace/utils.py b/auto_round/experimental/rotation_inplace/utils.pynew file mode 100644index 000000000..04bb18981--- /dev/null+++ b/auto_round/experimental/rotation_inplace/utils.py@@ -0,0 +1,786 @@+# # Copyright (C) 2026 Intel Corporation+# # SPDX-License-Identifier: Apache-2.0++"""Online Hadamard transform hooks.++After weight rotation, down_proj and o_proj require an online Hadamard+transform on their *input activations* at inference time. This module+provides the hooks and a helper to register them on the model.+"""++import math++import torch+import torch.nn as nn++try:+ import fast_hadamard_transform+except ImportError:+ fast_hadamard_transform = None+++def _resolve_compute_device(compute_device) -> torch.device:+ """Return *compute_device* if explicitly given, otherwise auto-detect GPU.++ When ``compute_device`` is ``None`` the function checks for CUDA / XPU+ availability and returns the first accelerator it finds so that heavy+ matrix operations are offloaded to GPU even when the model weights live+ on CPU. Falls back to ``torch.device("cpu")`` when no accelerator is+ present.+ """+ if compute_device is not None:+ return torch.device(compute_device) if not isinstance(compute_device, torch.device) else compute_device+ if torch.cuda.is_available():+ return torch.device("cuda:0")+ if hasattr(torch, "xpu") and torch.xpu.is_available():+ return torch.device("xpu:0")+ return torch.device("cpu")+++BUILTIN_ROTATION_PRESETS = {"quarot_hadamard", "hadamard", "random_hadamard"}++# Global cache for random Hadamard matrices keyed by dimension.+# Ensures the same shape always returns the exact same random matrix within+# a process, across all calls to ``_rotate_weights`` / ``_register_online_hooks``.+_RANDOM_HADAMARD_CACHE: dict = {}+++def get_or_create_random_hadamard(dim: int, device=None) -> torch.Tensor:+ """Return a random Hadamard matrix for *dim*, creating and caching it if needed.++ The matrix is cached globally in ``_RANDOM_HADAMARD_CACHE`` so that every+ caller that requests the same *dim* receives the identical matrix.+ """+ if dim in _RANDOM_HADAMARD_CACHE:+ mat = _RANDOM_HADAMARD_CACHE[dim]+ if device is not None:+ mat = mat.to(device)+ return mat+ mat = random_hadamard_matrix(dim, device or torch.device("cpu"))+ _RANDOM_HADAMARD_CACHE[dim] = mat+ return mat+++def clear_random_hadamard_cache():+ """Clear the global random Hadamard matrix cache.++ Call this when you want subsequent ``random_hadamard`` preset runs to+ generate fresh random matrices (e.g. between independent experiments).+ """+ _RANDOM_HADAMARD_CACHE.clear()+++def _normalize_rotation_matrix(rotation_matrix, group_size):+ """Normalize ``rotation_matrix`` into a ``(had_dict, use_fast_had, preset)`` tuple.++ Accepted inputs:+ * ``None`` → ``(None, False, None)`` — use built-in butterfly ``matmul_hadU``.+ * ``"quarot_hadamard"`` → ``(None, True, "quarot_hadamard")`` — fusable+ rotations use ``fast_hadamard_transform`` (random); non-fusable+ (online-paired) rotations use deterministic ``get_hadK``/``matmul_hadU``.+ * ``"hadamard"`` → ``(None, False, "hadamard")`` — all rotations use+ deterministic ``get_hadK``/``matmul_hadU``.+ * ``"random_hadamard"`` → ``(None, False, "random_hadamard")`` — all rotations use+ ``random_hadamard_matrix``.+ * A ``torch.Tensor`` of shape ``(n, n)`` → ``({n: tensor}, False, None)``.+ * A ``dict[int, Tensor]`` → ``(dict, False, None)`` — returned as-is.++ Returns:+ ``(had_dict, use_fast_had, preset)``++ Raises:+ ValueError: if a non-``str`` *rotation_matrix* is given but+ *group_size* is not a positive integer, or an unknown preset.+ """+ if rotation_matrix is None:+ return None, False, None++ if isinstance(rotation_matrix, str):+ if rotation_matrix not in BUILTIN_ROTATION_PRESETS:+ raise ValueError(+ f"Unknown rotation_matrix preset '{rotation_matrix}'. "+ f"Supported presets: {BUILTIN_ROTATION_PRESETS}."+ )+ if rotation_matrix == "quarot_hadamard":+ return None, True, "quarot_hadamard"+ elif rotation_matrix == "hadamard":+ return None, False, "hadamard"+ else: # "random_hadamard"+ return None, False, "random_hadamard"++ is_grouped = group_size is not None and group_size > 0+ if not is_grouped and not isinstance(rotation_matrix, dict):+ raise ValueError(+ "rotation_matrix (Tensor/dict) can only be used with a positive group_size. "+ f"Got group_size={group_size}."+ )++ if isinstance(rotation_matrix, torch.Tensor):+ assert (+ rotation_matrix.ndim == 2 and rotation_matrix.shape[0] == rotation_matrix.shape[1]+ ), f"rotation_matrix must be square, got shape {rotation_matrix.shape}"+ return {rotation_matrix.shape[0]: rotation_matrix}, False, None++ if isinstance(rotation_matrix, dict):+ for k, t in rotation_matrix.items():+ assert (+ isinstance(t, torch.Tensor) and t.ndim == 2 and t.shape[0] == t.shape[1]+ ), f"rotation_matrix[{k}] must be a square tensor, got shape {t.shape}"+ return rotation_matrix, False, None++ raise TypeError(+ f"rotation_matrix must be a Tensor, dict[int, Tensor], str, or None. " f"Got {type(rotation_matrix)}."+ )+++def _get_custom_had(had_dict, size):+ """Look up a custom Hadamard matrix for *size* from the normalized dict.++ Returns ``(had_tensor, True)`` if found, ``(None, False)`` otherwise.+ """+ if had_dict is None:+ return None, False+ if size in had_dict:+ return had_dict[size], True+ return None, False+++# ---------------------------------------------------------------------------+# Hook implementations+# ---------------------------------------------------------------------------+++class FullOnlineHadamardHook(nn.Module):+ """Pre-forward hook: full Hadamard on the entire last dimension (for ``down_proj``)."""++ def __init__(self, had_K, K, fp32_had=False, use_fast_had=True, had_matrix=None):+ super().__init__()+ self.custom_had = had_matrix is not None+ if had_matrix is not None:+ self.register_buffer("had_matrix", had_matrix)+ self.had_K = None+ self.K = None+ else:+ if had_K is not None:+ self.register_buffer("had_K", had_K)+ else:+ self.had_K = None+ self.K = K+ self.fp32_had = fp32_had+ self.use_fast_had = use_fast_had++ def __call__(self, module: nn.Module, args):+ x = args[0] if isinstance(args, tuple) else args+ x_dtype = x.dtype++ if self.custom_had:+ H = self.had_matrix.to(device=x.device, dtype=x.dtype)+ if self.fp32_had:+ H = self.had_matrix.to(device=x.device).float()+ x = (x.float() @ H.T).to(x_dtype)+ else:+ x = x @ H.T+ elif self.fp32_had:+ x = matmul_hadU_cuda(x.float(), self.had_K, self.K, use_fast_had=self.use_fast_had).to(x_dtype)+ else:+ x = matmul_hadU_cuda(x, self.had_K, self.K, use_fast_had=self.use_fast_had)++ if isinstance(args, tuple):+ return (x,) + args[1:]+ return x+++class CrossHeadOnlineHadamardHook(nn.Module):+ """Pre-forward hook: **cross-head** Hadamard on the ``num_heads`` dimension+ (for ``o_proj``).++ After offline rotation:+ - ``v_proj`` absorbed a per-head (within-head) Hadamard on ``head_dim``.+ - ``o_proj`` absorbed a full Hadamard on ``hidden_size``.++ Since ``H_full = H_cross ⊗ H_within`` (Kronecker decomposition) and the+ within-head part is already cancelled by ``v_proj`` through the attention+ path (``H_within² = I``), the online hook only needs to apply the residual+ **cross-head** Hadamard (``H_cross ⊗ I``):++ * reshape ``(*, hidden_size)`` → ``(*, num_heads, head_dim)``+ * transpose → ``(*, head_dim, num_heads)``+ * Hadamard on the **num_heads** axis (last dim)+ * transpose back and reshape+ """++ def __init__(self, had_K, K, head_dim, fp32_had=False, use_fast_had=True, had_matrix=None):+ """+ Args:+ had_K: Hadamard sub-matrix from ``get_hadK(num_heads)``.+ K: Block size from ``get_hadK(num_heads)``.+ head_dim: ``hidden_size // num_attention_heads``.+ fp32_had: Compute in fp32.+ use_fast_had: If True use fast_hadamard_transform; if False use matmul_hadU.+ had_matrix: Optional custom rotation matrix of shape ``(num_heads, num_heads)``.+ """+ super().__init__()+ self.custom_had = had_matrix is not None+ if had_matrix is not None:+ self.register_buffer("had_matrix", had_matrix)+ self.had_K = None+ self.K = None+ else:+ if had_K is not None:+ self.register_buffer("had_K", had_K)+ else:+ self.had_K = None+ self.K = K+ self.had_dim = head_dim+ self.fp32_had = fp32_had+ self.use_fast_had = use_fast_had++ def __call__(self, module: nn.Module, args):+ x = args[0] if isinstance(args, tuple) else args+ x_dtype = x.dtype++ if self.fp32_had:+ x = x.float()++ init_shape = x.shape+ num_heads = init_shape[-1] // self.had_dim++ if self.custom_had:+ H = self.had_matrix.to(device=x.device, dtype=x.dtype)+ # reshape (*, hidden) → (*, num_heads, head_dim), transpose → (*, head_dim, num_heads)+ x = x.reshape(-1, num_heads, self.had_dim).transpose(1, 2)+ # apply H on last dim (num_heads): x @ H.T+ x = (x @ H.T).transpose(1, 2)+ elif self.use_fast_had and fast_hadamard_transform is not None and self.K == 1:+ x = fast_hadamard_transform.hadamard_transform(+ x.reshape(-1, num_heads, self.had_dim).transpose(1, 2),+ scale=1 / math.sqrt(num_heads),+ ).transpose(1, 2)+ else:+ # Fallback: use matmul_hadU (pure butterfly + had_K, no fast_hadamard_transform)+ x = x.reshape(-1, num_heads, self.had_dim).transpose(1, 2)+ x = matmul_hadU(x.contiguous())+ x = x.transpose(1, 2)++ if self.fp32_had:+ x = x.to(x_dtype)+ x = x.reshape(init_shape)++ if isinstance(args, tuple):+ return (x,) + args[1:]+ return x+++# ---------------------------------------------------------------------------+# Registration helper+# ---------------------------------------------------------------------------+++def register_online_had_hooks(model, mapping=None, fp32_had=False, use_fast_had=True):+ """Register online Hadamard pre-forward hooks on ``down_proj`` and ``o_proj``.++ * **down_proj** (``online_full_had``): full Hadamard on ``intermediate_size``.+ Compensates ``apply_exact_had_to_linear(down_proj, had_dim=-1, output=False)``.++ * **o_proj** (``online cross-head had``): cross-head Hadamard on ``num_heads``.+ Compensates the residual after v_proj's within-head Hadamard cancels.++ Args:+ model: A HuggingFace model whose weights have already been rotated.+ mapping: A :class:`RotationMapping` (auto-inferred if ``None``).+ fp32_had: Whether to compute the Hadamard transform in fp32.+ use_fast_had: If True use fast_hadamard_transform; if False use matmul_hadU.++ Returns:+ list of hook handles (call ``handle.remove()`` to detach).+ """+ if mapping is None:+ from auto_round.experimental.rotation_inplace.model_config import infer_mapping_from_model++ mapping = infer_mapping_from_model(model)++ config = model.config+ num_heads = getattr(config, mapping.num_heads_attr)+ hidden_size = getattr(config, mapping.hidden_size_attr)+ intermediate_size = getattr(config, mapping.intermediate_size_attr)+ head_dim = mapping.attn_head_dim or (hidden_size // num_heads)++ # down_proj: full Hadamard on intermediate_size+ had_K_full, K_full = get_hadK(intermediate_size)++ # o_proj: cross-head Hadamard on num_heads+ had_K_head, K_head = get_hadK(num_heads)++ # Identify target module suffixes from mapping+ mlp_out_suffix = mapping.mlp_out.split(".")[-1] # e.g. "down_proj"+ attn_o_suffix = mapping.attn_o.split(".")[-1] # e.g. "o_proj"++ handles = []+ for name, module in model.named_modules():+ if name.endswith(mlp_out_suffix) and isinstance(module, nn.Linear):+ hook = FullOnlineHadamardHook(+ had_K=had_K_full,+ K=K_full,+ fp32_had=fp32_had,+ use_fast_had=use_fast_had,+ )+ h = module.register_forward_pre_hook(hook)+ handles.append(h)+ elif name.endswith(attn_o_suffix) and isinstance(module, nn.Linear):+ hook = CrossHeadOnlineHadamardHook(+ had_K=had_K_head,+ K=K_head,+ head_dim=head_dim,+ fp32_had=fp32_had,+ use_fast_had=use_fast_had,+ )+ h = module.register_forward_pre_hook(hook)+ handles.append(h)++ return handles+++def is_pow2(n):+ return (n & (n - 1) == 0) and (n > 0)+++# Adapted from https://github.com/Cornell-RelaxML/quip-sharp/blob/main/lib/utils/matmul_had.py+def get_hadK(n: int, transpose=False) -> (torch.Tensor, int):+ hadK, K = None, None++ if is_pow2(n):+ K = 1+ return hadK, K+ else:+ from auto_round.experimental.transform.utils.hadamard import _fetch_hadamard_divisor++ hadK = _fetch_hadamard_divisor(n, torch.float, torch.device("cpu"))+ if transpose:+ hadK = hadK.T+ if hadK is not None:+ return hadK, 1 if is_pow2(hadK.shape[0]) else hadK.shape[0]+ assert is_pow2(n)+++def matmul_hadU(X, transpose=False):+ n = X.shape[-1]+ hadK, K = get_hadK(n, transpose)+ input = X.clone().view(-1, n, 1)+ output = input.clone()+ while input.shape[1] > K:+ input = input.view(input.shape[0], input.shape[1] // 2, 2, input.shape[2])+ output = output.view(input.shape)+ output[:, :, 0, :] = input[:, :, 0, :] + input[:, :, 1, :]+ output[:, :, 1, :] = input[:, :, 0, :] - input[:, :, 1, :]+ output = output.view(input.shape[0], input.shape[1], -1)+ input, output = (output, input)+ del output++ if K > 1:+ # Do not explicitly repeat - OOM+ # input = torch.bmm(+ # hadK.repeat(len(input), 1, 1).to(input.device).to(input.dtype), input)+ # Use bcast instead+ input = hadK.view(1, K, K).to(input) @ input++ return input.view(X.shape) / torch.tensor(n).sqrt()+++def matmul_hadUt(X):+ return matmul_hadU(X, transpose=True)+++def random_hadamard_matrix(size, device):+ # See https://cornell-relaxml.github.io/quip-sharp/ , Section "Randomized Hadamard Transformation"+ Q = torch.randint(low=0, high=2, size=(size,)).to(torch.float64)+ Q = Q * 2 - 1+ Q = torch.diag(Q)+ return matmul_hadU(Q).to(device)+++def deterministic_hadamard_matrix(size, device):+ """Build a deterministic Hadamard matrix of the given *size*.++ Applies the butterfly ``matmul_hadU`` to an identity matrix so that the+ result is purely determined by ``get_hadK`` (no random sign flips).+ """+ Q = torch.eye(size, dtype=torch.float64)+ return matmul_hadU(Q).to(device)+++def matmul_hadU_cuda(X, hadK, K, use_fast_had=True):+ n = X.shape[-1]+ if not use_fast_had or fast_hadamard_transform is None:+ return matmul_hadU(X)+ if K == 1:+ return fast_hadamard_transform.hadamard_transform(X.contiguous(), 1.0 / torch.tensor(n).sqrt())+ # if transpose:+ # hadK = hadK.T.contiguous()+ input = X.view(*X.shape[:-1], K, n // K)+ input = fast_hadamard_transform.hadamard_transform(input.contiguous(), 1.0 / torch.tensor(n).sqrt())+ input = hadK.to(input.device).to(input.dtype) @ input+ return input.reshape(X.shape)+++def matmul_hadUt_cuda(X, hadK, K, use_fast_had=True):+ return matmul_hadU_cuda(X, hadK, K, use_fast_had=use_fast_had)+++def apply_exact_had_to_linear(+ module, had_dim=-1, output=False, use_fast_had=True, compute_device=None, had_matrix=None+):+ """Apply Hadamard rotation to a Linear layer's weight in-place.++ Args:+ module: ``nn.Linear`` layer.+ had_dim: Dimension of each Hadamard block (``-1`` for full dimension).+ output: If ``True`` rotate the output (row) side; otherwise input (col).+ use_fast_had: Use ``fast_hadamard_transform`` when available.+ compute_device: Device to run computation on.+ had_matrix: Optional custom rotation matrix. When ``had_dim == -1``+ this should be a square tensor whose size equals+ ``out_features`` (output) or ``in_features`` (input). When+ ``had_dim > 0`` the size should equal ``had_dim``.+ """+ assert isinstance(module, torch.nn.Linear)+ in_features, out_features = module.in_features, module.out_features++ if had_dim != -1 and had_matrix is None:+ assert is_pow2(had_dim), "Hadamard dimension must be a power of 2!"++ W_ = module.weight.data+ dtype = W_.dtype+ dev = W_.device+ init_shape = W_.shape+ compute_dev = _resolve_compute_device(compute_device)+ W_ = W_.double().to(compute_dev)++ if had_matrix is not None:+ H = had_matrix.to(device=compute_dev, dtype=torch.float64)+ if had_dim == -1:+ # Full-dimension custom matrix+ if output:+ # W.T = H @ W.T → W = (H @ W.T).T = W @ H.T+ W_ = W_ @ H.T+ else:+ # W = H @ W (rotate input columns: W_new[i,:] = sum H[i,j]*W[j,:])+ # Actually for input side: W_new = W @ H (each row is rotated)+ W_ = W_ @ H.T+ else:+ # Per-block custom matrix+ if output:+ W_ = W_.t()+ transposed_shape = W_.shape+ flat = W_.reshape(-1, had_dim)+ W_ = (flat @ H.T).reshape(transposed_shape).t()+ else:+ flat = W_.reshape(-1, had_dim)+ W_ = (flat @ H.T).reshape(init_shape)+ elif had_dim == -1:+ if output:+ had_K, K = get_hadK(out_features)+ W_ = matmul_hadU_cuda(W_.t(), had_K, K, use_fast_had=use_fast_had).t()+ if not output:+ had_K, K = get_hadK(in_features)+ W_ = matmul_hadU_cuda(W_, had_K, K, use_fast_had=use_fast_had)+ else:+ # Apply Hadamard to the last had_dim chunks of the weights+ if output:+ W_ = W_.t()+ transposed_shape = W_.shape+ if use_fast_had and fast_hadamard_transform is not None:+ W_ = (+ fast_hadamard_transform.hadamard_transform(+ W_.reshape(-1, transposed_shape[-1] // had_dim, had_dim), scale=1 / math.sqrt(had_dim)+ )+ .reshape(transposed_shape)+ .t()+ )+ else:+ W_ = matmul_hadU(W_.reshape(-1, had_dim)).reshape(transposed_shape).t()+ else:+ if use_fast_had and fast_hadamard_transform is not None:+ n = W_.shape[1]+ W_ = fast_hadamard_transform.hadamard_transform(+ W_.reshape(-1, n // had_dim, had_dim), scale=1 / math.sqrt(had_dim)+ ).reshape(init_shape)+ else:+ W_ = matmul_hadU(W_.reshape(-1, had_dim)).reshape(init_shape)+ module.weight.data = W_.to(device=dev, dtype=dtype)+++def apply_cross_head_had_to_linear(+ module, num_heads, head_dim, use_fast_had=True, compute_device=None, had_matrix=None+):+ """Apply a cross-head Hadamard rotation to a Linear layer's input side.++ The operation is equivalent to ``(H_cross ⊗ I_head_dim)`` applied to the+ input columns:++ * Reshape columns ``(hidden_size,)`` → ``(num_heads, head_dim)``+ * Transpose → ``(head_dim, num_heads)``+ * Hadamard on the ``num_heads`` axis+ * Transpose back and reshape++ This mirrors what :class:`CrossHeadOnlineHadamardHook` does at runtime.++ Args:+ module: ``nn.Linear`` layer whose ``in_features == num_heads * head_dim``.+ num_heads: Number of attention heads.+ head_dim: Per-head dimension.+ use_fast_had: Use ``fast_hadamard_transform`` when available.+ compute_device: Device to run computation on.+ had_matrix: Optional custom rotation matrix of shape ``(num_heads, num_heads)``.+ """+ assert isinstance(module, torch.nn.Linear)+ W_ = module.weight.data+ dtype = W_.dtype+ dev = W_.device+ compute_dev = _resolve_compute_device(compute_device)+ W_ = W_.double().to(compute_dev)++ out_f = W_.shape[0]+ # W shape: (out_features, hidden_size) where hidden_size = num_heads * head_dim+ # Reshape columns: (out_f, num_heads, head_dim)+ W_ = W_.reshape(out_f, num_heads, head_dim)+ # Transpose last two dims: (out_f, head_dim, num_heads)+ W_ = W_.transpose(1, 2).contiguous()++ if had_matrix is not None:+ H = had_matrix.to(device=compute_dev, dtype=torch.float64)+ # Apply H on last dim (num_heads): flat @ H.T+ flat = W_.reshape(-1, num_heads)+ W_ = (flat @ H.T).reshape(out_f, head_dim, num_heads)+ elif use_fast_had and fast_hadamard_transform is not None and is_pow2(num_heads):+ W_ = fast_hadamard_transform.hadamard_transform(W_, scale=1.0 / math.sqrt(num_heads))+ else:+ W_ = matmul_hadU(W_.reshape(-1, num_heads)).reshape(out_f, head_dim, num_heads)++ # Transpose back: (out_f, num_heads, head_dim) → (out_f, hidden_size)+ W_ = W_.transpose(1, 2).contiguous().reshape(out_f, num_heads * head_dim)+ module.weight.data = W_.to(device=dev, dtype=dtype)+++# ---------------------------------------------------------------------------+# Grouped (block-diagonal) Hadamard utilities+# ---------------------------------------------------------------------------+++class OnlineHadamardPostHook(nn.Module):+ """Forward hook (post-hook) adapter: wraps a pre-hook-style Hadamard+ transform to apply it on the layer's **output** instead of input.++ Used for v_proj per-head Hadamard on the output side when online+ rotation is not fused into weights.+ """++ def __init__(self, pre_hook):+ super().__init__()+ self.pre_hook = pre_hook++ def __call__(self, module, input, output):+ result = self.pre_hook(module, (output,))+ if isinstance(result, tuple):+ return result[0]+ return result+++class GroupOnlineHadamardHook(nn.Module):+ """Pre-forward hook: block-diagonal Hadamard with fixed ``group_size`` on last dim.++ Reshapes ``(*, D)`` → ``(*, D // group_size, group_size)``, applies Hadamard+ per group, then reshapes back. Much cheaper than a full-dimension Hadamard.+ """++ def __init__(self, group_size, fp32_had=False, use_fast_had=True, had_matrix=None):+ super().__init__()+ self.group_size = group_size+ self.fp32_had = fp32_had+ self.use_fast_had = use_fast_had+ self.custom_had = had_matrix is not None++ if had_matrix is not None:+ self.register_buffer("had_matrix", had_matrix)+ self.had_K = None+ self.K = None+ elif not is_pow2(group_size):+ had_K, K = get_hadK(group_size)+ if had_K is not None:+ self.register_buffer("had_K", had_K)+ else:+ self.had_K = None+ self.K = K+ else:+ self.had_K = None+ self.K = 1++ def __call__(self, module: nn.Module, args):+ x = args[0] if isinstance(args, tuple) else args+ x_dtype = x.dtype+ init_shape = x.shape+ gs = self.group_size++ if self.fp32_had:+ x = x.float()++ # Reshape: (*, D) → (*, D//gs, gs)+ x = x.reshape(*init_shape[:-1], init_shape[-1] // gs, gs)++ if self.custom_had:+ H = self.had_matrix.to(device=x.device, dtype=x.dtype)+ flat = x.reshape(-1, gs)+ x = (flat @ H.T).reshape(*init_shape[:-1], init_shape[-1] // gs, gs)+ elif self.use_fast_had and fast_hadamard_transform is not None and self.K == 1:+ x = fast_hadamard_transform.hadamard_transform(x, scale=1.0 / math.sqrt(gs))+ else:+ x = x.reshape(-1, gs)+ x = matmul_hadU(x)+ x = x.reshape(*init_shape[:-1], init_shape[-1] // gs, gs)++ x = x.reshape(init_shape)++ if self.fp32_had:+ x = x.to(x_dtype)++ if isinstance(args, tuple):+ return (x,) + args[1:]+ return x+++def _apply_grouped_had_to_weight(W, group_size, side="input", use_fast_had=True, had_matrix=None):+ """Apply block-diagonal Hadamard to a weight matrix.++ Args:+ W: Weight tensor, shape (out_features, in_features).+ group_size: Block size for the Hadamard rotation.+ side: ``'input'`` rotates columns (in_features dim),+ ``'output'`` rotates rows (out_features dim).+ use_fast_had: Use fast_hadamard_transform if available.+ had_matrix: Optional custom Hadamard matrix of shape ``(gs, gs)``+ to use instead of the built-in Hadamard.++ Returns:+ Rotated weight tensor.+ """+ gs = group_size+ dtype = W.dtype+ W = W.double()++ def _had_on_last_dim(X):+ """Apply Hadamard on the last dimension (size gs) of X shaped (..., gs)."""+ if had_matrix is not None:+ H = had_matrix.to(device=X.device, dtype=X.dtype)+ # X: (..., gs) → batch matmul with H^T → X @ H^T+ flat = X.reshape(-1, gs)+ return (flat @ H.T).reshape(X.shape)+ if use_fast_had and fast_hadamard_transform is not None and is_pow2(gs):+ return fast_hadamard_transform.hadamard_transform(X, scale=1.0 / math.sqrt(gs))+ orig_shape = X.shape+ return matmul_hadU(X.reshape(-1, gs)).reshape(orig_shape)++ if side == "input":+ out_f, in_f = W.shape+ W = W.reshape(out_f, in_f // gs, gs)+ W = _had_on_last_dim(W)+ W = W.reshape(out_f, in_f)+ else:+ out_f, in_f = W.shape+ Wt = W.t().contiguous()+ Wt = Wt.reshape(in_f, out_f // gs, gs)+ Wt = _had_on_last_dim(Wt)+ W = Wt.reshape(in_f, out_f).t().contiguous()++ return W.to(dtype)+++def _rotate_linear_grouped(module, group_size, side="input", use_fast_had=True, compute_device=None, had_matrix=None):+ """Apply block-diagonal Hadamard rotation to a Linear layer's weight.++ Args:+ module: ``nn.Linear`` layer.+ group_size: Block size.+ side: ``'input'`` or ``'output'``.+ use_fast_had: Use fast_hadamard_transform.+ compute_device: Device to run computation on. If None, auto-detects GPU.+ had_matrix: Optional custom Hadamard matrix of shape ``(gs, gs)``.+ """+ dtype = module.weight.data.dtype+ dev = module.weight.data.device+ compute_dev = _resolve_compute_device(compute_device)+ W = module.weight.data.to(device=compute_dev, dtype=torch.float64)+ W = _apply_grouped_had_to_weight(W, group_size, side=side, use_fast_had=use_fast_had, had_matrix=had_matrix)+ module.weight.data = W.to(device=dev, dtype=dtype)++ if side == "output" and module.bias is not None:+ bias = module.bias.data.to(device=compute_dev, dtype=torch.float64)+ gs = group_size+ bias = bias.reshape(-1, gs)+ if had_matrix is not None:+ H = had_matrix.to(device=compute_dev, dtype=torch.float64)+ bias = (bias @ H.T).reshape(-1)+ elif use_fast_had and fast_hadamard_transform is not None and is_pow2(gs):+ bias = (+ fast_hadamard_transform.hadamard_transform(bias.unsqueeze(0), scale=1.0 / math.sqrt(gs))+ .squeeze(0)+ .reshape(-1)+ )+ else:+ bias = matmul_hadU(bias).reshape(-1)+ module.bias.data = bias.to(device=dev, dtype=dtype)+++def _rotate_embedding_grouped(embedding, group_size, use_fast_had=True, compute_device=None, had_matrix=None):+ """Apply block-diagonal Hadamard rotation to an Embedding layer.++ Embedding weight: (vocab, hidden_size) → rotate on hidden_size (columns).+ """+ dtype = embedding.weight.data.dtype+ dev = embedding.weight.data.device+ compute_dev = _resolve_compute_device(compute_device)+ W = embedding.weight.data.to(device=compute_dev, dtype=torch.float64)+ W = _apply_grouped_had_to_weight(W, group_size, side="input", use_fast_had=use_fast_had, had_matrix=had_matrix)+ new_W = W.to(device=dev, dtype=dtype)+ del W+ embedding.weight.data = new_W+++def register_online_had_hooks_grouped(model, mapping, group_size, fp32_had=False, use_fast_had=True):+ """Register per-group online Hadamard hooks on ``down_proj`` and ``o_proj``.++ In grouped mode:+ - **down_proj**: block-diagonal Hadamard on ``intermediate_size`` with ``group_size``.+ - **o_proj**: block-diagonal Hadamard on ``hidden_size`` with ``group_size``.++ Args:+ model: HuggingFace model with rotated weights.+ mapping: RotationMapping.+ group_size: Block size for block-diagonal Hadamard.+ fp32_had: Compute in fp32.+ use_fast_had: Use fast_hadamard_transform.++ Returns:+ list of hook handles.+ """+ mlp_out_suffix = mapping.mlp_out.split(".")[-1]+ attn_o_suffix = mapping.attn_o.split(".")[-1]++ handles = []+ for name, module in model.named_modules():+ if name.endswith(mlp_out_suffix) and isinstance(module, nn.Linear):+ hook = GroupOnlineHadamardHook(+ group_size=group_size,+ fp32_had=fp32_had,+ use_fast_had=use_fast_had,+ )+ h = module.register_forward_pre_hook(hook)+ handles.append(h)+ elif name.endswith(attn_o_suffix) and isinstance(module, nn.Linear):+ hook = GroupOnlineHadamardHook(+ group_size=group_size,+ fp32_had=fp32_had,+ use_fast_had=use_fast_had,+ )+ h = module.register_forward_pre_hook(hook)+ handles.append(h)++ return handlesdiff --git a/auto_round/experimental/transform/apply.py b/auto_round/experimental/transform/apply.pyindex 4ae74f5fb..aafd4c1b0 100644--- a/auto_round/experimental/transform/apply.py+++ b/auto_round/experimental/transform/apply.py@@ -5,16 +5,16 @@ import torchimport tqdmfrom auto_round.experimental.qmodules.base import QModuleBase-from auto_round.experimental.transform.hadamard_config import HadamardConfigfrom auto_round.experimental.transform.hadamards import build_hadamard_transform-from auto_round.experimental.utils import is_triton_kernel_available, normalize_hadamard_config+from auto_round.experimental.transform.rotation_config import RotationConfig+from auto_round.experimental.utils import is_triton_kernel_available, normalize_rotation_config-__all__ = ["apply_hadamard_transform"]+__all__ = ["apply_rotation_transform"]-def apply_hadamard_transform(+def apply_rotation_transform(model: torch.nn.Module,- config: str | dict | HadamardConfig | None,+ config: str | dict | RotationConfig | None,location: str = "weight",use_tqdm=True,desc=None,@@ -31,13 +31,13 @@ def apply_hadamard_transform(* ``str``: A named/preset transform configuration. In this case,resolved to a concrete quantization/transform configuration.* ``dict``: A raw configuration mapping that will be normalized- (via :func:`normalize_hadamard_config`) and then passed to+ (via :func:`normalize_rotation_config`) and then passed to:class:`TransformConfig`.* :class:`TransformConfig`: An existing configuration instance.This will be used to construct the final configuration afternormalization.* ``None``: Uses the default behavior of- :func:`_normalize_hadamard_config` (for example, inferring a+ :func:`_normalize_rotation_config` (for example, inferring aconfiguration from ``data_type`` or other project defaults), ifsupported.:param data_type: quantization data type.@@ -48,9 +48,9 @@ def apply_hadamard_transform(``config.transform_type``."""- config = normalize_hadamard_config(config, data_type)- if not isinstance(config, HadamardConfig):- config = HadamardConfig(**config)+ config = normalize_rotation_config(config, data_type)+ if not isinstance(config, RotationConfig):+ config = RotationConfig(**config)modules_config = [(name, module, config)@@ -64,16 +64,19 @@ def apply_hadamard_transform(continue_apply_to_module(model, module, config, location, data_type)- # attach config to model for compression/serialization- setattr(model, "hadamard_config", config)+ # attach config to model for compression/serialization. Use a plain dict so+ # that downstream HF `save_pretrained` -> JSON works (RotationConfig is a+ # pydantic model and is not directly JSON serializable).+ setattr(model, "rotation_config", config.model_dump() if hasattr(config, "model_dump") else config)+ hooks = None- return model+ return model, hooksdef _apply_to_module(model: torch.nn.Module,module: torch.nn.Module,- config: HadamardConfig,+ config: RotationConfig,location: str = "weight",data_type: str = "mx_fp",):diff --git a/auto_round/experimental/transform/hadamard_config.py b/auto_round/experimental/transform/hadamard_config.pydeleted file mode 100644index b58cce9dc..000000000--- a/auto_round/experimental/transform/hadamard_config.py+++ /dev/null@@ -1,28 +0,0 @@-# # Copyright (C) 2026 Intel Corporation-# # SPDX-License-Identifier: Apache-2.0--from pydantic import BaseModel, Field, field_validator--__all__ = ["HadamardConfig"]---class HadamardConfig(BaseModel):- """- Configuration of transforms to be applied to a model. This config is to be- serialized within a model's `config.json` file- """-- block_size: int = Field(default=32)-- hadamard_type: str = Field(default="hadamard")-- # for random hadamard transform- random_seed: bool = Field(default=False, exclude=True)-- @field_validator("hadamard_type")- @classmethod- def validate_hadamard_type(cls, v: str) -> str:- allowed = {"hadamard", "random_hadamard"}- if v not in allowed:- raise ValueError(f"Unsupported hadamard_type: {v}. Supported values: {sorted(allowed)}")- return vdiff --git a/auto_round/experimental/transform/patch_modules.py b/auto_round/experimental/transform/patch_modules.pyindex 34cb2f1ce..e0f9adbae 100644--- a/auto_round/experimental/transform/patch_modules.py+++ b/auto_round/experimental/transform/patch_modules.py@@ -51,11 +51,11 @@ def patch_wrapperlinear_to_apply_transform(w_transform, inp_transform):orig_qdq_act = WrapperLinear._qdq_act- def _qdq_act_patched(self, x, act_max_scale, act_max=None):+ def _qdq_act_patched(self, x, act_min_scale, act_max_scale, act_max=None):x = inp_transform(x)- return orig_qdq_act(self, x, act_max_scale, act_max)+ return orig_qdq_act(self, x, act_min_scale, act_max_scale, act_max)WrapperLinear._qdq_weight = _qdq_weight_patchedWrapperLinear._qdq_act = _qdq_act_patcheddiff --git a/auto_round/experimental/transform/rotation_config.py b/auto_round/experimental/transform/rotation_config.pynew file mode 100644index 000000000..dfcfb5d45--- /dev/null+++ b/auto_round/experimental/transform/rotation_config.py@@ -0,0 +1,61 @@+# # Copyright (C) 2026 Intel Corporation+# # SPDX-License-Identifier: Apache-2.0++from typing import Optional++from pydantic import BaseModel, Field, field_validator++__all__ = ["RotationConfig"]+++class RotationConfig(BaseModel):+ """+ Unified configuration for Hadamard rotation/transform applied to a model.++ Two implementation paths are supported:++ * ``backend="inplace"`` -> ``auto_round.experimental.rotation_inplace``+ QuaRot-style residual-stream / per-layer rotation. Supports any+ weight/activation dtype (incl. INT4/INT8/FPx). Can optionally fuse+ the online Hadamard into weights (``fuse_online_to_weight=True``).+ * ``backend="transform"`` -> ``auto_round.experimental.transform``+ Per-Linear weight + activation Hadamard with a fused triton kernel.+ **Only supports MXFP4 / NVFP4** and **cannot fuse online to weight.**+ * ``backend="auto"`` (default)+ - If ``fuse_online_to_weight=True`` -> inplace (fused).+ - Else if ``data_type`` is MX-FP / NV-FP -> transform.+ - Otherwise -> inplace (unfused).++ Notes:+ * ``block_size`` is the group/block size for grouped Hadamard.+ For ``backend="inplace"`` it is forwarded as ``group_size`` (``None``+ / ``-1`` means full-dimension Hadamard).+ """++ # ---- shared ----+ backend: str = Field(default="auto")+ block_size: Optional[int] = Field(default=None)+ hadamard_type: str = Field(default="hadamard")++ # ---- inplace-only ----+ fuse_online_to_weight: Optional[bool] = Field(default=None)+ allow_online_rotation: bool = Field(default=True)++ # for random hadamard transform (transform path)+ random_seed: bool = Field(default=False, exclude=True)++ @field_validator("backend")+ @classmethod+ def validate_backend(cls, v: str) -> str:+ allowed = {"auto", "inplace", "transform"}+ if v not in allowed:+ raise ValueError(f"Unsupported backend: {v}. Supported values: {sorted(allowed)}")+ return v++ @field_validator("hadamard_type")+ @classmethod+ def validate_hadamard_type(cls, v: str) -> str:+ allowed = {"hadamard", "random_hadamard", "quarot_hadamard"}+ if v not in allowed:+ raise ValueError(f"Unsupported hadamard_type: {v}. Supported values: {sorted(allowed)}")+ return vdiff --git a/auto_round/experimental/utils.py b/auto_round/experimental/utils.pyindex 8527dbe00..ea1dade78 100644--- a/auto_round/experimental/utils.py+++ b/auto_round/experimental/utils.py@@ -17,8 +17,8 @@ from typing import Anyimport torchfrom auto_round.compressors.utils import is_mx_fp, is_nv_fp-from auto_round.experimental.transform.hadamard_config import HadamardConfigfrom auto_round.experimental.transform.hadamards import HADAMARDS+from auto_round.experimental.transform.rotation_config import RotationConfigfrom auto_round.utils import loggerSUPPORTED_QUANTIZATION_SCHEMES = ["MXFP8", "MXFP4", "NVFP4"]@@ -137,14 +137,38 @@ def is_triton_kernel_available(data_type: str) -> bool:return True-def normalize_hadamard_config(hadamard_config: str | dict | HadamardConfig | None, data_type: str) -> dict[str, Any]:+def dump_group_size_to_rotation_config(rotation_config: str | dict | RotationConfig, group_size: int):+ rotation_dict = to_dict_rotation_config(rotation_config)+ if rotation_dict.get("block_size", None) is None:+ rotation_dict["block_size"] = group_size+ return rotation_dict+++def to_dict_rotation_config(rotation_config: str | dict | RotationConfig):+ if isinstance(rotation_config, str):+ key = rotation_config.strip()+ if not key:+ return {}++ if key == "default":+ cfg_dict = {"hadamard_type": "hadamard"}+ else:+ cfg_dict = {"hadamard_type": key}+ elif isinstance(rotation_config, RotationConfig):+ cfg_dict = rotation_config.model_dump()+ else:+ cfg_dict = dict(rotation_config)+ return cfg_dict+++def normalize_rotation_config(rotation_config: str | dict | RotationConfig | None, data_type: str) -> dict[str, Any]:"""- Normalize and validate `hadamard_config`.+ Normalize and validate `rotation_config`.Supported input types:- None -> {}- - dict -> validated via HadamardConfig- - HadamardConfig -> validated & converted to dict+ - dict -> validated via RotationConfig+ - RotationConfig -> validated & converted to dict- str -> shorthand for `hadamard_type` in HADAMARDS keysAdditional behavior:@@ -171,7 +195,7 @@ def normalize_hadamard_config(hadamard_config: str | dict | HadamardConfig | Nonelse:logger.warning(f"block_size is not set and cannot be inferred for data_type {data_type!r}; "- "please set block_size explicitly in hadamard_config if needed."+ "please set block_size explicitly in rotation_config if needed.")else:if is_mx_fp(data_type) and block_size != 32:@@ -182,62 +206,16 @@ def normalize_hadamard_config(hadamard_config: str | dict | HadamardConfig | Nonreturn cfg_dict# 1) None -> {}- if hadamard_config is None:+ if rotation_config is None:return {}- # 2) HadamardConfig instance- if isinstance(hadamard_config, HadamardConfig):- raw_cfg_dict = hadamard_config.model_dump(exclude_unset=True)- block_size_explicitly_set = "block_size" in raw_cfg_dict-- cfg_dict = dict(raw_cfg_dict)- cfg_dict = _apply_data_type_block_size(cfg_dict, block_size_explicitly_set)-- try:- return HadamardConfig.model_validate(cfg_dict).model_dump()- except Exception as e:- raise ValueError(f"Invalid HadamardConfig: {e}") from e-- # 3) dict- if isinstance(hadamard_config, dict):- block_size_explicitly_set = "block_size" in hadamard_config-- cfg_dict = dict(hadamard_config)- cfg_dict = _apply_data_type_block_size(cfg_dict, block_size_explicitly_set)-- try:- return HadamardConfig.model_validate(cfg_dict).model_dump()- except Exception as e:- raise ValueError(f"Invalid hadamard_config dict: {e}") from e-- # 4) str -> shorthand for hadamard_type- if isinstance(hadamard_config, str):- key = hadamard_config.strip()- if not key:- return {}-- if key == "default":- cfg_dict = {}- cfg_dict = _apply_data_type_block_size(cfg_dict, block_size_explicitly_set=False)- try:- return HadamardConfig.model_validate(cfg_dict).model_dump()- except Exception as e:- raise ValueError(f"Invalid default hadamard_config after data_type adjustment: {e}") from e-- if key not in HADAMARDS:- raise ValueError(f"Invalid hadamard_config string: {key!r}. Expected one of {sorted(HADAMARDS.keys())}.")-- cfg_dict = {"hadamard_type": key}- cfg_dict = _apply_data_type_block_size(cfg_dict, block_size_explicitly_set=False)-- try:- return HadamardConfig.model_validate(cfg_dict).model_dump()- except Exception as e:- raise ValueError(f"hadamard_config built from string {key!r} is invalid for HadamardConfig: {e}") from e-- raise TypeError(- "hadamard_config must be one of: None, dict, HadamardConfig, or str " f"(got {type(hadamard_config).__name__})"- )+ rotation_dict = to_dict_rotation_config(rotation_config)+ block_size_explicitly_set = "block_size" in rotation_dict+ cfg_dict = _apply_data_type_block_size(rotation_dict, block_size_explicitly_set)+ try:+ return RotationConfig.model_validate(cfg_dict).model_dump()+ except Exception as e:+ raise ValueError(f"Invalid RotationConfig: {e}") from edef check_supported_schemes(scheme: str):diff --git a/auto_round/formats.py b/auto_round/formats.pyindex 40f984cca..68d392d48 100644--- a/auto_round/formats.py+++ b/auto_round/formats.py@@ -269,7 +269,7 @@ class OutputFormat(ABC):if ar.act_bits <= 8 and (not is_standard_fp(ar.act_data_type) or ar.act_dynamic) and not is_block_dynamic_fp8:logger.warning(- f"{self.format_name} format not support for current activation quantization configuration,"+ f"{self.format_name} format does not support the current activation quantization configuration,"" reset to fake format and save.")return "fake"diff --git a/auto_round/inference/convert_model.py b/auto_round/inference/convert_model.pyindex e769e1eca..e8a2779ec 100644--- a/auto_round/inference/convert_model.py+++ b/auto_round/inference/convert_model.py@@ -214,7 +214,7 @@ def get_layer_config(model, quantization_config):act_data_type = getattr(quantization_config, "act_data_type", None)act_dynamic = getattr(quantization_config, "act_dynamic", False)- hadamard_config = getattr(quantization_config, "hadamard_config", None)+ rotation_config = getattr(quantization_config, "rotation_config", None)default_quant_scheme = QuantizationScheme(bits=bits,@@ -226,7 +226,7 @@ def get_layer_config(model, quantization_config):act_sym=act_sym,act_data_type=act_data_type,act_dynamic=act_dynamic,- hadamard_config=hadamard_config,+ rotation_config=rotation_config,)# Determine the quantization block list@@ -680,19 +680,19 @@ def convert_hf_model(model: nn.Module, target_device: str = "cpu") -> tuple[nn.Mlayer_configs = get_layer_config(model, quantization_config)used_backends = _replace_by_quant_layers(model, layer_configs, backend, target_device, packing_format)- hadamard_config = getattr(quantization_config, "hadamard_config", None)- if hadamard_config is not None and hadamard_config:- from auto_round.experimental.transform.apply import apply_hadamard_transform- from auto_round.experimental.transform.hadamard_config import HadamardConfig+ rotation_config = getattr(quantization_config, "rotation_config", None)+ if rotation_config is not None and rotation_config:+ from auto_round.experimental.transform.apply import apply_rotation_transform+ from auto_round.experimental.transform.rotation_config import RotationConfig# apply forward hook- act_hadamard_config = HadamardConfig(- block_size=hadamard_config["block_size"],- hadamard_type=hadamard_config["hadamard_type"],+ act_rotation_config = RotationConfig(+ block_size=rotation_config["block_size"],+ hadamard_type=rotation_config["hadamard_type"],) # apply to activation- model = apply_hadamard_transform(+ model, _ = apply_rotation_transform(model,- act_hadamard_config,+ act_rotation_config,location="input",desc="Register pre forward hook for hadamard transform",data_type=quantization_config.data_type,diff --git a/auto_round/schemes.py b/auto_round/schemes.pyindex 0cfd757bb..0e9b922ba 100644--- a/auto_round/schemes.py+++ b/auto_round/schemes.py@@ -36,7 +36,7 @@ class QuantizationScheme:act_dynamic: Optional[bool] = Nonesuper_bits: Optional[int] = Nonesuper_group_size: Optional[int] = None- hadamard_config: Optional[dict] = None+ rotation_config: Optional[dict] = None@classmethoddef from_dict(cls, config: dict):@@ -88,7 +88,7 @@ class QuantizationScheme:continueself_val = getattr(self, field)other_val = getattr(other, field)- # Treat None and empty dict as equivalent for dict fields like hadamard_config+ # Treat None and empty dict as equivalent for dict fields like rotation_configif self_val != other_val:if isinstance(self_val, dict) and not self_val and other_val is None:continue@@ -206,6 +206,19 @@ MXFP4 = QuantizationScheme.from_dict(})+MXFP6 = QuantizationScheme.from_dict(+ {+ "bits": 6,+ "group_size": 32,+ "data_type": "mx_fp",+ "act_bits": 6,+ "act_data_type": "mx_fp",+ "act_group_size": 32,+ "act_sym": True,+ "act_dynamic": True,+ }+)+MXFP4_RCEIL = QuantizationScheme.from_dict({"bits": 4,@@ -330,6 +343,20 @@ INT8 = QuantizationScheme.from_dict(})+INT4 = QuantizationScheme.from_dict(+ {+ "bits": 4,+ "group_size": -1,+ "data_type": "int",+ "sym": True,+ "act_bits": 4,+ "act_group_size": -1,+ "act_data_type": "int",+ "act_dynamic": True,+ "act_sym": True,+ }+)+# For AutoScheme 16 bits optionsBF16 = QuantizationScheme.from_dict(@@ -348,6 +375,7 @@ PRESET_SCHEMES = {"W3A16": W3A16,"W8A16": W8A16,"MXFP4": MXFP4,+ "MXFP6": MXFP6,"MXFP4_RCEIL": MXFP4_RCEIL,"MXFP8": MXFP8,"MXFP8_RCEIL": MXFP8_RCEIL,@@ -358,6 +386,7 @@ PRESET_SCHEMES = {"FP8_STATIC": FP8_STATIC,"BF16": BF16,"W4A16_MIXED": W4A16,+ "INT4": INT4,"INT8": INT8,"INT8_W8A8": INT8,"FP8_BLOCK": FP8_BLOCK,diff --git a/auto_round/wrapper.py b/auto_round/wrapper.pyindex 782786a13..a48a654be 100644--- a/auto_round/wrapper.py+++ b/auto_round/wrapper.py@@ -12,12 +12,14 @@# See the License for the specific language governing permissions and# limitations under the License.+import mathfrom math import ceilimport torchimport transformersfrom torch.functional import F+import auto_round.envs as envsfrom auto_round.compressors.utils import is_nv_fpfrom auto_round.data_type import get_quant_func, reshape_pad_tensor_by_group_sizefrom auto_round.logger import logger@@ -92,6 +94,7 @@ class WrapperLinear(torch.nn.Module):"""super(WrapperLinear, self).__init__()self.orig_layer = orig_layer+ self.orig_layer.iters = kwargs.pop("iters", 200)self.disable_opt_rtn = disable_opt_rtnself.output_device = deviceself.device = self.orig_layer.tuning_device if hasattr(self.orig_layer, "tuning_device") else device@@ -169,18 +172,30 @@ class WrapperLinear(torch.nn.Module):self._init_params("max_scale", p_dtype, shape, 1.0, (self.enable_minmax_tuning and self.orig_layer.bits < 16))self.weight_quant_func, self.data_type = get_quant_func(- orig_layer.data_type, orig_layer.bits, orig_layer.sym, self.disable_opt_rtn, orig_layer.group_size+ orig_layer.data_type,+ orig_layer.bits,+ orig_layer.sym,+ self.disable_opt_rtn,+ orig_layer.group_size,+ iters=orig_layer.iters,)if self.enable_torch_compile:self.weight_quant_func = compile_func(self.weight_quant_func, self.device)if self.enable_act_quant:self.act_quant_func, self.act_data_type = get_quant_func(- orig_layer.act_data_type, orig_layer.act_bits, orig_layer.act_sym, self.disable_opt_rtn+ orig_layer.act_data_type,+ orig_layer.act_bits,+ orig_layer.act_sym,+ disable_opt_rtn=True,+ iters=orig_layer.iters,)if self.enable_torch_compile:self.act_quant_func = compile_func(self.act_quant_func, self.device)- self._init_params("act_max_scale", p_dtype, (1), 1.0, not orig_layer.act_dynamic)+ self._init_params(+ "act_max_scale", p_dtype, (1), 1.0, envs.AR_ENABLE_ACT_MINMAX_TUNING or (not orig_layer.act_dynamic)+ )+ self._init_params("act_min_scale", p_dtype, (1), 1.0, envs.AR_ENABLE_ACT_MINMAX_TUNING)# Bias tuningif self.enable_norm_bias_tuning:@@ -221,8 +236,8 @@ class WrapperLinear(torch.nn.Module):"""if self.orig_layer.bits >= 16:return self.orig_layer.weight, None, None- min_scale.data.clamp_(0, 1.0)- max_scale.data.clamp_(0, 1.0)+ min_scale.data.clamp_(0.0, 1.0)+ max_scale.data.clamp_(0.0, 1.0)weight = self.orig_layer.weightif weight.device.type == "meta":weight = self.orig_layer.get_weight().to(self.device)@@ -255,7 +270,7 @@ class WrapperLinear(torch.nn.Module):weight_q = weight_q.t()return weight_q, scale, zp- def _qdq_act(self, x, act_max_scale, act_max=None):+ def _qdq_act(self, x, act_min_scale=torch.tensor(1.0), act_max_scale=torch.tensor(1.0), act_max=None):"""Quantizes and dequantizes activations.Args:@@ -267,6 +282,8 @@ class WrapperLinear(torch.nn.Module):tuple: Quantized activation, scale, and zero point."""act_max_scale.data.clamp_(0, 1.0)+ act_min_scale.data.clamp_(0, 1.0)+ env_act_scale = envs.AR_ACT_SCALE # fixed activation ratio,prioritize to use this one if setx, scale, zp = self.act_quant_func(x,bits=self.orig_layer.act_bits,@@ -274,8 +291,9 @@ class WrapperLinear(torch.nn.Module):scale_dtype=self.orig_layer.scale_dtype,q_scale_thresh=self.q_scale_thresh,data_type=self.act_data_type,- max_scale=act_max_scale,- tensor_max=act_max,+ tensor_max=act_max, # for static+ max_scale=act_max_scale if math.isclose(env_act_scale, 1.0, rel_tol=1e-6) else env_act_scale,+ min_scale=act_min_scale if math.isclose(env_act_scale, 1.0, rel_tol=1e-6) else env_act_scale,global_scale=getattr(self, "input_global_scale", None),)return x, scale, zp@@ -394,7 +412,10 @@ class WrapperLinear(torch.nn.Module):elif self.orig_layer.act_group_size == -1:tmp_shape = (act_max.shape[0], 1)_, act_scale, _ = self._qdq_act(- torch.zeros(tmp_shape).to(self.device), act_max_scale=self.act_max_scale, act_max=act_max+ torch.zeros(tmp_shape).to(self.device),+ act_min_scale=self.act_min_scale,+ act_max_scale=self.act_max_scale,+ act_max=act_max,)self.orig_layer.act_max = self.orig_layer.act_max * act_max_scale.item()self.orig_layer.act_max = self.orig_layer.act_max.to("cpu")@@ -404,6 +425,8 @@ class WrapperLinear(torch.nn.Module):self.orig_layer.q_scale_thresh = self.q_scale_threshself.orig_layer.data_type = self.data_type+ self.orig_layer.act_min_scale = self.act_min_scale+ self.orig_layer.act_max_scale = self.act_max_scaleself.orig_layer.act_data_type = self.act_data_typeself.orig_layer.act_quant_func = self.act_quant_func@@ -477,8 +500,23 @@ class WrapperLinear(torch.nn.Module):weight_q, *_ = self._qdq_weight(self.value, self.min_scale, self.max_scale)if self.enable_act_quant:+ # Run orig_layer's forward_pre_hooks (e.g., online Hadamard transform)+ # BEFORE activation quantization, to match inference behavior in WrapperWALayer.+ for hook in self.orig_layer._forward_pre_hooks.values():+ result = hook(self.orig_layer, (x,))+ if result is not None:+ x = result[0] if isinstance(result, tuple) else resultact_max = self.orig_layer.act_max if hasattr(self.orig_layer, "act_max") else None- x, _, _ = self._qdq_act(x, act_max_scale=self.act_max_scale, act_max=act_max)+ x, _, _ = self._qdq_act(+ x, act_max_scale=self.act_max_scale, act_min_scale=self.act_min_scale, act_max=act_max+ )+ elif len(self.orig_layer._forward_pre_hooks) > 0:+ # Even without act_quant, run pre-hooks for online Hadamard correctness+ # (when fuse_online_to_weight=False, pre-hooks transform activations).+ for hook in self.orig_layer._forward_pre_hooks.values():+ result = hook(self.orig_layer, (x,))+ if result is not None:+ x = result[0] if isinstance(result, tuple) else result# pylint: disable=not-callablebias = self.orig_layer.bias@@ -488,6 +526,14 @@ class WrapperLinear(torch.nn.Module):bias, _, _ = self._qdq_bias(bias, self.bias_v)output = self.orig_forward(x, weight_q, bias).to(self.output_device)++ # Execute post-hooks from orig_layer (e.g., v_proj per-head Hadamard+ # when online rotation is not fused into weights).+ for hook in self.orig_layer._forward_hooks.values():+ hook_result = hook(self.orig_layer, (x,), output)+ if hook_result is not None:+ output = hook_result+return output@@ -504,6 +550,12 @@ class WrapperWALayer(torch.nn.Module):self.act_quant_func = compile_func(self.act_quant_func, self.device)self.extra_repr_org = orig_layer.extra_repr+ # Steal forward_pre_hooks from orig_layer (e.g., Hadamard transform hooks)+ # and remove them from orig_layer so they won't fire again inside orig_layer.forward().+ # We will run them explicitly in our forward() BEFORE activation quantization.+ self._stolen_pre_hooks = list(orig_layer._forward_pre_hooks.values())+ orig_layer._forward_pre_hooks.clear()+@propertydef weight(self):"""Exposes the weight of the wrapped layer for external access."""@@ -515,16 +567,42 @@ class WrapperWALayer(torch.nn.Module):return self.orig_layer.biasdef forward(self, x):+ # 1) Run stolen pre_hooks first (e.g., online Hadamard) → smooths activation+ for hook in self._stolen_pre_hooks:+ result = hook(self.orig_layer, (x,))+ if result is not None:+ x = result[0] if isinstance(result, tuple) else result++ # 2) Activation quantization on the smoothed activation+ import auto_round.envs as envs++ act_scale = envs.AR_ACT_SCALEact_max = self.orig_layer.act_max if hasattr(self.orig_layer, "act_max") else None- x, _, _ = self.orig_layer.act_quant_func(- x,- bits=self.orig_layer.act_bits,- group_size=self.orig_layer.act_group_size,- scale_dtype=self.orig_layer.scale_dtype,- q_scale_thresh=self.orig_layer.q_scale_thresh,- data_type=self.orig_layer.act_data_type,- tensor_max=act_max,- )++ max_scale = self.orig_layer.act_max_scale if math.isclose(act_scale, 1.0, rel_tol=1e-6) else act_scale+ min_scale = self.orig_layer.act_min_scale if math.isclose(act_scale, 1.0, rel_tol=1e-6) else act_scale+ if act_max is None:+ x, _, _ = self.orig_layer.act_quant_func(+ x,+ bits=self.orig_layer.act_bits,+ group_size=self.orig_layer.act_group_size,+ scale_dtype=self.orig_layer.scale_dtype,+ q_scale_thresh=self.orig_layer.q_scale_thresh,+ data_type=self.orig_layer.act_data_type,+ min_scale=min_scale,+ max_scale=max_scale,+ )+ else:+ x, _, _ = self.orig_layer.act_quant_func(+ x,+ bits=self.orig_layer.act_bits,+ group_size=self.orig_layer.act_group_size,+ scale_dtype=self.orig_layer.scale_dtype,+ q_scale_thresh=self.orig_layer.q_scale_thresh,+ data_type=self.orig_layer.act_data_type,+ act_max=act_max,+ )+ # 3) Linear computation via orig_layer (pre_hooks already removed, no double execution)return self.orig_layer.forward(x)def extra_repr(self):@@ -640,6 +718,7 @@ NORM_MAPPING["Qwen2RMSNorm"] = WrapperLlamaNormNORM_MAPPING["Phi3RMSNorm"] = WrapperLlamaNormNORM_MAPPING["MistralRMSNorm"] = WrapperLlamaNormNORM_MAPPING["Qwen3RMSNorm"] = WrapperLlamaNorm+NORM_MAPPING["Qwen3_5MoeRMSNorm"] = WrapperLlamaNormclass WrapperMultiblock(torch.nn.Module):diff --git a/docs/environments.md b/docs/environments.mdindex 242d660d7..ae1052976 100644--- a/docs/environments.md+++ b/docs/environments.md@@ -60,7 +60,7 @@ export AR_DISABLE_OFFLOAD=1```### AR_DISABLE_DATASET_SUBPROCESS-- **Description**: Disables the use of a subprocess for dataset preprocessing. By default, AutoRound uses a subprocess to ensure all temporary memory is reclaimed by the OS.+- **Description**: Only for research. Disables the use of a subprocess for dataset preprocessing. By default, AutoRound uses a subprocess to ensure all temporary memory is reclaimed by the OS.- **Default**: `False`- **Valid Values**: `"1"`, `"true"` (case-insensitive) for disabling; any other value for enabling- **Usage**: Set this to run dataset preprocessing in the main process@@ -69,6 +69,36 @@ export AR_DISABLE_OFFLOAD=1export AR_DISABLE_DATASET_SUBPROCESS=true```+### AR_ACT_SCALE+- **Description**: Only for research. Controls the scaling factor applied to activation min/max values during activation quantization. A value less than 1.0 shrinks the clipping range, which can reduce outlier impact.+- **Default**: `1.0`+- **Valid Values**: float>=0.0, e.g. `0.8`, `0.9`, `1.0`+- **Usage**: Set this to adjust the activation clipping range++```bash+export AR_ACT_SCALE=0.9+```++### AR_ENABLE_ACT_MINMAX_TUNING+- **Description**: Enables tuning of activation min/max scale parameters (`act_min_scale`, `act_max_scale`) during quantization optimization. When enabled, these scales become tunable instead of remaining fixed at `1.0`.+- **Default**: `False` (equivalent to `"0"`)+- **Valid Values**: `"1"`, `"true"`, `"yes"` (case-insensitive) for enabling tuning; any other value keeps tuning disabled+- **Usage**: Set this to enable activation min-max scale tuning++```bash+export AR_ENABLE_ACT_MINMAX_TUNING=1+```++### AR_SEARCH_SCALE_RATIO+- **Description**: Controls the search range ratio used by the symmetric INT scale search in `auto_round.data_type.int.search_scales`. The search bound is `nmax * AR_SEARCH_SCALE_RATIO`, where `nmax = 2^(bits-1)`. Smaller values restrict the search to a tighter neighborhood around the initial scale (faster, less thorough); larger values broaden the search (slower, may improve accuracy on outlier-heavy weights).+- **Default**: unset → falls back to the built-in default (`0.5`, i.e. `nmax/2`).+- **Valid Values**: positive float, e.g. `0.25`, `0.5`, `0.75`, `1.0`+- **Usage**: Set this to override the default scale-search range++```bash+export AR_SEARCH_SCALE_RATIO=0.75+```+## Usage Examples### Setting Environment Variablesdiff --git a/docs/environments_CN.md b/docs/environments_CN.mdindex 7407354cf..c360f5b99 100644--- a/docs/environments_CN.md+++ b/docs/environments_CN.md@@ -69,6 +69,36 @@ export AR_DISABLE_OFFLOAD=1export AR_DISABLE_DATASET_SUBPROCESS=true```+### AR_ACT_SCALE+- **描述**:只用于研究性质,控制激活量化时对激活值最小/最大值的缩放系数。小于 1.0 的值会缩小裁剪范围,有助于减小离群值的影响。+- **默认值**:`1.0`+- **有效值**:任意浮点数,如 `0.8`、`0.9`、`1.0`+- **用途**:调整激活裁剪范围++```bash+export AR_ACT_SCALE=0.9+```++### AR_ENABLE_ACT_MINMAX_TUNING+- **描述**:只用于研究性质,使用激活量化中最小/最大缩放参数(`act_min_scale`、`act_max_scale`)的调优。启用后,这些缩放参数将固定为 1.0。+- **默认值**:`False`(等同于 `"0"`)+- **有效值**:`"1"`、`"true"`、`"yes"`(不区分大小写)表示禁用调优;其他值表示保持调优+- **用途**:禁用激活最小-最大缩放参数的调优++```bash+export AR_ENABLE_ACT_MINMAX_TUNING=1+```++### AR_SEARCH_SCALE_RATIO+- **描述**:控制 `auto_round.data_type.int.search_scales` 中对称 INT 量化 scale 搜索的范围比例。搜索上界为 `nmax * AR_SEARCH_SCALE_RATIO`,其中 `nmax = 2^(bits-1)`。值越小搜索范围越窄(更快,但可能漏掉较优解);值越大搜索范围越广(更慢,对离群权重可能更准)。+- **默认值**:未设置 → 走内置默认值(`0.5`,即 `nmax/2`)。+- **有效值**:正浮点数,如 `0.25`、`0.5`、`0.75`、`1.0`+- **用途**:覆盖默认的 scale 搜索范围++```bash+export AR_SEARCH_SCALE_RATIO=0.75+```+## 使用示例### 设置环境变量diff --git a/docs/int_acc.md b/docs/int_acc.mdnew file mode 100644index 000000000..e3f09a39c--- /dev/null+++ b/docs/int_acc.md@@ -0,0 +1,59 @@++### Algorithm Details++| Alg | Rotation Type | Activation Clip Ratio | weight quantizer | w/a symmetric |+|-----------|---------------|-----------------------|------------------|--------------|+| Quarot | fixed | 0.9 | GPTQ | yes |+| FlatQuant | learnable | learnable | GPTQ | yes |+| Ours | fixed | 0.9 | SignRound | yes |++### Eval Details.++lm-eval 0.4.11 for all the data evaluation.++We report the average of 10 tasks (AVG10) as well as the individual performance on each task. The 10 tasks are: ARC-Challenge (arc_c), ARC-Easy (arc_e), BoolQ (boolq), HellaSwag (hellaswag), Lambada (lambada), MMLU (mmlu), OpenBookQA (obqa), PiQA (piqa), TriviaQA (tfq), and Winogrande (winog).++Following Quarot and FlatQuant, we report `acc_norm` instead of accuracy for tasks where acc_norm is available.++All results below `exclude` KV-cache quantization, as it is not yet supported in our implementation.++### How to Reproduce+~~~ bash+AR_ACT_SCALE=0.9 auto-round Qwen/Qwen3-8B --scheme INT4 --rotation_type hadamard --format fake --tasks mmlu+~~~+### Qwen3-8B++| Method | AVG10 | arc_c | arc_e | boolq | hellaswag | lambada | mmlu | obqa | piqa | tfq | winog |+|-----------|------------|------------|------------|------------|------------|------------|------------|-----------|------------|------------|------------|+| 16 bit | 0.6598 | 0.5640 | 0.8068 | 0.8670 | 0.7491 | 0.6420 | 0.7288 | 0.420 | 0.7764 | 0.3647 | 0.6788 |+| QuaRot | 0.6123 | 0.5060 | 0.7428 | 0.8220 | 0.6836 | 0.5762 | 0.6545 | 0.412 | 0.7383 | 0.3464 | 0.6409 |+| FlatQuant | 0.6511 | 0.5657 | 0.8072 | 0.8609 | 0.7256 | 0.6369 | 0.7032 | 0.410 | **0.7682** | 0.3525 | 0.6803 |+| Ours | **0.6533** | **0.5614** | **0.8035** | **0.8657** | **0.7336** | **0.6336** | **0.7188** | **0.408** | 0.7704 | **0.3562** | **0.6819** |++### Llama-3-8B++| Method | AVG10 | arc_c | arc_e | boolq | hellaswag | lambada | mmlu | obqa | piqa | tfq | winog |+|-----------|------------|------------|------------|------------|------------|------------|------------|-----------|------------|------------|------------|+| 16 bit | 0.6569 | 0.5392 | 0.7769 | 0.8190 | 0.7919 | 0.7545 | 0.6216 | 0.452 | 0.8069 | 0.2717 | 0.7356 |+| QuaRot | 0.5924 | 0.4326 | 0.7155 | 0.7526 | 0.7342 | 0.6810 | 0.5007 | 0.426 | 0.7492 | 0.2546 | 0.6772 |+| FlatQuant | 0.6442 | 0.5213 | 0.7706 | 0.8168 | 0.7724 | 0.7363 | 0.5875 | 0.438 | **0.7987** | **0.2754** | 0.7253 |+| Ours | **0.6521** | **0.5452** | **0.7862** | **0.8119** | **0.7832** | **0.7460** | **0.6107** | **0.436** | 0.8079 | 0.2668 | **0.7269** |++### Llama-2-7B++| Method | AVG10 | arc_c | arc_e | boolq | hellaswag | lambada | mmlu | obqa | piqa | tfq | winog |+|-----------|------------|------------|------------|------------|------------|------------|------------|-----------|------------|------------|------------|+| 16 bit | 0.6071 | 0.4505 | 0.7382 | 0.7939 | 0.7615 | 0.7367 | 0.4177 | 0.442 | 0.7878 | 0.2485 | 0.6946 |+| QuaRot | 0.5742 | 0.4352 | 0.6907 | 0.7388 | 0.7275 | 0.7093 | 0.3527 | 0.402 | 0.7715 | 0.2509 | 0.6630 |+| FlatQuant | 0.5954 | 0.4454 | 0.7239 | 0.7743 | 0.7452 | 0.7233 | **0.3920** | 0.430 | **0.7840** | 0.2448 | **0.6914** |+| Ours | **0.5991** | **0.4403** | **0.7281** | **0.7902** | **0.7495** | **0.7256** | 0.3915 | **0.450** | 0.7802 | **0.2497** | 0.6859 |++### Llama-2-13B++| Method | AVG10 | arc_c | arc_e | boolq | hellaswag | lambada | mmlu | obqa | piqa | tfq | winog |+|-----------|------------|------------|------------|------------|------------|------------|------------|-----------|------------|------------|------------|+| 16 bit | 0.6446 | 0.4881 | 0.7652 | 0.8217 | 0.7965 | 0.7652 | 0.5239 | 0.452 | 0.8030 | 0.2681 | 0.7621 |+| QuaRot | 0.6114 | 0.4573 | 0.7348 | 0.7872 | 0.7644 | 0.7444 | 0.4584 | 0.430 | 0.7818 | 0.2546 | 0.7009 |+| FlatQuant | 0.6308 | 0.4795 | 0.7534 | 0.8180 | 0.7817 | 0.7588 | 0.4985 | 0.446 | **0.7982** | **0.2583** | 0.7151 |+| Ours | **0.6369** | **0.4804** | **0.7626** | **0.8245** | **0.7886** | **0.7720** | **0.5106** | **0.450** | 0.8014 | 0.2497 | **0.7293** |+diff --git a/docs/step_by_step.md b/docs/step_by_step.mdindex 0327f4dc1..4ca147926 100644--- a/docs/step_by_step.md+++ b/docs/step_by_step.md@@ -659,8 +659,8 @@ from auto_round import AutoRoundmodel_name_or_path = "meta-llama/Llama-3.1-8B-Instruct"output_dir = "./Llama-3.1-8B-Instruct-mxfp4-ht"-# hadamard_config="default": block_size=32, hadamard_type="hadamard"-ar = AutoRound(model_name_or_path, scheme="MXFP4", hadamard_config="default")+# rotation_config="default": block_size=32, hadamard_type="hadamard"+ar = AutoRound(model_name_or_path, scheme="MXFP4", rotation_config="default")ar.quantize_and_save(output_dir=output_dir, format="auto_round")```diff --git a/docs/step_by_step_CN.md b/docs/step_by_step_CN.mdindex d535718a6..0e2c6bec1 100644--- a/docs/step_by_step_CN.md+++ b/docs/step_by_step_CN.md@@ -619,8 +619,8 @@ from auto_round import AutoRoundmodel_name_or_path = "meta-llama/Llama-3.1-8B-Instruct"output_dir = "./Llama-3.1-8B-Instruct-mxfp4-ht"-# hadamard_config="default": block_size=32, hadamard_type="hadamard"-ar = AutoRound(model_name_or_path, scheme="MXFP4", hadamard_config="default")+# rotation_config="default": block_size=32, hadamard_type="hadamard"+ar = AutoRound(model_name_or_path, scheme="MXFP4", rotation_config="default")ar.quantize_and_save(output_dir=output_dir, format="auto_round")```diff --git a/test/test_cuda/transform/test_mxfp4_transform.py b/test/test_cuda/transform/test_mxfp4_transform.pyindex fba84e678..16511edad 100644--- a/test/test_cuda/transform/test_mxfp4_transform.py+++ b/test/test_cuda/transform/test_mxfp4_transform.py@@ -36,7 +36,7 @@ class TestAutoRound:iters=0,seqlen=2,scheme=scheme,- hadamard_config="default",+ rotation_config="default",)compressed_model, _ = ar.quantize_and_save(output_dir=self.save_dir, format="auto_round")@@ -55,7 +55,7 @@ class TestAutoRound:iters=2,seqlen=2,scheme=scheme,- hadamard_config="default",+ rotation_config="default",)compressed_model, _ = ar.quantize_and_save(output_dir=self.save_dir, format="auto_round")@@ -74,7 +74,7 @@ class TestAutoRound:iters=0,seqlen=2,scheme=scheme,- hadamard_config="random_hadamard",+ rotation_config="random_hadamard",)compressed_model, _ = ar.quantize_and_save(output_dir=self.save_dir, format="auto_round")