PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
Rotation Transform — Details (Research)
⚠️ Research feature. Rotation transform is still in a research stage. Inference relies on forward hooks, which are currently only supported by the Hugging Face Transformers backend. As a result, inference may be slower compared to native (non-rotated) models.
This document is the in-depth reference for AutoRound's rotation transforms. For a concise getting-started guide, see the Rotation section in step_by_step.md.
Rotation redistributes outliers in weights and activations before quantization by
multiplying tensors with orthogonal (Hadamard) matrices. Because an orthogonal
rotation R satisfies R @ Rᵀ = I, a matched pair of rotations inserted around a
linear operation leaves the network's output unchanged (x @ R @ Rᵀ @ Wᵀ = x @ Wᵀ)
while making the intermediate distributions flatter and more quantization-friendly.
This is most useful for aggressive low-bit schemes such as MXFP4, NVFP4 and W4A4.
AutoRound provides two independent rotation implementations:
- QuaRot / SpinQuant — architecture-aware, full-model rotation applied at up to
four positions (R1–R4). Selected via
alg_configs=[..., "quarot"],alg_configs=[..., "spinquant"], or aSpinQuantConfiginstance. Recommended. - Per-Linear Block Rotation — an earlier, simpler implementation that applies a
block-diagonal Hadamard uniformly to every
nn.Linear. Selected viaalg_configs=[..., "hadamard"]/ a string Hadamard type, or aRotationConfiginstance. Also exposed through the--algorithm hadamardCLI.
1. QuaRot / SpinQuant
QuaRot applies deterministic Hadamard rotations at up to four positions in the transformer architecture. Unlike per-linear block rotation, it operates at the model architecture level — rotating the residual stream, attention heads, and MLP activations at specific positions for targeted outlier suppression.
The pipeline (implemented in auto_round/algorithms/transforms/spinquant/preprocessor.py,
SpinQuantPreprocessor) runs the following stages:
- Fuse RMSNorm scale parameters into the following linear layers (
fuse_rmsnorm). - Untie word embeddings if they are shared with the LM head (
untie_embeddings). - Initialize rotation matrices R1–R4.
- (Only if trainable) train the rotations and smooth values via KL/MSE loss.
- Fuse the offline rotations (R2, and R1 when
online_r1_rotation=False) into weights. - Register the online hooks (R1 when online, R3, R4) and clean up.
1.1 Rotation positions
| Position | Target | Dimension | Mode | Effect |
|---|---|---|---|---|
| R1 | Residual stream — inputs of q/k/v_proj, gate/up_proj; outputs of o_proj, down_proj, embed_tokens, lm_head |
hidden_size (or rotation_size) |
Online or Offline | Smooths weight outliers across all linear layers |
| R2 | v_proj output / o_proj input (per attention head) |
head_dim |
Offline (fused) | Balances per-head value distributions |
| R3 | Q/K after RoPE | head_dim |
Online (RoPE monkeypatch) | Improves KV-cache quantization friendliness |
| R4 | down_proj input (MLP activation) |
intermediate_size (or rotation_size) |
Online (hook) + weight fuse | Suppresses activation outliers in the FFN |
- Online: applied at runtime via forward hooks / a RoPE monkeypatch. Weights are pre-rotated so the paired inverse rotation happens at runtime; there is a small inference cost.
- Offline (fused): the rotation is absorbed into adjacent weight matrices, leaving no runtime overhead.
R1 is applied online by default (online_r1_rotation=True): the target weights are
rotated and a forward_pre_hook is registered on the same modules. When
online_r1_rotation=False, R1 is instead fully fused into embed_tokens,
q/k/v_proj, o_proj, gate/up/down_proj and lm_head, cancelling out with no
runtime cost.
1.2 SpinQuantConfig reference
Defined in auto_round/algorithms/transforms/spinquant/preprocessor.py.
| Field | Default | Description |
|---|---|---|
algorithm |
"spinquant" |
Registry key used by BaseRotation.from_config() to dispatch. |
r1 |
True |
Enable R1 (hidden_size residual-stream rotation). |
r2 |
True |
Enable R2 (head_dim per-head rotation, offline). |
r3 |
False |
Enable R3 (online Q/K rotation after RoPE). |
r4 |
False |
Enable R4 (online MLP activation rotation). |
rotation_size |
None (auto) |
Overrides the R1/R4 block size (R1 uses it instead of hidden_size, R4 instead of intermediate_size). R2 always uses head_dim; R3 has no custom size. Must be a positive power of 2. |
random_r1 / random_r2 / random_r3 / random_r4 |
False |
Use random Hadamard (H × diag(±1)) instead of the deterministic matrix at that position. Only relevant in QuaRot mode. |
online_r1_rotation |
True |
Apply R1 online via hook (True) or fuse it fully into weights (False). |
trainable_rotation |
False |
Learn the rotation matrices via Cayley SGD (SpinQuant mode). Research, requires a dataloader. |
trainable_smooth |
False |
Learn SmoothQuant-style smooth_values jointly via Adam. Research, requires a dataloader. |
iters |
200 |
Training iterations (trainable modes only). |
lr |
1e-4 |
SGDG (Cayley) learning rate for rotation matrices. |
smooth_lr |
1e-3 |
Adam learning rate for smooth values. |
batch_size |
1 |
Training batch size. |
loss_type |
"kl_top" |
Training loss: "kl_top", "kl_full", or "mse". |
kl_top_k |
1000 |
Top-k logits used by the kl_top loss. |
fuse_rmsnorm |
True |
Fuse RMSNorm scales into following linear layers. |
untie_embeddings |
True |
Untie shared embedding / LM-head weights before rotating. |
layerwise |
False |
Apply rotation per decoder block, in lock-step with AutoRound's block-wise quantization, instead of rotating the whole model up-front (see §1.8). Supported by QuaRot / SpinQuant and the per-Linear Hadamard transform backend; the Hadamard inplace backend ignores it and falls back to full-model rotation. |
dtype |
torch.float32 |
Numerical dtype used for rotation math. |
device |
None (auto) |
Defaults to "cuda" if available, else "cpu". |
1.3 String shortcuts
Both are registered algorithm aliases usable directly in alg_configs:
| Alias | Equivalent |
|---|---|
"quarot" |
SpinQuantConfig(trainable_rotation=False, trainable_smooth=False) — deterministic Hadamard, no training, no calibration data. |
"spinquant" |
SpinQuantConfig(trainable_rotation=True, trainable_smooth=True) — research, requires a dataloader. |
Both presets keep the default r1=True, r2=True, r3=False, r4=False.
⚠️ SpinQuant trainable rotation (
trainable_rotation=True) enables learnable rotation matrices optimized via Cayley SGD. This feature is a research feature and not fully validated on real models. Use"quarot"(fixed Hadamard) for production workloads.
1.4 Usage
from auto_round import AutoRound
from auto_round.algorithms.transforms.spinquant import SpinQuantConfig
model_name = "Qwen/Qwen3-0.6B"
# QuaRot preset: R1+R2 deterministic Hadamard, no training. Rotation configs are
# passed alongside a quantization algorithm through `alg_configs`.
ar = AutoRound(model_name, scheme="MXFP4", alg_configs=["auto_round", "quarot"])
# Choose how many positions to rotate by passing a SpinQuantConfig instance instead:
ar = AutoRound(model_name, scheme="MXFP4", alg_configs=["auto_round", SpinQuantConfig(r1=True)]) # R1 only
ar = AutoRound(model_name, scheme="MXFP4", alg_configs=["auto_round", SpinQuantConfig(r1=True, r2=True)]) # R1+R2
ar = AutoRound(
model_name,
scheme="MXFP4",
alg_configs=["auto_round", SpinQuantConfig(r1=True, r2=True, r3=True, r4=True)],
) # all four
ar.quantize_and_save(output_dir="./Qwen3-0.6B-mxfp4-quarot", format="auto_round")
1.5 Deterministic vs random Hadamard
Rotation matrices are constructed in
auto_round/algorithms/transforms/spinquant/rotation_utils.py:
- Deterministic (default):
deterministic_hadamard_matrix(size)builds a Sylvester Hadamard matrix (H → [[H, H], [H, −H]]) normalized by√size. The size must be a power of 2. Because the matrix is fully determined by its size, nothing needs to be stored. - Random:
random_hadamard_matrix(size)computesH @ diag(±1) / √N. It offers slightly better outlier suppression but the sign matrix must be persisted.
For non-power-of-2 dimensions, get_hadamard_K(n) factors n = K · 2^m and pulls a
known base Hadamard of size K (e.g. 12, 20, 28, 36, 40, 52, 60, 108, 140, 156, 172)
from known_hadamard.py, combining it with a Sylvester block.
# Random Hadamard at every enabled position
ar = AutoRound(
model_name,
scheme="MXFP4",
alg_configs=[
"auto_round",
SpinQuantConfig(
r1=True,
r2=True,
r3=True,
r4=True,
random_r1=True,
random_r2=True,
random_r3=True,
random_r4=True,
),
],
)
1.6 Trainable rotation (SpinQuant mode)
When trainable_rotation=True (and/or trainable_smooth=True), the rotations become
learnable and are optimized on calibration data:
cayley_optimizer.pyprovidesSGDG, an orthogonality-preserving optimizer that keeps each rotation on the Stiefel manifold using a Cayley/QR retraction.AdamAndSGDGcombines Adam (forsmooth_values) with SGDG (for thespinquant_R*matrices).training.pyruns a KL-/MSE-based distillation loop against a frozen clone of the original model (create_dual_optimizer(),run_training_loop()).SpinQuantPreprocessor.preprocess()requires a dataloader whenever either trainable flag is set.
1.7 Save & load
Quantized models with rotation are saved and loaded transparently. Serialization lives
in auto_round/algorithms/transforms/spinquant/serialize.py.
# Save (rotation matrices stored automatically when needed)
ar.quantize_and_save(output_dir="./my_model", format="auto_round")
# Load (rotation hooks restored automatically)
from transformers import AutoModelForCausalLM
model = AutoModelForCausalLM.from_pretrained("./my_model", device_map="auto")
What is stored per position (metadata keys prefixed with spinquant_r1 / spinquant_r4,
with a *_type code: 0 = deterministic, 1 = random, 2 = trained):
- Deterministic rotations: only metadata (type +
rotation_size) — the matrix is reconstructed on load. - Random rotations: an
int8(±1) matrix is stored (size ≈rotation_size²bytes). - Trained rotations: the full
float32matrix is stored. - Online rotations: rebuilt during model loading —
preregister_spinquant_buffers()creates empty buffers before thestate_dictload, andrebuild_spinquant_online()re-patches the QuantLinear forward (R1/R4) and re-applies the R3 RoPE monkeypatch. R3 is rebuilt purely fromconfig.json, not from stored buffers.
1.8 Block-wise (layer-wise) rotation
By default QuaRot / SpinQuant rotate the entire model up-front: R1 is fused into the embedding and all linears, R2 into attention, and R3/R4 are installed as online hooks — after which quantization runs. This requires the whole model to be resident at once, which is heavy for large models and decoupled from AutoRound's block-wise quantization loop.
Setting layerwise=True on the rotation config aligns rotation with block-wise
quantization: only the rotation matrices (R1–R4) are initialised up-front, and each
decoder block is rotated just before it is quantized, so only one block needs to be
on-device at a time.
from auto_round import AutoRound
from auto_round.algorithms.transforms.spinquant import SpinQuantConfig
# Block-wise QuaRot: rotate each decoder block in lock-step with quantization.
ar = AutoRound(
"Qwen/Qwen3-0.6B",
scheme="MXFP4",
alg_configs=["rtn", SpinQuantConfig(r1=True, r2=True, layerwise=True)],
)
ar.quantize_and_save(output_dir="./Qwen3-0.6B-mxfp4-quarot-blockwise", format="auto_round")
The per-Linear Hadamard rotation also supports block-wise execution. Because it fuses an independent Hadamard into each Linear (no cross-layer residual-stream coupling), rotating one decoder block at a time is mathematically identical to the full-model pass:
from auto_round import AutoRound
from auto_round.algorithms.transforms.hadamard.config import RotationConfig
# Block-wise Hadamard (per-Linear transform backend, MXFP4 / NVFP4).
ar = AutoRound(
"Qwen/Qwen3-0.6B",
scheme="MXFP4",
iters=0,
rotation_config=RotationConfig(hadamard_type="hadamard", layerwise=True),
)
ar.quantize_and_save(output_dir="./Qwen3-0.6B-mxfp4-hadamard-blockwise", format="auto_round")
Notes and limitations:
- Supported algorithms.
layerwiseis honoured by QuaRot / SpinQuant and by the per-Linear Hadamard transform backend (BaseRotation.supports_layerwise == True). The Hadamard inplace / QuaRot residual-stream backend (backend="inplace") couples consecutive layers, so it reportssupports_layerwise == Falseand transparently falls back to full-model rotation. Requestinglayerwise=Truefor the transform backend on a non-MXFP4/NVFP4 dtype raisesNotImplementedError(block-wise Hadamard needs the per-Linear kernel path). nblocksaware. When several decoder layers are fused into one scheduling group (nblocks > 1), each fused layer receives its correct globallayer_idx, so R2/R3/R4 head-dim math stays aligned with the model topology.- Single-block API. Layer-wise rotation is driven by AutoRound's internal block loop
and is not supported through the external single-block
quantize_block()API (e.g. LLM-Compressor); attempting it raisesNotImplementedError. Use the fullquantize()entry point, or setlayerwise=False. - Accuracy parity. Block-wise rotation is mathematically equivalent to full-model rotation for the R1/R2/R3/R4 positions it supports; see §4 for measured parity.
2. Per-Linear Block Rotation
⚠️ This is an earlier research implementation that applies block-diagonal Hadamard rotation per linear layer by patching every
nn.Linearmodule in the model. For most use cases the QuaRot / SpinQuant approach above is preferred — it provides architecture-aware rotation at specific positions (R1–R4) with better accuracy and lower overhead.
The per-linear block rotation iterates over all linear layers and applies a Hadamard of
a configurable block_size uniformly. It does not distinguish between residual stream,
attention and MLP layers, nor does it handle RoPE or attention-side rotation. It is
configured with RotationConfig
(auto_round/algorithms/transforms/hadamard/config.py), which is shared by three
backends:
backend |
Behaviour |
|---|---|
"inplace" |
QuaRot-style residual-stream rotation; works for any dtype and can fuse the online Hadamard into weights (fuse_online_to_weight=True). |
"transform" |
Per-Linear weight + activation Hadamard with a fused Triton kernel; supports only MXFP4 / NVFP4 and cannot fuse online to weight. |
"auto" (default) |
Picks inplace when a fused online rotation is requested, transform for MX/NV-FP data types, inplace otherwise. |
Application modes (hadamard/apply.py):
- Input mode: registers a
forward_pre_hookthat rotates the input activation before each linear layer (online). - Weight mode: fuses the Hadamard directly into the module weight (offline) and patches the calibration wrappers accordingly.
2.1 RotationConfig reference
| Field | Default | Description |
|---|---|---|
algorithm |
"hadamard" (frozen) |
Registry key. |
backend |
"auto" |
"auto", "inplace", or "transform". |
block_size |
None (auto) |
Grouped Hadamard block size. Auto-filled from the data type by normalize_rotation_config(): 32 for MXFP, 16 for NVFP. None/-1 means full-dimension Hadamard. |
hadamard_type |
"hadamard" |
One of hadamard, random_hadamard, inplace_quarot_hadamard, inplace_hadamard, inplace_random. Deterministic Hadamard uses Sylvester construction (block_size must be a power of 2); random_hadamard supports non-power-of-2 sizes from the known-matrix library. |
fuse_online_to_weight |
None |
Fuse online Hadamard rotation into weights when supported (inplace backend only). |
allow_online_rotation |
True |
Allow online activation rotation. |
layerwise |
False |
Apply the Hadamard rotation per decoder block, in lock-step with AutoRound's block-wise quantization (see §1.8). Supported by the per-Linear transform backend (MXFP4 / NVFP4); the inplace backend couples layers and falls back to full-model rotation. |
2.2 Usage
from auto_round import AutoRound
model_name_or_path = "meta-llama/Llama-3.1-8B-Instruct"
# "hadamard": block_size auto (32 for MXFP), hadamard_type="hadamard"
ar = AutoRound(model_name_or_path, scheme="MXFP4", alg_configs=["auto_round", "hadamard"])
ar.quantize_and_save(output_dir="./Llama-3.1-8B-Instruct-mxfp4-ht", format="auto_round")
The per-linear rotation is also exposed on the CLI via --algorithm hadamard:
auto-round --model meta-llama/Llama-3.1-8B-Instruct --scheme MXFP4 \
--algorithm hadamard --rotation_type hadamard --rotation_backend auto
CLI flags: --rotation_type {hadamard,random_hadamard,quarot_hadamard},
--rotation_backend {auto,inplace,transform}, --rotation_block_size N,
--fuse_online_to_weight / --no-fuse_online_to_weight,
--allow_online_rotation / --no-allow_online_rotation.
3. How rotation configs are dispatched
A rotation config is passed alongside a quantization algorithm through alg_configs
(e.g. alg_configs=["auto_round", "hadamard"] or
alg_configs=["auto_round", SpinQuantConfig(...)]). It is not accepted as a separate
top-level AutoRound(...) parameter. Once resolved, it is normalized and dispatched by
the unified apply_rotation() entry point in
auto_round/algorithms/transforms/__init__.py:
- A
BaseRotationConfiginstance (SpinQuantConfigorRotationConfig) is used directly. - The registered aliases
"quarot"/"spinquant"resolve to theSpinQuantConfigshortcuts above. - The registered aliases
"hadamard","random_hadamard","quarot_hadamard"resolve to the per-linearRotationConfigpresets.
4. Example Results
Enabling rotation consistently recovers accuracy lost to aggressive low-bit quantization. The tables below compare three configurations for each model:
- BF16 — the unquantized baseline (reference, 100%).
- MXFP4 — MXFP4 quantization without rotation.
- MXFP4 + rotation — MXFP4 quantization with the QuaRot-style rotation shown in the Rotation column.
All numbers were measured with lm-eval on four tasks — GSM8K, MMLU, PIQA and HellaSwag. Avg is the mean across the four tasks and Ratio is that average relative to the BF16 baseline. Higher is better. Across all three model sizes (8B, 32B and 72B), rotation improves the MXFP4 average and closes the gap to BF16.
Qwen3-8B — rotation lifts MXFP4 from 93.77% to 95.73% of BF16 (+1.96 pts), with the largest gains on the quantization-sensitive MMLU and PIQA tasks.
| Model | Scheme | Rotation | GSM8K (strict, acc) | MMLU (acc) | PIQA (acc) | HellaSwag (acc) | Avg | Ratio |
|---|---|---|---|---|---|---|---|---|
| Qwen/Qwen3-8B | BF16 | - | 0.9136 | 0.7290 | 0.7671 | 0.5708 | 0.7451 | 100.00% |
| Qwen/Qwen3-8B | MXFP4 | - | 0.8560 | 0.6775 | 0.7323 | 0.5289 | 0.6987 | 93.77% |
| Qwen/Qwen3-8B | MXFP4 | R1 (online, 32) + R2 (offline) + R4 (online, 32) | 0.8575 | 0.7009 | 0.7595 | 0.5352 | 0.7133 | 95.73% |
Qwen3-32B — rotation improves MXFP4 from 98.36% to 98.53% of BF16, recovering most of the remaining PIQA and MMLU drop with R1 + R4 alone.
| Model | Scheme | Rotation | GSM8K (strict, acc) | MMLU (acc) | PIQA (acc_norm) | HellaSwag (acc) | Avg | Ratio |
|---|---|---|---|---|---|---|---|---|
| Qwen/Qwen3-32B | BF16 | - | 0.9287 | 0.8076 | 0.8118 | 0.6389 | 0.7968 | 100.00% |
| Qwen/Qwen3-32B | MXFP4 | - | 0.9272 | 0.7876 | 0.7900 | 0.6300 | 0.7837 | 98.36% |
| Qwen/Qwen3-32B | MXFP4 | R1 (online, 32) + R4 (online, 32) | 0.9249 | 0.7908 | 0.8030 | 0.6215 | 0.7851 | 98.53% |
Qwen2.5-72B — rotation improves MXFP4 from 97.60% to 98.58% of BF16 (+0.98 pts), with gains across every task.
| Model | Scheme | Rotation | GSM8K (strict, acc) | MMLU (acc) | PIQA (acc_norm) | HellaSwag (acc) | Avg | Ratio |
|---|---|---|---|---|---|---|---|---|
| Qwen/Qwen2.5-72B | BF16 | - | 0.9454 | 0.8346 | 0.8335 | 0.7035 | 0.8293 | 100.00% |
| Qwen/Qwen2.5-72B | MXFP4 | - | 0.9242 | 0.8104 | 0.8226 | 0.6803 | 0.8094 | 97.60% |
| Qwen/Qwen2.5-72B | MXFP4 | R1 (online, 32) + R4 (online, 32) | 0.9333 | 0.8204 | 0.8292 | 0.6871 | 0.8175 | 98.58% |
Note: Exact numbers depend on the model, task set, lm-eval version and hardware. Treat these as illustrative of the trend — rotation improving low-bit accuracy — rather than as guaranteed values.