Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

21.2 KB
Raw Blame

Rotation Transform — Details (Research)

⚠️ Research feature. Rotation transform is still in a research stage. Inference relies on forward hooks, which are currently only supported by the Hugging Face Transformers backend. As a result, inference may be slower compared to native (non-rotated) models.

This document is the in-depth reference for AutoRound's rotation transforms. For a concise getting-started guide, see the Rotation section in step_by_step.md.

Rotation redistributes outliers in weights and activations before quantization by multiplying tensors with orthogonal (Hadamard) matrices. Because an orthogonal rotation R satisfies R @ Rᵀ = I, a matched pair of rotations inserted around a linear operation leaves the network's output unchanged (x @ R @ Rᵀ @ Wᵀ = x @ Wᵀ) while making the intermediate distributions flatter and more quantization-friendly. This is most useful for aggressive low-bit schemes such as MXFP4, NVFP4 and W4A4.

AutoRound provides two independent rotation implementations:

  • QuaRot / SpinQuant — architecture-aware, full-model rotation applied at up to four positions (R1–R4). Selected via alg_configs=[..., "quarot"], alg_configs=[..., "spinquant"], or a SpinQuantConfig instance. Recommended.
  • Per-Linear Block Rotation — an earlier, simpler implementation that applies a block-diagonal Hadamard uniformly to every nn.Linear. Selected via alg_configs=[..., "hadamard"] / a string Hadamard type, or a RotationConfig instance. Also exposed through the --algorithm hadamard CLI.

1. QuaRot / SpinQuant

QuaRot applies deterministic Hadamard rotations at up to four positions in the transformer architecture. Unlike per-linear block rotation, it operates at the model architecture level — rotating the residual stream, attention heads, and MLP activations at specific positions for targeted outlier suppression.

The pipeline (implemented in auto_round/algorithms/transforms/spinquant/preprocessor.py, SpinQuantPreprocessor) runs the following stages:

  1. Fuse RMSNorm scale parameters into the following linear layers (fuse_rmsnorm).
  2. Untie word embeddings if they are shared with the LM head (untie_embeddings).
  3. Initialize rotation matrices R1–R4.
  4. (Only if trainable) train the rotations and smooth values via KL/MSE loss.
  5. Fuse the offline rotations (R2, and R1 when online_r1_rotation=False) into weights.
  6. Register the online hooks (R1 when online, R3, R4) and clean up.

1.1 Rotation positions

Position Target Dimension Mode Effect
R1 Residual stream — inputs of q/k/v_proj, gate/up_proj; outputs of o_proj, down_proj, embed_tokens, lm_head hidden_size (or rotation_size) Online or Offline Smooths weight outliers across all linear layers
R2 v_proj output / o_proj input (per attention head) head_dim Offline (fused) Balances per-head value distributions
R3 Q/K after RoPE head_dim Online (RoPE monkeypatch) Improves KV-cache quantization friendliness
R4 down_proj input (MLP activation) intermediate_size (or rotation_size) Online (hook) + weight fuse Suppresses activation outliers in the FFN
  • Online: applied at runtime via forward hooks / a RoPE monkeypatch. Weights are pre-rotated so the paired inverse rotation happens at runtime; there is a small inference cost.
  • Offline (fused): the rotation is absorbed into adjacent weight matrices, leaving no runtime overhead.

R1 is applied online by default (online_r1_rotation=True): the target weights are rotated and a forward_pre_hook is registered on the same modules. When online_r1_rotation=False, R1 is instead fully fused into embed_tokens, q/k/v_proj, o_proj, gate/up/down_proj and lm_head, cancelling out with no runtime cost.

1.2 SpinQuantConfig reference

Defined in auto_round/algorithms/transforms/spinquant/preprocessor.py.

Field Default Description
algorithm "spinquant" Registry key used by BaseRotation.from_config() to dispatch.
r1 True Enable R1 (hidden_size residual-stream rotation).
r2 True Enable R2 (head_dim per-head rotation, offline).
r3 False Enable R3 (online Q/K rotation after RoPE).
r4 False Enable R4 (online MLP activation rotation).
rotation_size None (auto) Overrides the R1/R4 block size (R1 uses it instead of hidden_size, R4 instead of intermediate_size). R2 always uses head_dim; R3 has no custom size. Must be a positive power of 2.
random_r1 / random_r2 / random_r3 / random_r4 False Use random Hadamard (H × diag(±1)) instead of the deterministic matrix at that position. Only relevant in QuaRot mode.
online_r1_rotation True Apply R1 online via hook (True) or fuse it fully into weights (False).
trainable_rotation False Learn the rotation matrices via Cayley SGD (SpinQuant mode). Research, requires a dataloader.
trainable_smooth False Learn SmoothQuant-style smooth_values jointly via Adam. Research, requires a dataloader.
iters 200 Training iterations (trainable modes only).
lr 1e-4 SGDG (Cayley) learning rate for rotation matrices.
smooth_lr 1e-3 Adam learning rate for smooth values.
batch_size 1 Training batch size.
loss_type "kl_top" Training loss: "kl_top", "kl_full", or "mse".
kl_top_k 1000 Top-k logits used by the kl_top loss.
fuse_rmsnorm True Fuse RMSNorm scales into following linear layers.
untie_embeddings True Untie shared embedding / LM-head weights before rotating.
layerwise False Apply rotation per decoder block, in lock-step with AutoRound's block-wise quantization, instead of rotating the whole model up-front (see §1.8). Supported by QuaRot / SpinQuant and the per-Linear Hadamard transform backend; the Hadamard inplace backend ignores it and falls back to full-model rotation.
dtype torch.float32 Numerical dtype used for rotation math.
device None (auto) Defaults to "cuda" if available, else "cpu".

1.3 String shortcuts

Both are registered algorithm aliases usable directly in alg_configs:

Alias Equivalent
"quarot" SpinQuantConfig(trainable_rotation=False, trainable_smooth=False) — deterministic Hadamard, no training, no calibration data.
"spinquant" SpinQuantConfig(trainable_rotation=True, trainable_smooth=True) — research, requires a dataloader.

Both presets keep the default r1=True, r2=True, r3=False, r4=False.

⚠️ SpinQuant trainable rotation (trainable_rotation=True) enables learnable rotation matrices optimized via Cayley SGD. This feature is a research feature and not fully validated on real models. Use "quarot" (fixed Hadamard) for production workloads.

1.4 Usage

from auto_round import AutoRound
from auto_round.algorithms.transforms.spinquant import SpinQuantConfig

model_name = "Qwen/Qwen3-0.6B"

# QuaRot preset: R1+R2 deterministic Hadamard, no training. Rotation configs are
# passed alongside a quantization algorithm through `alg_configs`.
ar = AutoRound(model_name, scheme="MXFP4", alg_configs=["auto_round", "quarot"])

# Choose how many positions to rotate by passing a SpinQuantConfig instance instead:
ar = AutoRound(model_name, scheme="MXFP4", alg_configs=["auto_round", SpinQuantConfig(r1=True)])  # R1 only
ar = AutoRound(model_name, scheme="MXFP4", alg_configs=["auto_round", SpinQuantConfig(r1=True, r2=True)])  # R1+R2
ar = AutoRound(
    model_name,
    scheme="MXFP4",
    alg_configs=["auto_round", SpinQuantConfig(r1=True, r2=True, r3=True, r4=True)],
)  # all four

ar.quantize_and_save(output_dir="./Qwen3-0.6B-mxfp4-quarot", format="auto_round")

1.5 Deterministic vs random Hadamard

Rotation matrices are constructed in auto_round/algorithms/transforms/spinquant/rotation_utils.py:

  • Deterministic (default): deterministic_hadamard_matrix(size) builds a Sylvester Hadamard matrix (H → [[H, H], [H, −H]]) normalized by √size. The size must be a power of 2. Because the matrix is fully determined by its size, nothing needs to be stored.
  • Random: random_hadamard_matrix(size) computes H @ diag(±1) / √N. It offers slightly better outlier suppression but the sign matrix must be persisted.

For non-power-of-2 dimensions, get_hadamard_K(n) factors n = K · 2^m and pulls a known base Hadamard of size K (e.g. 12, 20, 28, 36, 40, 52, 60, 108, 140, 156, 172) from known_hadamard.py, combining it with a Sylvester block.

# Random Hadamard at every enabled position
ar = AutoRound(
    model_name,
    scheme="MXFP4",
    alg_configs=[
        "auto_round",
        SpinQuantConfig(
            r1=True,
            r2=True,
            r3=True,
            r4=True,
            random_r1=True,
            random_r2=True,
            random_r3=True,
            random_r4=True,
        ),
    ],
)

1.6 Trainable rotation (SpinQuant mode)

When trainable_rotation=True (and/or trainable_smooth=True), the rotations become learnable and are optimized on calibration data:

  • cayley_optimizer.py provides SGDG, an orthogonality-preserving optimizer that keeps each rotation on the Stiefel manifold using a Cayley/QR retraction. AdamAndSGDG combines Adam (for smooth_values) with SGDG (for the spinquant_R* matrices).
  • training.py runs a KL-/MSE-based distillation loop against a frozen clone of the original model (create_dual_optimizer(), run_training_loop()).
  • SpinQuantPreprocessor.preprocess() requires a dataloader whenever either trainable flag is set.

1.7 Save & load

Quantized models with rotation are saved and loaded transparently. Serialization lives in auto_round/algorithms/transforms/spinquant/serialize.py.

# Save (rotation matrices stored automatically when needed)
ar.quantize_and_save(output_dir="./my_model", format="auto_round")

# Load (rotation hooks restored automatically)
from transformers import AutoModelForCausalLM

model = AutoModelForCausalLM.from_pretrained("./my_model", device_map="auto")

What is stored per position (metadata keys prefixed with spinquant_r1 / spinquant_r4, with a *_type code: 0 = deterministic, 1 = random, 2 = trained):

  • Deterministic rotations: only metadata (type + rotation_size) — the matrix is reconstructed on load.
  • Random rotations: an int8 (±1) matrix is stored (size ≈ rotation_size² bytes).
  • Trained rotations: the full float32 matrix is stored.
  • Online rotations: rebuilt during model loading — preregister_spinquant_buffers() creates empty buffers before the state_dict load, and rebuild_spinquant_online() re-patches the QuantLinear forward (R1/R4) and re-applies the R3 RoPE monkeypatch. R3 is rebuilt purely from config.json, not from stored buffers.

1.8 Block-wise (layer-wise) rotation

By default QuaRot / SpinQuant rotate the entire model up-front: R1 is fused into the embedding and all linears, R2 into attention, and R3/R4 are installed as online hooks — after which quantization runs. This requires the whole model to be resident at once, which is heavy for large models and decoupled from AutoRound's block-wise quantization loop.

Setting layerwise=True on the rotation config aligns rotation with block-wise quantization: only the rotation matrices (R1–R4) are initialised up-front, and each decoder block is rotated just before it is quantized, so only one block needs to be on-device at a time.

from auto_round import AutoRound
from auto_round.algorithms.transforms.spinquant import SpinQuantConfig

# Block-wise QuaRot: rotate each decoder block in lock-step with quantization.
ar = AutoRound(
    "Qwen/Qwen3-0.6B",
    scheme="MXFP4",
    alg_configs=["rtn", SpinQuantConfig(r1=True, r2=True, layerwise=True)],
)
ar.quantize_and_save(output_dir="./Qwen3-0.6B-mxfp4-quarot-blockwise", format="auto_round")

The per-Linear Hadamard rotation also supports block-wise execution. Because it fuses an independent Hadamard into each Linear (no cross-layer residual-stream coupling), rotating one decoder block at a time is mathematically identical to the full-model pass:

from auto_round import AutoRound
from auto_round.algorithms.transforms.hadamard.config import RotationConfig

# Block-wise Hadamard (per-Linear transform backend, MXFP4 / NVFP4).
ar = AutoRound(
    "Qwen/Qwen3-0.6B",
    scheme="MXFP4",
    iters=0,
    rotation_config=RotationConfig(hadamard_type="hadamard", layerwise=True),
)
ar.quantize_and_save(output_dir="./Qwen3-0.6B-mxfp4-hadamard-blockwise", format="auto_round")

Notes and limitations:

  • Supported algorithms. layerwise is honoured by QuaRot / SpinQuant and by the per-Linear Hadamard transform backend (BaseRotation.supports_layerwise == True). The Hadamard inplace / QuaRot residual-stream backend (backend="inplace") couples consecutive layers, so it reports supports_layerwise == False and transparently falls back to full-model rotation. Requesting layerwise=True for the transform backend on a non-MXFP4/NVFP4 dtype raises NotImplementedError (block-wise Hadamard needs the per-Linear kernel path).
  • nblocks aware. When several decoder layers are fused into one scheduling group (nblocks > 1), each fused layer receives its correct global layer_idx, so R2/R3/R4 head-dim math stays aligned with the model topology.
  • Single-block API. Layer-wise rotation is driven by AutoRound's internal block loop and is not supported through the external single-block quantize_block() API (e.g. LLM-Compressor); attempting it raises NotImplementedError. Use the full quantize() entry point, or set layerwise=False.
  • Accuracy parity. Block-wise rotation is mathematically equivalent to full-model rotation for the R1/R2/R3/R4 positions it supports; see §4 for measured parity.

2. Per-Linear Block Rotation

⚠️ This is an earlier research implementation that applies block-diagonal Hadamard rotation per linear layer by patching every nn.Linear module in the model. For most use cases the QuaRot / SpinQuant approach above is preferred — it provides architecture-aware rotation at specific positions (R1–R4) with better accuracy and lower overhead.

The per-linear block rotation iterates over all linear layers and applies a Hadamard of a configurable block_size uniformly. It does not distinguish between residual stream, attention and MLP layers, nor does it handle RoPE or attention-side rotation. It is configured with RotationConfig (auto_round/algorithms/transforms/hadamard/config.py), which is shared by three backends:

backend Behaviour
"inplace" QuaRot-style residual-stream rotation; works for any dtype and can fuse the online Hadamard into weights (fuse_online_to_weight=True).
"transform" Per-Linear weight + activation Hadamard with a fused Triton kernel; supports only MXFP4 / NVFP4 and cannot fuse online to weight.
"auto" (default) Picks inplace when a fused online rotation is requested, transform for MX/NV-FP data types, inplace otherwise.

Application modes (hadamard/apply.py):

  • Input mode: registers a forward_pre_hook that rotates the input activation before each linear layer (online).
  • Weight mode: fuses the Hadamard directly into the module weight (offline) and patches the calibration wrappers accordingly.

2.1 RotationConfig reference

Field Default Description
algorithm "hadamard" (frozen) Registry key.
backend "auto" "auto", "inplace", or "transform".
block_size None (auto) Grouped Hadamard block size. Auto-filled from the data type by normalize_rotation_config(): 32 for MXFP, 16 for NVFP. None/-1 means full-dimension Hadamard.
hadamard_type "hadamard" One of hadamard, random_hadamard, inplace_quarot_hadamard, inplace_hadamard, inplace_random. Deterministic Hadamard uses Sylvester construction (block_size must be a power of 2); random_hadamard supports non-power-of-2 sizes from the known-matrix library.
fuse_online_to_weight None Fuse online Hadamard rotation into weights when supported (inplace backend only).
allow_online_rotation True Allow online activation rotation.
layerwise False Apply the Hadamard rotation per decoder block, in lock-step with AutoRound's block-wise quantization (see §1.8). Supported by the per-Linear transform backend (MXFP4 / NVFP4); the inplace backend couples layers and falls back to full-model rotation.

2.2 Usage

from auto_round import AutoRound

model_name_or_path = "meta-llama/Llama-3.1-8B-Instruct"

# "hadamard": block_size auto (32 for MXFP), hadamard_type="hadamard"
ar = AutoRound(model_name_or_path, scheme="MXFP4", alg_configs=["auto_round", "hadamard"])
ar.quantize_and_save(output_dir="./Llama-3.1-8B-Instruct-mxfp4-ht", format="auto_round")

The per-linear rotation is also exposed on the CLI via --algorithm hadamard:

auto-round --model meta-llama/Llama-3.1-8B-Instruct --scheme MXFP4 \
  --algorithm hadamard --rotation_type hadamard --rotation_backend auto

CLI flags: --rotation_type {hadamard,random_hadamard,quarot_hadamard}, --rotation_backend {auto,inplace,transform}, --rotation_block_size N, --fuse_online_to_weight / --no-fuse_online_to_weight, --allow_online_rotation / --no-allow_online_rotation.


3. How rotation configs are dispatched

A rotation config is passed alongside a quantization algorithm through alg_configs (e.g. alg_configs=["auto_round", "hadamard"] or alg_configs=["auto_round", SpinQuantConfig(...)]). It is not accepted as a separate top-level AutoRound(...) parameter. Once resolved, it is normalized and dispatched by the unified apply_rotation() entry point in auto_round/algorithms/transforms/__init__.py:

  • A BaseRotationConfig instance (SpinQuantConfig or RotationConfig) is used directly.
  • The registered aliases "quarot" / "spinquant" resolve to the SpinQuantConfig shortcuts above.
  • The registered aliases "hadamard", "random_hadamard", "quarot_hadamard" resolve to the per-linear RotationConfig presets.

4. Example Results

Enabling rotation consistently recovers accuracy lost to aggressive low-bit quantization. The tables below compare three configurations for each model:

  • BF16 — the unquantized baseline (reference, 100%).
  • MXFP4 — MXFP4 quantization without rotation.
  • MXFP4 + rotation — MXFP4 quantization with the QuaRot-style rotation shown in the Rotation column.

All numbers were measured with lm-eval on four tasks — GSM8K, MMLU, PIQA and HellaSwag. Avg is the mean across the four tasks and Ratio is that average relative to the BF16 baseline. Higher is better. Across all three model sizes (8B, 32B and 72B), rotation improves the MXFP4 average and closes the gap to BF16.

Qwen3-8B — rotation lifts MXFP4 from 93.77% to 95.73% of BF16 (+1.96 pts), with the largest gains on the quantization-sensitive MMLU and PIQA tasks.

Model Scheme Rotation GSM8K (strict, acc) MMLU (acc) PIQA (acc) HellaSwag (acc) Avg Ratio
Qwen/Qwen3-8B BF16 - 0.9136 0.7290 0.7671 0.5708 0.7451 100.00%
Qwen/Qwen3-8B MXFP4 - 0.8560 0.6775 0.7323 0.5289 0.6987 93.77%
Qwen/Qwen3-8B MXFP4 R1 (online, 32) + R2 (offline) + R4 (online, 32) 0.8575 0.7009 0.7595 0.5352 0.7133 95.73%

Qwen3-32B — rotation improves MXFP4 from 98.36% to 98.53% of BF16, recovering most of the remaining PIQA and MMLU drop with R1 + R4 alone.

Model Scheme Rotation GSM8K (strict, acc) MMLU (acc) PIQA (acc_norm) HellaSwag (acc) Avg Ratio
Qwen/Qwen3-32B BF16 - 0.9287 0.8076 0.8118 0.6389 0.7968 100.00%
Qwen/Qwen3-32B MXFP4 - 0.9272 0.7876 0.7900 0.6300 0.7837 98.36%
Qwen/Qwen3-32B MXFP4 R1 (online, 32) + R4 (online, 32) 0.9249 0.7908 0.8030 0.6215 0.7851 98.53%

Qwen2.5-72B — rotation improves MXFP4 from 97.60% to 98.58% of BF16 (+0.98 pts), with gains across every task.

Model Scheme Rotation GSM8K (strict, acc) MMLU (acc) PIQA (acc_norm) HellaSwag (acc) Avg Ratio
Qwen/Qwen2.5-72B BF16 - 0.9454 0.8346 0.8335 0.7035 0.8293 100.00%
Qwen/Qwen2.5-72B MXFP4 - 0.9242 0.8104 0.8226 0.6803 0.8094 97.60%
Qwen/Qwen2.5-72B MXFP4 R1 (online, 32) + R4 (online, 32) 0.9333 0.8204 0.8292 0.6871 0.8175 98.58%

Note: Exact numbers depend on the model, task set, lm-eval version and hardware. Treat these as illustrative of the trend — rotation improving low-bit accuracy — rather than as guaranteed values.