Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

update rotation doc. (#1999)

lkk5ee45db7b38f
diff --git a/docs/rotation_details.md b/docs/rotation_details.md
new file mode 100644
index 000000000..5a9aa8ffc
--- /dev/null
+++ b/docs/rotation_details.md
@@ -0,0 +1,331 @@
+# Rotation Transform — Details (Experimental)
+
+> ⚠️ **Experimental feature.** Rotation transform is still in an experimental stage. Inference relies on forward hooks, which are currently only supported by the Hugging Face Transformers backend. As a result, inference may be slower compared to native (non-rotated) models.
+
+This document is the in-depth reference for AutoRound's rotation transforms. For a
+concise getting-started guide, see the *Rotation* section in
+[step_by_step.md](./step_by_step.md).
+
+Rotation redistributes outliers in weights and activations before quantization by
+multiplying tensors with orthogonal (Hadamard) matrices. Because an orthogonal
+rotation `R` satisfies `R @ Rᵀ = I`, a matched pair of rotations inserted around a
+linear operation leaves the network's output unchanged (`x @ R @ Rᵀ @ Wᵀ = x @ Wᵀ`)
+while making the intermediate distributions flatter and more quantization-friendly.
+This is most useful for aggressive low-bit schemes such as MXFP4, NVFP4 and W4A4.
+
+AutoRound provides two independent rotation implementations:
+
+- **QuaRot / SpinQuant** — architecture-aware, full-model rotation applied at up to
+ four positions (R1–R4). Selected via `rotation_config="quarot"`,
+ `rotation_config="spinquant"`, or a `SpinQuantConfig` instance. **Recommended.**
+- **Per-Linear Block Rotation** — an earlier, simpler implementation that applies a
+ block-diagonal Hadamard uniformly to every `nn.Linear`. Selected via
+ `rotation_config="default"` / a string Hadamard type, or a `RotationConfig`
+ instance. Also exposed through the `--algorithm hadamard` CLI.
+
+---
+
+## 1. QuaRot / SpinQuant
+
+QuaRot applies deterministic Hadamard rotations at up to four positions in the
+transformer architecture. Unlike per-linear block rotation, it operates at the model
+architecture level — rotating the residual stream, attention heads, and MLP
+activations at specific positions for targeted outlier suppression.
+
+The pipeline (implemented in `auto_round/algorithms/transforms/spinquant/preprocessor.py`,
+`SpinQuantPreprocessor`) runs the following stages:
+
+1. Fuse RMSNorm scale parameters into the following linear layers (`fuse_rmsnorm`).
+2. Untie word embeddings if they are shared with the LM head (`untie_embeddings`).
+3. Initialize rotation matrices R1–R4.
+4. (Only if trainable) train the rotations and smooth values via KL/MSE loss.
+5. Fuse the offline rotations (R2, and R1 when `online_r1_rotation=False`) into weights.
+6. Register the online hooks (R1 when online, R3, R4) and clean up.
+
+### 1.1 Rotation positions
+
+| Position | Target | Dimension | Mode | Effect |
+|----------|--------|-----------|------|--------|
+| **R1** | Residual stream — inputs of `q/k/v_proj`, `gate/up_proj`; outputs of `o_proj`, `down_proj`, `embed_tokens`, `lm_head` | `hidden_size` (or `rotation_size`) | Online *or* Offline | Smooths weight outliers across all linear layers |
+| **R2** | `v_proj` output / `o_proj` input (per attention head) | `head_dim` | Offline (fused) | Balances per-head value distributions |
+| **R3** | Q/K after RoPE | `head_dim` | Online (RoPE monkeypatch) | Improves KV-cache quantization friendliness |
+| **R4** | `down_proj` input (MLP activation) | `intermediate_size` (or `rotation_size`) | Online (hook) + weight fuse | Suppresses activation outliers in the FFN |
+
+- **Online**: applied at runtime via forward hooks / a RoPE monkeypatch. Weights are
+ pre-rotated so the paired inverse rotation happens at runtime; there is a small
+ inference cost.
+- **Offline (fused)**: the rotation is absorbed into adjacent weight matrices, leaving
+ no runtime overhead.
+
+R1 is applied online by default (`online_r1_rotation=True`): the target weights are
+rotated and a `forward_pre_hook` is registered on the same modules. When
+`online_r1_rotation=False`, R1 is instead fully fused into `embed_tokens`,
+`q/k/v_proj`, `o_proj`, `gate/up/down_proj` and `lm_head`, cancelling out with no
+runtime cost.
+
+### 1.2 `SpinQuantConfig` reference
+
+Defined in `auto_round/algorithms/transforms/spinquant/preprocessor.py`.
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `algorithm` | `"spinquant"` | Registry key used by `BaseRotation.from_config()` to dispatch. |
+| `r1` | `True` | Enable R1 (hidden_size residual-stream rotation). |
+| `r2` | `True` | Enable R2 (head_dim per-head rotation, offline). |
+| `r3` | `False` | Enable R3 (online Q/K rotation after RoPE). |
+| `r4` | `False` | Enable R4 (online MLP activation rotation). |
+| `rotation_size` | `None` (auto) | Overrides the R1/R4 block size (R1 uses it instead of `hidden_size`, R4 instead of `intermediate_size`). R2 always uses `head_dim`; R3 has no custom size. Must be a positive power of 2. |
+| `random_r1` / `random_r2` / `random_r3` / `random_r4` | `False` | Use random Hadamard (`H × diag(±1)`) instead of the deterministic matrix at that position. Only relevant in QuaRot mode. |
+| `online_r1_rotation` | `True` | Apply R1 online via hook (`True`) or fuse it fully into weights (`False`). |
+| `trainable_rotation` | `False` | Learn the rotation matrices via Cayley SGD (SpinQuant mode). **Experimental**, requires a dataloader. |
+| `trainable_smooth` | `False` | Learn SmoothQuant-style `smooth_values` jointly via Adam. **Experimental**, requires a dataloader. |
+| `iters` | `200` | Training iterations (trainable modes only). |
+| `lr` | `1e-4` | SGDG (Cayley) learning rate for rotation matrices. |
+| `smooth_lr` | `1e-3` | Adam learning rate for smooth values. |
+| `batch_size` | `1` | Training batch size. |
+| `loss_type` | `"kl_top"` | Training loss: `"kl_top"`, `"kl_full"`, or `"mse"`. |
+| `kl_top_k` | `1000` | Top-k logits used by the `kl_top` loss. |
+| `fuse_rmsnorm` | `True` | Fuse RMSNorm scales into following linear layers. |
+| `untie_embeddings` | `True` | Untie shared embedding / LM-head weights before rotating. |
+| `dtype` | `torch.float32` | Numerical dtype used for rotation math. |
+| `device` | `None` (auto) | Defaults to `"cuda"` if available, else `"cpu"`. |
+
+### 1.3 String shortcuts
+
+| Value | Equivalent |
+|-------|-----------|
+| `"quarot"` | `SpinQuantConfig(trainable_rotation=False, trainable_smooth=False)` — deterministic Hadamard, no training, no calibration data. |
+| `"spinquant"` | `SpinQuantConfig(trainable_rotation=True, trainable_smooth=True)` — **experimental**, requires a dataloader. |
+
+Both presets keep the default `r1=True, r2=True, r3=False, r4=False`. A dict with
+`algorithm="spinquant"` is also parsed into a `SpinQuantConfig`.
+
+> ⚠️ **SpinQuant trainable rotation** (`trainable_rotation=True`) enables learnable
+> rotation matrices optimized via Cayley SGD. This feature is experimental and not
+> fully validated on real models. Use `"quarot"` (fixed Hadamard) for production
+> workloads.
+
+### 1.4 Usage
+
+```python
+from auto_round import AutoRound
+from auto_round.algorithms.transforms.spinquant import SpinQuantConfig
+
+model_name = "Qwen/Qwen3-0.6B"
+
+# QuaRot preset: R1+R2 deterministic Hadamard, no training
+ar = AutoRound(model_name, scheme="MXFP4", rotation_config="quarot")
+
+# Choose how many positions to rotate:
+ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True)) # R1 only
+ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True, r2=True)) # R1+R2
+ar = AutoRound(
+ model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True, r2=True, r3=True, r4=True)
+) # all four
+
+ar.quantize_and_save(output_dir="./Qwen3-0.6B-mxfp4-quarot", format="auto_round")
+```
+
+### 1.5 Deterministic vs random Hadamard
+
+Rotation matrices are constructed in
+`auto_round/algorithms/transforms/spinquant/rotation_utils.py`:
+
+- **Deterministic** (default): `deterministic_hadamard_matrix(size)` builds a Sylvester
+ Hadamard matrix (`H → [[H, H], [H, −H]]`) normalized by `√size`. The size must be a
+ power of 2. Because the matrix is fully determined by its size, nothing needs to be
+ stored.
+- **Random**: `random_hadamard_matrix(size)` computes `H @ diag(±1) / √N`. It offers
+ slightly better outlier suppression but the sign matrix must be persisted.
+
+For non-power-of-2 dimensions, `get_hadamard_K(n)` factors `n = K · 2^m` and pulls a
+known base Hadamard of size `K` (e.g. 12, 20, 28, 36, 40, 52, 60, 108, 140, 156, 172)
+from `known_hadamard.py`, combining it with a Sylvester block.
+
+```python
+# Random Hadamard at every enabled position
+ar = AutoRound(
+ model_name,
+ scheme="MXFP4",
+ rotation_config=SpinQuantConfig(
+ r1=True,
+ r2=True,
+ r3=True,
+ r4=True,
+ random_r1=True,
+ random_r2=True,
+ random_r3=True,
+ random_r4=True,
+ ),
+)
+```
+
+### 1.6 Trainable rotation (SpinQuant mode)
+
+When `trainable_rotation=True` (and/or `trainable_smooth=True`), the rotations become
+learnable and are optimized on calibration data:
+
+- `cayley_optimizer.py` provides `SGDG`, an orthogonality-preserving optimizer that
+ keeps each rotation on the Stiefel manifold using a Cayley/QR retraction.
+ `AdamAndSGDG` combines Adam (for `smooth_values`) with SGDG (for the `spinquant_R*`
+ matrices).
+- `training.py` runs a KL-/MSE-based distillation loop against a frozen clone of the
+ original model (`create_dual_optimizer()`, `run_training_loop()`).
+- `SpinQuantPreprocessor.preprocess()` **requires a dataloader** whenever either
+ trainable flag is set.
+
+### 1.7 Save & load
+
+Quantized models with rotation are saved and loaded transparently. Serialization lives
+in `auto_round/algorithms/transforms/spinquant/serialize.py`.
+
+```python
+# Save (rotation matrices stored automatically when needed)
+ar.quantize_and_save(output_dir="./my_model", format="auto_round")
+
+# Load (rotation hooks restored automatically)
+from transformers import AutoModelForCausalLM
+
+model = AutoModelForCausalLM.from_pretrained("./my_model", device_map="auto")
+```
+
+What is stored per position (metadata keys prefixed with `spinquant_r1` / `spinquant_r4`,
+with a `*_type` code: `0` = deterministic, `1` = random, `2` = trained):
+
+- **Deterministic rotations**: only metadata (type + `rotation_size`) — the matrix is
+ reconstructed on load.
+- **Random rotations**: an `int8` (±1) matrix is stored (size ≈ `rotation_size²` bytes).
+- **Trained rotations**: the full `float32` matrix is stored.
+- **Online rotations**: rebuilt during model loading — `preregister_spinquant_buffers()`
+ creates empty buffers before the `state_dict` load, and `rebuild_spinquant_online()`
+ re-patches the QuantLinear forward (R1/R4) and re-applies the R3 RoPE monkeypatch.
+ R3 is rebuilt purely from `config.json`, not from stored buffers.
+
+---
+
+## 2. Per-Linear Block Rotation
+
+> ⚠️ This is an earlier experimental implementation that applies block-diagonal
+> Hadamard rotation **per linear layer** by patching every `nn.Linear` module in the
+> model. For most use cases the QuaRot / SpinQuant approach above is preferred — it
+> provides architecture-aware rotation at specific positions (R1–R4) with better
+> accuracy and lower overhead.
+
+The per-linear block rotation iterates over all linear layers and applies a Hadamard of
+a configurable `block_size` uniformly. It does not distinguish between residual stream,
+attention and MLP layers, nor does it handle RoPE or attention-side rotation. It is
+configured with `RotationConfig`
+(`auto_round/algorithms/transforms/hadamard/config.py`), which is shared by three
+backends:
+
+| `backend` | Behaviour |
+|-----------|-----------|
+| `"inplace"` | QuaRot-style residual-stream rotation; works for any dtype and can fuse the online Hadamard into weights (`fuse_online_to_weight=True`). |
+| `"transform"` | Per-Linear weight + activation Hadamard with a fused Triton kernel; supports only MXFP4 / NVFP4 and cannot fuse online to weight. |
+| `"auto"` (default) | Picks `inplace` when a fused online rotation is requested, `transform` for MX/NV-FP data types, `inplace` otherwise. |
+
+Application modes (`hadamard/apply.py`):
+
+- **Input mode**: registers a `forward_pre_hook` that rotates the input activation
+ before each linear layer (online).
+- **Weight mode**: fuses the Hadamard directly into the module weight (offline) and
+ patches the calibration wrappers accordingly.
+
+### 2.1 `RotationConfig` reference
+
+| Field | Default | Description |
+|-------|---------|-------------|
+| `algorithm` | `"hadamard"` (frozen) | Registry key. |
+| `backend` | `"auto"` | `"auto"`, `"inplace"`, or `"transform"`. |
+| `block_size` | `None` (auto) | Grouped Hadamard block size. Auto-filled from the data type by `normalize_rotation_config()`: **32** for MXFP, **16** for NVFP. `None`/`-1` means full-dimension Hadamard. |
+| `hadamard_type` | `"hadamard"` | One of `hadamard`, `random_hadamard`, `inplace_quarot_hadamard`, `inplace_hadamard`, `inplace_random`. Deterministic Hadamard uses Sylvester construction (`block_size` must be a power of 2); `random_hadamard` supports non-power-of-2 sizes from the known-matrix library. |
+| `fuse_online_to_weight` | `None` | Fuse online Hadamard rotation into weights when supported (`inplace` backend only). |
+| `allow_online_rotation` | `True` | Allow online activation rotation. |
+
+### 2.2 Usage
+
+```python
+from auto_round import AutoRound
+
+model_name_or_path = "meta-llama/Llama-3.1-8B-Instruct"
+
+# rotation_config="default": block_size auto (32 for MXFP), hadamard_type="hadamard"
+ar = AutoRound(model_name_or_path, scheme="MXFP4", rotation_config="default")
+ar.quantize_and_save(output_dir="./Llama-3.1-8B-Instruct-mxfp4-ht", format="auto_round")
+```
+
+The per-linear rotation is also exposed on the CLI via `--algorithm hadamard`:
+
+```bash
+auto-round --model meta-llama/Llama-3.1-8B-Instruct --scheme MXFP4 \
+ --algorithm hadamard --rotation_type hadamard --rotation_backend auto
+```
+
+CLI flags: `--rotation_type {hadamard,random_hadamard,quarot_hadamard}`,
+`--rotation_backend {auto,inplace,transform}`, `--rotation_block_size N`,
+`--fuse_online_to_weight` / `--no-fuse_online_to_weight`,
+`--allow_online_rotation` / `--no-allow_online_rotation`.
+
+---
+
+## 3. How `rotation_config` is dispatched
+
+`rotation_config` is accepted by `AutoRound` and normalized by the unified
+`apply_rotation()` entry point in
+`auto_round/algorithms/transforms/__init__.py`:
+
+- A `BaseRotationConfig` instance (`SpinQuantConfig` or `RotationConfig`) is used
+ directly.
+- The strings `"quarot"` / `"spinquant"` map to the `SpinQuantConfig` shortcuts above.
+- A dict with `algorithm="spinquant"` becomes a `SpinQuantConfig`.
+- Any other string (e.g. `"default"`, `"hadamard"`, `"random_hadamard"`) or dict maps to
+ the per-linear `RotationConfig`.
+
+---
+
+## 4. Example Results
+
+Enabling rotation consistently recovers accuracy lost to aggressive low-bit
+quantization. The tables below compare three configurations for each model:
+
+- **BF16** — the unquantized baseline (reference, 100%).
+- **MXFP4** — MXFP4 quantization without rotation.
+- **MXFP4 + rotation** — MXFP4 quantization with the QuaRot-style rotation shown in
+ the *Rotation* column.
+
+All numbers were measured with [lm-eval](https://github.com/EleutherAI/lm-evaluation-harness)
+on four tasks — **GSM8K**, **MMLU**, **PIQA** and **HellaSwag**. **Avg** is the mean
+across the four tasks and **Ratio** is that average relative to the BF16 baseline.
+Higher is better. Across all three model sizes (8B, 32B and 72B), rotation improves the
+MXFP4 average and closes the gap to BF16.
+
+**Qwen3-8B** — rotation lifts MXFP4 from 93.77% to 95.73% of BF16 (+1.96 pts), with
+the largest gains on the quantization-sensitive MMLU and PIQA tasks.
+
+| Model | Scheme | Rotation | GSM8K (strict, acc) | MMLU (acc) | PIQA (acc) | HellaSwag (acc) | Avg | Ratio |
+|---------|---------|---------|---------:|---------:|---------:|---------:|---------:|---------:|
+| Qwen/Qwen3-8B | BF16 | - | 0.9136 | 0.7290 | 0.7671 | 0.5708 | 0.7451 | 100.00% |
+| Qwen/Qwen3-8B | MXFP4 | - | 0.8560 | 0.6775 | 0.7323 | 0.5289 | 0.6987 | 93.77% |
+| Qwen/Qwen3-8B | MXFP4 | R1 (online, 32) + R2 (offline) + R4 (online, 32) | 0.8575 | 0.7009 | 0.7595 | 0.5352 | 0.7133 | 95.73% |
+
+**Qwen3-32B** — rotation improves MXFP4 from 98.36% to 98.53% of BF16, recovering
+most of the remaining PIQA and MMLU drop with R1 + R4 alone.
+
+| Model | Scheme | Rotation | GSM8K (strict, acc) | MMLU (acc) | PIQA (acc_norm) | HellaSwag (acc) | Avg | Ratio |
+|---------|---------|---------|---------:|---------:|---------:|---------:|---------:|---------:|
+| Qwen/Qwen3-32B | BF16 | - | 0.9287 | 0.8076 | 0.8118 | 0.6389 | 0.7968 | 100.00% |
+| Qwen/Qwen3-32B | MXFP4 | - | 0.9272 | 0.7876 | 0.7900 | 0.6300 | 0.7837 | 98.36% |
+| Qwen/Qwen3-32B | MXFP4 | R1 (online, 32) + R4 (online, 32) | 0.9249 | 0.7908 | 0.8030 | 0.6215 | 0.7851 | 98.53% |
+
+**Qwen2.5-72B** — rotation improves MXFP4 from 97.60% to 98.58% of BF16 (+0.98 pts),
+with gains across every task.
+
+| Model | Scheme | Rotation | GSM8K (strict, acc) | MMLU (acc) | PIQA (acc_norm) | HellaSwag (acc) | Avg | Ratio |
+|---------|---------|---------|---------:|---------:|---------:|---------:|---------:|---------:|
+| Qwen/Qwen2.5-72B | BF16 | - | 0.9454 | 0.8346 | 0.8335 | 0.7035 | 0.8293 | 100.00% |
+| Qwen/Qwen2.5-72B | MXFP4 | - | 0.9242 | 0.8104 | 0.8226 | 0.6803 | 0.8094 | 97.60% |
+| Qwen/Qwen2.5-72B | MXFP4 | R1 (online, 32) + R4 (online, 32) | 0.9333 | 0.8204 | 0.8292 | 0.6871 | 0.8175 | 98.58% |
+
+> **Note:** Exact numbers depend on the model, task set, lm-eval version and hardware.
+> Treat these as illustrative of the *trend* — rotation improving low-bit accuracy —
+> rather than as guaranteed values.
diff --git a/docs/step_by_step.md b/docs/step_by_step.md
index bbd30f98a..e040dc854 100644
--- a/docs/step_by_step.md
+++ b/docs/step_by_step.md
@@ -35,8 +35,6 @@ This document presents step-by-step instructions for auto-round llm quantization
- [Enable multiple gpus calibration in lm_head quantization](#enable-multiple-gpus-calibration-in-lm_head-quantization)
+ [Adjust Hyperparameters](#adjust-hyperparameters)
+ [Rotation (Experimental)](#rotation-experimental)
- - [QuaRot / SpinQuant](#quarot--spinquant)
- - [Per-Linear Block Rotation (Experimental)](#per-linear-block-rotation-experimental)
* [4 Inference](#4-inference)
+ [CPU](#cpu)
+ [Intel GPU](#intel-gpu)
@@ -898,154 +896,27 @@ autoround.save_quantized(format="auto_awq", output_dir="tmp_autoround")
### Rotation (Experimental)
-> ⚠️ **Experimental feature**: Rotation transform is still in an experimental stage. Inference relies on forward hooks, which are currently only supported by the Hugging Face Transformers backend. As a result, inference may be slower compared to native (non-rotated) models.
+> ⚠️ **Experimental feature**: Rotation transform is still experimental. Inference relies on forward hooks, which are currently only supported by the Hugging Face Transformers backend, so rotated models may run slower than native (non-rotated) models.
-AutoRound supports rotation-based transforms to improve quantization accuracy. Rotation redistributes outliers in weights and activations before quantization, making the distribution more uniform and quantization-friendly.
+Rotation redistributes outliers in weights and activations before quantization, making the distribution more uniform and quantization-friendly. It is most useful for aggressive low-bit schemes such as MXFP4, NVFP4 and W4A4.
-Two rotation approaches are available:
+AutoRound applies rotation through the `rotation_config` argument. The `"quarot"` preset — deterministic Hadamard rotation (QuaRot / SpinQuant), no training and no calibration data — is recommended for most use cases.
-- **QuaRot / SpinQuant** — full-model rotation across multiple positions (R1–R4), providing comprehensive outlier suppression for aggressive quantization (e.g., MXFP4, NVFP4, W4A4). **Recommended for most use cases.**
-- **Per-Linear Block Rotation** — block-diagonal rotation applied uniformly to every linear layer. An earlier experimental implementation; the QuaRot/SpinQuant approach above is generally preferred.
-
-#### QuaRot / SpinQuant
-
-QuaRot applies deterministic Hadamard rotations at up to 4 positions in the transformer architecture. Unlike the per-linear block rotation, it operates at the model architecture level — rotating the residual stream, attention heads, and MLP activations at specific positions for targeted outlier suppression.
-
-##### Rotation Positions
-
-| Position | Target | Mode | Effect |
-|----------|--------|------|--------|
-| **R1** | Residual stream (hidden_size) | Online or Offline | Smooths weight outliers across all linear layers |
-| **R2** | V/O projections (head_dim) | Offline (fused) | Balances per-head value distributions |
-| **R3** | Q/K after RoPE (head_dim) | Online (hook) | Improves KV-cache quantization friendliness |
-| **R4** | MLP up/down (intermediate_size) | Online (hook) | Suppresses activation outliers in FFN |
-
-- **Online**: Applied at runtime via forward hooks (no weight modification, zero-cost at save time)
-- **Offline (fused)**: Absorbed into adjacent weight matrices (no runtime overhead)
-
-##### Quick Start
+#### API Usage
```python
from auto_round import AutoRound
model_name = "Qwen/Qwen3-0.6B"
-output_dir = "./Qwen3-0.6B-mxfp4-quarot"
-# QuaRot preset: R1+R2+R3+R4 with deterministic Hadamard
+# QuaRot preset: deterministic Hadamard, no training
ar = AutoRound(model_name, scheme="MXFP4", rotation_config="quarot")
-ar.quantize_and_save(output_dir=output_dir, format="auto_round")
-```
-
-##### Rotation Levels
-
-Choose how many positions to rotate based on your accuracy needs:
-
-```python
-from auto_round import AutoRound
-from auto_round.algorithms.transforms.spinquant import SpinQuantConfig
-
-# R1 only (fast, good baseline improvement)
-ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True))
-
-# R1 + R2 (better, no runtime overhead after fuse)
-ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True, r2=True))
-
-# R1 + R2 + R3 + R4 (best accuracy, slight runtime overhead from hooks)
-ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True, r2=True, r3=True, r4=True))
-```
-
-##### String Shortcuts
-
-| Value | Equivalent |
-|-------|-----------|
-| `"quarot"` | `SpinQuantConfig(r1=True, r2=True, trainable_rotation=False, trainable_smooth=False)` — deterministic Hadamard, no training |
-| `"spinquant"` | `SpinQuantConfig(r1=True, r2=True, trainable_rotation=True, trainable_smooth=True)` — **experimental** (requires a dataloader) |
-
-> ⚠️ **SpinQuant trainable rotation** (`trainable_rotation=True`) enables learnable rotation matrices optimized via Cayley SGD. This feature is **experimental** and not fully validated. Use `"quarot"` (fixed Hadamard) for production workloads.
-
-##### Deterministic vs Random Hadamard
-
-```python
-# Deterministic (default): fixed Hadamard matrix, no extra storage needed
-ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True, r2=True, r3=True, r4=True))
-
-# Random: H × diag(±1), slightly better outlier suppression, requires saving the rotation matrix
-ar = AutoRound(
- model_name,
- scheme="MXFP4",
- rotation_config=SpinQuantConfig(
- r1=True, r2=True, r3=True, r4=True, random_r1=True, random_r2=True, random_r3=True, random_r4=True
- ),
-)
+ar.quantize_and_save(output_dir="./Qwen3-0.6B-mxfp4-quarot", format="auto_round")
```
-##### Key Parameters
-
-| Parameter | Default | Description |
-|-----------|---------|-------------|
-| `r1` / `r2` / `r3` / `r4` | `True / True / False / False` | Enable rotation at each position |
-| `online_r1_rotation` | `True` | R1 via hook (`True`) or fused into weights (`False`) |
-| `random_r1` / `random_r2` / `random_r3` / `random_r4` | `False` | Use random Hadamard (H×diag(±1)) instead of deterministic |
-| `rotation_size` | `None` (auto) | Block rotation dimension; auto-detected from model dimensions |
-| `trainable_rotation` | `False` | Enable SpinQuant learnable rotation (**experimental**, requires dataloader) |
-| `trainable_smooth` | `False` | Enable learnable smooth values (**experimental**, requires dataloader) |
-
-##### Save & Load
-
-Quantized models with rotation are saved and loaded transparently:
-
-```python
-# Save (rotation matrices stored automatically if needed)
-ar.quantize_and_save(output_dir="./my_model", format="auto_round")
-
-# Load (rotation hooks restored automatically)
-from transformers import AutoModelForCausalLM
-
-model = AutoModelForCausalLM.from_pretrained("./my_model", device_map="auto")
-```
-
-- **Deterministic rotations**: Only metadata (type + rotation_size) is stored — matrices are reconstructed on load
-- **Random rotations**: An `int8` (±1) rotation matrix is stored (size ~rotation_size² bytes)
-- **Online rotations**: Rebuilt during model loading (R1/R4 via QuantLinear forward patching; R3 via monkeypatch from config)
-
-#### Per-Linear Block Rotation (Experimental)
-
-> ⚠️ This is an earlier experimental implementation that applies block-diagonal Hadamard rotation **per linear layer** by patching every `nn.Linear` module in the model. For most use cases, the [QuaRot / SpinQuant](#quarot--spinquant) approach above is preferred — it provides architecture-aware rotation at specific positions (R1–R4) with better accuracy and lower overhead.
-
-The per-linear block rotation works by iterating over all linear layers and either:
-- **Weight mode**: Fusing the Hadamard matrix directly into the weight tensor (offline)
-- **Input mode**: Registering a forward pre-hook that rotates the input activation before each linear layer (online)
-
-This approach uses a configurable `block_size` (default 32) applied uniformly to every linear layer. It is simpler than QuaRot but less targeted — it does not distinguish between residual stream, attention, and MLP layers, nor does it handle RoPE or activation-side rotation. Use `rotation_config="default"` to enable it.
-
-##### Usage
-
-```python
-from auto_round import AutoRound
-
-model_name_or_path = "meta-llama/Llama-3.1-8B-Instruct"
-output_dir = "./Llama-3.1-8B-Instruct-mxfp4-ht"
-
-# rotation_config="default": block_size=32, hadamard_type="hadamard"
-ar = AutoRound(model_name_or_path, scheme="MXFP4", rotation_config="default")
-
-ar.quantize_and_save(output_dir=output_dir, format="auto_round")
-```
-
-##### Types
-
-| Type | Description |
-|------|-------------|
-| `hadamard` (default) | Deterministic Hadamard matrix (Sylvester construction, block_size must be power of 2) |
-| `random_hadamard` | Randomly signed Hadamard from known matrix library; supports non-power-of-2 sizes |
-
-##### Parameters
+Quantized models with rotation are saved and loaded transparently — rotation matrices and hooks are restored automatically on load, so inference needs no extra steps.
-| Parameter | Default | Description |
-|-----------|---------|-------------|
-| `block_size` | `32` | Size of the Hadamard block applied to each linear layer |
-| `hadamard_type` | `"hadamard"` | `"hadamard"` or `"random_hadamard"` |
-| `seed` | `None` | Random seed (for `random_hadamard` only) |
+For rotation positions (R1–R4), full configuration options, deterministic vs random Hadamard, trainable SpinQuant, the per-linear block rotation variant, and save/load internals, see [Rotation Details](./rotation_details.md).
## 4 Inference
diff --git a/docs/step_by_step_CN.md b/docs/step_by_step_CN.md
index 018cbeef8..660077ef6 100644
--- a/docs/step_by_step_CN.md
+++ b/docs/step_by_step_CN.md
@@ -36,8 +36,6 @@
- [手动配置设备映射](#手动配置设备映射)
+ [超参数调整](#超参数调整)
+ [旋转(Rotation)(实验性)](#旋转rotation实验性)
- - [QuaRot / SpinQuant](#quarot--spinquant)
- - [逐线性层块旋转(实验性)](#逐线性层块旋转实验性)
* [4 推理部署](#4-推理部署)
+ [CPU](#cpu)
+ [英特尔 GPU](#英特尔-gpu)
@@ -866,154 +864,27 @@ auto-round --model_name Qwen/Qwen3-0.6B --scheme "W4A16" --quant_lm_head --form
### 旋转(Rotation)(实验性)
-> ⚠️ **实验性功能**:旋转变换仍处于实验阶段。推理依赖 forward hook 机制,目前仅支持 Hugging Face Transformers 后端,因此相比非旋转模型,推理速度可能较慢。
+> ⚠️ **实验性功能**:旋转变换仍处于实验阶段。推理依赖 forward hook 机制,目前仅支持 Hugging Face Transformers 后端,因此相比非旋转模型,旋转后的模型推理速度可能较慢。
-AutoRound 支持基于旋转的变换技术来提升量化精度。旋转在量化前对权重和激活中的离群点进行重分布,使分布更加均匀,从而对量化更友好。
+旋转在量化前对权重和激活中的离群点进行重分布,使分布更加均匀、对量化更友好。它对 MXFP4、NVFP4、W4A4 等激进的低比特方案最为有效。
-目前提供两种旋转方式:
+AutoRound 通过 `rotation_config` 参数应用旋转。推荐在大多数场景中使用 `"quarot"` 预设——确定性 Hadamard 旋转(QuaRot / SpinQuant),无需训练、无需校准数据。
-- **QuaRot / SpinQuant** — 在多个位置(R1–R4)进行全模型旋转,为激进量化(如 MXFP4、NVFP4、W4A4)提供全面的离群点抑制。**推荐在大多数场景中使用。**
-- **逐线性层块旋转(Per-Linear Block Rotation)** — 对每个线性层均匀应用块对角旋转。为较早期的实验性实现,通常建议使用上述 QuaRot/SpinQuant 方式。
-
-#### QuaRot / SpinQuant
-
-QuaRot 在 Transformer 架构的最多 4 个位置应用确定性 Hadamard 旋转。与逐线性层块旋转不同,它在模型架构级别操作——在特定位置对残差流、注意力头和 MLP 激活进行旋转,以实现针对性的离群点抑制。
-
-##### 旋转位置
-
-| 位置 | 目标 | 模式 | 效果 |
-|------|------|------|------|
-| **R1** | 残差流(hidden_size) | 在线或离线 | 平滑所有线性层的权重离群点 |
-| **R2** | V/O 投影(head_dim) | 离线(融合) | 平衡各注意力头的值分布 |
-| **R3** | RoPE 后的 Q/K(head_dim) | 在线(hook) | 提升 KV-cache 的量化友好性 |
-| **R4** | MLP up/down(intermediate_size) | 在线(hook) | 抑制 FFN 中的激活离群点 |
-
-- **在线(Online)**:通过 forward hook 在运行时应用(不修改权重,保存时零开销)
-- **离线/融合(Offline/Fused)**:吸收到相邻权重矩阵中(无运行时开销)
-
-##### 快速开始
+#### API 用法
```python
from auto_round import AutoRound
model_name = "Qwen/Qwen3-0.6B"
-output_dir = "./Qwen3-0.6B-mxfp4-quarot"
-# QuaRot 预设:R1+R2+R3+R4,确定性 Hadamard
+# QuaRot 预设:确定性 Hadamard,无需训练
ar = AutoRound(model_name, scheme="MXFP4", rotation_config="quarot")
-ar.quantize_and_save(output_dir=output_dir, format="auto_round")
-```
-
-##### 旋转级别
-
-根据精度需求选择旋转的位置数量:
-
-```python
-from auto_round import AutoRound
-from auto_round.algorithms.transforms.spinquant import SpinQuantConfig
-
-# 仅 R1(速度快,良好的基准提升)
-ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True))
-
-# R1 + R2(更好,融合后无运行时开销)
-ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True, r2=True))
-
-# R1 + R2 + R3 + R4(最佳精度,hook 带来少许运行时开销)
-ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True, r2=True, r3=True, r4=True))
-```
-
-##### 字符串快捷方式
-
-| 值 | 等价配置 |
-|----|---------|
-| `"quarot"` | `SpinQuantConfig(r1=True, r2=True, trainable_rotation=False, trainable_smooth=False)` — 确定性 Hadamard,无需训练 |
-| `"spinquant"` | `SpinQuantConfig(r1=True, r2=True, trainable_rotation=True, trainable_smooth=True)` — **实验性**(需要提供 dataloader) |
-
-> ⚠️ **SpinQuant 可训练旋转**(`trainable_rotation=True`)启用通过 Cayley SGD 优化的可学习旋转矩阵。此功能为**实验性**,尚未完全验证。生产环境建议使用 `"quarot"`(固定 Hadamard)。
-
-##### 确定性与随机 Hadamard
-
-```python
-# 确定性(默认):固定 Hadamard 矩阵,无需额外存储
-ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True, r2=True, r3=True, r4=True))
-
-# 随机:H × diag(±1),离群点抑制效果略好,需保存旋转矩阵
-ar = AutoRound(
- model_name,
- scheme="MXFP4",
- rotation_config=SpinQuantConfig(
- r1=True, r2=True, r3=True, r4=True, random_r1=True, random_r2=True, random_r3=True, random_r4=True
- ),
-)
+ar.quantize_and_save(output_dir="./Qwen3-0.6B-mxfp4-quarot", format="auto_round")
```
-##### 关键参数
-
-| 参数 | 默认值 | 描述 |
-|------|--------|------|
-| `r1` / `r2` / `r3` / `r4` | `True / True / False / False` | 启用各位置的旋转 |
-| `online_r1_rotation` | `True` | R1 通过 hook 应用(`True`)或融合到权重中(`False`) |
-| `random_r1` / `random_r2` / `random_r3` / `random_r4` | `False` | 使用随机 Hadamard(H×diag(±1))而非确定性 |
-| `rotation_size` | `None`(自动) | 块旋转维度;从模型维度自动检测 |
-| `trainable_rotation` | `False` | 启用 SpinQuant 可学习旋转(**实验性**,需要 dataloader) |
-| `trainable_smooth` | `False` | 启用可学习 smooth 值(**实验性**,需要 dataloader) |
-
-##### 保存与加载
-
-带旋转的量化模型可透明地保存和加载:
-
-```python
-# 保存(旋转矩阵在需要时自动存储)
-ar.quantize_and_save(output_dir="./my_model", format="auto_round")
-
-# 加载(旋转 hook 自动恢复)
-from transformers import AutoModelForCausalLM
-
-model = AutoModelForCausalLM.from_pretrained("./my_model", device_map="auto")
-```
-
-- **确定性旋转**:仅存储元数据(类型 + rotation_size)——矩阵在加载时重建
-- **随机旋转**:以 `int8`(±1)旋转矩阵形式存储(大小约为 rotation_size² 字节)
-- **在线旋转**:在模型加载时重建(R1/R4 通过 QuantLinear forward patch;R3 通过配置触发的 monkeypatch)
-
-#### 逐线性层块旋转(实验性)
-
-> ⚠️ 这是较早期的实验性实现,通过对模型中每个 `nn.Linear` 模块应用块对角 Hadamard 旋转来工作。对于大多数场景,推荐使用上述 [QuaRot / SpinQuant](#quarot--spinquant) 方式——它在特定位置(R1–R4)提供架构感知的旋转,具有更好的精度和更低的开销。
-
-逐线性层块旋转通过遍历所有线性层实现,可选择以下模式:
-- **权重模式**:将 Hadamard 矩阵直接融合到权重张量中(离线)
-- **输入模式**:注册 forward pre-hook 在每个线性层前旋转输入激活(在线)
-
-该方式使用可配置的 `block_size`(默认 32),对每个线性层统一应用。它比 QuaRot 更简单,但不够精确——不区分残差流、注意力和 MLP 层,也不处理 RoPE 或激活侧旋转。使用 `rotation_config="default"` 启用。
-
-##### 使用方式
-
-```python
-from auto_round import AutoRound
-
-model_name_or_path = "meta-llama/Llama-3.1-8B-Instruct"
-output_dir = "./Llama-3.1-8B-Instruct-mxfp4-ht"
-
-# rotation_config="default": block_size=32, hadamard_type="hadamard"
-ar = AutoRound(model_name_or_path, scheme="MXFP4", rotation_config="default")
-
-ar.quantize_and_save(output_dir=output_dir, format="auto_round")
-```
-
-##### 类型
-
-| 类型 | 描述 |
-|------|------|
-| `hadamard`(默认) | 确定性 Hadamard 矩阵(Sylvester 构造,block_size 必须为 2 的幂次) |
-| `random_hadamard` | 随机签名 Hadamard,来自已知矩阵库;支持非 2 的幂次尺寸 |
-
-##### 参数说明
+带旋转的量化模型可透明地保存和加载——旋转矩阵和 hook 会在加载时自动恢复,推理无需额外步骤。
-| 参数 | 默认值 | 描述 |
-|------|--------|------|
-| `block_size` | `32` | 应用于每个线性层的 Hadamard 块大小 |
-| `hadamard_type` | `"hadamard"` | `"hadamard"` 或 `"random_hadamard"` |
-| `seed` | `None` | 随机种子(仅用于 `random_hadamard`) |
+关于旋转位置(R1–R4)、完整配置选项、确定性与随机 Hadamard、可训练的 SpinQuant、逐线性层块旋转变体以及保存/加载的内部细节,请参阅 [Rotation Details](./rotation_details.md)(英文)。
## 4 推理部署