PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
update rotation doc. (#1999)
lkk5ee45db7b38f
diff --git a/docs/rotation_details.md b/docs/rotation_details.mdnew file mode 100644index 000000000..5a9aa8ffc--- /dev/null+++ b/docs/rotation_details.md@@ -0,0 +1,331 @@+# Rotation Transform — Details (Experimental)++> ⚠️ **Experimental feature.** Rotation transform is still in an experimental stage. Inference relies on forward hooks, which are currently only supported by the Hugging Face Transformers backend. As a result, inference may be slower compared to native (non-rotated) models.++This document is the in-depth reference for AutoRound's rotation transforms. For a+concise getting-started guide, see the *Rotation* section in+[step_by_step.md](./step_by_step.md).++Rotation redistributes outliers in weights and activations before quantization by+multiplying tensors with orthogonal (Hadamard) matrices. Because an orthogonal+rotation `R` satisfies `R @ Rᵀ = I`, a matched pair of rotations inserted around a+linear operation leaves the network's output unchanged (`x @ R @ Rᵀ @ Wᵀ = x @ Wᵀ`)+while making the intermediate distributions flatter and more quantization-friendly.+This is most useful for aggressive low-bit schemes such as MXFP4, NVFP4 and W4A4.++AutoRound provides two independent rotation implementations:++- **QuaRot / SpinQuant** — architecture-aware, full-model rotation applied at up to+ four positions (R1–R4). Selected via `rotation_config="quarot"`,+ `rotation_config="spinquant"`, or a `SpinQuantConfig` instance. **Recommended.**+- **Per-Linear Block Rotation** — an earlier, simpler implementation that applies a+ block-diagonal Hadamard uniformly to every `nn.Linear`. Selected via+ `rotation_config="default"` / a string Hadamard type, or a `RotationConfig`+ instance. Also exposed through the `--algorithm hadamard` CLI.++---++## 1. QuaRot / SpinQuant++QuaRot applies deterministic Hadamard rotations at up to four positions in the+transformer architecture. Unlike per-linear block rotation, it operates at the model+architecture level — rotating the residual stream, attention heads, and MLP+activations at specific positions for targeted outlier suppression.++The pipeline (implemented in `auto_round/algorithms/transforms/spinquant/preprocessor.py`,+`SpinQuantPreprocessor`) runs the following stages:++1. Fuse RMSNorm scale parameters into the following linear layers (`fuse_rmsnorm`).+2. Untie word embeddings if they are shared with the LM head (`untie_embeddings`).+3. Initialize rotation matrices R1–R4.+4. (Only if trainable) train the rotations and smooth values via KL/MSE loss.+5. Fuse the offline rotations (R2, and R1 when `online_r1_rotation=False`) into weights.+6. Register the online hooks (R1 when online, R3, R4) and clean up.++### 1.1 Rotation positions++| Position | Target | Dimension | Mode | Effect |+|----------|--------|-----------|------|--------|+| **R1** | Residual stream — inputs of `q/k/v_proj`, `gate/up_proj`; outputs of `o_proj`, `down_proj`, `embed_tokens`, `lm_head` | `hidden_size` (or `rotation_size`) | Online *or* Offline | Smooths weight outliers across all linear layers |+| **R2** | `v_proj` output / `o_proj` input (per attention head) | `head_dim` | Offline (fused) | Balances per-head value distributions |+| **R3** | Q/K after RoPE | `head_dim` | Online (RoPE monkeypatch) | Improves KV-cache quantization friendliness |+| **R4** | `down_proj` input (MLP activation) | `intermediate_size` (or `rotation_size`) | Online (hook) + weight fuse | Suppresses activation outliers in the FFN |++- **Online**: applied at runtime via forward hooks / a RoPE monkeypatch. Weights are+ pre-rotated so the paired inverse rotation happens at runtime; there is a small+ inference cost.+- **Offline (fused)**: the rotation is absorbed into adjacent weight matrices, leaving+ no runtime overhead.++R1 is applied online by default (`online_r1_rotation=True`): the target weights are+rotated and a `forward_pre_hook` is registered on the same modules. When+`online_r1_rotation=False`, R1 is instead fully fused into `embed_tokens`,+`q/k/v_proj`, `o_proj`, `gate/up/down_proj` and `lm_head`, cancelling out with no+runtime cost.++### 1.2 `SpinQuantConfig` reference++Defined in `auto_round/algorithms/transforms/spinquant/preprocessor.py`.++| Field | Default | Description |+|-------|---------|-------------|+| `algorithm` | `"spinquant"` | Registry key used by `BaseRotation.from_config()` to dispatch. |+| `r1` | `True` | Enable R1 (hidden_size residual-stream rotation). |+| `r2` | `True` | Enable R2 (head_dim per-head rotation, offline). |+| `r3` | `False` | Enable R3 (online Q/K rotation after RoPE). |+| `r4` | `False` | Enable R4 (online MLP activation rotation). |+| `rotation_size` | `None` (auto) | Overrides the R1/R4 block size (R1 uses it instead of `hidden_size`, R4 instead of `intermediate_size`). R2 always uses `head_dim`; R3 has no custom size. Must be a positive power of 2. |+| `random_r1` / `random_r2` / `random_r3` / `random_r4` | `False` | Use random Hadamard (`H × diag(±1)`) instead of the deterministic matrix at that position. Only relevant in QuaRot mode. |+| `online_r1_rotation` | `True` | Apply R1 online via hook (`True`) or fuse it fully into weights (`False`). |+| `trainable_rotation` | `False` | Learn the rotation matrices via Cayley SGD (SpinQuant mode). **Experimental**, requires a dataloader. |+| `trainable_smooth` | `False` | Learn SmoothQuant-style `smooth_values` jointly via Adam. **Experimental**, requires a dataloader. |+| `iters` | `200` | Training iterations (trainable modes only). |+| `lr` | `1e-4` | SGDG (Cayley) learning rate for rotation matrices. |+| `smooth_lr` | `1e-3` | Adam learning rate for smooth values. |+| `batch_size` | `1` | Training batch size. |+| `loss_type` | `"kl_top"` | Training loss: `"kl_top"`, `"kl_full"`, or `"mse"`. |+| `kl_top_k` | `1000` | Top-k logits used by the `kl_top` loss. |+| `fuse_rmsnorm` | `True` | Fuse RMSNorm scales into following linear layers. |+| `untie_embeddings` | `True` | Untie shared embedding / LM-head weights before rotating. |+| `dtype` | `torch.float32` | Numerical dtype used for rotation math. |+| `device` | `None` (auto) | Defaults to `"cuda"` if available, else `"cpu"`. |++### 1.3 String shortcuts++| Value | Equivalent |+|-------|-----------|+| `"quarot"` | `SpinQuantConfig(trainable_rotation=False, trainable_smooth=False)` — deterministic Hadamard, no training, no calibration data. |+| `"spinquant"` | `SpinQuantConfig(trainable_rotation=True, trainable_smooth=True)` — **experimental**, requires a dataloader. |++Both presets keep the default `r1=True, r2=True, r3=False, r4=False`. A dict with+`algorithm="spinquant"` is also parsed into a `SpinQuantConfig`.++> ⚠️ **SpinQuant trainable rotation** (`trainable_rotation=True`) enables learnable+> rotation matrices optimized via Cayley SGD. This feature is experimental and not+> fully validated on real models. Use `"quarot"` (fixed Hadamard) for production+> workloads.++### 1.4 Usage++```python+from auto_round import AutoRound+from auto_round.algorithms.transforms.spinquant import SpinQuantConfig++model_name = "Qwen/Qwen3-0.6B"++# QuaRot preset: R1+R2 deterministic Hadamard, no training+ar = AutoRound(model_name, scheme="MXFP4", rotation_config="quarot")++# Choose how many positions to rotate:+ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True)) # R1 only+ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True, r2=True)) # R1+R2+ar = AutoRound(+ model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True, r2=True, r3=True, r4=True)+) # all four++ar.quantize_and_save(output_dir="./Qwen3-0.6B-mxfp4-quarot", format="auto_round")+```++### 1.5 Deterministic vs random Hadamard++Rotation matrices are constructed in+`auto_round/algorithms/transforms/spinquant/rotation_utils.py`:++- **Deterministic** (default): `deterministic_hadamard_matrix(size)` builds a Sylvester+ Hadamard matrix (`H → [[H, H], [H, −H]]`) normalized by `√size`. The size must be a+ power of 2. Because the matrix is fully determined by its size, nothing needs to be+ stored.+- **Random**: `random_hadamard_matrix(size)` computes `H @ diag(±1) / √N`. It offers+ slightly better outlier suppression but the sign matrix must be persisted.++For non-power-of-2 dimensions, `get_hadamard_K(n)` factors `n = K · 2^m` and pulls a+known base Hadamard of size `K` (e.g. 12, 20, 28, 36, 40, 52, 60, 108, 140, 156, 172)+from `known_hadamard.py`, combining it with a Sylvester block.++```python+# Random Hadamard at every enabled position+ar = AutoRound(+ model_name,+ scheme="MXFP4",+ rotation_config=SpinQuantConfig(+ r1=True,+ r2=True,+ r3=True,+ r4=True,+ random_r1=True,+ random_r2=True,+ random_r3=True,+ random_r4=True,+ ),+)+```++### 1.6 Trainable rotation (SpinQuant mode)++When `trainable_rotation=True` (and/or `trainable_smooth=True`), the rotations become+learnable and are optimized on calibration data:++- `cayley_optimizer.py` provides `SGDG`, an orthogonality-preserving optimizer that+ keeps each rotation on the Stiefel manifold using a Cayley/QR retraction.+ `AdamAndSGDG` combines Adam (for `smooth_values`) with SGDG (for the `spinquant_R*`+ matrices).+- `training.py` runs a KL-/MSE-based distillation loop against a frozen clone of the+ original model (`create_dual_optimizer()`, `run_training_loop()`).+- `SpinQuantPreprocessor.preprocess()` **requires a dataloader** whenever either+ trainable flag is set.++### 1.7 Save & load++Quantized models with rotation are saved and loaded transparently. Serialization lives+in `auto_round/algorithms/transforms/spinquant/serialize.py`.++```python+# Save (rotation matrices stored automatically when needed)+ar.quantize_and_save(output_dir="./my_model", format="auto_round")++# Load (rotation hooks restored automatically)+from transformers import AutoModelForCausalLM++model = AutoModelForCausalLM.from_pretrained("./my_model", device_map="auto")+```++What is stored per position (metadata keys prefixed with `spinquant_r1` / `spinquant_r4`,+with a `*_type` code: `0` = deterministic, `1` = random, `2` = trained):++- **Deterministic rotations**: only metadata (type + `rotation_size`) — the matrix is+ reconstructed on load.+- **Random rotations**: an `int8` (±1) matrix is stored (size ≈ `rotation_size²` bytes).+- **Trained rotations**: the full `float32` matrix is stored.+- **Online rotations**: rebuilt during model loading — `preregister_spinquant_buffers()`+ creates empty buffers before the `state_dict` load, and `rebuild_spinquant_online()`+ re-patches the QuantLinear forward (R1/R4) and re-applies the R3 RoPE monkeypatch.+ R3 is rebuilt purely from `config.json`, not from stored buffers.++---++## 2. Per-Linear Block Rotation++> ⚠️ This is an earlier experimental implementation that applies block-diagonal+> Hadamard rotation **per linear layer** by patching every `nn.Linear` module in the+> model. For most use cases the QuaRot / SpinQuant approach above is preferred — it+> provides architecture-aware rotation at specific positions (R1–R4) with better+> accuracy and lower overhead.++The per-linear block rotation iterates over all linear layers and applies a Hadamard of+a configurable `block_size` uniformly. It does not distinguish between residual stream,+attention and MLP layers, nor does it handle RoPE or attention-side rotation. It is+configured with `RotationConfig`+(`auto_round/algorithms/transforms/hadamard/config.py`), which is shared by three+backends:++| `backend` | Behaviour |+|-----------|-----------|+| `"inplace"` | QuaRot-style residual-stream rotation; works for any dtype and can fuse the online Hadamard into weights (`fuse_online_to_weight=True`). |+| `"transform"` | Per-Linear weight + activation Hadamard with a fused Triton kernel; supports only MXFP4 / NVFP4 and cannot fuse online to weight. |+| `"auto"` (default) | Picks `inplace` when a fused online rotation is requested, `transform` for MX/NV-FP data types, `inplace` otherwise. |++Application modes (`hadamard/apply.py`):++- **Input mode**: registers a `forward_pre_hook` that rotates the input activation+ before each linear layer (online).+- **Weight mode**: fuses the Hadamard directly into the module weight (offline) and+ patches the calibration wrappers accordingly.++### 2.1 `RotationConfig` reference++| Field | Default | Description |+|-------|---------|-------------|+| `algorithm` | `"hadamard"` (frozen) | Registry key. |+| `backend` | `"auto"` | `"auto"`, `"inplace"`, or `"transform"`. |+| `block_size` | `None` (auto) | Grouped Hadamard block size. Auto-filled from the data type by `normalize_rotation_config()`: **32** for MXFP, **16** for NVFP. `None`/`-1` means full-dimension Hadamard. |+| `hadamard_type` | `"hadamard"` | One of `hadamard`, `random_hadamard`, `inplace_quarot_hadamard`, `inplace_hadamard`, `inplace_random`. Deterministic Hadamard uses Sylvester construction (`block_size` must be a power of 2); `random_hadamard` supports non-power-of-2 sizes from the known-matrix library. |+| `fuse_online_to_weight` | `None` | Fuse online Hadamard rotation into weights when supported (`inplace` backend only). |+| `allow_online_rotation` | `True` | Allow online activation rotation. |++### 2.2 Usage++```python+from auto_round import AutoRound++model_name_or_path = "meta-llama/Llama-3.1-8B-Instruct"++# rotation_config="default": block_size auto (32 for MXFP), hadamard_type="hadamard"+ar = AutoRound(model_name_or_path, scheme="MXFP4", rotation_config="default")+ar.quantize_and_save(output_dir="./Llama-3.1-8B-Instruct-mxfp4-ht", format="auto_round")+```++The per-linear rotation is also exposed on the CLI via `--algorithm hadamard`:++```bash+auto-round --model meta-llama/Llama-3.1-8B-Instruct --scheme MXFP4 \+ --algorithm hadamard --rotation_type hadamard --rotation_backend auto+```++CLI flags: `--rotation_type {hadamard,random_hadamard,quarot_hadamard}`,+`--rotation_backend {auto,inplace,transform}`, `--rotation_block_size N`,+`--fuse_online_to_weight` / `--no-fuse_online_to_weight`,+`--allow_online_rotation` / `--no-allow_online_rotation`.++---++## 3. How `rotation_config` is dispatched++`rotation_config` is accepted by `AutoRound` and normalized by the unified+`apply_rotation()` entry point in+`auto_round/algorithms/transforms/__init__.py`:++- A `BaseRotationConfig` instance (`SpinQuantConfig` or `RotationConfig`) is used+ directly.+- The strings `"quarot"` / `"spinquant"` map to the `SpinQuantConfig` shortcuts above.+- A dict with `algorithm="spinquant"` becomes a `SpinQuantConfig`.+- Any other string (e.g. `"default"`, `"hadamard"`, `"random_hadamard"`) or dict maps to+ the per-linear `RotationConfig`.++---++## 4. Example Results++Enabling rotation consistently recovers accuracy lost to aggressive low-bit+quantization. The tables below compare three configurations for each model:++- **BF16** — the unquantized baseline (reference, 100%).+- **MXFP4** — MXFP4 quantization without rotation.+- **MXFP4 + rotation** — MXFP4 quantization with the QuaRot-style rotation shown in+ the *Rotation* column.++All numbers were measured with [lm-eval](https://github.com/EleutherAI/lm-evaluation-harness)+on four tasks — **GSM8K**, **MMLU**, **PIQA** and **HellaSwag**. **Avg** is the mean+across the four tasks and **Ratio** is that average relative to the BF16 baseline.+Higher is better. Across all three model sizes (8B, 32B and 72B), rotation improves the+MXFP4 average and closes the gap to BF16.++**Qwen3-8B** — rotation lifts MXFP4 from 93.77% to 95.73% of BF16 (+1.96 pts), with+the largest gains on the quantization-sensitive MMLU and PIQA tasks.++| Model | Scheme | Rotation | GSM8K (strict, acc) | MMLU (acc) | PIQA (acc) | HellaSwag (acc) | Avg | Ratio |+|---------|---------|---------|---------:|---------:|---------:|---------:|---------:|---------:|+| Qwen/Qwen3-8B | BF16 | - | 0.9136 | 0.7290 | 0.7671 | 0.5708 | 0.7451 | 100.00% |+| Qwen/Qwen3-8B | MXFP4 | - | 0.8560 | 0.6775 | 0.7323 | 0.5289 | 0.6987 | 93.77% |+| Qwen/Qwen3-8B | MXFP4 | R1 (online, 32) + R2 (offline) + R4 (online, 32) | 0.8575 | 0.7009 | 0.7595 | 0.5352 | 0.7133 | 95.73% |++**Qwen3-32B** — rotation improves MXFP4 from 98.36% to 98.53% of BF16, recovering+most of the remaining PIQA and MMLU drop with R1 + R4 alone.++| Model | Scheme | Rotation | GSM8K (strict, acc) | MMLU (acc) | PIQA (acc_norm) | HellaSwag (acc) | Avg | Ratio |+|---------|---------|---------|---------:|---------:|---------:|---------:|---------:|---------:|+| Qwen/Qwen3-32B | BF16 | - | 0.9287 | 0.8076 | 0.8118 | 0.6389 | 0.7968 | 100.00% |+| Qwen/Qwen3-32B | MXFP4 | - | 0.9272 | 0.7876 | 0.7900 | 0.6300 | 0.7837 | 98.36% |+| Qwen/Qwen3-32B | MXFP4 | R1 (online, 32) + R4 (online, 32) | 0.9249 | 0.7908 | 0.8030 | 0.6215 | 0.7851 | 98.53% |++**Qwen2.5-72B** — rotation improves MXFP4 from 97.60% to 98.58% of BF16 (+0.98 pts),+with gains across every task.++| Model | Scheme | Rotation | GSM8K (strict, acc) | MMLU (acc) | PIQA (acc_norm) | HellaSwag (acc) | Avg | Ratio |+|---------|---------|---------|---------:|---------:|---------:|---------:|---------:|---------:|+| Qwen/Qwen2.5-72B | BF16 | - | 0.9454 | 0.8346 | 0.8335 | 0.7035 | 0.8293 | 100.00% |+| Qwen/Qwen2.5-72B | MXFP4 | - | 0.9242 | 0.8104 | 0.8226 | 0.6803 | 0.8094 | 97.60% |+| Qwen/Qwen2.5-72B | MXFP4 | R1 (online, 32) + R4 (online, 32) | 0.9333 | 0.8204 | 0.8292 | 0.6871 | 0.8175 | 98.58% |++> **Note:** Exact numbers depend on the model, task set, lm-eval version and hardware.+> Treat these as illustrative of the *trend* — rotation improving low-bit accuracy —+> rather than as guaranteed values.diff --git a/docs/step_by_step.md b/docs/step_by_step.mdindex bbd30f98a..e040dc854 100644--- a/docs/step_by_step.md+++ b/docs/step_by_step.md@@ -35,8 +35,6 @@ This document presents step-by-step instructions for auto-round llm quantization- [Enable multiple gpus calibration in lm_head quantization](#enable-multiple-gpus-calibration-in-lm_head-quantization)+ [Adjust Hyperparameters](#adjust-hyperparameters)+ [Rotation (Experimental)](#rotation-experimental)- - [QuaRot / SpinQuant](#quarot--spinquant)- - [Per-Linear Block Rotation (Experimental)](#per-linear-block-rotation-experimental)* [4 Inference](#4-inference)+ [CPU](#cpu)+ [Intel GPU](#intel-gpu)@@ -898,154 +896,27 @@ autoround.save_quantized(format="auto_awq", output_dir="tmp_autoround")### Rotation (Experimental)-> ⚠️ **Experimental feature**: Rotation transform is still in an experimental stage. Inference relies on forward hooks, which are currently only supported by the Hugging Face Transformers backend. As a result, inference may be slower compared to native (non-rotated) models.+> ⚠️ **Experimental feature**: Rotation transform is still experimental. Inference relies on forward hooks, which are currently only supported by the Hugging Face Transformers backend, so rotated models may run slower than native (non-rotated) models.-AutoRound supports rotation-based transforms to improve quantization accuracy. Rotation redistributes outliers in weights and activations before quantization, making the distribution more uniform and quantization-friendly.+Rotation redistributes outliers in weights and activations before quantization, making the distribution more uniform and quantization-friendly. It is most useful for aggressive low-bit schemes such as MXFP4, NVFP4 and W4A4.-Two rotation approaches are available:+AutoRound applies rotation through the `rotation_config` argument. The `"quarot"` preset — deterministic Hadamard rotation (QuaRot / SpinQuant), no training and no calibration data — is recommended for most use cases.-- **QuaRot / SpinQuant** — full-model rotation across multiple positions (R1–R4), providing comprehensive outlier suppression for aggressive quantization (e.g., MXFP4, NVFP4, W4A4). **Recommended for most use cases.**-- **Per-Linear Block Rotation** — block-diagonal rotation applied uniformly to every linear layer. An earlier experimental implementation; the QuaRot/SpinQuant approach above is generally preferred.--#### QuaRot / SpinQuant--QuaRot applies deterministic Hadamard rotations at up to 4 positions in the transformer architecture. Unlike the per-linear block rotation, it operates at the model architecture level — rotating the residual stream, attention heads, and MLP activations at specific positions for targeted outlier suppression.--##### Rotation Positions--| Position | Target | Mode | Effect |-|----------|--------|------|--------|-| **R1** | Residual stream (hidden_size) | Online or Offline | Smooths weight outliers across all linear layers |-| **R2** | V/O projections (head_dim) | Offline (fused) | Balances per-head value distributions |-| **R3** | Q/K after RoPE (head_dim) | Online (hook) | Improves KV-cache quantization friendliness |-| **R4** | MLP up/down (intermediate_size) | Online (hook) | Suppresses activation outliers in FFN |--- **Online**: Applied at runtime via forward hooks (no weight modification, zero-cost at save time)-- **Offline (fused)**: Absorbed into adjacent weight matrices (no runtime overhead)--##### Quick Start+#### API Usage```pythonfrom auto_round import AutoRoundmodel_name = "Qwen/Qwen3-0.6B"-output_dir = "./Qwen3-0.6B-mxfp4-quarot"-# QuaRot preset: R1+R2+R3+R4 with deterministic Hadamard+# QuaRot preset: deterministic Hadamard, no trainingar = AutoRound(model_name, scheme="MXFP4", rotation_config="quarot")-ar.quantize_and_save(output_dir=output_dir, format="auto_round")-```--##### Rotation Levels--Choose how many positions to rotate based on your accuracy needs:--```python-from auto_round import AutoRound-from auto_round.algorithms.transforms.spinquant import SpinQuantConfig--# R1 only (fast, good baseline improvement)-ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True))--# R1 + R2 (better, no runtime overhead after fuse)-ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True, r2=True))--# R1 + R2 + R3 + R4 (best accuracy, slight runtime overhead from hooks)-ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True, r2=True, r3=True, r4=True))-```--##### String Shortcuts--| Value | Equivalent |-|-------|-----------|-| `"quarot"` | `SpinQuantConfig(r1=True, r2=True, trainable_rotation=False, trainable_smooth=False)` — deterministic Hadamard, no training |-| `"spinquant"` | `SpinQuantConfig(r1=True, r2=True, trainable_rotation=True, trainable_smooth=True)` — **experimental** (requires a dataloader) |--> ⚠️ **SpinQuant trainable rotation** (`trainable_rotation=True`) enables learnable rotation matrices optimized via Cayley SGD. This feature is **experimental** and not fully validated. Use `"quarot"` (fixed Hadamard) for production workloads.--##### Deterministic vs Random Hadamard--```python-# Deterministic (default): fixed Hadamard matrix, no extra storage needed-ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True, r2=True, r3=True, r4=True))--# Random: H × diag(±1), slightly better outlier suppression, requires saving the rotation matrix-ar = AutoRound(- model_name,- scheme="MXFP4",- rotation_config=SpinQuantConfig(- r1=True, r2=True, r3=True, r4=True, random_r1=True, random_r2=True, random_r3=True, random_r4=True- ),-)+ar.quantize_and_save(output_dir="./Qwen3-0.6B-mxfp4-quarot", format="auto_round")```-##### Key Parameters--| Parameter | Default | Description |-|-----------|---------|-------------|-| `r1` / `r2` / `r3` / `r4` | `True / True / False / False` | Enable rotation at each position |-| `online_r1_rotation` | `True` | R1 via hook (`True`) or fused into weights (`False`) |-| `random_r1` / `random_r2` / `random_r3` / `random_r4` | `False` | Use random Hadamard (H×diag(±1)) instead of deterministic |-| `rotation_size` | `None` (auto) | Block rotation dimension; auto-detected from model dimensions |-| `trainable_rotation` | `False` | Enable SpinQuant learnable rotation (**experimental**, requires dataloader) |-| `trainable_smooth` | `False` | Enable learnable smooth values (**experimental**, requires dataloader) |--##### Save & Load--Quantized models with rotation are saved and loaded transparently:--```python-# Save (rotation matrices stored automatically if needed)-ar.quantize_and_save(output_dir="./my_model", format="auto_round")--# Load (rotation hooks restored automatically)-from transformers import AutoModelForCausalLM--model = AutoModelForCausalLM.from_pretrained("./my_model", device_map="auto")-```--- **Deterministic rotations**: Only metadata (type + rotation_size) is stored — matrices are reconstructed on load-- **Random rotations**: An `int8` (±1) rotation matrix is stored (size ~rotation_size² bytes)-- **Online rotations**: Rebuilt during model loading (R1/R4 via QuantLinear forward patching; R3 via monkeypatch from config)--#### Per-Linear Block Rotation (Experimental)--> ⚠️ This is an earlier experimental implementation that applies block-diagonal Hadamard rotation **per linear layer** by patching every `nn.Linear` module in the model. For most use cases, the [QuaRot / SpinQuant](#quarot--spinquant) approach above is preferred — it provides architecture-aware rotation at specific positions (R1–R4) with better accuracy and lower overhead.--The per-linear block rotation works by iterating over all linear layers and either:-- **Weight mode**: Fusing the Hadamard matrix directly into the weight tensor (offline)-- **Input mode**: Registering a forward pre-hook that rotates the input activation before each linear layer (online)--This approach uses a configurable `block_size` (default 32) applied uniformly to every linear layer. It is simpler than QuaRot but less targeted — it does not distinguish between residual stream, attention, and MLP layers, nor does it handle RoPE or activation-side rotation. Use `rotation_config="default"` to enable it.--##### Usage--```python-from auto_round import AutoRound--model_name_or_path = "meta-llama/Llama-3.1-8B-Instruct"-output_dir = "./Llama-3.1-8B-Instruct-mxfp4-ht"--# rotation_config="default": block_size=32, hadamard_type="hadamard"-ar = AutoRound(model_name_or_path, scheme="MXFP4", rotation_config="default")--ar.quantize_and_save(output_dir=output_dir, format="auto_round")-```--##### Types--| Type | Description |-|------|-------------|-| `hadamard` (default) | Deterministic Hadamard matrix (Sylvester construction, block_size must be power of 2) |-| `random_hadamard` | Randomly signed Hadamard from known matrix library; supports non-power-of-2 sizes |--##### Parameters+Quantized models with rotation are saved and loaded transparently — rotation matrices and hooks are restored automatically on load, so inference needs no extra steps.-| Parameter | Default | Description |-|-----------|---------|-------------|-| `block_size` | `32` | Size of the Hadamard block applied to each linear layer |-| `hadamard_type` | `"hadamard"` | `"hadamard"` or `"random_hadamard"` |-| `seed` | `None` | Random seed (for `random_hadamard` only) |+For rotation positions (R1–R4), full configuration options, deterministic vs random Hadamard, trainable SpinQuant, the per-linear block rotation variant, and save/load internals, see [Rotation Details](./rotation_details.md).## 4 Inferencediff --git a/docs/step_by_step_CN.md b/docs/step_by_step_CN.mdindex 018cbeef8..660077ef6 100644--- a/docs/step_by_step_CN.md+++ b/docs/step_by_step_CN.md@@ -36,8 +36,6 @@- [手动配置设备映射](#手动配置设备映射)+ [超参数调整](#超参数调整)+ [旋转(Rotation)(实验性)](#旋转rotation实验性)- - [QuaRot / SpinQuant](#quarot--spinquant)- - [逐线性层块旋转(实验性)](#逐线性层块旋转实验性)* [4 推理部署](#4-推理部署)+ [CPU](#cpu)+ [英特尔 GPU](#英特尔-gpu)@@ -866,154 +864,27 @@ auto-round --model_name Qwen/Qwen3-0.6B --scheme "W4A16" --quant_lm_head --form### 旋转(Rotation)(实验性)-> ⚠️ **实验性功能**:旋转变换仍处于实验阶段。推理依赖 forward hook 机制,目前仅支持 Hugging Face Transformers 后端,因此相比非旋转模型,推理速度可能较慢。+> ⚠️ **实验性功能**:旋转变换仍处于实验阶段。推理依赖 forward hook 机制,目前仅支持 Hugging Face Transformers 后端,因此相比非旋转模型,旋转后的模型推理速度可能较慢。-AutoRound 支持基于旋转的变换技术来提升量化精度。旋转在量化前对权重和激活中的离群点进行重分布,使分布更加均匀,从而对量化更友好。+旋转在量化前对权重和激活中的离群点进行重分布,使分布更加均匀、对量化更友好。它对 MXFP4、NVFP4、W4A4 等激进的低比特方案最为有效。-目前提供两种旋转方式:+AutoRound 通过 `rotation_config` 参数应用旋转。推荐在大多数场景中使用 `"quarot"` 预设——确定性 Hadamard 旋转(QuaRot / SpinQuant),无需训练、无需校准数据。-- **QuaRot / SpinQuant** — 在多个位置(R1–R4)进行全模型旋转,为激进量化(如 MXFP4、NVFP4、W4A4)提供全面的离群点抑制。**推荐在大多数场景中使用。**-- **逐线性层块旋转(Per-Linear Block Rotation)** — 对每个线性层均匀应用块对角旋转。为较早期的实验性实现,通常建议使用上述 QuaRot/SpinQuant 方式。--#### QuaRot / SpinQuant--QuaRot 在 Transformer 架构的最多 4 个位置应用确定性 Hadamard 旋转。与逐线性层块旋转不同,它在模型架构级别操作——在特定位置对残差流、注意力头和 MLP 激活进行旋转,以实现针对性的离群点抑制。--##### 旋转位置--| 位置 | 目标 | 模式 | 效果 |-|------|------|------|------|-| **R1** | 残差流(hidden_size) | 在线或离线 | 平滑所有线性层的权重离群点 |-| **R2** | V/O 投影(head_dim) | 离线(融合) | 平衡各注意力头的值分布 |-| **R3** | RoPE 后的 Q/K(head_dim) | 在线(hook) | 提升 KV-cache 的量化友好性 |-| **R4** | MLP up/down(intermediate_size) | 在线(hook) | 抑制 FFN 中的激活离群点 |--- **在线(Online)**:通过 forward hook 在运行时应用(不修改权重,保存时零开销)-- **离线/融合(Offline/Fused)**:吸收到相邻权重矩阵中(无运行时开销)--##### 快速开始+#### API 用法```pythonfrom auto_round import AutoRoundmodel_name = "Qwen/Qwen3-0.6B"-output_dir = "./Qwen3-0.6B-mxfp4-quarot"-# QuaRot 预设:R1+R2+R3+R4,确定性 Hadamard+# QuaRot 预设:确定性 Hadamard,无需训练ar = AutoRound(model_name, scheme="MXFP4", rotation_config="quarot")-ar.quantize_and_save(output_dir=output_dir, format="auto_round")-```--##### 旋转级别--根据精度需求选择旋转的位置数量:--```python-from auto_round import AutoRound-from auto_round.algorithms.transforms.spinquant import SpinQuantConfig--# 仅 R1(速度快,良好的基准提升)-ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True))--# R1 + R2(更好,融合后无运行时开销)-ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True, r2=True))--# R1 + R2 + R3 + R4(最佳精度,hook 带来少许运行时开销)-ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True, r2=True, r3=True, r4=True))-```--##### 字符串快捷方式--| 值 | 等价配置 |-|----|---------|-| `"quarot"` | `SpinQuantConfig(r1=True, r2=True, trainable_rotation=False, trainable_smooth=False)` — 确定性 Hadamard,无需训练 |-| `"spinquant"` | `SpinQuantConfig(r1=True, r2=True, trainable_rotation=True, trainable_smooth=True)` — **实验性**(需要提供 dataloader) |--> ⚠️ **SpinQuant 可训练旋转**(`trainable_rotation=True`)启用通过 Cayley SGD 优化的可学习旋转矩阵。此功能为**实验性**,尚未完全验证。生产环境建议使用 `"quarot"`(固定 Hadamard)。--##### 确定性与随机 Hadamard--```python-# 确定性(默认):固定 Hadamard 矩阵,无需额外存储-ar = AutoRound(model_name, scheme="MXFP4", rotation_config=SpinQuantConfig(r1=True, r2=True, r3=True, r4=True))--# 随机:H × diag(±1),离群点抑制效果略好,需保存旋转矩阵-ar = AutoRound(- model_name,- scheme="MXFP4",- rotation_config=SpinQuantConfig(- r1=True, r2=True, r3=True, r4=True, random_r1=True, random_r2=True, random_r3=True, random_r4=True- ),-)+ar.quantize_and_save(output_dir="./Qwen3-0.6B-mxfp4-quarot", format="auto_round")```-##### 关键参数--| 参数 | 默认值 | 描述 |-|------|--------|------|-| `r1` / `r2` / `r3` / `r4` | `True / True / False / False` | 启用各位置的旋转 |-| `online_r1_rotation` | `True` | R1 通过 hook 应用(`True`)或融合到权重中(`False`) |-| `random_r1` / `random_r2` / `random_r3` / `random_r4` | `False` | 使用随机 Hadamard(H×diag(±1))而非确定性 |-| `rotation_size` | `None`(自动) | 块旋转维度;从模型维度自动检测 |-| `trainable_rotation` | `False` | 启用 SpinQuant 可学习旋转(**实验性**,需要 dataloader) |-| `trainable_smooth` | `False` | 启用可学习 smooth 值(**实验性**,需要 dataloader) |--##### 保存与加载--带旋转的量化模型可透明地保存和加载:--```python-# 保存(旋转矩阵在需要时自动存储)-ar.quantize_and_save(output_dir="./my_model", format="auto_round")--# 加载(旋转 hook 自动恢复)-from transformers import AutoModelForCausalLM--model = AutoModelForCausalLM.from_pretrained("./my_model", device_map="auto")-```--- **确定性旋转**:仅存储元数据(类型 + rotation_size)——矩阵在加载时重建-- **随机旋转**:以 `int8`(±1)旋转矩阵形式存储(大小约为 rotation_size² 字节)-- **在线旋转**:在模型加载时重建(R1/R4 通过 QuantLinear forward patch;R3 通过配置触发的 monkeypatch)--#### 逐线性层块旋转(实验性)--> ⚠️ 这是较早期的实验性实现,通过对模型中每个 `nn.Linear` 模块应用块对角 Hadamard 旋转来工作。对于大多数场景,推荐使用上述 [QuaRot / SpinQuant](#quarot--spinquant) 方式——它在特定位置(R1–R4)提供架构感知的旋转,具有更好的精度和更低的开销。--逐线性层块旋转通过遍历所有线性层实现,可选择以下模式:-- **权重模式**:将 Hadamard 矩阵直接融合到权重张量中(离线)-- **输入模式**:注册 forward pre-hook 在每个线性层前旋转输入激活(在线)--该方式使用可配置的 `block_size`(默认 32),对每个线性层统一应用。它比 QuaRot 更简单,但不够精确——不区分残差流、注意力和 MLP 层,也不处理 RoPE 或激活侧旋转。使用 `rotation_config="default"` 启用。--##### 使用方式--```python-from auto_round import AutoRound--model_name_or_path = "meta-llama/Llama-3.1-8B-Instruct"-output_dir = "./Llama-3.1-8B-Instruct-mxfp4-ht"--# rotation_config="default": block_size=32, hadamard_type="hadamard"-ar = AutoRound(model_name_or_path, scheme="MXFP4", rotation_config="default")--ar.quantize_and_save(output_dir=output_dir, format="auto_round")-```--##### 类型--| 类型 | 描述 |-|------|------|-| `hadamard`(默认) | 确定性 Hadamard 矩阵(Sylvester 构造,block_size 必须为 2 的幂次) |-| `random_hadamard` | 随机签名 Hadamard,来自已知矩阵库;支持非 2 的幂次尺寸 |--##### 参数说明+带旋转的量化模型可透明地保存和加载——旋转矩阵和 hook 会在加载时自动恢复,推理无需额外步骤。-| 参数 | 默认值 | 描述 |-|------|--------|------|-| `block_size` | `32` | 应用于每个线性层的 Hadamard 块大小 |-| `hadamard_type` | `"hadamard"` | `"hadamard"` 或 `"random_hadamard"` |-| `seed` | `None` | 随机种子(仅用于 `random_hadamard`) |+关于旋转位置(R1–R4)、完整配置选项、确定性与随机 Hadamard、可训练的 SpinQuant、逐线性层块旋转变体以及保存/加载的内部细节,请参阅 [Rotation Details](./rotation_details.md)(英文)。## 4 推理部署