PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
update ARK README (#1922)
Luo Yuf42a04216eb0
diff --git a/auto_round_extension/ark/README.md b/auto_round_extension/ark/README.mdindex a9d8eb63a..6e7a7254c 100644--- a/auto_round_extension/ark/README.md+++ b/auto_round_extension/ark/README.md@@ -1,246 +1,187 @@-## What is AutoRound Kernel?+## What is AutoRound Kernel (ARK)?-AutoRound Kernel (ARK) is a low-bit acceleration library for Intel platform, providing high-performance kernels for LLM inference including weight-only quantization, flash attention (with [SageAttention](https://github.com/thu-ml/SageAttention) v1 support), and Mixture-of-Experts (MOE) computation.+AutoRound Kernel (ARK) is a low-bit acceleration library for Intel platform, providing three categories of optimized operators for LLM inference.-The kernels are optimized for the following CPUs:-* Intel Xeon Scalable processor (formerly Sapphire Rapids, and Emerald Rapids)-* Intel Xeon 6 processors (formerly Sierra Forest and Granite Rapids)+| Operator Category | CPU | XPU (Battlemage) |+|:------------------|:---:|:----------------:|+| **Weight-Only Quantized Linear** (INT4/INT8/FP8/FP4) | ✅ | ✅ |+| **MoE Grouped GEMM** | ❌ | ✅ |+| **SageAttention** (SDPA / SAGE) | ❌ | ✅ |-The kernels are optimized for the following GPUs:-* Intel Arc B-Series Graphics and Intel Arc Pro B-Series Graphics- (formerly Battlemage)+**Validated CPU:** Intel Xeon Scalable (Sapphire Rapids / Emerald Rapids), Intel Xeon 6 (Sierra Forest / Granite Rapids)<br>+**Validated GPU:** Intel Arc B-Series / Arc Pro B-Series (Battlemage)-## Key Features+### Highlights — Ecosystem Integration-AutoRound Kernel provides the following computational capabilities for LLM inference:+ARK kernels are integrated into the following projects:-### Weight-Only Quantization (WOQ) Linear-| Weight dtype | Compute dtype | Scale dtype | Algorithm<sup>[1]</sup> |-|------------------|:--------------------------------:|:-----------------:|:-----------------------:|-| INT8 | INT8<sup>[2]</sup> / BF16 / FP32 | BF16 / FP32 | sym / asym |-| INT4 | INT8 / BF16 / FP32 | BF16 / FP32 | sym / asym |-| INT3 | INT8 / BF16 / FP32 | BF16 / FP32 | sym / asym |-| INT2 | INT8 / BF16 / FP32 | BF16 / FP32 | sym / asym |-| INT5 | INT8 / BF16 / FP32 | BF16 / FP32 | sym / asym |-| INT6 | INT8 / BF16 / FP32 | BF16 / FP32 | sym / asym |-| INT7 | INT8 / BF16 / FP32 | BF16 / FP32 | sym / asym |-| INT1 | INT8 / BF16 / FP32 | BF16 / FP32 | sym / asym |-| FP8 (E4M3, E5M2) | BF16 / FP32 | FP32 / FP8 (E8M0) | NA |-| FP4 (E2M1) | BF16 / FP32 | BF16 / FP32 | NA |+| Project | Integration | Description |+|---------|-------------|-------------|+| [vllm](https://github.com/vllm-project/vllm) | [`inc_wna16_linear.py`](https://github.com/vllm-project/vllm/blob/main/vllm/model_executor/layers/quantization/inc/schemes/inc_wna16_linear.py#L379-L470) | `INCXPUARKLinearMethod` — weight-only quantized linear on XPU via `auto_round_kernel.qlinear.QuantLinear`. The goal is to support all AutoRound-quantized models. The plan is tracked in [Intel Quantization Support Roadmap](https://github.com/vllm-project/vllm/issues/37979) |+| [vllm-omni](https://github.com/vllm-project/vllm-omni) | [`sage_attn.py`](https://github.com/vllm-project/vllm-omni/blob/f8340d078e4e9c3b793bd92d55d891b29703f0a8/vllm_omni/diffusion/attention/backends/sage_attn.py#L27) | `SageAttentionBackend` — diffusion model attention on XPU via `ARK.sagev1` |+| [Transformers](https://github.com/huggingface/transformers) (via [auto-round](https://github.com/intel/auto-round)) | [`backend.py`](https://github.com/intel/auto-round/blob/main/auto_round/inference/backend.py#L531) | All models quantized by AutoRound automatically use ARK on CPU/XPU by default; no additional configuration required. 6 backends registered: `auto_round_kernel[_xpu]` (GPTQ no-zp), `auto_round_kernel_zp[_xpu]` (GPTQ +zp), `auto_round_kernel_awq[_xpu]` (AWQ).<br>**CPU:** INT2/INT4/INT8; **XPU:** INT4/INT8 |-### XPU Weight-Only Quantization-| Weight dtype | Compute dtype | Scale dtype | Algorithm |-|------------------|:--------------:|:-----------------:|:----------:|-| INT8 | INT8 / FP16 | FP16 | sym |-| INT4 | INT8 / FP16 | FP16 | sym |-| INT2 | INT8 / FP16 | FP16 | sym |-| FP8 (E4M3, E5M2) | FP16 | FP16 / FP8 (E8M0) | NA |+----<sup>[1]</sup>: Quantization algorithms for integer types: symmetric or asymmetric.-<sup>[2]</sup>: Includes dynamic activation quantization; results are dequantized to floating-point formats.+## 1. Linear (Weight-Only Quantized GEMM)-### Flash Attention (XPU only)+Low-bit weight-only linear for LLM inference. Both CPU and XPU are supported.-ARK provides multiple attention backends for prefill and decode, with [SageAttention](https://github.com/thu-ml/SageAttention) v1 support — offering INT8-quantized attention variants for higher throughput on Intel Arc GPUs. (SageAttention v2/v3 are under development.)+### API-| Backend | Description | Q/K/V dtype | Head dim | Features |-|---------|-------------|-------------|----------|----------|-| `sdpa` | Standard flash attention (FP16/BF16) | FP16 / BF16 | 64, 96, 128, 192 | Causal mask, additive mask, GQA |-| `sagev1` | **SageAttention v1** with INT8 Q/K quantization | FP16 / BF16 | 64, 128 | Block-wise INT8 QK, PV in half |-| `sagev1_pvi8` | **SageAttention v1** with INT8 Q/K/V quantization | FP16 / BF16 | 64, 128 | Block-wise INT8 QK + INT8 PV |-| `sage` | Low-level SageAttention with pre-quantized INT8 Q/K | INT8 (Q/K), FP16 (V) | 64, 128 | External Q/K scales |-| `sage_pvi8` | Low-level SageAttention with pre-quantized INT8 Q/K/V | INT8 (Q/K/V) | 64, 128 | External Q/K/V scales |-| `sage_dynquant` | SageAttention with fused dynamic INT8 quantization | FP16 | 64, 128 | Auto-quantizes Q/K internally |+| API | Description | Platform |+|-----|-------------|----------|+| `QuantLinear` ([example ↓](#example)) | Unified PyTorch module (GPTQ/AWQ/raw quantized checkpoint) | CPU / XPU |+| `QuantLinearGPTQ` | GPTQ-format checkpoint loader | CPU / XPU |+| `QuantLinearAWQ` | AWQ-format checkpoint loader | CPU / XPU |+| `QuantLinearFP8` | FP8 weight-only linear | CPU / XPU |+| `woqgemm` | Low-level weight-only GEMM (packed format) | CPU / XPU |+| `woqgemm_s8` | Low-level INT8-weight GEMM with scale | CPU / XPU |+| `_repack_quantized_weight` | Repack raw qweight/qzero/scale → ARK format | CPU / XPU |+| `_unpack_weight` | Unpack ARK-format weight back to full precision | CPU / XPU |-All attention backends support both `HND` (`[B, H, N, D]`) and `NHD` (`[B, N, H, D]`) tensor layouts, as well as non-contiguous (sliced) input tensors.+### Key Features-### MOE GEMM (XPU only)+> **W4A8 / W2A8 Rescale (QQQ-style)** — On XPU, ARK supports \~[QQQ](https://github.com/HandH1998/QQQ)-style compute: low-bit weights (INT2/INT4) are re-scaled to INT8 and computed via INT8 GEMM, avoiding FP16 dequantization for better throughput. Enabled automatically via environment variable `ARK_AUTO_S8`; see [xpu_wrapper.hpp](auto_round_kernel/wrapper/include/xpu_wrapper.hpp).-Grouped GEMM for Mixture-of-Experts layers, supporting FP16/BF16 with variable token counts per expert.+### Supported Data Types-## Installation--### 1. Install via pip-```bash-pip install auto-round-lib-```--### 2. Install from Source--Requires a sourced oneAPI environment (2025.3+ recommended for SYCL TLA support).--```bash-# Source oneAPI environment-source /opt/intel/oneapi/setvars.sh--# Build and install-pip install . --no-build-isolation-# or-python setup.py bdist_wheel; pip install dist/*-```+#### CPU-The build system automatically detects the oneAPI version. SYCL TLA (Tensor Linear Algebra) kernels are enabled when oneAPI >= 2025.3.+| Weight dtype | Compute dtype | Scale dtype | Algorithm |+|--------------|:-------------:|:-----------:|:---------:|+| INT1–INT8 | INT8<sup>[1]</sup> / BF16 / FP32 | BF16 / FP32 | sym / asym |+| FP8 (E4M3, E5M2) | BF16 / FP32 | FP32 / FP8 (E8M0) | NA |+| FP4 (E2M1) | BF16 / FP32 | BF16 / FP32 | NA |-### Validated Hardware Environment+#### XPU-#### CPU based on [Intel 64 architecture or compatible processors](https://en.wikipedia.org/wiki/X86-64):-* Intel Xeon Scalable processor (Granite Rapids)+| Weight dtype | Compute dtype | Scale dtype | Algorithm |+|--------------|:-------------:|:-----------:|:---------:|+| INT4, INT8 | INT8 / FP16 | FP16 | sym |+| FP8 (E4M3, E5M2) | FP16 | FP16 / FP8 (E8M0) | NA |-#### GPU built on Intel's Xe architecture:-* Intel Arc B-Series Graphics (Battlemage)+<sup>[1]</sup> INT8 compute includes dynamic activation quantization; results are dequantized to floating-point.-## API Reference--### QuantLinear (Weight-Only Quantization)--ARK exposes a unified weight-only linear interface through `QuantLinear`, `QuantLinearGPTQ`, `QuantLinearAWQ`, and `QuantLinearFP8`. Please refer to the [QLinear](auto_round_kernel/qlinear.py) for more integration details.--The expected lifecycle is: create the module, load quantized tensors from the checkpoint, call `post_init()` once to repack weights into the ARK-friendly layout, and then call `forward()` during inference.--Minimal usage:-```python-from auto_round_kernel.qlinear import QuantLinear--qlinear = QuantLinear(- bits=4,- group_size=128,- sym=True,- in_features=in_features,- out_features=out_features,- bias=bias is not None,- weight_dtype=weight_dtype,-)-# Load qweight, qzeros, scales, and bias from checkpoint.-qlinear.post_init()--# Run inference-y = qlinear(x)-```--### Attention APIs--#### `ark.sdpa` — Standard Flash Attention+### Example```pythonimport auto_round_kernel as ark-output = ark.sdpa(- query,- key,- value,- attn_mask=None,- dropout_p=0.0,- is_causal=False,- scale=None,- tensor_layout="HND", # "HND" or "NHD"+# Prepare quantized weight: qweight [K, N] int4/int2, scale [K/G, N] fp16/fp32, zp [K/G, N] int4/int2+packw = ark.repack_quantized_weight(+ qweight,+ scale,+ zp,+ blocksize=128,+ compute_type="fp16",+ weight_type="int4",+ scale_type="fp16",+ asym=False,)-```-- Q/K/V dtype: FP16 or BF16-- Supported head dims: 64, 96, 128, 192-- Supports GQA (grouped query attention) via different Hq/Hkv--#### `ark.sagev1` — SAGE v1 Attention+# Run weight-only quantized GEMM: activation [M, K] → output [M, N]+output = ark.woqgemm(+ activation, # [M, K] fp16/bf16+ packw, # packed weight blob (INT8)+ bias, # [1, N] optional bias+ n, # output features+ k, # input features+ groupsize=128,+ compute_type="fp16",+ weight_type="int4",+ scale_type="fp16",+ asym=False,+)-```python-output = ark.sagev1(- query,- key,- value,- attn_mask=None,- dropout_p=0.0,- is_causal=False,- scale=None,- enable_gqa=False,- quant_block_size=64, # block size for INT8 QK quantization- tensor_layout="HND",+# Decompose back to full precision for verification+decompressed = ark.unpack_weight(+ packw,+ dtype=torch.float16,+ n=n,+ k=k,+ groupsize=128,+ compute_type="fp16",+ weight_type="int4",+ scale_type="fp16",+ asym=False,)```-- Q/K/V dtype: FP16 or BF16-- Supported head dims: 64, 128-- Internally quantizes Q/K to INT8 per block; PV computed in half precision-- Falls back to `ark.sdpa` when `quant_block_size <= 0`+See [test_weightonly.py](test/test_weightonly.py) for an end-to-end example of weight repack, verification, and woqgemm execution on CPU and XPU.-#### `ark.sagev1_pvi8` — SAGE v1 with INT8 PV+----Same interface as `sagev1` but also quantizes V to INT8 internally for higher throughput.+## 2. MoE (Mixture-of-Experts Grouped GEMM)-#### `ark.sage_dynquant` — SAGE with Fused Dynamic Quantization+Grouped GEMM for MoE layers where different experts process varying numbers of tokens.-```python-output = ark.sage_dynquant(- query,- key,- value,- attn_mask=None,- dropout_p=0.0,- is_causal=False,- scale=None,- enable_gqa=False,- quant_block_size=64,-)-```+### API++| Function | Description | Platform | Activation Dtype | Weight Dtype |+|----------|-------------|----------|:----------------:|:------------:|+| `ark.moe_gemm(...)` ([example ↓](#example-1)) | Grouped GEMM across experts | XPU | FP16 / BF16 | FP16 / BF16 |+| `ark.moe_gemm(...)` (WIP) | Grouped GEMM with INT4 weight | XPU | FP16 / BF16 | INT4 🚧 |+| `ark.moe_gemm(...)` (WIP) | Grouped GEMM with INT2 weight | XPU | FP16 / BF16 | INT2 🚧 |+| `ark.moe_gemm(...)` (WIP) | Grouped GEMM with INT8 weight | XPU | FP16 / BF16 | INT8 🚧 |-- Takes FP16 Q/K/V, performs fused block-wise INT8 quantization of Q/K via SYCL kernel-- Supports quant_block_size: 1 (per-token), 32, 64, 128, 256-- Auto-pads sequence lengths for block alignment+> 🚧 INT2 / INT4 / INT8 weight support is under active development. See [#PR](https://github.com/intel/auto-round/pull).-#### `ark.sageattn` — SageAttention-Compatible Dispatcher+### Details-ARK provides a drop-in replacement for the `sageattention.sageattn` API, enabling seamless integration with existing SageAttention workflows:+| Parameter | Shape | Dtype |+|-----------|-------|-------|+| activations | `[total_tokens, K]` | FP16 / BF16 |+| weights | `[num_experts, K, N]` (row-major) | FP16 / BF16 |+| num_tokens_per_expert | `[num_experts]` | INT32 |+| scales (optional) | `[num_experts, N]` | FP16 / BF16 |+| **output** | `[total_tokens, N]` | same as activations |++### Example```python-import auto_round_kernel as ark+# FP16/BF16 MoE+output = ark.moe_gemm(activations, weights, num_tokens_per_expert)-# Drop-in replacement for sageattention.sageattn-output = ark.sageattn(- q,- k,- v,- tensor_layout="HND",- is_causal=False,- sm_scale=None,- return_lse=False,- kernel="v1_pvhalf", # or "v1_pvi8"- **kwargs,-)+# INT4 MoE (coming soon)+# output = ark.moe_gemm(activations, q4_weights, num_tokens_per_expert, scales=scales)```-This mirrors the [SageAttention](https://github.com/thu-ml/SageAttention) interface for drop-in compatibility, allowing models using SageAttention to run on Intel Arc GPUs without code changes.+Build requirement: `ARK_SYCL_TLA=ON`. See [test_moe.py](test/test_moe.py).-### MOE GEMM+----```python-output = ark.moe_gemm(- activations, # [total_tokens, K] FP16/BF16- weights, # [num_experts, K, N] FP16/BF16- num_tokens_per_expert, # [num_experts] int32- scales=None, # optional [num_experts, N] FP16/BF16-)-```+## 3. SageAttention (XPU SDPA Acceleration)-### Patching torch SDPA+ARK provides a full family of scaled dot-product attention kernels on XPU, ranging from vanilla FP16 SDPA to INT8-quantized SageAttention variants.-ARK can globally replace `torch.nn.functional.scaled_dot_product_attention` for evaluation, including SageAttention backends:+### API Overview-```python-import auto_round_kernel as ark--# Patch with standard SDPA backend-ark.patch_torch_sdpa(backend="sdpa")+| Function | Description | Q/K/V Input | PV Precision | Head Dim |+|----------|-------------|-------------|:------------:|:--------:|+| `ark.sdpa` ([example ↓](#drop-in-sdpa-replacement)) | FP16/BF16 SDPA (flash attention) | FP16 / BF16 | FP16 | 64, 96, 128, 192 |+| `ark.sage` | Low-level INT8 SAGE (pre-quantized Q/K) | INT8 (Q/K), FP16 (V) | FP16 | 64, 128 |+| `ark.sage_pvi8` | Low-level INT8 SAGE (pre-quantized Q/K/V) | INT8 | INT8 | 64, 128 |+| `ark.sagev1` | High-level FP16 → internal Q/K quant → SAGE | FP16 / BF16 | FP16 | 64, 128 |+| `ark.sagev1_pvi8` | High-level FP16 → internal Q/K/V quant → SAGE PV INT8 | FP16 / BF16 | INT8 | 64, 128 |+| `ark.sageattn` | Dispatcher (sageattention-compatible API) | FP16 / BF16 | FP16 / INT8 | 64, 128 |+| `ark.sage_dynquant` | Dynamic INT8 block-wise Q/K quant → SAGE (drop-in SDPA replacement) | FP16 / BF16 | FP16 | 64, 128 |-# Patch with SageAttention v1 backend (INT8 QK, PV half)-ark.patch_torch_sdpa(backend="sagev1", quant_block_size=64)+### Comparison-# Patch with SageAttention v1 + INT8 PV backend-ark.patch_torch_sdpa(backend="sagev1_pvi8", quant_block_size=64)+| Feature | `sdpa` | `sagev1` | `sagev1_pvi8` | `sage_dynquant` |+|---------|:------:|:--------:|:-------------:|:---------------:|+| Q/K quantization | None | Internal INT8 | Internal INT8 | Internal INT8 |+| PV quantization | None | None | Internal INT8 | None |+| quant_block_size | N/A | 1 / ≥32 | 1 / ≥32 | 1 / ≥32 |+| Additive mask | ✅ [B,1,Sq,Skv] FP32 | ✅ | ✅ | ✅ |+| Causal mask | ✅ | ✅ | ✅ | ✅ |+| GQA | ✅ | ✅ | ✅ | ✅ |+| Tensor layout | HND / NHD | HND / NHD | HND / NHD | HND |-# Restore original-ark.unpatch_torch_sdpa()-```+### Drop-in SDPA Replacement-Or use the helper launcher for lm-eval:+Replace `torch.nn.functional.scaled_dot_product_attention` globally for lm-eval:```bashcd /path/to/auto_round_extension/ark@@ -248,46 +189,52 @@ PYTHONPATH=$PWD python tools/lm_eval_with_ark_sdpa.py \--model hf \--model_args pretrained=/path/to/model,trust_remote_code=True,dtype=bfloat16 \--tasks hellaswag,piqa,winogrande \- --device xpu:0 \- --batch_size 1+ --device xpu:0 --batch_size 1```-### Low-Level Matrix Operations+The patching logic (in [`auto_round_kernel/torch_sdpa_patch.py`](auto_round_kernel/torch_sdpa_patch.py)) routes to ARK on XPU when constraints are met; otherwise falls back to PyTorch SDPA.-```python-# FP16/BF16 matrix multiply with bias-C = ark.matmul(A, B, bias)+### Constraints++| Constraint | `sdpa` | `sagev1` / `sagev1_pvi8` / `sage_dynquant` |+|------------|:------:|:------------------------------------------:|+| Q/K/V dtype | FP16, BF16 | FP16, BF16 |+| Head dim | 64, 96, 128, 192 | 64, 128 |+| `dropout_p` | must be 0.0 | must be 0.0 |+| Boolean mask | falls back to torch | falls back to torch |+| Additive mask shape | `[B, 1, Sq, Skv]` FP32 | `[B, 1, Sq, Skv]` FP32 |+| quant_block_size | N/A | 1 (per-token) or ≥32 |-# INT8 matrix multiply (s8s8s32)-C = ark.igemm_s8s8s32(A, B)+---++## Installation-# Weight-only quantized GEMM with INT8 weights-C = ark.woqgemm_s8(A, B, scaleB, bias)+### Install via pip+```bash+pip install auto-round-lib+```-# General weight-only quantized GEMM-C = ark.woqgemm(A, B, bias, n, k, groupsize, compute_type, weight_type, scale_type, asym)+### Install from Source+```bash+pip install . --no-build-isolation+# or+python setup.py bdist_wheel; pip install dist/*```-## Testing--Unit tests are available in the [test](test/) directory:--| Test file | Description |-|-----------|-------------|-| `test_weightonly.py` | Weight-only quantized GEMM (CPU + XPU) |-| `test_flash_attn.py` | Flash attention (sdpa) correctness |-| `test_sdpa.py` | SDPA benchmark suite |-| `test_sdpa_parity.py` | SDPA parity with non-contiguous inputs and layouts |-| `test_sage_dynquant.py` | SageDynQuant block-wise benchmark |-| `test_bench_bmg.py` | BMG comparison benchmark |-| `test_matmul.py` | General matrix multiply |-| `test_packq.py` | Weight packing/unpacking |-| `test_moe.py` | MOE GEMM correctness |--## Notes--- The SDPA patch only routes calls to ARK on XPU when inputs match kernel constraints; otherwise it falls back to the original torch SDPA.-- Supported Q/K/V dtypes for attention are FP16 and BF16 (except SAGE variants which may use INT8 internally).-- `dropout_p` must be 0.0 for all ARK attention paths.-- Additive masks are supported when they can be normalized to `[B, 1, Sq, Skv]`; boolean masks fall back to torch.-- Non-contiguous (sliced) input tensors are supported for all attention backends.+Build with MoE / SageAttention support requires `ARK_SYCL_TLA=ON`.++---++## Tests++| Test | Description |+|------|-------------|+| [test_weightonly.py](test/test_weightonly.py) | WOQ GEMM pack/unpack/run on CPU & XPU |+| [test_moe.py](test/test_moe.py) | MoE grouped GEMM |+| [test_flash_attn.py](test/test_flash_attn.py) | SDPA (flash attention) prefill |+| [test_sdpa.py](test/test_sdpa.py) | SDPA benchmark suite |+| [test_sdpa_parity.py](test/test_sdpa_parity.py) | SDPA vs PyTorch parity check |+| [test_sage_dynquant.py](test/test_sage_dynquant.py) | SageAttention dynamic INT8 quant benchmarks |+| [test_bench_bmg.py](test/test_bench_bmg.py) | BMG SDPA / SageAttention benchmarking |+| [test_matmul.py](test/test_matmul.py) | Low-level matmul |+| [test_packq.py](test/test_packq.py) | Weight packing utilities |