PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
Step-by-Step
English | 简体中文
This document presents step-by-step instructions for auto-round llm quantization. You can refer to vlms user guide for vlms quantization and diffusions user guide for diffusions quantization.
- 1 Prerequisite
- 2 Prepare Calibration Dataset
- 3 Quantization
- Supported Quantization Schemes
- Supported Export Formats
- Hardware Compatibility
- Environment Configuration
- Command Line Usage
- API usage
- AutoScheme
- OPT RTN mode
- AWQ Algorithm-Experimental
- SVDQuant Algorithm-Experimental
- Model-Free Mode
- GGUF format
- Quantization Costs
- Device/Multi-GPU setting in Quantization
- Adjust Hyperparameters
- Rotation (Research)
- 4 Inference
- 5 Evaluation
- 6 Known Issues
1 Prerequisite
Install auto-round or install from source
pip install auto-round
2 Prepare Calibration Dataset
Default Dataset
The NeelNanda/pile-10k in huggingface is adopted as the default calibration data and will be downloaded automatically from the datasets Hub. Other available datasets include:
fineweb-edufrom Hugging Face. If Hugging Face is inaccessible, installmodelscope, setAR_USE_MODELSCOPE=1, and use the same alias to load the ModelScope mirrorBAAI/CCI3-HQfor Chinesecodeparrot/github-code-cleanfor codeHuggingFaceH4/ultrachat_200kfor chat datamadao33/new-title-chinesefor Chinesembppfor codenvidia/OpenCodeInstructfor code instruction dataopenbmb/Ultra-FineWebopenbmb/UltraData-SFT-Agent-2609(experimental) for agent/tool-calling data. Each sample carries amessagesconversation (withtool_calls/tool turns) and atoolsschema that is rendered through the chat template (apply_chat_templateis always enabled), so the quantized model sees full tool-calling traces. All configs (Code-Agent,Search-Agent,General-Agent,Tool-Use) are loaded and interleavedqwedsacf/competition_math(experimental) for math (theproblemandsolutionfields are concatenated, and short samples are packed into fullseqlensequences by default)
Customized Dataset
Option 1: Pass a local json file path to dataset argument
Option 2: Register your dataset following the code and pass the new dataset and split args to initialize AutoRound object, e.g. autoround=Autoround(dataset="NeelNanda/pile-10k:train", ...)
Option 3: pass list of string or list of input_ids to dataset.
def customized_data(): # Important Notice!!! AutoRound will drop data < args.seqlen and truncate data to args.seqlen data = ["AutoRound is an advanced quantization algorithm for low-bits LLM inference" * 240] return data def customized_data_with_tokenizer(tokenizer, seqlen=2048): # Import notice!!! AutoRound will drop data < args.seqlen data = ["AutoRound is an advanced quantization algorithm for low-bits LLM inference" * 240] tokens = [] for d in data: token = tokenizer(d, truncation=True, max_length=seqlen, return_tensors="pt").data tokens.append(token) return tokens
Dataset operations
Dataset combination:We support combination of different datasets and parametrization of calibration datasets by
using --dataset ./tmp.json,NeelNanda/pile-10k:num=256,mbpp:num=128. Both local calibration file
and huggingface dataset are supported. You could specify splits of a dataset by setting split=split1+split2.
Samples concatenation: An optional setting allows users to concatenate calibration samples
using --dataset NeelNanda/pile-10k:concat=True.
All samples will be concatenated first, then split into chunks of seqlen length.
For the generic loader (any HuggingFace dataset, or one specified with fields/template),
concat=True performs sample-level packing instead: whole samples are accumulated until the buffer
reaches seqlen, a full-length sequence is emitted, and the buffer is reset so the next sample starts
a fresh concatenation. This prevents datasets whose individual samples are shorter than seqlen
(e.g. qwedsacf/competition_math) from being dropped by the length filter.
Apply chat template: Using --dataset NeelNanda/pile-10k:apply_chat_template enables application of a chat template
to the calibration data before tokenization. This is commonly used for instruct-style models during generation. To
customize the system prompt,
use:--dataset 'NeelNanda/pile-10k:apply_chat_template:system_prompt="You are a helpful assistant."'
Generic dataset loading (experimental): Any HuggingFace dataset can be used directly without prior registration. The text field
is auto-detected by inspecting column names and sampling values. For example:
--dataset my-org/my-dataset:split=train:num=512
Note: Generic dataset loading (including
fields,template,separator,timeout, auto field detection, and theCalibDatasetAPI below) is experimental and its behavior/interface may change in future releases.
Field selection: Use fields to explicitly specify the text column(s). A single field:
--dataset my-org/my-dataset:fields=text. Multiple fields are concatenated with a separator:
--dataset my-org/my-dataset:fields=question+answer
Template: Use template with {field} placeholders to build custom text from multiple columns.
Takes precedence over fields when both are set.
--dataset 'my-org/my-dataset:template={question} {answer}'
Separator: When concatenating multiple fields, customize the separator (default "\n\n"):
--dataset 'my-org/my-dataset:fields=q+a:separator=\n'
Timeout: For streaming datasets, set timeout (seconds) to bound data collection. If the dataset
cannot provide enough samples within the timeout, a warning is logged and the collected samples are used.
--dataset my-org/my-dataset:timeout=600
CalibDataset (API): For programmatic usage, the CalibDataset dataclass provides a typed interface:
~~~python
from auto_round.utils.dataset_utils import CalibDataset
from auto_round import AutoRound
# Single field
spec = CalibDataset("my-org/my-dataset", split="train", num=1000, fields="text")
# Multiple fields concatenated
spec = CalibDataset("my-org/my-dataset", fields=["question", "answer"], separator="\n")
# Template
spec = CalibDataset("my-org/my-dataset", template="{question} {answer}")
# Pass directly to AutoRound
ar = AutoRound(model="Qwen/Qwen3-8B", dataset=spec, scheme="W4A16")
ar.quantize_and_save()
~~~
Note: If the concatenation option is not enabled, samples shorter than args.seqlen will be dropped.
Please use ',' to split datasets, ':' to split parameters of a dataset and '+' to add values for one targeted parameter.
3 Quantization
Supported Quantization Schemes
AutoRound supports several Schemes:
- W4A16(bits:4,group_size:128,sym:True,act_bits:16)
- W8A16(bits:8,group_size:128,sym:True,act_bits:16)
- W7A16(bits:7,group_size:128,sym:True,act_bits:16)
- W6A16(bits:6,group_size:128,sym:True,act_bits:16)
- W5A16(bits:5,group_size:128,sym:True,act_bits:16)
- W3A16(bits:3,group_size:128,sym:True,act_bits:16)
- W2A16(bits:2,group_size:128,sym:True,act_bits:16)
- Group-size variants
W{2..8}A16G64/W{2..8}A16G32-llm_compressor(W2-W8);auto_round/auto_gptq(W2 only) - Asymmetric quantization (
--asym) supports weight bits <= 7 forauto_round/auto_gptq/auto_awqexports: vLLM serves W8 GPTQ-format weights symmetric-only and Marlin supports zero points at 4 bits only. Thellm_compressorformat supports 8-bit asym - vLLM serves it via Machete (SM90+, group sizes 64/128/-1) or Conch (SM80+, group size 128 or channelwise). The rule applies uniformly to AutoScheme options and fixed layer pins (e.g. lm_head): a run exporting tollm_compressorkeeps its 8-bit entries asymmetric, other formats pin them back to symmetric. Setting the environment variableAR_ALLOW_W8_ASYM=1lifts the restriction for every format (for serving stacks beyond stock vLLM); artifacts produced with it may not load in vLLM. - GGUF:Q4_K_M(all Q*_K,Q*_0,Q*_1 provided by llamacpp are supported)
- Mixed Bits Weight only
- NVFP4(Experimental feature, recommend exporting to
llm_compressorformat.data_type nvfp4,act_data_type nvfp4,static_global_scale,group_size 16) - MXFP4(Research feature, no real kernel, Standard MXFP4, data_type mxfp,act_data_type mxfp,bits 4, act_bits 4, group_size 32)
- MXINT4(Research feature, no real kernel, Standard MXINT4, data_type mxint,act_data_type mxint,bits 4, act_bits 4, group_size 32)
- MXFP4_RCEIL(Research feature,no real kernel, NVIDIA's variant, data_type mxfp,act_data_type mxfp_rceil,bits 4, act_bits 4, group_size 32)
- MXFP8(Research feature, no real kernel, data_type mxfp,act_data_type mxfp_rceil,group_size 32)
- FPW8A16(Research feature, no real kernel, data_type fp8,group_size 0->per tensor )
- FP8_STATIC(Research feature, no real kernel, data_type:fp8,act_data_type:fp8,group_size -1 ->per channel, act_group_size=0->per tensor)
Besides, you could modify the group_size, bits, sym and many other configs you want, though there are maybe no real kernels.
Supported Export Formats
You can use command auto_round list format to show all supported formats with support scheme.
AutoRound Format: This format is well-suited for CPU, Intel GPU, CUDA and HPU devices, 2 bits, as well as mixed-precision
inference. [2,3,4,5,6,7,8] bits are supported. Please set --format auto_round
About 5/6/7 bits: these widths do not divide 32 evenly, so each value is stored in a contiguous little-endian bit-stream laid over blocks of 32 values (
in_features * bits / 32int32 words). This is a strict generalization of the existing layouts — 2/4/8-bit and 3-bit checkpoints keep byte-identicalqweight/qzeros. 5/6/7 bits work withauto_round,auto_round:auto_gptqandauto_round:auto_awq, and can be freely combined in mixed-bit recipes. Upstream AutoGPTQ/AutoAWQ kernels cannot read them, so the plainauto_gptq/auto_awqformats still reject 5/6/7 bits. Bothin_featuresandout_featuresmust be multiples of 32. For inference, install the humming kernels (pip install git+https://github.com/inclusionAI/humming.git); otherwise AutoRound falls back to the slowtorchdequantize-and-matmul backend.
GGUF Format: Experimental feature. This format is well-suited for CPU devices and is widely adopted by the
community. q*_k,q*_0,q*_1 are supported. Please set --format gguf:q4_k_m, --format gguf:q2_k_s, etc
AutoGPTQ Format: This format is well-suited for symmetric quantization on CUDA devices and is widely adopted by the
community, [2,3,4,8] bits are supported. However, the
asymmetric kernel has issues that can cause considerable accuracy drops, particularly at 2-bit quantization and small
models. Besides, recently 3 bits may have some accuracy issues in Transformers. Please set --format auto_gptq
AutoAWQ Format: This format is well-suited for asymmetric 4-bit quantization on CUDA devices and is widely
adopted within the community, only 4-bits quantization is supported. Please set --format auto_awq
(the AutoRound-flavoured auto_round:auto_awq additionally supports 5/6/7 bits).
LLM-Compressor Format: NVFP4, MXFP4(kernel in WIP), MXFP8 are supported. Please set --format llm_compressor
MLX Format[Experimental Feature]: This format targets Apple Silicon (M1/M2/M3/...) and is loaded directly by mlx-lm (text-only LLM) or mlx-vlm (vision/audio + language).
- Supports 2, 3, 4, 5, 6, 8 bits (5/6 bits are MLX-exclusive — GPTQ/AWQ have no standard packing for them).
- Native mixed-bit / mixed-group_size via
layer_configor AutoScheme (--schemes "..." --bits 3.5); - Use
--format mlxfor a native MLX checkpoint; use--format auto_round:mlxif you want HuggingFacetransformers+ AutoRound to load it (post-init repacks each layer into MLXQuantLinearon Darwin). - Limitation: embedding layer quantization has not supported
Format and scheme support matrix
Gray indicates the absence of a kernel or the presence of only an inefficient/reference kernel. BF16 is mainly for AutoScheme
| Format | Supported Schemes |
|---|---|
| auto_round | W4A16, W2A16, W3A16, W5A16, W6A16, W7A16, W8A16, W2A16G64, W2A16G32, MXFP4, MXFP8, MXFP4_RCEIL, MXFP8_RCEIL, NVFP4, FPW8A16, FP8_STATIC, FP8_BLOCK, BF16, MXINT4 |
| llm_compressor | NVFP4, MXFP4, MXFP8, FPW8A16, FP8_STATIC, FP8_BLOCK, W2A16, W3A16, W4A16, W5A16, W6A16, W7A16, W8A16 and G64/G32 variants of W2-W8 |
| gguf | GGUF:Q4_K_M, GGUF:Q2_K_S, GGUF:Q3_K_S, GGUF:Q3_K_M, GGUF:Q3_K_L, GGUF:Q4_K_S, GGUF:Q5_K_S, GGUF:Q5_K_M, GGUF:Q6_K, GGUF:Q4_0, GGUF:Q4_1, GGUF:Q5_0, GGUF:Q5_1,GGUF:Q8_0 |
| mlx / auto_round:mlx | W2A16, W3A16, W4A16, W5A16, W6A16, W8A16, BF16, mixed-bit / mixed-group_size (Apple Silicon only) |
| auto_round:auto_awq | W4A16, W5A16, W6A16, W7A16, BF16 |
| auto_round:auto_gptq | W4A16, W2A16, W3A16, W5A16, W6A16, W7A16, W8A16, W2A16G64, W2A16G32, BF16 |
| auto_awq | W4A16, BF16 |
| auto_gptq | W4A16, W2A16, W3A16, W8A16,W2A16G64, W2A16G32, BF16 |
| fp8 | FP8_BLOCK |
| fake | all schemes (only for research) |
Hardware Compatibility
CPU, Intel GPU, HPU and CUDA for both quantization and inference. The MLX format is exclusive to Apple Silicon (macOS / Darwin) at inference time; quantization (export) itself can be done on any platform.
Environment Configuration
Before starting quantization, you may want to configure AutoRound's environment variables for optimal performance. For detailed information about available environment variables (logging levels, ModelScope integration, workspace settings, etc.), please refer to the Environment Variables Guide.
Command Line Usage
AutoRound recipe:
This setting offers a better trade-off between accuracy and tuning cost, and is recommended in all scenarios.
auto-round --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_gptq,auto_awq,auto_round"AutoRoundBest recipe:
This setting provides the best accuracy in most scenarios but is 4–5× slower than the standard AutoRound recipe. It is especially recommended for 2-bit quantization and is a good choice if sufficient resources are available.
auto-round-best --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_gptq,auto_awq,auto_round"AutoRoundLight Settings:
This setting offers the best speed (2-3X faster than AutoRound), but it may cause a significant accuracy drop for small models and 2-bit quantization. It is recommended for 4-bit settings and models larger than 3B
auto-round-light --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_gptq,auto_awq,auto_round"AutoRoundOptRTN recipe (optimized RTN, without gradient computation):
This setting runs the optimized RTN (Round-To-Nearest) path (
iters=0withdisable_opt_rtn=False). It is calibration-free and several times faster than the default AutoRound recipe, while still applying AutoRound's RTN-side optimizations (e.g. improved scale/zero-point search and llamacpp-style refinements for GGUF). Recommended as a fast baseline when calibration data or tuning time is limited. See the OPT RTN Mode section for details.auto-round-opt-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"AutoRoundRTN recipe (pure RTN, calibration-free, no optimization):
This setting runs pure RTN (
iters=0withdisable_opt_rtn=True), without any AutoRound optimization. It is the fastest path and uses the least memory, but typically yields lower accuracy thanauto-round-opt-rtn. When combined with a supported INT WOQ scheme, it is automatically routed through Model-Free Mode for minimal memory usage. Use this as a quick sanity-check or when you want a calibration-free baseline equivalent to traditional RTN.auto-round-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"
API usage
AutoRound API Usage
This setting offers a better trade-off between accuracy and tuning cost, and is recommended in all scenarios.
from auto_round import AutoRound
model_name_or_path = "Qwen/Qwen3-0.6B"
ar = AutoRound(
model_name_or_path,
scheme="W4A16",
# torch.compile is enabled by default except on Windows. On Windows, set True to force enable it.
)
output_dir = "./tmp_autoround"
# format= 'auto_round'(default), 'auto_gptq', 'auto_awq'
ar.quantize_and_save(output_dir, format="auto_gptq,auto_awq,auto_round")
Mixed Bits Usage
AutoRound(>0.8) offers auto-scheme to generate mixed bits recipe autocially, please refer to AutoScheme section for more details.
Auto-GPTQ and Auto-AWQ only support a limited set of mixed-bit configurations. If you're unsure about the details, we recommend using the AutoRound format.
vLLM and SGLang fuse MoE and QKV layers, so it's recommended not to assign different bit widths to these layers.
from auto_round import AutoRound
model_name_or_path = "Qwen/Qwen3-0.6B"
layer_config = { # Supports both full layer names and fuzzy (partial) matching
"model.decoder.layers.6.self_attn.out_proj": {"bits": 8, "group_size": 32},
"model.decoder.layers.*k_proj": {"bits": 2, "group_size": 32},
}
ar = AutoRound(
model_name_or_path,
layer_config=layer_config,
)
output_dir = "./tmp_autoround"
ar.quantize_and_save(output_dir, format="auto_round")
AutoRoundBest recipe
This setting provides the best accuracy in most scenarios but is 4–5× slower than the standard AutoRound recipe. It is especially recommended for 2-bit quantization and is a good choice if sufficient resources are available.
from auto_round import AutoRound
model_name_or_path = "Qwen/Qwen3-0.6B"
ar = AutoRound(model=model_name_or_path, scheme="W4A16", nsamples=512, iters=1000, low_gpu_mem_usage=True)
output_dir = "./tmp_autoround"
ar.quantize_and_save(output_dir, format="auto_round")
AutoRoundLight recipe
This setting offers the best speed (2 - 3X faster than AutoRound), but it may cause a significant accuracy drop for small models and 2-bit quantization. It is recommended for 4-bit settings and models larger than 3B.
from auto_round import AutoRound
model_name_or_path = "Qwen/Qwen3-0.6B"
ar = AutoRound(
model=model_name_or_path,
scheme="W4A16",
iters=50,
lr=5e-3,
)
output_dir = "./tmp_autoround"
ar.quantize_and_save(output_dir, format="auto_round")
Recipe recommendation
In conclusion, we recommend using auto-round for W4A16 and auto-round-best for W2A16. However, you may adjust the configuration to suit your specific requirements and available resources.
Recipe Configuration Details
| Recipe | batch_size | iters | seqlen | nsamples | lr | disable_opt_rtn |
|---|---|---|---|---|---|---|
| default | 8 | 200 | 2048 | 128 | None | False |
| best | 8 | 1000 | 2048 | 512 | None | False |
| light | 8 | 50 | 2048 | 128 | 5e-3 | False |
| opt_rtn | 8 | 0 | 2048 | 128 | None | False |
| rtn | 8 | 0 | 2048 | 0 | None | True |
W4G128 Average Accuracy of 13 tasks and Time Cost Results(Testing was conducted on the Nvidia A100 80G using the version of PyTorch 2.6.0 with enable_torch_compile):
| Model | Qwen2.5-0.5B-Instruct | Falcon3-3B | Qwen2.5-7B-Instruct | Meta-Llama-3.1-8B-Instruct | Falcon3-10B | Qwen2.5-72B-Instruct |
|---|---|---|---|---|---|---|
| 16bits | 0.4192 | 0.5203 | 0.6470 | 0.6212 | 0.6151 | 0.7229 |
| Best | 0.4137(7m) | 0.5142(23m) | 0.6426(58m) | 0.6116(65m) | 0.6092(81m) | 0.7242(575m) |
| Default | 0.4129(2m) | 0.5133(6m) | 0.6441(13m) | 0.6106(13m) | 0.6080(18m) | 0.7252(118m) |
| Light | 0.4052(2m) | 0.5108(3m) | 0.6453(5m) | 0.6104(6m) | 0.6063(6m) | 0.7243(37m) |
W2G64 results
W2G64 Average Accuracy of 13 tasks and Time Cost Results(Testing was conducted on the Nvidia A100 80G using the version of PyTorch 2.6.0 with enable_torch_compile). We recommend using higher precision for the head, tail, and non-expert modules to alleviate the significant accuracy drop.| Model | Qwen2.5-0.5B-Instruct | Falcon3-3B | Qwen2.5-7B-Instruct | Falcon3-10B | Qwen2.5-72B-Instruct |
|---|---|---|---|---|---|
| 16bits | 0.4192 | 0.5203 | 0.6470 | 0.6151 | 0.7229 |
| Best | 0.2989(6m) | 0.4267(24m) | 0.5343(56m) | 0.5207(79m) | 0.6715(564m) |
| Default | 0.2878(2m) | 0.4219(6m) | 0.5209(13m) | 0.5133(18m) | 0.6713(122m) |
| Light | 0.2760(2m) | 0.4063(3m) | 0.4764(5m) | 0.4810(7m) | 0.6581(38m) |
AWQ Algorithm
Experimental feature: AWQ weight clipping is optional. Enable it with --awq_apply_clip when you want to match the original AWQ flow more closely.
AWQ (Activation-Aware Weight Quantization) is available as an alternative quantization algorithm. AWQ protects salient weight channels by analyzing activation patterns and applying channel-wise scaling before standard RTN quantization.
The canonical AWQ deployment path is W4A16 served by vLLM's AWQ/Marlin CUDA kernels. INT8 is AutoRound's W8A8 scheme and can use AWQ smoothing before RTN quantization for vLLM's compressed_tensors backend (cutlass INT8 GEMM).
AWQ can also be composed with AutoRound optimization (--algorithm awq,auto_round). See AWQ algorithm results for accuracy and cost comparisons across W4A16, MXFP4, and INT8.
CLI Usage
auto-round --model Qwen/Qwen3-0.6B --scheme "W4A16" --algorithm awq --format "auto_round"
AWQ can also be paired with AutoRound optimization:
auto-round --model Qwen/Qwen3-0.6B --scheme "W4A16" --algorithm awq,auto_round
Recommended INT8/W8A8 recipe using AWQ smoothing:
auto-round \
--model Qwen/Qwen3-0.6B \
--scheme INT8 \
--algorithm awq \
--nsamples 256 \
--awq_seqlen 512 \
--awq_apply_clip \
--format auto_round:llm_compressor
The explicit --nsamples 256 and --awq_seqlen 512 settings are recommended for W8A8 AWQ calibration. The default
AutoRound values are tuned for AutoRound optimization, not plain AWQ smoothing.
AWQ-specific options:
--awq_duo_scaling: Use both activations and weights for scaling. Options:true,false, orboth(searches both modes and picks the best). (default: True).--awq_n_grid: Number of grid points for scaling ratio search (default: 20).--awq_apply_clip: Search and apply AWQ weight clipping after smoothing.--awq_seqlen: Maximum sequence length used by AWQ calibration, including activation statistics, smoothing scale search, and clip-search input features. This is separate from the global--seqlen, which controls calibration sample construction. Set a value<= 0to use the full calibration sequence.
API-only AWQ options:
AWQConfig(skip_moe=True): Skips routed MoE experts during AWQ smoothing while keeping attention and dense/shared paths. Explicitmappingsare used as provided.
API Usage
For default AWQ settings, the string alias is sufficient:
from auto_round import AutoRound
ar = AutoRound(model, tokenizer, alg_configs="awq", scheme="W4A16")
Use AWQConfig when you need AWQ-specific options such as apply_clip=True:
from auto_round import AWQConfig, AutoRound
ar = AutoRound(
"Qwen/Qwen3-0.6B",
scheme="W4A16",
alg_configs=AWQConfig(apply_clip=True),
)
output_dir = "./tmp_awq"
ar.quantize_and_save(output_dir, format="auto_round:llm_compressor")
alg_configs="awq" or alg_configs=AWQConfig() selects the AWQ algorithm. This is independent from export
format selection such as format="auto_awq". For example:
alg_configs="awq"+format="auto_round": AWQ smoothing with AutoRound packing.alg_configs="signround"+format="auto_awq": AutoAWQ packing without AWQ smoothing.
SVDQuant Algorithm
Experimental feature: the end-to-end workflow has currently been validated with FLUX.1-dev only.
SVDQuant decomposes each Linear weight into a quantized residual branch and a small floating-point low-rank branch. It can be combined with RTN or SignRound to produce an MXFP4 model for Nunchaku inference.
RTN (recommended starting point):
auto-round-rtn --model /path/to/FLUX.1-dev --model_dtype bf16 \
--scheme MXFP4 --algorithm svdquant --device 0 \
--format svdquant_nunchaku \
--output_dir ./flux-dev-mxfp4-svdquant-rtn
SignRound:
auto-round --model /path/to/FLUX.1-dev --model_dtype bf16 \
--scheme MXFP4 --algorithm svdquant,auto_round \
--format svdquant_nunchaku \
--dataset /path/to/captions.tsv --batch_size 1 --device 0 \
--output_dir ./flux-dev-mxfp4-svdquant-signround
FLUX.1-dev quality results using Nunchaku commit 4de4986:
| Configuration | CLIP | CLIP-IQA | ImageReward |
|---|---|---|---|
| BF16 | 26.0189 | 0.954360 | 1.018340 |
| MXFP4, smooth + SVDQuant + SignRound | 26.1039 | 0.962655 | 1.021020 |
| MXFP4, no smooth + SVDQuant + SignRound | 26.0727 | 0.959363 | 1.002380 |
| MXFP4, smooth + SVDQuant + RTN | 25.9719 | 0.947763 | 0.939392 |
| MXFP4, no smooth + SVDQuant + RTN | 25.9624 | 0.946939 | 0.934579 |
SignRound used 128 calibration samples, 50 inference steps, 200 tuning iterations, rank 32, and 20 residual iterations.
See SVDQuant Details for smooth search, residual iterations, export, and inference.
AutoScheme
AutoScheme automatically generates adaptive mixed-bit and mixed-data-type quantization recipes. For accuracy results, see AutoScheme Accuracy Report.
We recommend exporting to the llm_compressor format for now, as it can be easily deployed with vLLM.
Note: Mixed-data-types are supported during tuning, but cannot be exported to real models at this time.
CLI Usage
--iters 0: RTN. Fast (seconds to minutes).--iters 200: Tuning-aware scheme selection. More accurate but much slower.
Providing multiple --schemes enables AutoScheme; --bits is then the average target bits.
auto_round \
--model_name $model_name \
--schemes "mxfp4,mxfp8" \
--bits 6 \
--ignore_scale_zp_bits \
--iters 0 \
--format fake
API Usage
ar = AutoRound(model=model_name, schemes=("W2A16G64", "W4A16", "W8A16"), bits=3.0, iters=0, nsamples=1)
ar.quantize_and_save()
Hyperparameters in AutoScheme
bits(float) Target average bits for the whole model; only layers to be quantized will be counted in the average bits calculation. Without schemes, bits is the plain weight bit width and must be an integer.
schemes(Union[str, list[Union[QuantizationScheme, str]]) the candidate quantization schemes to choose from, e.g. a string like "W4A16,W8A16", or a list of strings or QuantizationScheme objects. Providing schemes enables AutoScheme.
ignore_scale_zp_bits(bool) Whether to ignore the bits of scale and zero point in average bits calculation. Default is False.
device_map (Optional[str,dict,torch.device]) only supported in API now, as auto-scheme used more VRAM than auto-round tuning, so you could set a different device_map for it.
shared_layers (Optional[Iterable[Iterable[str]]]) only supported in API now
batch_size (Optional[int]) could be set to 1 to reduce VRAM but increase time cost
low_gpu_mem_usage(bool=True) whether to reduce gpu memory usage at the cost of more time cost
calibration_data_device (str) placement of the block calibration data in multi-GPU runs: auto (default) | cpu | off | 1,2 | cuda:1,cuda:2. CLI: --calibration_data_device.
In some serving frameworks, certain layers (e.g., QKV or MoE) are fused to accelerate inference. These fused layers may require the same data type and bit configuration. The shared_layers option simplifies this setup by supporting both regex and full-name matching. Note that regex matching is applied in a block-wise manner.
MoE expert layers are automatically grouped per block — all expert projections (gate/up/down across all experts) within the same transformer block are treated as a single entity during DP optimization. This means they share the same quantization scheme and their losses/numel are summed together. No manual shared_layers configuration is needed for expert layers.
from auto_round import AutoRound
shared_layers = [
["*.self_attn.k_proj", "v_proj", "q_proj", "out_proj"],
("model.decoder.layers.6.fc1", "model.decoder.layers.6.fc2"),
("fc1", "fc2"),
]
model_name = "Qwen/Qwen3-0.6B"
ar = AutoRound(model=model_name, schemes=("W4A16", "MXFP8"), bits=5.0, shared_layers=shared_layers, iters=0, nsamples=1)
model, layer_config = ar.quantize()
Besides, if you want to fix the scheme for some layers, you could set it via layer_config in AutoRound API.
from auto_round import AutoRound
model_name = "Qwen/Qwen3-8B"
layer_config = {"lm_head": "GGUF:Q6_K"}
ar = AutoRound(
model=model_name,
schemes=("GGUF:Q2_K_S", "GGUF:Q4_K_S"),
bits=3.0,
ignore_scale_zp_bits=True,
layer_config=layer_config,
iters=0,
)
ar.quantize_and_save()
AutoScheme Cost
We tested it on Nvidia A100 80G using torch v2.8.
RAM usage has been optimized since v0.14.2.
| Models | Scheme | VRAM Cost | Time Cost |
|---|---|---|---|
| Qwen3-8B | W2A16 / W4A16 / W8A16 | 14G | 60s * len of options |
| Qwen3-8B | MXFP4 / MXFP8 | 18G | 60s * len of options |
| Qwen3-8B | GGUF* | 14G | 80s * len of options |
| Qwen3-32B | W2A16 / W4A16 / W8A16 | 29G | 180s* len of options |
| Qwen3-32B | MXFP4 / MXFP8 | 29G | 180s* len of options |
| Qwen3-32B | GGUF* | 18G | 300s * len of options |
| Llama-3.3-70B | W2A16 / W4A16 / W8A16 | 32G | 420s * len of options |
Cost w/o low_gpu_mem_usage
| Models | Scheme | VRAM Cost (torch compile) |
Time Cost torch compile |
VRAM Cost wo torch compile |
Time Cost wo torch compile |
|---|---|---|---|---|---|
| Qwen3-8B | W2A16/W4A16/W8A16 | 34G | 30s * len of options | 61G | 40s * len of options |
| Qwen3-8B | MXFP4/MXFP8 | 36G | 60s * len of options | 54G | 120s * len of options |
| Qwen3-8B | GGUF* | 54G | 30s * len of options | 50G | 23S * len of options |
| Qwen3-32B | W2A16/W4A16/W8A16 | OOM with 240G | --- | OOM with 240G | --- |
| Qwen3-32B | MXFP4/MXFP8 | 160G | 200s * len of options | 200G | 240s * len of options |
| Qwen3-32B | GGUF* | 210G | 80s * len of options | 200G | 60s * len of options |
Limitations
Embedding layer is not supported in AutoScheme, it will use the best scheme in options.
When using AutoScheme with model_free=True, only INT (W2A16/W4A16/W8A16) and MXFP (MXFP4/MXFP8) option families are supported. Options like W3A16, GGUF:*, and NVFP4 will raise a ValueError. INT and MXFP families cannot be mixed in the same AutoScheme.
OPT RTN Mode
AutoRound also supports Optimized RTN (Round-To-Nearest) mode for fast, calibration-free baseline quantization. Setting iters=0 tp enable it and we recommend using group_size=32 for better results. Check accuracy comparison between RTN and OPT RTN mode
For the GGUF format, we have optimized the RTN algorithm inspired by llamacpp. To use the original (pure) RTN algorithm instead, enable the --disable_opt_rtn option.
Passing --enable_neuqi opts the optimized path into the NeUQI grid search (arXiv 2505.17595): asymmetric layers run a joint (scale, integer zero-point) search and symmetric layers a two-stage signed scale search, both weighted by the activation imatrix on the zero-shot path, which is collected automatically (with iters > 0 the anchor uses the imatrix when the tuning path collects one, running unweighted otherwise). With iters > 0, the search result anchors the SignRound tuning grid (frozen init). Grid sizes are tunable via AR_NEUQI_COARSE/AR_NEUQI_FINE (see environments); the unpinned default is backend-aware (wide only on the Triton/torch.compile lanes). Accuracy and wall-time results are shown in NeUQI accuracy validation.
CLI Usage
Two dedicated CLI entry points are provided as shortcuts:
auto-round-opt-rtn— equivalent toauto-round --iters 0 --enable_opt_rtn(optimized RTN, recommended).auto-round-rtn— equivalent toauto-round --iters 0 --disable_opt_rtn(pure RTN, no optimization; auto-routes to Model-Free Mode for supported INT WOQ schemes).
# Optimized RTN (recommended fast baseline)
auto-round-opt-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"
# Pure RTN (fastest, lowest memory; baseline quality)
auto-round-rtn --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_round"
API Usage
from auto_round import AutoRound
model_name_or_path = "Qwen/Qwen3-0.6B"
ar = AutoRound(
model=model_name_or_path,
scheme="W4A16",
iters=0,
)
output_dir = "./tmp_autoround"
ar.quantize_and_save(output_dir, format="auto_round")
Model-Free Mode
Model-free mode performs calibration-free WOQ quantization without loading the full model into memory. It downloads safetensors files directly, quantizes each Linear weight tensor shard-by-shard, and saves the packed result. This is useful when you want fast, no-calibration quantization with minimal resource requirements.
Auto-enabled by default. As of v0.13, when you pass
--iters 0 --disable_opt_rtntogether with a supported INT WOQ or MXFP scheme, the CLI automatically takes the model-free path. This is bit-exactly equivalent to the regular--iters 0 --disable_opt_rtnflow but uses far less memory. Use--disable_model_freeto opt out and force the original flow.
When using--model_freeexplicitly, INT WOQ always uses plain RTN (opt_rtn is disabled for INT WOQ to preserve accuracy); MXFP schemes use optimized RTN (opt_rtn) by default — pass--disable_opt_rtnto use plain RTN for MXFP.
Key features:
- No model object required – only
config.jsonand safetensors files are needed - Low disk memory required (If no local model files) – downloads and quantizes one shard at a time, deleting the source shard after processing
- Per-layer configuration – supports
--layer_configfor per-layer bit-width overrides and--ignore_layersto keep specific layers in full precision - Predefined ignore layers – automatically skips model-specific layers (e.g., MoE gates, MTP layers) based on config detection
- Optimized RTN (opt_rtn) for MXFP — MXFP schemes use opt_rtn by default; pass
--disable_opt_rtnfor plain RTN. INT WOQ always uses plain RTN (opt_rtn disabled for INT to preserve accuracy). - AutoScheme integration – pass an
AutoSchemeobject asschemeto get automatic mixed-bit selection followed by shard-by-shard packing (two-phase: score with model briefly loaded, then free and pack)
Model-free Parallelism Benchmarks (Rounded Minutes)
Model-free mode quantizes multiple weight shards in parallel worker processes. Set the parallelism with the AR_MODEL_FREE_SHARD_PARALLELISM environment variable (see the Environment Variables Guide):
- Default (unset):
shard_count // 4, min 1, max 10. For example, 8 shards use 2 workers and 40+ shards use 10 workers. - Manual: e.g.
export AR_MODEL_FREE_SHARD_PARALLELISM=20. The effective value never exceeds the shard count. - Higher parallelism is usually faster but increases peak GPU (and host) memory; lower it when resources are tight.
Runtimes are rounded up to the next full minute.
| Model | Device | Scheme | Parallelism | Runtime (min) | Speedup | Peak Memory (G) |
|---|---|---|---|---|---|---|
| Qwen3-Next-80B-A3B-Instruct | A100 | W4A16 | 1 -> 10 | N/A -> 7 | N/A | 2 -> 8 |
| Qwen3-235B-A22B-Instruct-2507 | A100 | W4A16 | 1 -> 10 | 17 -> 5 | 3.40x | 2 -> 8 |
| GLM-5.2 | B200 | MXFP4-Mixed | 1 -> 10 | 60 -> 16 | 3.75x | 2 -> 27 |
| GLM-5.2 | B200 | W4A16 | 1 -> 10 / 20 | 30 -> 10 / 8 | 3.00x / 3.75x | 3 -> 16 / 32 |
| MiniMax-M2.7 (FP8) | B200 | W4A16 | 1 -> 10 | 18 -> 5 | 3.60x | 2 -> 10 |
| DeepSeek-V4-Pro (MXFP) | B200 | W4A16 | 1 -> 10 | 80 -> 13 | 6.15x | 6 -> 50 |
Key takeaway: model-free quantization usually gets about 3x-6x runtime speedup with higher parallelism, while peak memory usage increases significantly.
Click to expand supported schemes and examples
Supported schemes
Model-free mode supports the following preset schemes:
Integer weight-only (packed in auto_round:auto_gptq format):
| Preset | Bits | Group size | Sym |
|---|---|---|---|
W2A16 |
2 | 128 | true |
W2A16G32 |
2 | 32 | true |
W2A16G64 |
2 | 64 | true |
W4A16 (default) |
4 | 128 | true |
W4A16_MIXED |
4 | 128 | true |
W8A16 |
8 | 128 | true |
All of the above presets also support asymmetric quantization (sym=False) for 2-bit and 8-bit variants (W2A16, W2A16G32, W2A16G64, W8A16), producing auto_round:auto_gptq-packed output with bit-exact parity to the regular flow. For 4-bit asymmetric quantization the regular flow uses auto_round:auto_awq packing as suggested; use the standard AutoRound flow for that case.
You can also pass a custom QuantizationScheme(bits=N, group_size=G, sym=True/False, data_type="int", act_bits=16) with bits ∈ {2, 4, 8} and any group_size / sym configuration.
MXFP (Microscaling Floating Point) (packed in mxfp4-pack-quantized / mxfp8-quantized format, compatible with compressed-tensors / vLLM):
| Preset | Bits | Group size | Format |
|---|---|---|---|
MXFP4 |
4 | 32 | mxfp4-pack-quantized |
MXFP8 |
8 | 32 | mxfp8-quantized |
Schemes that require special packing kernels (W3A16, FPW8A16, BF16, MXINT4, NVFP4, FP8_BLOCK, FP8_STATIC, INT8_W8A8, GGUF:*, ...) are not supported in model-free mode and will raise ValueError. Use the regular AutoRound flow for those.
NVFP4 source checkpoints → MXFP4
Goal: convert published NVFP4 checkpoints to MXFP4 to serve 4-bit LLMs with high throughput on Intel GPUs, which support MXFP4 but have no native NVFP4 acceleration path.
Supported source tensor layouts:
| Source layout | Tensors | Status |
|---|---|---|
| compressed-tensors (NVFP4) | .weight_packed + .weight_scale + .weight_global_scale + .input_global_scale |
Supported |
| ModelOpt / Quark (NVFP4) | .weight (uint8) + .weight_scale (FP8) + .weight_scale_2 + .input_scale |
Supported |
inkling_mm_model type |
X (uint8) + X.scale + X.scale2 (may live in different shards) |
Supported |
Status of commonly requested checkpoints:
| Model | Source format | Status | gsm8k strict-match (MXFP4) | gsm8k flexible-extract (MXFP4) |
|---|---|---|---|---|
RedHatAI/Qwen3.8-27B-NVFP4 |
compressed-tensors, NVFP4 + FP8 | Verified | 0.9636 ± 0.0052 | 0.9636 |
nvidia/Qwen3.8-27B-NVFP4 |
ModelOpt, NVFP4 + FP8 | Verified | 0.9644 ± 0.0051 | 0.9659 |
RedHatAI/Qwen3.6-35B-A3B-NVFP4 |
compressed-tensors, NVFP4 | Verified | 0.9575 ± 0.0056 | 0.9575 |
amd/MiniMax-M2.7-NVFP4 |
Quark, NVFP4 | Verified | 0.9568 ± 0.0056 | 0.9591 |
nvidia/MiniMax-M2.7-NVFP4 |
ModelOpt, NVFP4 | Verified | 0.9477 ± 0.0061 | 0.9522 |
thinkingmachines/Inkling-Small-NVFP4 |
Special format (inkling_mm_model), NVFP4 |
Verified | 0.9477 ± 0.0061 | 0.9545 |
gsm8k is evaluated with 5-shot on the converted MXFP4 models, with thinking disabled. These results only demonstrate that the converted models work correctly; the numbers are not guaranteed to be accurate.
Running --scheme MXFP4 --model_free on an NVFP4 checkpoint converts only the NVFP4 layers to MXFP4. The source is detected as NVFP4 from the model name (containing nvfp4) or its quantization config (config.json / hf_quant_config.json), and a warning is printed:
- NVFP4 layers: detected automatically in each shard and converted to MXFP4.
- Unquantized layers (BF16,
lm_head, router, ...): kept as-is and recorded as unquantized inquantization_config. To quantize some of them as well, list them in--layer_config, e.g.--layer_config "{lm_head:mxfp4}". - FP8 layers (mixed-precision sources, e.g. FP8 attention + NVFP4 MLP): kept as-is. The source quantization config (compressed-tensors; ModelOpt and Quark configs are first converted to compressed-tensors) is reused with only its NVFP4 groups rewritten to MXFP4, so the output is a
mixed-precisionquantization_config. - Unrecognized source configs trigger a warning and are skipped.
- Currently only
--format llm_compressoris supported.
auto_round nvidia/MiniMax-M2.7-NVFP4 --model_free --scheme MXFP4 --format llm_compressor --output_dir ./mxfp4-model
CLI Usage
# Easiest: --iters 0 --disable_opt_rtn auto-routes to model-free (plain RTN)
auto_round meta-llama/Llama-3.2-1B-Instruct \
--scheme W4A16 \
--iters 0 --disable_opt_rtn \
--output_dir ./int4-llama
# Explicit model_free (INT WOQ always uses plain RTN; bit-exact with the auto-route above)
auto_round meta-llama/Llama-3.2-1B-Instruct \
--model_free \
--scheme W4A16 \
--output_dir ./int4-llama
# Opt out of auto-routing and use the regular flow instead
auto_round meta-llama/Llama-3.2-1B-Instruct \
--scheme W4A16 \
--iters 0 --disable_opt_rtn --disable_model_free \
--output_dir ./int4-llama
# With per-layer configuration and ignored layers
auto_round meta-llama/Llama-3.2-1B-Instruct \
--model_free \
--scheme W4A16 \
--group_size 32 \
--asym \
--layer_config "{k_proj:{bits:8},v_proj:{bits:8}}" \
--ignore_layers "mlp" \
--output_dir ./int4-llama
# MXFP4 quantization
auto_round meta-llama/Llama-3.2-1B-Instruct \
--model_free \
--scheme MXFP4 \
--output_dir ./mxfp4-llama
# MXFP8 quantization
auto_round meta-llama/Llama-3.2-1B-Instruct \
--model_free \
--scheme MXFP8 \
--output_dir ./mxfp8-llama
API Usage
from auto_round import AutoRound
AutoRound(
model="meta-llama/Llama-3.2-1B-Instruct",
scheme="W4A16", # Or a QuantizationScheme instance for custom group_size / sym.
layer_config={
".*k_proj": {"bits": 8, "group_size": 32},
".*v_proj": {"bits": 8, "group_size": 32},
},
ignore_layers="mlp",
model_free=True,
).quantize_and_save("./int4-llama")
Note: Model-free mode always uses plain RTN for INT WOQ (opt_rtn is disabled to preserve accuracy). For MXFP schemes, optimized RTN (opt_rtn) is enabled by default; pass
disable_opt_rtn=Truefor plain RTN. INT schemes output inauto_round:auto_gptqformat; MXFP schemes output in compressed-tensors format (mxfp4-pack-quantized/mxfp8-quantized). For higher-quality quantization or schemes outside the supported list, use the standard AutoRound flow.
GGUF format
Experimental feature. This format is well-suited for CPU devices and is widely adopted by the community.
The optimized RTN mode is suggested (--iters 0) for all bits other than 3 bits.
from auto_round import AutoRound
model_name_or_path = "Qwen/Qwen3-0.6B"
ar = AutoRound(
model=model_name_or_path,
)
output_dir = "./tmp_autoround"
ar.quantize_and_save(output_dir, format="gguf:q4_k_m") # gguf:q*_k_s,gguf:q*_k_0,gguf:q*_k_1,
Quantization Costs
Testing was conducted on the Nvidia A100 80G using the nightly version of PyTorch 2.6.0.dev20241029+cu124. Please note that data loading and packing costs have been excluded from the evaluation. We recommend enabling torch.compile for PyTorch versions 2.6 and above.
To optimize GPU memory usage, in addition to activating low_gpu_mem_usage, you can set gradient_accumulate_steps=8
and a
batch_size=1, though this may increase tuning time.
The 3B and 14B models were evaluated on Qwen 2.5, the 8X7B model is Mixtral, while the remaining models utilized LLaMA 3.1.
| Torch version/Config W4G128 | 3B | 8B | 14B | 70B | 8X7B |
|---|---|---|---|---|---|
| 2.6 with torch compile | 7min 10GB |
12min 18GB |
23min 22GB |
120min 42GB |
28min 46GB |
| 2.6 with torch compile low_gpu_mem_usage=True |
12min 6GB |
19min 10GB |
33min 11GB |
140min 25GB |
38min 36GB |
| 2.6 with torch compile low_gpu_mem_usage=True gradient_accumulate_steps=8,bs=1 |
15min 3GB |
25min 6GB |
45min 7GB |
187min 19GB |
75min 36GB |
| 2.5 w/o torch compile | 8min 10GB |
16min 20GB |
30min 25GB |
140min 49GB |
50min 49GB |
W4G128 Quantization Time and Memory Usage (Intel GPU B60 24G) Testing was conducted on the Intel GPU B60 24G using the release version of PyTorch 2.11.0+xpu. Please note that data loading and packing costs have been excluded from the evaluation. Time and memory usage were measured using Qwen2.5-series models.
| Torch version/Config W4G128 | 0.5B | 1.5B | 3B | 7B |
|---|---|---|---|---|
| 2.11.0+xpu with torch compile | 6min 2.9GB |
13min 5.4GB |
22min 7.1GB |
40min 14.9GB |
| 2.11.0+xpu with torch compile low_gpu_mem_usage=True |
10min 1.7GB |
17min 3.3GB |
30min 4.3GB |
50min 8.5GB |
| 2.11.0+xpu with torch compile low_gpu_mem_usage=True gradient_accumulate_steps=8,bs=1 |
14min 0.4GB |
22min 1.1GB |
38min 1.5GB |
1h 4min 4.1GB |
| 2.11.0+xpu w/o torch compile | 6min 2.9GB |
14min 5.7GB |
26min 7.6GB |
51min 15.5GB |
Device/Multi-GPU setting in Quantization
The tuning device is specified using the device_map argument in AutoRound API, not through the device_map
parameter used by Transformers.from_pretrained.
AutoRound tunes the model in a block-by-block manner. Although the block size is much smaller than the model size, it still requires a significant amount of GPU memory for tuning—typically 10 times the block size. This can lead to out-of-memory (OOM) errors when working with extremely large models.
For strategies to reduce GPU memory usage, please refer to the [Reduced GPU Memory Usage](###Adjust Hyperparameters) section below, where you can adjust hyperparameters to optimize memory consumption.
If adjusting hyperparameters does not resolve the issue a, a simple solution is just adding more devices in device_map, for example,
from auto_round import AutoRound
model_name_or_path = "Qwen/Qwen3-0.6B"
ar = AutoRound(
model=model_name_or_path,
device_map="0,1,2,3"
)
or
CUDA_VISIBLE_DEVICES=0,1,2,3 auto-round --model "Qwen/Qwen3-0.6B" --scheme "W4A16" --device_map "auto"
There are typically two scenarios that require multi-GPU tuning: one is the calibration phase mainly for lm-head quantization, and the other is quantizing extremely large models (e.g., models larger than 100 GB).
Parallel block tuning across GPUs
When a tuning block fits on one GPU, --data_parallel uses the other GPUs of the same process for
data-parallel tuning: the block's tuning loop runs once per GPU (a rank, one thread each) on that GPU's copy of the
block and its share of the calibration samples, and the ranks average their gradients every iteration. This is the
same contract as the multi-process torchrun path (sync_gradients()), so the algorithm's loop itself is unchanged.
CUDA_VISIBLE_DEVICES=0,1,2,3 auto-round --model "Qwen/Qwen3-0.6B" --scheme "W4A16" --data_parallel auto
off(default) keeps the serial loop.Npins the number of ranks: at least 2, at most the visible device count, and a power of two when--iters > 0(the gradient all-reduce is a halving-doubling exchange); checked at startup.autotakes all visible devices at--iters 0and the largest power of two that fits them at--iters > 0, and announces the choice in the log.- The block must sit whole on one GPU: run without
--device_mapor with a single-device map. A block spread across several GPUs stops with that reason; blocks larger than one GPU are not covered. - Every rank holds a full copy of the block plus its tuning state. Free-VRAM estimates only warn (naming the device); an actual shortage surfaces as an out-of-memory error.
--parallel_batch split(default) keeps--batch_sizeas the global batch of an iteration; every rank drawsbatch_size / worldsamples (the division must be exact).--parallel_batch per_replicalets every rank draw the full--batch_size, so the global batch isbatch_size x world.- A block whose sample count or batch does not split evenly over the ranks, or whose per-rank batch exceeds its share of the samples, tunes serially and logs the reason. LFQ, dict-shaped (diffusion-style) calibration pools and an active torchrun group are not supported.
- The calibration forwards (reference outputs, the quantized-input chain, imatrix and activation statistics) are
sharded across the GPUs too, also at
--iters 0. Quantizers declare the statistics their hooks write and how they reduce (calibration_stats); a hook pass with undeclared statistics stays serial. - Each rank samples from its own share of the calibration pool, so per-iteration loss lines and tuned weights differ from a serial run. Compare runs at the model level (accuracy, KL divergence against the base model).
- To exclude devices, set the backend's visibility variable (e.g.
CUDA_VISIBLE_DEVICES) before launch.
Enable multiple gpus calibration in lm_head quantization
For LM head tuning, AutoRound needs to cache the inputs to the lm-head, which requires the entire model to reside on the GPU for efficient calibration. If there is no enough VRAM, some layers will fallback to RTN mode
Manually set the device_map
Customized device map
If device_map=auto does not correctly map the model, we also support mapping different layers within a block to different devices by setting the `device_map` argument in the AutoRound API. For reference, we provide an example of quantizing the DeepSeekV3-BF16 (1.4T) model using five 80GB GPUs.import torch
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "opensourcerelease/DeepSeek-R1-bf16"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name, trust_remote_code=True, torch_dtype="auto")
block = model.model.layers
device_map = {}
for n, m in block.named_modules():
if type(m) == torch.nn.Linear:
if "experts" in n and ("shared_experts" not in n) and int(n.split(".")[-2]) < 63:
device = "cuda:1"
elif (
"experts" in n
and ("shared_experts" not in n)
and int(n.split(".")[-2]) >= 63
and int(n.split(".")[-2]) < 128
):
device = "cuda:2"
elif (
"experts" in n
and ("shared_experts" not in n)
and int(n.split(".")[-2]) >= 128
and int(n.split(".")[-2]) < 192
):
device = "cuda:3"
elif "experts" in n and ("shared_experts" not in n) and int(n.split(".")[-2]) >= 192:
device = "cuda:4"
else:
device = "cuda:0"
n = n[2:]
device_map.update({n: device})
from auto_round import AutoRound
autoround = AutoRound(
model=model,
tokenizer=tokenizer,
device_map=device_map,
nsamples=512,
batch_size=4,
low_gpu_mem_usage=True,
seqlen=2048,
)
autoround.quantize()
autoround.save_quantized(format="auto_awq", output_dir="tmp_autoround")
Adjust Hyperparameters
Reduced GPU Memory Usage:
keep
torch.compileenabled by default on non-Windows platforms; on Windows, passenable_torch_compile=Trueto opt inenable
low_gpu_mem_usage(more tuning cost)set
--bs 1 --gradient_accumulate_steps 8(more tuning cost)reduce the
bsto 4(potential accuracy drop)reduce the
seqlento 512 (potential accuracy drop)or combine them
Reduced CPU Memory Usage :
low_cpu_mem_usage(experimental) is enabled by default. In CLI, use--disable_low_cpu_mem_usageto turn it off (--low_cpu_mem_usageis retained only for compatibility). Only one export format is supported. The quantized model is saved immediately after each block is packed, reducing peak CPU memory usage.Trigger immediate packing: Packing will be triggered immediately when using the command-line interface or the quantize_and_save API, as long as only one export format is specified.
Speedup the tuning:
keep
torch.compileenabled by default on non-Windows platforms; on Windows, passenable_torch_compile=Trueto opt inuse
auto-round-lightconfigurationreduce the seqlen to 512(potential large accuracy drop for some scenarios)
reduce the train bs to 4(little accuracy drop. )
or combine them
torch.compileis disabled by default on Windows because TorchInductor needs a compatible compiler toolchain. Windows users can passenable_torch_compile=Trueto the Python API or use--enable_torch_compileto force enable it. On other platforms, passenable_torch_compile=Falseor use--disable_torch_compileto opt out.For NVIDIA GPUs on Windows, install a CUDA-enabled PyTorch build and a compatible
triton-windowsversion in the same Python environment as AutoRound before enabling compilation. Follow the Triton for Windows installation guide for the PyTorch/Triton version matrix, supported GPUs, and compiler requirements. Do not simply install the newest Triton version with an older PyTorch build. Enabling--enable_torch_compilealone does not install these dependencies; aTritonMissingerror means the compilation environment still needs attention. Keep compilation disabled until the prerequisites are met.Enable quantized lm-head:
Currently only support in AutoRound format inference for this config
auto-round --model_name Qwen/Qwen3-0.6B --scheme "W4A16" --quant_lm_head --format "auto_round"Utilize the AdamW Optimizer:
Include the flag
--adam. Note that AdamW is less effective than sign gradient descent in many scenarios we tested.
Rotation (Research)
⚠️ Research feature: Rotation transform is still a research feature. Inference relies on forward hooks, which are currently only supported by the Hugging Face Transformers backend, so rotated models may run slower than native (non-rotated) models.
Rotation redistributes outliers in weights and activations before quantization, making the distribution more uniform and quantization-friendly. It is most useful for aggressive low-bit schemes such as MXFP4, NVFP4 and W4A4.
AutoRound applies rotation through alg_configs, alongside a quantization algorithm. The "quarot" preset — deterministic Hadamard rotation (QuaRot / SpinQuant), no training and no calibration data — is recommended for most use cases.
API Usage
from auto_round import AutoRound
model_name = "Qwen/Qwen3-0.6B"
# QuaRot preset: deterministic Hadamard, no training
ar = AutoRound(model_name, scheme="MXFP4", alg_configs=["auto_round", "quarot"])
ar.quantize_and_save(output_dir="./Qwen3-0.6B-mxfp4-quarot", format="auto_round")
Quantized models with rotation are saved and loaded transparently — rotation matrices and hooks are restored automatically on load, so inference needs no extra steps.
For rotation positions (R1–R4), full configuration options, deterministic vs random Hadamard, trainable SpinQuant, the per-linear block rotation variant, and save/load internals, see Rotation Details.
4 Inference
AutoRound automatically selects the best available backend based on the installed libraries and prompts the user to install additional libraries when a better backend is found.
Please avoid manually moving the quantized model to a different device (e.g., model.to('cpu')) during inference, as this may cause unexpected exceptions.
CPU
Supports 2, 4, and 8 bits. We recommend using auto-round-lib (ark) for inference. When using the ark backend, ensure that your PyTorch version is >= 2.8.0.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "OPEA/Qwen2.5-1.5B-Instruct-int4-sym-inc"
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="cpu", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
text = "There is a girl who likes adventure,"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))
Intel GPU
Supports 4,8 bits. When using the ark backend, PyTorch (torch) >= 2.8.0 is required. We recommend using auto-round-lib (ark) for inference.
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "OPEA/Qwen2.5-1.5B-Instruct-int4-sym-inc"
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="xpu", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
text = "There is a girl who likes adventure,"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))
CUDA
Supports 2, 3, 4, 5, 6, 7 and 8 bits. We recommend using GPTQModel for 4 and 8 bits inference, and humming for 5/6/7 bits (it is also faster than Triton for the other widths).
pip install git+https://github.com/inclusionAI/humming.git
from transformers import AutoModelForCausalLM, AutoTokenizer
model_name = "OPEA/Qwen2.5-1.5B-Instruct-int4-sym-inc"
model = AutoModelForCausalLM.from_pretrained(model_name, device_map="cuda", torch_dtype="auto")
tokenizer = AutoTokenizer.from_pretrained(model_name)
text = "There is a girl who likes adventure,"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))
HPU
docker image with Gaudi Software Stack is recommended. More details can be found in Gaudi Guide.
import habana_frameworks.torch.core as htcore
import habana_frameworks.torch.hpu as hthpu
from transformers import AutoModelForCausalLM, AutoTokenizer
import torch
model_name = "Intel/Qwen2-7B-int4-inc"
tokenizer = AutoTokenizer.from_pretrained(model_name)
model = AutoModelForCausalLM.from_pretrained(model_name).to("hpu").to(torch.bfloat16)
text = "There is a girl who likes adventure,"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))
Specify Inference Backend
AutoRound automatically selects the backend for each layer based on compatibility. In general, the priority order is Marlin > ExLLaMAV2 > Triton, but the final choice depends on factors such as group size, bit width, packing format, hardware device, and other implementation details.
The backend may not always be the most suitable for certain devices. You can specify your preferred backend such as "ark" for CPU and Intel GPU, "marlin/exllamav2/triton" for CUDA, according to your needs or hardware compatibility. Please note that additional corresponding libraries may be required.
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoRoundConfig
model_name = "OPEA/Qwen2.5-1.5B-Instruct-int4-sym-inc"
quantization_config = AutoRoundConfig(backend="ark")
model = AutoModelForCausalLM.from_pretrained(
model_name, device_map="cpu", quantization_config=quantization_config, torch_dtype="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
text = "There is a girl who likes adventure,"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))
| Name | Devices | Bits | Dtypes | Priority | Packing format | Requirements |
|---|---|---|---|---|---|---|
| ark | cpu | 2,4,8 | FP32/FP16/BF16 | 6 | gptq/gptq_zp+-1 | auto-round-lib torch>=2.8.0 |
| ark | cpu | 4 | FP32/FP16/BF16 | 6 | awq | auto-round-lib torch>=2.8.0 |
| ark | xpu | 4,8 | FP32/FP16/BF16 | 6 | gptq/gptq_zp+-1 | auto-round-lib torch>=2.8.0 |
| ark | xpu | 4 | FP32/FP16/BF16 | 6 | awq | auto-round-lib torch>=2.8.0 |
| marlin | cuda | 4,8 | BF16/FP16 | 6 | gptq/gptq_zp+-1 | gptqmodel |
| exllamav2 or gptqmodel:exllamav2 |
cuda | 4 | BF16/FP16 | 5 | gptq/gptq_zp+-1 | gptqmodel |
| exllamav2 or gptq:exllamav2 |
cuda | 4 | FP16 | 3 | gptq_zp+-1 | auto-gptq transformers<5.0.0 |
| gptq:cuda | cuda | 2,3,4,8 | FP16 | 1 | gptq_zp+-1 | auto-gptq transformers<5.0.0 |
| triton | xpu/cuda | 2,4,8 | BF16/FP16 | 2 | gptq/gptq_zp+-1 | auto-round |
| awq | cuda | 4 | FP16 | 5 | awq | auto-awq transformers<4.57.0 |
| gptqmodel:awq or gptqmodel:awq_exllamav2 |
cuda | 4 | BF16/FP16 | 6 | awq | gptqmodel |
| gptqmodel:awq_marlin | cuda | 4,8 | FP16 | 5 | awq | gptqmodel |
| gptqmodel:awq_gemm | cuda | 4 | FP16 | 3 | awq | gptqmodel |
| gptqmodel:awq_torch | cuda/cpu | 4 | FP16 | 2 | awq | gptqmodel |
| humming | cuda | 2,3,4,5,6,7,8 | BF16/FP16 | 4 | gptq/gptq_zp+-1 | humming-kernels |
| humming | cuda | 2,3,4,5,6,7,8 | BF16/FP16 | 4 | awq | humming-kernels |
| hpu | hpu | 4 | BF16 | 0 | gptq/gptq_zp+-1 | auto-round |
| torch | xpu/cpu/cuda | 2,3,4,5,6,7,8 | BF16/FP16 | 0 | gptq/gptq_zp+-1 | auto-round |
Convert GPTQ/AWQ to AutoRound
Most GPTQ/AWQ models can be converted to the AutoRound format for better compatibility and support with Intel devices. Please note that the quantization config will be changed if the model is serialized.
from transformers import AutoModelForCausalLM, AutoTokenizer, AutoRoundConfig
model_name = "ybelkada/opt-125m-gptq-4bit"
quantization_config = AutoRoundConfig()
model = AutoModelForCausalLM.from_pretrained(
model_name, device_map="cpu", quantization_config=quantization_config, torch_dtype="auto"
)
tokenizer = AutoTokenizer.from_pretrained(model_name)
text = "There is a girl who likes adventure,"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))
5 Evaluation
AutoRound leverages lm-eval-harness for evaluation. If --tasks is not specified, a set of default tasks (typically 10+ common benchmarks) will be automatically used.
Single GPU Evaluation
HF Backend (default):
auto-round --model Qwen/Qwen3-0.6B --bits 4 --format "auto_round,auto_gptq" --tasks mmlu
vLLM Backend:
auto-round --model Qwen/Qwen3-0.6B --bits 4 --format "auto_round,auto_gptq" --tasks mmlu --eval_backend vllm
Multi-GPU Evaluation
HF Backend:
auto-round --model="your_model_path" --eval --device_map 0,1 --tasks lambada_openai --eval_bs 16
vLLM Backend (Option 1 - using --device_map):
auto-round "your_model_path" --eval --device_map 0,1 --tasks lambada_openai --eval_backend vllm
vLLM Backend (Option 2 - manual configuration):
CUDA_VISIBLE_DEVICES=0,1 auto-round "your_model_path" --eval --tasks lambada_openai --eval_backend vllm --vllm_args="tensor_parallel_size=2,gpu_memory_utilization=0.8"
Important Notes
- Use the
--evalflag to evaluate models directly. This supports both original and quantized models. - The
--eval_task_by_taskoption helps handle task failures by evaluating tasks sequentially. This only applies to the HF backend. - Use
--num_fewshot,--eval_gen_kwargs, and--fewshot_as_multiturnto pass few-shot and generation options through to lm-eval. - When multiple formats are exported, the last format in the list will be used for evaluation.
- For vLLM backend, you can use
--device 0,1,2to specify GPU devices. This will automatically setCUDA_VISIBLE_DEVICESand configuretensor_parallel_sizebased on the number of devices. Alternatively, you can manually set these via environment variables and--vllm_args.
6 Known Issues
Randomness in quantization may affect tuning results for some models, set enable_deterministic_algorithms=True to ensure reproducibility.
Some VLMs require manual support.
Mamba is not supported.