PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
refine readme (#978)
Wenhua Cheng3f7bdaceec7e
diff --git a/auto_round/compressors/base.py b/auto_round/compressors/base.pyindex da325c989..bed19c3e9 100644--- a/auto_round/compressors/base.py+++ b/auto_round/compressors/base.py@@ -187,7 +187,7 @@ class BaseCompressor(object):super_group_size, super_bits, scale_dtype ("fp16" etc.),nblocks, to_quant_block_names,enable_norm_bias_tuning, enable_quanted_input,- disable_deterministic_algorithms, mllm, static_kv_dtype+ disable_deterministic_algorithms, mllm, static_kv_dtype,enable_deterministic_algorithmsRaises:ValueError: If invalid device is provided or tokenizer is missing for non-str model with iters > 0.RuntimeError: If model parameters are on meta device.diff --git a/docs/step_by_step.md b/docs/step_by_step.mdindex 6efbc85e7..d2e46008f 100644--- a/docs/step_by_step.md+++ b/docs/step_by_step.md@@ -73,16 +73,14 @@ calibration data and will be downloaded automatically from the datasets Hub. Oth~~~pythondef customized_data():- ##Important Notice!!! Autoround will drop data < args.seqlen and truncate data to args.seqlen- data = ["AutoRound is an advanced weight-only quantization algorithm for low-bits LLM inference" * 240]- data.append("AutoRound is an advanced weight-only quantization algorithm for low-bits LLM inference")+ # Important Notice!!! AutoRound will drop data < args.seqlen and truncate data to args.seqlen+ data = ["AutoRound is an advanced quantization algorithm for low-bits LLM inference" * 240]return datadef customized_data_with_tokenizer(tokenizer, seqlen=2048):- ##Import notice!!! Autoround will drop data < args.seqlen- data = ["AutoRound is an advanced weight-only quantization algorithm for low-bits LLM inference" * 240]- data.append("AutoRound is an advanced weight-only quantization algorithm for low-bits LLM inference")+ # Import notice!!! AutoRound will drop data < args.seqlen+ data = ["AutoRound is an advanced quantization algorithm for low-bits LLM inference" * 240]tokens = []for d in data:token = tokenizer(d, truncation=True, max_length=seqlen, return_tensors="pt").data@@ -121,9 +119,10 @@ AutoRound supports several Schemes:- **W2A16**(bits:2,group_size:128,sym:True,act_bits:16)- **GGUF:Q4_K_M**(all Q*_K,Q*_0,Q*_1 are supported)- **Mixed Bits Weight only**-- **NVFP4**(Experimental feature, recommend exporting to llm-compressor format. data_type:nvfp4,act_data_type:nvfp4,static_global_scale,group_size 16)-- **MXFP4**(**Research feature,no real kernel**, data_type:mxfp4,act_data_type:mxfp4,rceil,group_size 32)-- **FPW8A16**(**Research feature,no real kernel**, data_type:fp8,act_data_type 16:,group_size 0->per tensor )+- **NVFP4**(Experimental feature, recommend exporting to `llm_compressor` format.data_type nvfp4,act_data_type nvfp4,static_global_scale,group_size 16)+- **MXFP4**(**Research feature,no real kernel**, data_type mxfp,act_data_type mxfp_rceil,bits 4, act_bits 4, group_size 32)+- **MXFP8**(**Research feature,no real kernel**, data_type mxfp,act_data_type mxfp_rceil,group_size 32)+- **FPW8A16**(**Research feature,no real kernel**, data_type fp8,group_size 0->per tensor )- **FP8_STATIC**(**Research feature,no real kernel**, data_type:fp8,act_data_type:fp8,group_size -1 ->per channel, act_group_size=0->per tensor)Besides, you could modify the `group_size`, `bits`, `sym` and many other configs you want, though there are maybe no real kernels.@@ -131,20 +130,20 @@ Besides, you could modify the `group_size`, `bits`, `sym` and many other configs### Supported export Formats**AutoRound Format**: This format is well-suited for CPU, Intel GPU, CUDA and HPU devices, 2 bits, as well as mixed-precision-inference. **[2,3,4,8] bits are supported**.+inference. **[2,3,4,8] bits are supported**. Please set `--format auto_round`**GGUF** Format: Experimental feature. This format is well-suited for CPU devices and is widely adopted by the-community. `q*_k`,`q*_0`,`q*_1` are supported.+community. `q*_k`,`q*_0`,`q*_1` are supported. Please set `--format gguf:q4_k_m`, `--format gguf:q2_k_s`, etc**AutoGPTQ Format**: This format is well-suited for symmetric quantization on CUDA devices and is widely adopted by thecommunity, **[2,3,4,8] bits are supported**. However, **theasymmetric kernel has issues** that can cause considerable accuracy drops, particularly at 2-bit quantization and small-models. Besides, recently 3 bits may have some accuracy issues in Transformers.+models. Besides, recently 3 bits may have some accuracy issues in Transformers. Please set `--format auto_gptq`**AutoAWQ Format**: This format is well-suited for asymmetric 4-bit quantization on CUDA devices and is widely-adopted within the community, **only 4-bits quantization is supported**.+adopted within the community, **only 4-bits quantization is supported**. Please set `--format auto_awq`-**LLM-Compressor Format**:** NVFP4, MXFP(Kernel is WIP), INT8 are supported**.+**LLM-Compressor Format**: **NVFP4, MXFP4(kernel in WIP), MXFP8 are supported**. Please set `--format llm_compressor`### Hardware Compatibility@@ -158,7 +157,7 @@ CPU, Intel GPU, HPU and CUDA for both quantization and inference.This setting offers a better trade-off between accuracy and tuning cost, and is recommended in all scenarios.```bash- auto-round --model facebook/opt-125m --scheme "W4A16" --format "auto_gptq,auto_awq,auto_round"+ auto-round --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_gptq,auto_awq,auto_round"```- **AutoRoundBest recipe:**@@ -166,7 +165,7 @@ CPU, Intel GPU, HPU and CUDA for both quantization and inference.This setting provides the best accuracy in most scenarios but is 4–5× slower than the standard AutoRound recipe. It is especially recommended for 2-bit quantization and is a good choice if sufficient resources are available.```bash- auto-round-best --model facebook/opt-125m --scheme "W4A16" --format "auto_gptq,auto_awq,auto_round"+ auto-round-best --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_gptq,auto_awq,auto_round"```- **AutoRoundLight Settings:**@@ -174,7 +173,7 @@ CPU, Intel GPU, HPU and CUDA for both quantization and inference.This setting offers the best speed (2-3X faster than AutoRound), but it may cause a significant accuracy drop for small models and 2-bit quantization. It is recommended for 4-bit settings and models larger than 3B```bash- auto-round-light --model facebook/opt-125m --scheme "W4A16" --format "auto_gptq,auto_awq,auto_round"+ auto-round-light --model Qwen/Qwen3-0.6B --scheme "W4A16" --format "auto_gptq,auto_awq,auto_round"```### API usage@@ -184,7 +183,7 @@ This setting offers a better trade-off between accuracy and tuning cost, and is```pythonfrom auto_round import AutoRound-model_name_or_path = "facebook/opt-125m"+model_name_or_path = "Qwen/Qwen3-0.6B"ar = AutoRound(model_name_or_path,scheme="W4A16",@@ -206,7 +205,7 @@ vLLM and SGLang fuse MoE and QKV layers, so it's recommended not to assign diffe```pythonfrom auto_round import AutoRound-model_name_or_path = "facebook/opt-125m"+model_name_or_path = "Qwen/Qwen3-0.6B"layer_config = { # Supports both full layer names and fuzzy (partial) matching"model.decoder.layers.6.self_attn.out_proj": {"bits": 8, "group_size": 32},@@ -226,7 +225,7 @@ This setting provides the best accuracy in most scenarios but is 4–5× slower```pythonfrom auto_round import AutoRound-model_name_or_path = "facebook/opt-125m"+model_name_or_path = "Qwen/Qwen3-0.6B"ar = AutoRound(model=model_name_or_path, scheme="W4A16", nsamples=512, iters=1000, low_gpu_mem_usage=True)output_dir = "./tmp_autoround"@@ -238,7 +237,7 @@ This setting offers the best speed (2 - 3X faster than AutoRound), but it may ca```pythonfrom auto_round import AutoRound-model_name_or_path = "facebook/opt-125m"+model_name_or_path = "Qwen/Qwen3-0.6B"ar = AutoRound(model=model_name_or_path,@@ -331,7 +330,7 @@ shared_layers = [("fc1", "fc2"),]target_bits = 5.0-model_name = "facebook/opt-125m"+model_name = "Qwen/Qwen3-0.6B"scheme = AutoScheme(avg_bits=target_bits, options=("W4A16", "MXFP8"), shared_layers=shared_layers)ar = AutoRound(model=model_name, scheme=scheme, iters=0, nsamples=1)model, layer_config = ar.quantize()@@ -391,7 +390,7 @@ For the GGUF format, we have optimized the RTN algorithm inspired by llamacpp. T```pythonfrom auto_round import AutoRound-model_name_or_path = "facebook/opt-125m"+model_name_or_path = "Qwen/Qwen3-0.6B"ar = AutoRound(model=model_name_or_path,scheme="W4A16",@@ -408,7 +407,7 @@ This format is well-suited for CPU devices and is widely adopted by the communit```pythonfrom auto_round import AutoRound-model_name_or_path = "facebook/opt-125m"+model_name_or_path = "Qwen/Qwen3-0.6B"ar = AutoRound(model=model_name_or_path,)@@ -454,7 +453,7 @@ If adjusting hyperparameters does not resolve the issue a, a simple solution is~~~pythonfrom auto_round import AutoRound-model_name_or_path = "facebook/opt-125m"+model_name_or_path = "Qwen/Qwen3-0.6B"ar = AutoRound(model=model_name_or_path,device_map="0,1,2,3"@@ -464,7 +463,7 @@ ar = AutoRound(or~~~bash-CUDA_VISIBLE_DEVICES=0,1,2,3 auto-round --model "facebook/opt-125m" --scheme "W4A16" --device_map "auto"+CUDA_VISIBLE_DEVICES=0,1,2,3 auto-round --model "Qwen/Qwen3-0.6B" --scheme "W4A16" --device_map "auto"~~~@@ -576,7 +575,7 @@ autoround.save_quantized(format="auto_awq", output_dir="tmp_autoround")Currently only support in AutoRound format inference for this config```bash- auto-round --model_name facebook/opt-125m --scheme "W4A16" --quant_lm_head --format "auto_round"+ auto-round --model_name Qwen/Qwen3-0.6B --scheme "W4A16" --quant_lm_head --format "auto_round"```@@ -717,7 +716,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=Fal- We leverage lm-eval-harnessing for the evaluation.If not explicitly specify '--task', the default value will be used (typically covering 10+ common tasks).~~~bash- auto-round --model facebook/opt-125m --bits 4 --format "auto_round,auto_gptq" --tasks mmlu+ auto-round --model Qwen/Qwen3-0.6B --bits 4 --format "auto_round,auto_gptq" --tasks mmlu~~~The last format will be used in evaluation if multiple formats have been exported.@@ -755,7 +754,12 @@ Note: To use the vllm backend, please add `--vllm` into the upper command.## 6 Known Issues-* Random quantization results in tuning some models-* ChatGlm-V1 is not supported+Randomness in quantization may affect tuning results for some models, set `enable_deterministic_algorithms=True` to ensure reproducibility.+++Some VLMs require manual support.+++Mamba is not supported.