PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
MXFP4/NVFP4 Results
MXFP4
| Model | ARC-c | ARC-e | BoolQ | Hella | LAMB | MMLU | OBQA | PIQA | Truth. | Wino | GSM8K strict | AVG. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-8BBF16 | 55.46 | 83.38 | 86.73 | 57.04 | 64.00 | 72.96 | 31.20 | 76.61 | 36.72 | 68.27 | 87.57 | 65.45 |
| RTN | 49.23 | 76.30 | 85.63 | 51.93 | 58.45 | 66.18 | 27.40 | 73.12 | 33.29 | 64.48 | 79.98 | 60.54 |
| OPT-RTN(-2,-1,1,2) | 50.17 | 79.25 | 84.98 | 53.18 | 60.20 | 66.88 | 28.80 | 73.83 | 34.88 | 65.43 | 82.03 | 61.78 |
| OPT-RTN(-1,1) (Our's default) | 49.83 | 79.04 | 85.54 | 53.02 | 60.00 | 67.26 | 28.60 | 73.67 | 35.01 | 63.77 | 82.94 | 61.70 |
| Llama3.1-8B-I BF16 | 51.54 | 81.78 | 84.04 | 59.13 | 73.14 | 67.96 | 33.00 | 80.03 | 37.21 | 73.80 | 75.28 | 65.17 |
| orig-RTN | 46.08 | 76.64 | 80.37 | 55.60 | 61.42 | 57.11 | 32.00 | 75.52 | 28.15 | 69.69 | 46.70 | 57.21 |
| OPT-RTN(-2,-1,1,2) | 45.48 | 77.78 | 80.83 | 56.10 | 64.62 | 58.81 | 32.40 | 76.44 | 29.01 | 69.69 | 52.08 | 58.48 |
| OPT-RNT(-1,1) (Our's default) | 46.25 | 76.39 | 80.86 | 55.88 | 65.17 | 58.77 | 33.60 | 77.64 | 30.48 | 70.56 | 54.89 | 59.14 |
NVFP4
| Model | ARC-c | ARC-e | BoolQ | Hella | LAMB | MMLU | OBQA | PIQA | Truth. | Wino | GSM8K strict | AVG. |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| Qwen3-8B | 55.46 | 83.38 | 86.73 | 57.04 | 64.00 | 72.96 | 31.20 | 76.61 | 36.72 | 68.27 | 87.57 | 65.45 |
| RTN | 54.35 | 81.65 | 86.02 | 55.36 | 62.31 | 70.93 | 30.20 | 75.79 | 34.76 | 68.59 | 84.15 | 64.01 |
| opt-rtn | 54.69 | 82.24 | 85.72 | 55.09 | 63.77 | 71.33 | 31.60 | 75.52 | 35.74 | 66.30 | 86.35 | 64.40 |
| Llama3.1-8B-I | 51.54 | 81.78 | 84.04 | 59.13 | 73.14 | 67.96 | 33.00 | 80.03 | 37.21 | 73.80 | 75.28 | 65.17 |
| RTN | 49.15 | 79.25 | 83.24 | 58.29 | 71.20 | 64.05 | 35.00 | 78.62 | 33.41 | 72.85 | 64.06 | 62.65 |
| opt-rtn | 51.02 | 80.13 | 84.53 | 58.40 | 71.34 | 64.76 | 35.60 | 79.43 | 35.62 | 72.93 | 69.29 | 63.91 |
🧮 Evaluation Results (LM-Eval)
For 2/3bit, we strongly recommend not using iter=0 except for GGUF:Q2_K_S which has a different quantization algorithm.
4BIT=W4A16 3BIT=W3A16 2BIT=W2A16G64
RTN mode
auto-round --model xxx --disable_opt_rtn --iters 0
OPT RTN mode
auto-round --model xxx --iters 0
| Model | RNT/OPT | AVG | HellaSwag | LAMBADA | MMLU | PIQA | WinoGrande |
|---|---|---|---|---|---|---|---|
| Meta-Llama-3.1-8B-Instruct | RTN-4BIT | 0.69328 | 0.5896 | 0.7013 | 0.6538 | 0.7987 | 0.7230 |
| OPT-4BIT | 0.69560 | 0.5882 | 0.7074 | 0.6631 | 0.7916 | 0.7277 | |
| RTN-3BIT | 0.64562 | 0.5410 | 0.6695 | 0.5449 | 0.7742 | 0.6985 | |
| OPT-3BIT | 0.65970 | 0.5490 | 0.6893 | 0.5711 | 0.7677 | 0.7214 | |
| RTN-2BIT | 0.33008 | 0.2918 | 0.0474 | 0.2321 | 0.5740 | 0.5051 | |
| OPT-2BIT | 0.38908 | 0.3241 | 0.1560 | 0.2822 | 0.6235 | 0.5596 | |
| Qwen2.5-7B-Instruct | RTN-4BIT | 0.69560 | 0.6114 | 0.6713 | 0.7011 | 0.7878 | 0.7064 |
| OPT-4BIT | 0.70034 | 0.6143 | 0.6945 | 0.7115 | 0.7845 | 0.6969 | |
| RTN-3BIT | 0.64144 | 0.5585 | 0.6092 | 0.6455 | 0.7476 | 0.6464 | |
| OPT-3BIT | 0.66764 | 0.5756 | 0.7013 | 0.6597 | 0.7481 | 0.6535 | |
| RTN-2BIT | 0.31856 | 0.2804 | 0.0351 | 0.2379 | 0.5256 | 0.5138 | |
| OPT-2BIT | 0.45146 | 0.3645 | 0.2992 | 0.4043 | 0.6415 | 0.5478 | |
| Qwen3-8B | RTN-4BIT | 0.66240 | 0.5619 | 0.6150 | 0.7077 | 0.7573 | 0.6701 |
| OPT-4BIT | 0.66992 | 0.5619 | 0.6346 | 0.7102 | 0.7633 | 0.6796 | |
| RTN-3BIT | 0.57322 | 0.4992 | 0.4260 | 0.6002 | 0.7361 | 0.6046 | |
| OPT-3BIT | 0.63698 | 0.5226 | 0.5814 | 0.6718 | 0.7437 | 0.6654 | |
| RTN-2BIT | 0.31150 | 0.2679 | 0.0041 | 0.2536 | 0.5283 | 0.5036 | |
| OPT-2BIT | 0.44254 | 0.3749 | 0.2005 | 0.4202 | 0.6670 | 0.5501 | |
| Qwen3-14B | RTN-4BIT | 0.70448 | 0.5999 | 0.6511 | 0.7565 | 0.7998 | 0.7151 |
| OPT-4BIT | 0.70798 | 0.6031 | 0.6627 | 0.7534 | 0.8009 | 0.7198 | |
| RTN-3BIT | 0.65876 | 0.5746 | 0.5467 | 0.7065 | 0.7628 | 0.7032 | |
| OPT-3BIT | 0.68610 | 0.5683 | 0.6633 | 0.7258 | 0.7699 | 0.7032 | |
| RTN-2BIT | 0.39398 | 0.3764 | 0.0607 | 0.3836 | 0.6480 | 0.5012 | |
| OPT-2BIT | 0.50080 | 0.4554 | 0.2451 | 0.4899 | 0.7138 | 0.5998 |