Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

5.8 KB
Raw Blame

MXFP4/NVFP4 Results

MXFP4

Model ARC-c ARC-e BoolQ Hella LAMB MMLU OBQA PIQA Truth. Wino GSM8K strict AVG.
Qwen3-8BBF16 55.46 83.38 86.73 57.04 64.00 72.96 31.20 76.61 36.72 68.27 87.57 65.45
RTN 49.23 76.30 85.63 51.93 58.45 66.18 27.40 73.12 33.29 64.48 79.98 60.54
OPT-RTN(-2,-1,1,2) 50.17 79.25 84.98 53.18 60.20 66.88 28.80 73.83 34.88 65.43 82.03 61.78
OPT-RTN(-1,1) (Our's default) 49.83 79.04 85.54 53.02 60.00 67.26 28.60 73.67 35.01 63.77 82.94 61.70
Llama3.1-8B-I BF16 51.54 81.78 84.04 59.13 73.14 67.96 33.00 80.03 37.21 73.80 75.28 65.17
orig-RTN 46.08 76.64 80.37 55.60 61.42 57.11 32.00 75.52 28.15 69.69 46.70 57.21
OPT-RTN(-2,-1,1,2) 45.48 77.78 80.83 56.10 64.62 58.81 32.40 76.44 29.01 69.69 52.08 58.48
OPT-RNT(-1,1) (Our's default) 46.25 76.39 80.86 55.88 65.17 58.77 33.60 77.64 30.48 70.56 54.89 59.14

NVFP4

Model ARC-c ARC-e BoolQ Hella LAMB MMLU OBQA PIQA Truth. Wino GSM8K strict AVG.
Qwen3-8B 55.46 83.38 86.73 57.04 64.00 72.96 31.20 76.61 36.72 68.27 87.57 65.45
RTN 54.35 81.65 86.02 55.36 62.31 70.93 30.20 75.79 34.76 68.59 84.15 64.01
opt-rtn 54.69 82.24 85.72 55.09 63.77 71.33 31.60 75.52 35.74 66.30 86.35 64.40
Llama3.1-8B-I 51.54 81.78 84.04 59.13 73.14 67.96 33.00 80.03 37.21 73.80 75.28 65.17
RTN 49.15 79.25 83.24 58.29 71.20 64.05 35.00 78.62 33.41 72.85 64.06 62.65
opt-rtn 51.02 80.13 84.53 58.40 71.34 64.76 35.60 79.43 35.62 72.93 69.29 63.91

🧮 Evaluation Results (LM-Eval)

For 2/3bit, we strongly recommend not using iter=0 except for GGUF:Q2_K_S which has a different quantization algorithm.

4BIT=W4A16 3BIT=W3A16 2BIT=W2A16G64

RTN mode

auto-round --model xxx --disable_opt_rtn --iters 0 

OPT RTN mode

auto-round --model xxx  --iters 0 
Model RNT/OPT AVG HellaSwag LAMBADA MMLU PIQA WinoGrande
Meta-Llama-3.1-8B-Instruct RTN-4BIT 0.69328 0.5896 0.7013 0.6538 0.7987 0.7230
OPT-4BIT 0.69560 0.5882 0.7074 0.6631 0.7916 0.7277
RTN-3BIT 0.64562 0.5410 0.6695 0.5449 0.7742 0.6985
OPT-3BIT 0.65970 0.5490 0.6893 0.5711 0.7677 0.7214
RTN-2BIT 0.33008 0.2918 0.0474 0.2321 0.5740 0.5051
OPT-2BIT 0.38908 0.3241 0.1560 0.2822 0.6235 0.5596
Qwen2.5-7B-Instruct RTN-4BIT 0.69560 0.6114 0.6713 0.7011 0.7878 0.7064
OPT-4BIT 0.70034 0.6143 0.6945 0.7115 0.7845 0.6969
RTN-3BIT 0.64144 0.5585 0.6092 0.6455 0.7476 0.6464
OPT-3BIT 0.66764 0.5756 0.7013 0.6597 0.7481 0.6535
RTN-2BIT 0.31856 0.2804 0.0351 0.2379 0.5256 0.5138
OPT-2BIT 0.45146 0.3645 0.2992 0.4043 0.6415 0.5478
Qwen3-8B RTN-4BIT 0.66240 0.5619 0.6150 0.7077 0.7573 0.6701
OPT-4BIT 0.66992 0.5619 0.6346 0.7102 0.7633 0.6796
RTN-3BIT 0.57322 0.4992 0.4260 0.6002 0.7361 0.6046
OPT-3BIT 0.63698 0.5226 0.5814 0.6718 0.7437 0.6654
RTN-2BIT 0.31150 0.2679 0.0041 0.2536 0.5283 0.5036
OPT-2BIT 0.44254 0.3749 0.2005 0.4202 0.6670 0.5501
Qwen3-14B RTN-4BIT 0.70448 0.5999 0.6511 0.7565 0.7998 0.7151
OPT-4BIT 0.70798 0.6031 0.6627 0.7534 0.8009 0.7198
RTN-3BIT 0.65876 0.5746 0.5467 0.7065 0.7628 0.7032
OPT-3BIT 0.68610 0.5683 0.6633 0.7258 0.7699 0.7032
RTN-2BIT 0.39398 0.3764 0.0607 0.3836 0.6480 0.5012
OPT-2BIT 0.50080 0.4554 0.2451 0.4899 0.7138 0.5998