Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

2.6 KB
Raw Blame

If you are evaluating LLaMA models with recent versions of Transformers, please remove @use_kernel_forward_from_hub("RMSNorm") in modeling_llama.py and enable add_bos_token(this is set as default in AutoRound) in lm-eval to stabilize the accuracy. These adjustments affect the quantized model but not the BF16 model for the tasks evaluated in the AutoRoundv2 paper.

All other settings follow the default configurations of AutoRound and lm-eval.

Qwen3-8B W2G64 Avg. arc_challenge hellaswag gsm8k lambada_openai mmlu mmlupro truthfulqa_mc1 winogrande
AutoRound 0.4373 0.4019 0.4437 0.4215 0.4826 0.5474 0.2630 0.3072 0.6314
AutoRound+alg_ext 0.4787 0.4275 0.4516 0.5944 0.5181 0.5773 0.2807 0.3305 0.6496
AutoRoundBest+alg_ext lr 2e-3 0.4937 0.4505 0.474 0.5906 0.5556 0.6028 0.3127 0.3109 0.6527
AutoRoundBest_alg_ext lr 4e-3 seqlen4096 nsamples 4096 dataset=finewebedu 0.4977 0.4966 0.4906 0.5171 0.5397 0.6218 0.3381 0.3195 0.6582
AutoRound+alg_ext iters 2000 nsamples 1024 lr 2e-3 0.5100 0.4548 0.4803 0.6505 0.5529 0.6161 0.3272 0.3256 0.6725
Llama3.1-8B-Instruct W2G64 Avg. arc_challenge hellaswag gsm8k lambada_openai mmlu mmlupro truthfulqa_mc1 winogrande
AutoRound 0.3820 0.3635 0.4562 0.1622 0.5069 0.4411 0.1661 0.3207 0.6393
AutoRound+alg_ext 0.4166 0.3712 0.4729 0.2039 0.5946 0.4981 0.2163 0.3011 0.6748
AutoRoundBest+alg_ext lr 2e-3 0.4539 0.4138 0.4999 0.3071 0.6233 0.5279 0.2364 0.3231 0.6993