Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

2.2 KB
Raw Blame

Below accuracy results are got by lm-eval.

cmd:

# model quantization
auto-round --model model_name_or_path --scheme FP8_BLOCK --iters 0 --format fp8 # RTN
auto-round --model model_name_or_path --scheme FP8_BLOCK --format fp8 # Tuning


# accuracy evaluation
# --apply_chat_template --fewshot_as_multiturn are required for gsm8k_llama,mmlu_llama,mmlu_pro_llama tasks
# `add_bos_token=true` is only required in model_args for llama evaluation
lm_eval --model vllm --model_args pretrained=model_path,add_bos_token=true,tensor_parallel_size=1,gpu_memory_utilization=0.8,max_model_len=8192 --tasks tasks --batch_size 128

LLaMa-3-8B-Instruct

Scheme arc_challenge arc_easy boolq hellaswag lambada_openai openbookqa piqa truthfulqa_mc1 winogrande gsm8k_llama mmlu_llama mmlu_pro_llama Avg.
BF16 0.5358 0.8232 0.8459 0.5795 0.7266 0.3540 0.7786 0.3721 0.7135 0.7369 0.6741 0.4334 0.6311
FP8_BLOCK RTN 0.5316 0.8173 0.8468 0.5787 0.7256 0.3560 0.7824 0.3684 0.7103 0.7369 0.6661 0.4358 0.6297
FP8_BLOCK Tuning 0.5401 0.8228 0.8459 0.5786 0.7275 0.3480 0.7769 0.3684 0.7103 0.7437 0.6727 0.4315 0.6305

Qwen3-8B

Scheme arc_challenge arc_easy boolq hellaswag lambada_openai openbookqa piqa truthfulqa_mc1 winogrande gsm8k mmlu mmlu_pro Avg.
BF16 0.5580 0.8342 0.8667 0.5712 0.6517 0.3120 0.7688 0.3647 0.6788 0.8726 0.7292 0.6214 0.6524
FP8_BLOCK RTN 0.5538 0.8350 0.8691 0.5713 0.6468 0.3140 0.7661 0.3721 0.6780 0.8704 0.7269 0.6204 0.6520
FP8_BLOCK Tuning 0.5520 0.8384 0.8703 0.5710 0.6519 0.3260 0.7671 0.3635 0.6725 0.8681 0.7302 0.6168 0.6523