Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

update auto_scheme_acc doc (#2000)

Heng Guo5683ea15c940
diff --git a/docs/auto_scheme_acc.md b/docs/auto_scheme_acc.md
index 8b6981831..864de5c5e 100644
--- a/docs/auto_scheme_acc.md
+++ b/docs/auto_scheme_acc.md
@@ -1,4 +1,4 @@
-### The GGUF results
+### Fake quantization results
We tested with a fake model because the main branch currently has layer name mismatches between Transformers and GGUF.
@@ -30,6 +30,99 @@ evalscope eval --model qwen3-8b --api-url http://127.0.0.1:8001/v1 --api-key E
| qwen3.5-4b: Fake quantized model | 0.505 | 0.3172 | 0.5948 |
+### GGUF model evaluation results
+
+**Unless explicitly stated otherwise, no AutoRound tuning is applied.**
+
+#### Evaluation with lm-eval
+
+```bash
+lm_eval --model hf --model_args pretrained=test_gguf/,gguf_file=Qwen3-8B-Q4_K_M.gguf --device cuda:0 --tasks arc_challenge,arc_easy,boolq,hellaswag,lambada_openai,mmlu,openbookqa,piqa,truthfulqa_mc1,winogrande
+```
+| Qwen3-8B | Quant Type | Actual Bits | Avg Acc | arc_challenge | arc_easy | boolq | hellaswag | lambada_openai | openbookqa | piqa | truthfulqa_mc1 | winogrande | mmlu |
+|--------|------------|-------------|---------|---------------|----------|-------|-----------|----------------|------------|------|----------------|------------|------|
+| f16 | f16 | 16 | 0.6326 | 0.5546 | 0.8333 | 0.8664 | 0.5717 | 0.6416 | 0.3140 | 0.7671 | 0.3672 | 0.6796 | 0.7301 |
+| unsloth | Q2_K | 3.20 | 0.6075 | 0.5145 | 0.8051 | 0.8526 | 0.5309 | 0.6255 | 0.3166 | 0.7530 | 0.3452 | 0.6590 | 0.6729 |
+| bartowski | Q2_K | 3.20 | 0.6071 | 0.5128 | 0.8018 | 0.8511 | 0.5312 | 0.6257 | 0.3100 | 0.7568 | 0.3366 | 0.6677 | 0.6772 |
+| auto_round | 2.5b mixed[q2,q3,q4] | 3.23 | 0.6153 | 0.5239 | 0.8102 | 0.8615 | 0.5242 | 0.6598 | 0.3140 | 0.7503 | 0.3390 | 0.6756 | 0.6943 |
+| auto_round | 2.5b mixed[q2,q3,q4,q6] | 3.24 | 0.6074 | 0.5034 | 0.8131 | 0.8508 | 0.5109 | 0.6571 | 0.3020 | 0.7470 | 0.3317 | 0.6685 | 0.6897 |
+| auto_round | 2.5b mixed[q2,q3,q4] lm_head Q6K | 3.25 | 0.6093 | 0.5060 | 0.8093 | 0.8538 | 0.5119 | 0.6598 | 0.3040 | 0.7459 | 0.3256 | 0.6835 | 0.6929 |
+| auto_round | 2.5b mixed[q2,q3,q4,q6] iters 200 | 3.25 | 0.6133 | 0.5060 | 0.8136 | 0.8541 | 0.5234 | 0.6565 | 0.3040 | 0.7606 | 0.3403 | 0.6827 | 0.6919 |
+| unsloth | Q2_K_L | 3.34 | 0.6078 | 0.5128 | 0.8068 | 0.8529 | 0.5295 | 0.6276 | 0.3189 | 0.7546 | 0.3427 | 0.6567 | 0.6750 |
+| bartowski | Q2_K_L | 3.79 | 0.6076 | 0.5162 | 0.8018 | 0.8449 | 0.5306 | 0.6266 | 0.3120 | 0.7606 | 0.3305 | 0.6717 | 0.6806 |
+| unsloth | Q3_K_M | 4.02 | 0.6260 | 0.5503 | 0.8270 | 0.8596 | 0.5589 | 0.6301 | 0.3120 | 0.7655 | 0.3770 | 0.6677 | 0.7115 |
+| bartowski | Q3_K_M | 4.02 | 0.6246 | 0.5478 | 0.8304 | 0.8606 | 0.5554 | 0.6321 | 0.3200 | 0.7661 | 0.3647 | 0.6575 | 0.7118 |
+| auto_round | 3.5b mixed[q2,q3,q4] | 4.17 | 0.6284 | 0.5478 | 0.8300 | 0.8648 | 0.5651 | 0.6429 | 0.3100 | 0.7612 | 0.3574 | 0.6819 | 0.7225 |
+| auto_round | 3.5b mixed[q2,q3,q4] lm_head Q6K | 4.19 | 0.6251 | 0.5384 | 0.8338 | 0.8609 | 0.5581 | 0.6354 | 0.3160 | 0.7612 | 0.3574 | 0.6725 | 0.7175 |
+| auto_round | 3.5b mixed[q2,q3,q4,q6] | 4.19 | 0.6246 | 0.5401 | 0.8259 | 0.8624 | 0.5582 | 0.6387 | 0.3080 | 0.7628 | 0.3586 | 0.6725 | 0.7191 |
+| auto_round | 3.5b mixed[q2,q3,q4,q6] iters 200 | 4.19 | 0.6318 | 0.5520 | 0.8346 | 0.8676 | 0.5624 | 0.6404 | 0.3160 | 0.7682 | 0.3696 | 0.6890 | 0.7183 |
+| unsloth | Q4_K_M | 4.90 | 0.6308 | 0.5520 | 0.8329 | 0.8691 | 0.5685 | 0.6437 | 0.3180 | 0.7568 | 0.3623 | 0.6803 | 0.7248 |
+| bartowski | Q4_K_M | 4.90 | 0.6301 | 0.5546 | 0.8359 | 0.8654 | 0.5655 | 0.6443 | 0.3120 | 0.7612 | 0.3599 | 0.6748 | 0.7269 |
+
+#### Evaluation with llama-server and EvalScope
+
+```bash
+llama-server \
+ --model test_gguf/Qwen3-8B-Q2_K_S.gguf \
+ --host 127.0.0.1 \
+ --port 8001 \
+ --alias qwen \
+ --ctx-size 65536 \
+ --gpu-layers all \
+ --parallel 16 \
+ --predict 2048 \
+ --reasoning off \
+ --no-webui \
+```
+```bash
+evalscope eval \
+ --model qwen \
+ --api-url http://127.0.0.1:8001/v1 \
+ --api-key EMPTY \
+ --datasets math_500 gpqa_diamond \
+ --eval-batch-size 16
+```
+
+| Qwen3-8B | Quant Type | Actual Bits | gpqa_diamond | math_500 |
+|--------|------------|-------------|--------------|----------|
+| f16 | f16 | 16 | 0.4596 | 0.798 |
+| unsloth | Q2_K | 3.20 | 0.3131 | 0.726 |
+| auto_round | 2.5b mixed[q2,q3,q4] | 3.23 | 0.3636 | 0.706 |
+| auto_round | 2.5b mixed[q2,q3,q4,q6] | 3.24 | 0.3687 | 0.698 |
+| auto_round | 2.5b mixed[q2,q3,q4] lm_head Q6K | 3.25 | 0.3485 | 0.714 |
+| auto_round | 2.5b mixed[q2,q3,q4,q6] iters 200 | 3.25 | 0.3990 | 0.700 |
+| unsloth | Q2_K_L | 3.34 | 0.3788 | 0.732 |
+| unsloth | Q3_K_M | 4.02 | 0.4192 | 0.778 |
+| auto_round | 3.5b mixed[q2,q3,q4] | 4.17 | 0.4495 | 0.816 |
+| auto_round | 3.5b mixed[q2,q3,q4] lm_head Q6K | 4.19 | 0.4495 | 0.788 |
+| auto_round | 3.5b mixed[q2,q3,q4,q6] | 4.19 | 0.4444 | 0.814 |
+| auto_round | 3.5b mixed[q2,q3,q4,q6] iters 200 | 4.19 | 0.5051 | 0.788 |
+| unsloth | Q4_K_M | 4.90 | 0.4545 | 0.792 |
+
+| Qwen3.5-4B | Quant Type | Actual Bits | gpqa_diamond | math_500 |
+|-------------|------------------------------------------------------|-------------|--------------|----------|
+| f16 | f16 | 16 | 0.4697 | 0.79 |
+| unsloth | UD-IQ2_M | 3.33 | 0.2273 | 0.594 |
+| auto_round | 2.5 bits mixed[q2,q3,q4,q6] | 3.9 | 0.4242 | 0.72 |
+| auto_round | 2.5 bits mixed[q2,q3,q4,q6] tuning iters 200 | 3.9 | 0.3838 | 0.754 |
+| auto_round | Q3_K_M | 4.17 | 0.4042 | 0.772 |
+| unsloth | Q3_K_M | 4.34 | 0.4040 | 0.776 |
+| auto_round | 3.5 bits mixed[q2,q3,q4,q6] | 4.86 | 0.4192 | 0.804 |
+| auto_round | 3.5 bits mixed[q2,q3,q4,q6] tuning iters 200 | 4.86 | 0.4192 | 0.786 |
+| auto_round | Q4_K_M | 5.12 | 0.4545 | 0.792 |
+| auto_round | Q4_K_M (repeat 5) | 5.12 | 0.4323 | 0.7958 |
+| unsloth | Q4_K_M | 5.19 | 0.3838 | 0.794 |
+| unsloth | Q4_K_M (repeat 5) | 5.19 | 0.4424 | 0.7906 |
+
+| Gemma-4-12B-IT | Quant Type | Actual Bits | gpqa_diamond | math_500 |
+|-----------------|------------------------------------------------------|-------------|--------------|----------|
+| unsloth | Q3_K_M | 3.81 | 0.5253 | 0.886 |
+| auto_round | 3.5 bits mixed[q2,q3,q4,q6] | 3.99 | 0.4545 | 0.818 |
+| auto_round | 3.5 bits mixed[q2,q3,q4,q6] tuning iters 200 | 3.99 | 0.4949 | 0.87 |
+| auto_round | Q3_K_M | 4.08 | 0.5354 | 0.874 |
+| unsloth | Q4_K_M | 4.77 | 0.5758 | 0.912 |
+| auto_round | Q4_K_M | 4.85 | 0.5808 | 0.9 |
+
### Other results
We use **lm-eval** for evaluation. For LLaMA, we enabled `add_bos_token` and