Fake quantization results
We tested with a fake model because the main branch currently has layer name mismatches between Transformers and GGUF.
python3 -m auto_round Qwen/Qwen3-8B --schemes "gguf:q2_k_s,gguf:q4_k_s" --bits 3.5 --ignore_scale_zp_bits --iters 0 --format fake --output_dir "./test_gguf"
eval
# Start vLLM serve
vllm serve ./test_gguf/Qwen3-8B-w2g16/ --port 8000 --max-model-len 8192 --host 127.0.0.1 --served-model-name qwen3-8b
# Perform five repeated evaluations on math_500 and gpqa_diamond.
evalscope eval --model qwen3-8b --api-url http://127.0.0.1:8000/v1 --api-key EMPTY --datasets math_500 gpqa_diamond --eval-batch-size 64 --generation-config "{\"n\":5}"
# Evaluate mmlu_pro
evalscope eval --model qwen3-8b --api-url http://127.0.0.1:8001/v1 --api-key EMPTY --datasets mmlu_pro --eval-batch-size 128
| evalscope, options q2ks,q4ks avgbits 3.5, ignore_scale_zp |
math_500 (repeat=5) |
gpqa_diamond (repeat=5) |
mmlu_pro |
| qwen3-8b: BF16 |
0.8083 |
0.4586 |
0.6934 |
| qwen3-8b: Fake quantized model |
0.7924 |
0.4313 |
0.6751 |
| evalscope, options q2ks,q4ks avgbits 3.5, ignore_scale_zp |
math_500 (repeat=5) |
gpqa_diamond (repeat=5) |
mmlu_pro |
| qwen3.5-4b: BF16 |
0.5365 |
0.3263 |
0.5891 |
| qwen3.5-4b: Fake quantized model |
0.505 |
0.3172 |
0.5948 |
GGUF model evaluation results
Unless explicitly stated otherwise, no AutoRound tuning is applied.
Evaluation with lm-eval
lm_eval --model hf --model_args pretrained=test_gguf/,gguf_file=Qwen3-8B-Q4_K_M.gguf --device cuda:0 --tasks arc_challenge,arc_easy,boolq,hellaswag,lambada_openai,mmlu,openbookqa,piqa,truthfulqa_mc1,winogrande
| Qwen3-8B |
Quant Type |
Actual Bits |
Avg Acc |
arc_challenge |
arc_easy |
boolq |
hellaswag |
lambada_openai |
openbookqa |
piqa |
truthfulqa_mc1 |
winogrande |
mmlu |
| f16 |
f16 |
16 |
0.6326 |
0.5546 |
0.8333 |
0.8664 |
0.5717 |
0.6416 |
0.3140 |
0.7671 |
0.3672 |
0.6796 |
0.7301 |
| unsloth |
Q2_K |
3.20 |
0.6075 |
0.5145 |
0.8051 |
0.8526 |
0.5309 |
0.6255 |
0.3166 |
0.7530 |
0.3452 |
0.6590 |
0.6729 |
| bartowski |
Q2_K |
3.20 |
0.6071 |
0.5128 |
0.8018 |
0.8511 |
0.5312 |
0.6257 |
0.3100 |
0.7568 |
0.3366 |
0.6677 |
0.6772 |
| auto_round |
2.5b mixed[q2,q3,q4] |
3.23 |
0.6153 |
0.5239 |
0.8102 |
0.8615 |
0.5242 |
0.6598 |
0.3140 |
0.7503 |
0.3390 |
0.6756 |
0.6943 |
| auto_round |
2.5b mixed[q2,q3,q4,q6] |
3.24 |
0.6074 |
0.5034 |
0.8131 |
0.8508 |
0.5109 |
0.6571 |
0.3020 |
0.7470 |
0.3317 |
0.6685 |
0.6897 |
| auto_round |
2.5b mixed[q2,q3,q4] lm_head Q6K |
3.25 |
0.6093 |
0.5060 |
0.8093 |
0.8538 |
0.5119 |
0.6598 |
0.3040 |
0.7459 |
0.3256 |
0.6835 |
0.6929 |
| auto_round |
2.5b mixed[q2,q3,q4,q6] iters 200 |
3.25 |
0.6133 |
0.5060 |
0.8136 |
0.8541 |
0.5234 |
0.6565 |
0.3040 |
0.7606 |
0.3403 |
0.6827 |
0.6919 |
| unsloth |
Q2_K_L |
3.34 |
0.6078 |
0.5128 |
0.8068 |
0.8529 |
0.5295 |
0.6276 |
0.3189 |
0.7546 |
0.3427 |
0.6567 |
0.6750 |
| bartowski |
Q2_K_L |
3.79 |
0.6076 |
0.5162 |
0.8018 |
0.8449 |
0.5306 |
0.6266 |
0.3120 |
0.7606 |
0.3305 |
0.6717 |
0.6806 |
| unsloth |
Q3_K_M |
4.02 |
0.6260 |
0.5503 |
0.8270 |
0.8596 |
0.5589 |
0.6301 |
0.3120 |
0.7655 |
0.3770 |
0.6677 |
0.7115 |
| bartowski |
Q3_K_M |
4.02 |
0.6246 |
0.5478 |
0.8304 |
0.8606 |
0.5554 |
0.6321 |
0.3200 |
0.7661 |
0.3647 |
0.6575 |
0.7118 |
| auto_round |
3.5b mixed[q2,q3,q4] |
4.17 |
0.6284 |
0.5478 |
0.8300 |
0.8648 |
0.5651 |
0.6429 |
0.3100 |
0.7612 |
0.3574 |
0.6819 |
0.7225 |
| auto_round |
3.5b mixed[q2,q3,q4] lm_head Q6K |
4.19 |
0.6251 |
0.5384 |
0.8338 |
0.8609 |
0.5581 |
0.6354 |
0.3160 |
0.7612 |
0.3574 |
0.6725 |
0.7175 |
| auto_round |
3.5b mixed[q2,q3,q4,q6] |
4.19 |
0.6246 |
0.5401 |
0.8259 |
0.8624 |
0.5582 |
0.6387 |
0.3080 |
0.7628 |
0.3586 |
0.6725 |
0.7191 |
| auto_round |
3.5b mixed[q2,q3,q4,q6] iters 200 |
4.19 |
0.6318 |
0.5520 |
0.8346 |
0.8676 |
0.5624 |
0.6404 |
0.3160 |
0.7682 |
0.3696 |
0.6890 |
0.7183 |
| unsloth |
Q4_K_M |
4.90 |
0.6308 |
0.5520 |
0.8329 |
0.8691 |
0.5685 |
0.6437 |
0.3180 |
0.7568 |
0.3623 |
0.6803 |
0.7248 |
| bartowski |
Q4_K_M |
4.90 |
0.6301 |
0.5546 |
0.8359 |
0.8654 |
0.5655 |
0.6443 |
0.3120 |
0.7612 |
0.3599 |
0.6748 |
0.7269 |
Evaluation with llama-server and EvalScope
llama-server \
--model test_gguf/Qwen3-8B-Q2_K_S.gguf \
--host 127.0.0.1 \
--port 8001 \
--alias qwen \
--ctx-size 65536 \
--gpu-layers all \
--parallel 16 \
--predict 2048 \
--reasoning off \
--no-webui \
evalscope eval \
--model qwen \
--api-url http://127.0.0.1:8001/v1 \
--api-key EMPTY \
--datasets math_500 gpqa_diamond \
--eval-batch-size 16
| Qwen3-8B |
Quant Type |
Actual Bits |
gpqa_diamond |
math_500 |
| f16 |
f16 |
16 |
0.4596 |
0.798 |
| unsloth |
Q2_K |
3.20 |
0.3131 |
0.726 |
| auto_round |
2.5b mixed[q2,q3,q4] |
3.23 |
0.3636 |
0.706 |
| auto_round |
2.5b mixed[q2,q3,q4,q6] |
3.24 |
0.3687 |
0.698 |
| auto_round |
2.5b mixed[q2,q3,q4] lm_head Q6K |
3.25 |
0.3485 |
0.714 |
| auto_round |
2.5b mixed[q2,q3,q4,q6] iters 200 |
3.25 |
0.3990 |
0.700 |
| unsloth |
Q2_K_L |
3.34 |
0.3788 |
0.732 |
| unsloth |
Q3_K_M |
4.02 |
0.4192 |
0.778 |
| auto_round |
3.5b mixed[q2,q3,q4] |
4.17 |
0.4495 |
0.816 |
| auto_round |
3.5b mixed[q2,q3,q4] lm_head Q6K |
4.19 |
0.4495 |
0.788 |
| auto_round |
3.5b mixed[q2,q3,q4,q6] |
4.19 |
0.4444 |
0.814 |
| auto_round |
3.5b mixed[q2,q3,q4,q6] iters 200 |
4.19 |
0.5051 |
0.788 |
| unsloth |
Q4_K_M |
4.90 |
0.4545 |
0.792 |
| Qwen3.5-4B |
Quant Type |
Actual Bits |
gpqa_diamond |
math_500 |
| f16 |
f16 |
16 |
0.4697 |
0.79 |
| unsloth |
UD-IQ2_M |
3.33 |
0.2273 |
0.594 |
| auto_round |
2.5 bits mixed[q2,q3,q4,q6] |
3.9 |
0.4242 |
0.72 |
| auto_round |
2.5 bits mixed[q2,q3,q4,q6] tuning iters 200 |
3.9 |
0.3838 |
0.754 |
| auto_round |
Q3_K_M |
4.17 |
0.4042 |
0.772 |
| unsloth |
Q3_K_M |
4.34 |
0.4040 |
0.776 |
| auto_round |
3.5 bits mixed[q2,q3,q4,q6] |
4.86 |
0.4192 |
0.804 |
| auto_round |
3.5 bits mixed[q2,q3,q4,q6] tuning iters 200 |
4.86 |
0.4192 |
0.786 |
| auto_round |
Q4_K_M |
5.12 |
0.4545 |
0.792 |
| auto_round |
Q4_K_M (repeat 5) |
5.12 |
0.4323 |
0.7958 |
| unsloth |
Q4_K_M |
5.19 |
0.3838 |
0.794 |
| unsloth |
Q4_K_M (repeat 5) |
5.19 |
0.4424 |
0.7906 |
| Gemma-4-12B-IT |
Quant Type |
Actual Bits |
gpqa_diamond |
math_500 |
| unsloth |
Q3_K_M |
3.81 |
0.5253 |
0.886 |
| auto_round |
3.5 bits mixed[q2,q3,q4,q6] |
3.99 |
0.4545 |
0.818 |
| auto_round |
3.5 bits mixed[q2,q3,q4,q6] tuning iters 200 |
3.99 |
0.4949 |
0.87 |
| auto_round |
Q3_K_M |
4.08 |
0.5354 |
0.874 |
| unsloth |
Q4_K_M |
4.77 |
0.5758 |
0.912 |
| auto_round |
Q4_K_M |
4.85 |
0.5808 |
0.9 |
Other results
We use lm-eval for evaluation. For LLaMA, we enabled add_bos_token and
removed @use_kernel_forward_from_hub("RMSNorm")
in modeling_llama.py
to stabilize accuracy during evaluation. All other settings follow the default configurations of AutoRound and lm-eval.
We ignore the scale and zp bits in the tables below. The accuracy may change a little as we modified a little of the
implementation. We will rerun all the experiments.
For mxfp experiment, we use fake model while for weight only model we use real model. No tuning is applied unless
explicitly stated.
Average accuracy across lambada_openai, hellaswag, piqa, winogrande, and mmlu.
Table 1 MXFP4/8 mixed accuracy.
| Average bits |
Llama3.1-8B-I |
Qwen2.5-7B-I |
Qwen3-8B |
Qwen3-32B |
| BF16 |
0.7076 (100%) |
0.7075 (100%) |
0.6764 (100%) |
0.7321 (100%) |
| Pure 4-bit |
0.6626 (93.6%) |
0.6550 (92.6%) |
0.6316 (93.4%) |
0.6901 (94.3%) |
| Ours 4.5-bit |
0.6808 (96.2%) |
0.6776 (95.8%) |
0.6550 (96.8%) |
0.7176 (98.0%) |
| Ours 5-bit |
0.6857 (96.9%) |
0.6823 (96.4%) |
0.6594 (97.5%) |
0.7201 (98.3%) |
| Ours 6-bit |
0.6975 (98.6%) |
0.6970 (98.5%) |
0.6716 (99.3%) |
0.7303 (99.8%) |
We compare the proposed method against naive layer-wise bit allocation strategies, such as assigning higher
precision to the network’s head((near lm-head) or tailad(close to embedding)) layers, to demonstrate its relative
performance advantages.
Table 2 Comparison with other recipes at an average of 5 bits of mxfp datatype
| Avg. bits = 5 |
Llama3.1-8B-I |
Qwen2.5-7B-I |
Qwen3-8B |
| Tail layers 8-bit |
0.6671 (94.3%) |
0.6616 (93.5%) |
0.6410 (94.8%) |
| Head layers 8-bit |
0.6657 (94.1%) |
0.6686 (94.5%) |
0.6356 (94.0%) |
| Ours |
0.6857 (96.9%) |
0.6823 (96.4%) |
0.6594 (97.5%) |
Table 3 Comparison with other recipes at an average of 4.5 bits of mxfp datatype
| Avg. bits = 4.5 |
Llama3.1-8B-I |
Qwen2.5-7B-I |
Qwen3-8B |
| Tail layers 8-bit |
0.6614 (93.5%) |
0.6535 (92.4%) |
0.6373 (94.2%) |
| Head layers 8-bit |
0.6568 (92.8%) |
0.6642 (93.9%) |
0.6305 (93.2%) |
| Ours |
0.6808 (96.2%) |
0.6776 (95.5%) |
0.6550 (95.8%) |
Table4 Comparison with other recipes at an average of 3 bits of W2G128 and W4G128
| Avg. bits = 4.5 |
Llama3.1-8B-I |
Qwen2.5-7B-I |
Qwen3-8B |
| Tail layers 4-bit |
0.6058 |
0.3798 |
0.4536 |
| Head layers 4-bit |
0.3198 |
0.3270 |
0.3196 |
| Ours |
0.6148 |
0.4058 |
0.4862 |