Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

7.2 KB
Raw Blame

Calibration Dataset Study (WIP)

Current Recommendation: For 4-bit quantization, current results suggest 75% UltraChat + 25% OpenCode (Mix2 recipe). With the default nsamples=128, this recipe can be applied using:

--dataset 'HuggingFaceH4/ultrachat_200k:apply_chat_template=true:num=96,nvidia/OpenCodeInstruct:num=32'

Qwen3.8-27B-W4A16

Calibration Dataset AIME26 GPQA Diamond IFBench Strict MMLU-Pro LiveCodeBench SWE-mini Terminal_Bench
cci3hq 96.67
96.67 / 90 / 100 / 100 / 96.67
90.40
91.92 / 90.4 / 88.89 / 91.41 / 89.39
74.92
73.67 / 76.16
85.03 91.18 – –
OpenCode 98.00
96.67 / 96.67 / 100 / 100 / 96.67
89.27
89.39 / 87.88 / 90.4 / 89.39 / –
75.23
74 / 76.45
85.26 92.51 58.00 49.40
pile10k 97.33
100 / 93.33 / 100 / 93.33 / 100
89.09
87.88 / 88.38 / 88.89 / 89.9 / 90.4
73.67
72.33 / 75
85.00 91.37 – –
UltraChat 98.00
100 / 93.33 / 100 / 100 / 96.67
90.78
90.91 / 91.41 / 90.4 / 90.4 / –
74.75
73.33 / 76.16
85.20 91.75 56.00 41.60
UltraFineWeb 97.33
96.67 / 100 / 100 / 96.67 / 93.33
89.90
89.39 / 90.4 / 90.4 / 90.4 / 88.89
71.63
70 / 73.26
84.96 91.75 – –
mix_a (UltraChat 50% + OpenCode 25% + cci3hq 12.5% + pile10k 12.5%) 99.34
100 / 100 / 100 / 96.7 / 100
90.10
90.9 / 89.4 / 90.9 / 90.4 / 88.9
73.65
72.3 / 75
85.40 92.70 60.00 –
mix_b (UltraChat 75% + OpenCode 25%) 98.02
96.7 / 100 / 96.7 / 100 / 96.7
90.40
87.9 / 90.9 / 89.9 / 91.9 / 91.4
76.00
74.7 / 77.3
85.50 92.20 56.90 –
mix_b (UltraChat 75%, apply_chat_template=true + OpenCode 25%) 99.18
100 / 100 / 96.7 / 100 / –
89.53
87.9 / 90.4 / 89.4 / 90.4 / –
– – – – –
mix_c (UltraChat 87.5% + OpenCode 12.5%) 96.67
93.3 / 100 / 96.7 / – / –
88.40
88.4 / 90.4 / 86.4 / – / –
73.00
71.3 / 74.7
– – – –

Gemma-4-12B-it-W4A16

Calibration Dataset AIME26 GPQA Diamond IFBench Strict
cci3hq 49.34
46.7 / 46.7 / 50 / 50 / 53.3
38.70
42.4 / 35.4 / 36.9 / 40.4 / 38.4
58.70
59 / 58.4
OpenCode 66.00
63.3 / 76.7 / 60 / 60 / 70
52.00
51.5 / 50.5 / 54 / – / –
64.25
63.7 / 64.8
pile10k 58.66
53.3 / 53.3 / 66.7 / 56.7 / 63.3
– 65.95
65.3 / 66.6
UltraChat 68.00
66.7 / 63.3 / 66.7 / 70 / 73.3
62.60
61.6 / 61.1 / 65.1 / – / –
68.75
67.7 / 69.8
UltraFineWeb 57.34
56.7 / 60 / 60 / 56.7 / 53.3
– 67.80
66.7 / 68.9
mix_a (UltraChat 50% + OpenCode 25% + cci3hq 12.5% + pile10k 12.5%) 56.70 53.50 67.15
66.3 / 68
mix_b (UltraChat 75% + OpenCode 25%) 76.70 58.60 67.95
mix_b (UltraChat 75%, apply_chat_template=true + OpenCode 25%) 70.00
73.3 / 66.7 / 76.7 / 80 / 53.3
64.32
65.1 / 63.6 / 65.7 / 62.1 / 65.1
69.05
68 / 70.1
mix_c (UltraChat 87.5% + OpenCode 12.5%) 66.68
66.7 / 66.7 / 66.7 / 63.3 / 70
62.82
60.1 / 61.6 / 66.7 / 64.6 / 61.1
68.75
67.7 / 69.8

Reproduction Setup

(1) Environment

  • Quantization & vLLM: Python 3.11.17, AutoRound 0.16.0, PyTorch 2.13.0+cu130, vLLM 0.29.0
  • Standard Evaluation: Python 3.11.17, EvalScope 1.12.0
  • Agent Evaluation: Python 3.12.15, EvalScope 1.12.0

(2) Model Configurations

Qwen

  • Hardware: H200 / B200
  • Quantization: W4A16g128
  • Generation: temperature=1, top_p=1, top_k=20, min_p=0
  • Max tokens: 131,072
  • Evaluation: batch_size=32, timeout=14,400s, seed=42
  • Thinking: enabled, preserve_thinking=true
  • Streaming: disabled

Gemma

  • Model: /models/gemma-4-12B-it
  • Served model name: gemma4_bf16
  • Hardware: A100
  • Dtype: bfloat16
  • Thinking: enabled
  • Reasoning parser: gemma4
  • Generation: temperature=1, top_p=0.95, top_k=64, min_p=0
  • Repetition penalty: 1
  • Max tokens: 32,768
  • Evaluation: batch_size=16, timeout=7,200s
  • Enforce eager: enabled
  • Max model length: not explicitly set

Benchmark Datasets

Benchmark Samples
AIME26 30
GPQA Diamond 198
IFBench (Prompts & Instructions; Strict Prompt Strict, Instruction Strict = avg) 300 & 344
LiveCodeBench (release_v6) 1,055
MMLU-Pro 12,032
SWE-bench Verified Mini 50