PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
AWQ Algorithm Results
This document summarizes an experimental comparison between native AutoRound flows and AWQ-composed flows. The goal is to evaluate whether applying AWQ smoothing before RTN or AutoRound improves accuracy for W4A16, MXFP4, and INT8 (W8A8) schemes.
All table results are based on the AutoRound 0.15.0 release, measured with --format fake, and evaluated on mmlu,gsm8k,piqa,hellaswag,winogrande. These five widely used LLM benchmarks cover complementary error modes: MMLU for factual knowledge and broad understanding, GSM8K for mathematical reasoning, PIQA for physical commonsense, HellaSwag for commonsense sentence completion, and Winogrande for pronoun and coreference resolution. Reporting the average score across this task set reduces sensitivity to single-task variance and provides a more stable view of quantization-induced accuracy changes. Unless noted otherwise, the remaining quantization settings follow the AutoRound defaults.
AWQ Algorithm Characteristics
AWQ is an activation-aware weight smoothing method. It searches for per-channel scaling factors from calibration activations, applies the scaling to reduce quantization-sensitive weight channels, and then hands the transformed model to the terminal quantization algorithm.
AWQ changes the pre-quantization weight distribution, while RTN, AutoRound, and AutoRound2 decide how the final quantized values are produced. In this design, AWQ can be composed with different quantizers without changing their core optimization logic.
From the resource perspective, AWQ does not introduce a persistent RAM or VRAM burden in the quantized model. During calibration, AWQ_RTN stays around the same host RAM level as the other flows and uses much less VRAM than iterative AutoRound flows. When AWQ is followed by AutoRound or AutoRound2, the peak VRAM is mainly determined by the downstream AutoRound optimization rather than by AWQ itself.
The main added cost is calibration runtime. This is a one-time offline cost and is acceptable when AWQ provides measurable accuracy gains, especially for MXFP4 and selected W4A16 cases.
The configurations in the tables map to the following CLI usage:
# Pure RTN baseline
auto-round-rtn --model <model> --scheme <scheme> --format fake \
--tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"
# Optimized RTN baseline
auto-round-opt-rtn --model <model> --scheme <scheme> --format fake \
--tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"
# Native AutoRound / AutoRound with algorithm extension
auto-round --model <model> --scheme <scheme> --format fake \
--algorithm auto_round \
--tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"
auto-round --model <model> --scheme <scheme> --format fake \
--algorithm auto_round_v2 \
--tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"
# AWQ + RTN / AWQ + AutoRound / AWQ + AutoRound with algorithm extension
auto-round --model <model> --scheme <scheme> --format fake \
--algorithm awq \
--tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"
auto-round --model <model> --scheme <scheme> --format fake \
--algorithm awq,auto_round \
--tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"
auto-round --model <model> --scheme <scheme> --format fake \
--algorithm awq,auto_round_v2 \
--tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"
Calibration Controls
AWQ has its own calibration budget for activation statistics and for the smoothing and clipping searches. These controls affect AWQ preprocessing only; the downstream RTN or AutoRound quantizer retains the global calibration cache and performs its own final scale search.
| CLI option | AWQConfig field |
Default | Purpose |
|---|---|---|---|
--awq-nsamples |
awq_nsamples |
128 |
Maximum number of calibration samples used by AWQ activation statistics, smoothing, and clipping. |
--awq_seqlen |
awq_seqlen |
512 |
Maximum sequence length used by AWQ calibration; <= 0 disables truncation. |
--awq_smooth_batch_size |
smooth_batch_size |
None |
Micro-batch size for replaying the parent module during the AWQ scale search; lower positive values reduce peak VRAM, while <= 0 disables micro-batching. |
--awq-v2-scale-search |
v2_scale_search |
auto |
Scale-search policy for AWQ's internal QDQ when the terminal quantizer is AutoRoundV2. |
--awq-nsamples is independent of the global --nsamples. AWQ selects a deterministic random subset (seed 42) from the cached calibration inputs and reuses the same subset for every block. The global cache is not modified, so the downstream RTN/AutoRound quantizer and the FP reference forward still use all cached samples. Set --awq-nsamples greater than or equal to --nsamples to use all of them.
--awq-v2-scale-search takes effect only when the terminal quantizer is AutoRoundV2, which is enabled with --enable_alg_ext and --algorithm auto_round (or awq,auto_round). With an RTN terminal quantizer, the option is ignored. The policy changes only the QDQ used to score AWQ candidates; the terminal AutoRoundV2 quantizer always performs its own initialization search.
auto(default): useofffor INT schemes andfullfor non-INT schemes such as MXFP and NVFP.off: use plain RTN for AWQ's candidate loss. This is the fastest option and the recommended default for INT.full: reproduce the terminal V2 optimized init-scale search for every AWQ candidate. This is the slowest and most faithful option.reuse: run the optimized init-scale search once per balance layer and reuse it for every AWQ candidate. This is a middle ground betweenoffandfull.
When smoothing or hard clipping changes a block, AWQ requests a fresh FP reference forward before quantization, so a stale reference is never used as the quantizer target or passed to the next block.
# INT: auto resolves to off; AWQ uses at most 128 samples while the quantizer uses 256.
auto-round --model <model> --scheme W4A16 --format fake \
--algorithm awq,auto_round --enable_alg_ext \
--nsamples 256 --awq-nsamples 128
# MXFP4: auto resolves to full; use reuse to cut AWQ calibration time.
auto-round --model <model> --scheme MXFP4 --format fake \
--algorithm awq,auto_round --enable_alg_ext \
--awq-v2-scale-search reuse
# INT with an explicit full optimized search for every AWQ candidate.
auto-round --model <model> --scheme W4A16 --format fake \
--algorithm awq,auto_round --enable_alg_ext \
--awq-v2-scale-search full
The same settings are available through AWQConfig:
from auto_round import AutoRound, AWQConfig, SignRoundV2Config
ar = AutoRound(
model="<model>",
scheme="W4A16",
alg_configs=[
AWQConfig(awq_nsamples=128, awq_seqlen=512, v2_scale_search="auto"),
SignRoundV2Config(iters=200),
],
)
In the result tables, AR2 denotes AutoRound with --algorithm auto_round_v2 (formerly --enable_alg_ext), and AWQ_AR2 denotes AWQ followed by AutoRound with the same algorithm extension. Accuracy columns are reported as percentages, so a raw score such as 0.7058 is shown as 70.58.
The AWQ rows use the default AWQ smoothing flow. They do not include AWQ weight clipping unless --awq_apply_clip is explicitly enabled.
In the observation tables, AVG Win Rate counts how often the AWQ-composed method improves model-level AVG accuracy for that scheme, Max AVG Delta Gain is the largest absolute AVG improvement in percentage points, and Max AVG Rel. Gain is the largest relative AVG improvement. For AWQ_RTN, the comparison includes both RTN and opt_rtn when both baselines are available.
MXFP4
| Model | Config | GSM8K (%) | HellaSwag (%) | MMLU (%) | PIQA (%) | WinoGrande (%) | AVG (%) | Time cost (s) | RAM | VRAM |
| Llama-3.1-8B-I | BF16 | 70.51 | 59.71 | 68.37 | 80.09 | 73.40 | 70.42 | - | - | - |
| RTN | 50.11 | 55.54 | 57.34 | 76.01 | 70.01 | 61.80 | 88.00 | 9.83GB | 2.99GB | |
| opt_rtn | 52.08 | 55.82 | 57.86 | 76.88 | 71.11 | 62.75 | 100.00 | 10.02GB | 3.84GB | |
| AR | 60.73 | 55.29 | 61.34 | 77.53 | 70.01 | 64.98 | 976.75 | 24.33GB | 16.86GB | |
| AR2 | 61.87 | 56.59 | 61.60 | 77.97 | 71.11 | 65.83 | 1151.81 | 24.38GB | 17.97GB | |
| AWQ_RTN | 50.27 | 55.51 | 58.55 | 76.93 | 70.56 | 62.36 | 993.51 | 23.36GB | 7.91GB | |
| AWQ_AR | 60.05 | 56.00 | 61.96 | 78.07 | 72.14 | 65.64 | 2042.67 | 24.35GB | 18.80GB | |
| AWQ_AR2 | 62.02 | 56.40 | 61.84 | 78.62 | 72.14 | 66.20 | 2164.04 | 24.40GB | 17.94GB | |
| Qwen3-8B | BF16 | 87.41 | 57.16 | 72.92 | 76.71 | 67.72 | 72.38 | - | - | - |
| RTN | 81.27 | 52.24 | 66.13 | 73.01 | 63.14 | 67.16 | 119.00 | 7.81GB | 2.66GB | |
| opt_rtn | 83.55 | 52.60 | 66.10 | 73.50 | 64.40 | 68.03 | 125.00 | 7.99GB | 3.42GB | |
| AR | 80.82 | 52.64 | 67.98 | 74.81 | 66.30 | 68.51 | 999.52 | 22.31GB | 15.43GB | |
| AR2 | 83.70 | 53.41 | 68.51 | 75.57 | 67.56 | 69.75 | 1240.32 | 22.34GB | 16.18GB | |
| AWQ_RTN | 82.79 | 53.54 | 67.48 | 75.19 | 67.17 | 69.23 | 1073.95 | 21.33GB | 7.14GB | |
| AWQ_AR | 79.91 | 53.07 | 68.99 | 75.35 | 67.32 | 68.93 | 2095.77 | 22.32GB | 16.72GB | |
| AWQ_AR2 | 83.70 | 53.89 | 69.54 | 75.57 | 68.19 | 70.18 | 2244.87 | 22.37GB | 16.03GB |
MXFP4 Observations
| Comparison | AVG Win Rate | Max AVG Delta Gain | Max AVG Rel. Gain |
|---|---|---|---|
AWQ_RTN vs RTN / opt_rtn |
3/4 (75%) | +2.07 pts | +3.08% |
AWQ_AR vs AR |
2/2 (100%) | +0.66 pts | +1.02% |
AWQ_AR2 vs AR2 |
2/2 (100%) | +0.43 pts | +0.62% |
- MXFP4 remains the clearest case for AWQ composition in these experiments. AWQ improves AVG for both AutoRound and AutoRound2 on both models.
AWQ_AR2gives the highest MXFP4 AVG for both models, reaching 66.20 on Llama and 70.18 on Qwen.AWQ_RTNimproves over plainRTNon both models. This suggests AWQ smoothing is useful, while optimized RTN remains a strong low-cost MXFP4 baseline.- The main tradeoff is one-time runtime.
AWQ_ARandAWQ_AR2take roughly 1.8x-2.1x the corresponding native AutoRound runtime, while peak RAM/VRAM stays in the same range as the downstream AutoRound optimizer.
W4A16
| Model | Config | GSM8K (%) | HellaSwag (%) | MMLU (%) | PIQA (%) | WinoGrande (%) | AVG (%) | Time cost (s) | RAM | VRAM |
| Llama-3.1-8B-I | BF16 | 70.51 | 59.71 | 68.37 | 80.09 | 73.40 | 70.42 | - | - | - |
| RTN | 71.80 | 59.14 | 65.46 | 79.43 | 72.77 | 69.72 | 60.00 | 25.08GB | 0.56GB | |
| opt_rtn | 67.63 | 58.99 | 65.67 | 78.94 | 73.95 | 69.04 | 162.30 | 28.31GB | 7.90GB | |
| AR | 67.40 | 58.93 | 66.81 | 79.87 | 73.72 | 69.35 | 769.27 | 24.29GB | 12.32GB | |
| AR2 | 69.60 | 59.13 | 66.81 | 79.87 | 73.09 | 69.70 | 918.59 | 24.04GB | 13.26GB | |
| AWQ_RTN | 70.58 | 59.09 | 65.89 | 79.87 | 72.85 | 69.66 | 1075.44 | 24.07GB | 7.68GB | |
| AWQ_AR | 73.24 | 59.11 | 66.97 | 80.14 | 72.93 | 70.48 | 1813.51 | 24.07GB | 13.32GB | |
| AWQ_AR2 | 70.13 | 59.09 | 67.11 | 79.87 | 73.56 | 69.95 | 3533.26 | 24.07GB | 13.17GB | |
| Qwen3-8B | BF16 | 87.41 | 57.16 | 72.92 | 76.71 | 67.72 | 72.38 | - | - | - |
| RTN | 86.43 | 56.18 | 70.84 | 75.52 | 67.17 | 71.23 | 97.00 | 23.10GB | 0.50GB | |
| opt_rtn | 87.26 | 55.96 | 71.58 | 76.33 | 68.43 | 71.91 | 180.17 | 27.17GB | 7.38GB | |
| AR | 88.02 | 56.03 | 72.13 | 76.22 | 67.72 | 72.02 | 779.03 | 21.73GB | 11.41GB | |
| AR2 | 86.96 | 55.95 | 72.38 | 76.71 | 68.27 | 72.05 | 1028.79 | 22.00GB | 12.48GB | |
| AWQ_RTN | 86.13 | 56.04 | 71.91 | 75.63 | 67.56 | 71.45 | 1108.32 | 22.02GB | 6.93GB | |
| AWQ_AR | 86.73 | 56.13 | 71.51 | 76.22 | 69.06 | 71.93 | 1957.19 | 21.94GB | 12.07GB | |
| AWQ_AR2 | 86.50 | 56.07 | 72.23 | 76.55 | 69.14 | 72.10 | 3673.00 | 22.03GB | 12.79GB |
W4A16 Observations
| Comparison | AVG Win Rate | Max AVG Delta Gain | Max AVG Rel. Gain |
|---|---|---|---|
AWQ_RTN vs RTN / opt_rtn |
2/4 (50%) | +0.62 pts | +0.90% |
AWQ_AR vs AR |
1/2 (50%) | +1.13 pts | +1.63% |
AWQ_AR2 vs AR2 |
2/2 (100%) | +0.25 pts | +0.36% |
- W4A16 shows model-dependent behavior. The clearest gain is Llama
AWQ_ARoverAR, with +1.13 average points and +1.63% relative AVG gain. AWQ_RTNis not a strong W4A16 default. It wins 2/4 AVG comparisons againstRTNandopt_rtn.AWQ_AR2improves AVG on both models, but the gain is small compared with the added runtime.
INT8 / W8A8
| Model | Config | GSM8K (%) | HellaSwag (%) | MMLU (%) | PIQA (%) | WinoGrande (%) | AVG (%) | Time cost (s) | RAM | VRAM |
| Llama-3.1-8B-I | BF16 | 70.51 | 59.71 | 68.37 | 80.09 | 73.40 | 70.42 | - | - | - |
| RTN | 72.25 | 59.73 | 68.10 | 79.71 | 74.82 | 70.92 | 98.00 | 9.53GB | 0.85GB | |
| AR | 68.99 | 59.63 | 67.87 | 79.71 | 74.11 | 70.06 | 988.81 | 23.87GB | 13.38GB | |
| AWQ_RTN | 72.63 | 59.54 | 68.13 | 79.82 | 73.72 | 70.77 | 1001.96 | 22.98GB | 7.55GB | |
| AWQ_AR | 70.05 | 59.65 | 68.20 | 79.87 | 74.11 | 70.38 | 1879.53 | 23.95GB | 14.21GB | |
| Qwen3-8B | BF16 | 87.41 | 57.16 | 72.92 | 76.71 | 67.72 | 72.38 | - | - | - |
| RTN | 87.49 | 56.93 | 72.53 | 76.01 | 63.95 | 71.38 | 88.00 | 7.58GB | 0.75GB | |
| AR | 88.10 | 56.74 | 72.46 | 76.06 | 68.11 | 72.29 | 905.88 | 21.83GB | 12.37GB | |
| AWQ_RTN | 86.58 | 57.05 | 72.57 | 76.77 | 68.35 | 72.26 | 1046.69 | 20.93GB | 6.89GB | |
| AWQ_AR | 86.81 | 56.77 | 72.58 | 76.61 | 68.11 | 72.18 | 2277.47 | 21.92GB | 13.30GB |
INT8 Observations
| Comparison | AVG Win Rate | Max AVG Delta Gain | Max AVG Rel. Gain |
|---|---|---|---|
AWQ_RTN vs RTN |
1/2 (50%) | +0.88 pts | +1.23% |
AWQ_AR vs AR |
1/2 (50%) | +0.32 pts | +0.46% |
- INT8/W8A8 is already a high-accuracy setting in these experiments. Most INT8 recipes are close to the BF16 reference, and some recipes slightly exceed BF16 AVG.
- For AWQ-composed INT8, the AVG movement is small and model-dependent.
AWQ_RTNimproves overRTNon Qwen but is slightly belowRTNon Llama;AWQ_ARimproves overARon Llama but is slightly belowARon Qwen. - Given the strong INT8 baseline and small AWQ deltas, AWQ is better positioned as an optional exploration path for INT8/W8A8 rather than a primary default.
Summary
AWQ is most useful for schemes where quantization is sensitive to activation-aware weight smoothing, such as MXFP4. It can improve scheme accuracy, especially when combined with AutoRound plus the algorithm extension (--algorithm auto_round_v2).
AWQ does not add persistent memory overhead to the quantized model and is orthogonal to terminal quantization algorithms, so it can be composed with RTN, AutoRound, or AutoRound2 without changing their core quantization logic.