Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

20.2 KB
Raw Blame

AWQ Algorithm Results

This document summarizes an experimental comparison between native AutoRound flows and AWQ-composed flows. The goal is to evaluate whether applying AWQ smoothing before RTN or AutoRound improves accuracy for W4A16, MXFP4, and INT8 (W8A8) schemes.

All table results are based on the AutoRound 0.15.0 release, measured with --format fake, and evaluated on mmlu,gsm8k,piqa,hellaswag,winogrande. These five widely used LLM benchmarks cover complementary error modes: MMLU for factual knowledge and broad understanding, GSM8K for mathematical reasoning, PIQA for physical commonsense, HellaSwag for commonsense sentence completion, and Winogrande for pronoun and coreference resolution. Reporting the average score across this task set reduces sensitivity to single-task variance and provides a more stable view of quantization-induced accuracy changes. Unless noted otherwise, the remaining quantization settings follow the AutoRound defaults.

AWQ Algorithm Characteristics

AWQ is an activation-aware weight smoothing method. It searches for per-channel scaling factors from calibration activations, applies the scaling to reduce quantization-sensitive weight channels, and then hands the transformed model to the terminal quantization algorithm.

AWQ changes the pre-quantization weight distribution, while RTN, AutoRound, and AutoRound2 decide how the final quantized values are produced. In this design, AWQ can be composed with different quantizers without changing their core optimization logic.

From the resource perspective, AWQ does not introduce a persistent RAM or VRAM burden in the quantized model. During calibration, AWQ_RTN stays around the same host RAM level as the other flows and uses much less VRAM than iterative AutoRound flows. When AWQ is followed by AutoRound or AutoRound2, the peak VRAM is mainly determined by the downstream AutoRound optimization rather than by AWQ itself.

The main added cost is calibration runtime. This is a one-time offline cost and is acceptable when AWQ provides measurable accuracy gains, especially for MXFP4 and selected W4A16 cases.

The configurations in the tables map to the following CLI usage:

# Pure RTN baseline
auto-round-rtn --model <model> --scheme <scheme> --format fake \
  --tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"

# Optimized RTN baseline
auto-round-opt-rtn --model <model> --scheme <scheme> --format fake \
  --tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"

# Native AutoRound / AutoRound with algorithm extension
auto-round --model <model> --scheme <scheme> --format fake \
  --algorithm auto_round \
  --tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"

auto-round --model <model> --scheme <scheme> --format fake \
  --algorithm auto_round_v2 \
  --tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"

# AWQ + RTN / AWQ + AutoRound / AWQ + AutoRound with algorithm extension
auto-round --model <model> --scheme <scheme> --format fake \
  --algorithm awq \
  --tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"

auto-round --model <model> --scheme <scheme> --format fake \
  --algorithm awq,auto_round \
  --tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"

auto-round --model <model> --scheme <scheme> --format fake \
  --algorithm awq,auto_round_v2 \
  --tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"

Calibration Controls

AWQ has its own calibration budget for activation statistics and for the smoothing and clipping searches. These controls affect AWQ preprocessing only; the downstream RTN or AutoRound quantizer retains the global calibration cache and performs its own final scale search.

CLI option AWQConfig field Default Purpose
--awq-nsamples awq_nsamples 128 Maximum number of calibration samples used by AWQ activation statistics, smoothing, and clipping.
--awq_seqlen awq_seqlen 512 Maximum sequence length used by AWQ calibration; <= 0 disables truncation.
--awq_smooth_batch_size smooth_batch_size None Micro-batch size for replaying the parent module during the AWQ scale search; lower positive values reduce peak VRAM, while <= 0 disables micro-batching.
--awq-v2-scale-search v2_scale_search auto Scale-search policy for AWQ's internal QDQ when the terminal quantizer is AutoRoundV2.

--awq-nsamples is independent of the global --nsamples. AWQ selects a deterministic random subset (seed 42) from the cached calibration inputs and reuses the same subset for every block. The global cache is not modified, so the downstream RTN/AutoRound quantizer and the FP reference forward still use all cached samples. Set --awq-nsamples greater than or equal to --nsamples to use all of them.

--awq-v2-scale-search takes effect only when the terminal quantizer is AutoRoundV2, which is enabled with --enable_alg_ext and --algorithm auto_round (or awq,auto_round). With an RTN terminal quantizer, the option is ignored. The policy changes only the QDQ used to score AWQ candidates; the terminal AutoRoundV2 quantizer always performs its own initialization search.

  • auto (default): use off for INT schemes and full for non-INT schemes such as MXFP and NVFP.
  • off: use plain RTN for AWQ's candidate loss. This is the fastest option and the recommended default for INT.
  • full: reproduce the terminal V2 optimized init-scale search for every AWQ candidate. This is the slowest and most faithful option.
  • reuse: run the optimized init-scale search once per balance layer and reuse it for every AWQ candidate. This is a middle ground between off and full.

When smoothing or hard clipping changes a block, AWQ requests a fresh FP reference forward before quantization, so a stale reference is never used as the quantizer target or passed to the next block.

# INT: auto resolves to off; AWQ uses at most 128 samples while the quantizer uses 256.
auto-round --model <model> --scheme W4A16 --format fake \
  --algorithm awq,auto_round --enable_alg_ext \
  --nsamples 256 --awq-nsamples 128

# MXFP4: auto resolves to full; use reuse to cut AWQ calibration time.
auto-round --model <model> --scheme MXFP4 --format fake \
  --algorithm awq,auto_round --enable_alg_ext \
  --awq-v2-scale-search reuse

# INT with an explicit full optimized search for every AWQ candidate.
auto-round --model <model> --scheme W4A16 --format fake \
  --algorithm awq,auto_round --enable_alg_ext \
  --awq-v2-scale-search full

The same settings are available through AWQConfig:

from auto_round import AutoRound, AWQConfig, SignRoundV2Config

ar = AutoRound(
    model="<model>",
    scheme="W4A16",
    alg_configs=[
        AWQConfig(awq_nsamples=128, awq_seqlen=512, v2_scale_search="auto"),
        SignRoundV2Config(iters=200),
    ],
)

In the result tables, AR2 denotes AutoRound with --algorithm auto_round_v2 (formerly --enable_alg_ext), and AWQ_AR2 denotes AWQ followed by AutoRound with the same algorithm extension. Accuracy columns are reported as percentages, so a raw score such as 0.7058 is shown as 70.58.

The AWQ rows use the default AWQ smoothing flow. They do not include AWQ weight clipping unless --awq_apply_clip is explicitly enabled.

In the observation tables, AVG Win Rate counts how often the AWQ-composed method improves model-level AVG accuracy for that scheme, Max AVG Delta Gain is the largest absolute AVG improvement in percentage points, and Max AVG Rel. Gain is the largest relative AVG improvement. For AWQ_RTN, the comparison includes both RTN and opt_rtn when both baselines are available.

MXFP4

Model Config GSM8K (%) HellaSwag (%) MMLU (%) PIQA (%) WinoGrande (%) AVG (%) Time cost (s) RAM VRAM
Llama-3.1-8B-I BF16 70.51 59.71 68.37 80.09 73.40 70.42 - - -
RTN 50.11 55.54 57.34 76.01 70.01 61.80 88.00 9.83GB 2.99GB
opt_rtn 52.08 55.82 57.86 76.88 71.11 62.75 100.00 10.02GB 3.84GB
AR 60.73 55.29 61.34 77.53 70.01 64.98 976.75 24.33GB 16.86GB
AR2 61.87 56.59 61.60 77.97 71.11 65.83 1151.81 24.38GB 17.97GB
AWQ_RTN 50.27 55.51 58.55 76.93 70.56 62.36 993.51 23.36GB 7.91GB
AWQ_AR 60.05 56.00 61.96 78.07 72.14 65.64 2042.67 24.35GB 18.80GB
AWQ_AR2 62.02 56.40 61.84 78.62 72.14 66.20 2164.04 24.40GB 17.94GB
Qwen3-8B BF16 87.41 57.16 72.92 76.71 67.72 72.38 - - -
RTN 81.27 52.24 66.13 73.01 63.14 67.16 119.00 7.81GB 2.66GB
opt_rtn 83.55 52.60 66.10 73.50 64.40 68.03 125.00 7.99GB 3.42GB
AR 80.82 52.64 67.98 74.81 66.30 68.51 999.52 22.31GB 15.43GB
AR2 83.70 53.41 68.51 75.57 67.56 69.75 1240.32 22.34GB 16.18GB
AWQ_RTN 82.79 53.54 67.48 75.19 67.17 69.23 1073.95 21.33GB 7.14GB
AWQ_AR 79.91 53.07 68.99 75.35 67.32 68.93 2095.77 22.32GB 16.72GB
AWQ_AR2 83.70 53.89 69.54 75.57 68.19 70.18 2244.87 22.37GB 16.03GB

MXFP4 Observations

Comparison AVG Win Rate Max AVG Delta Gain Max AVG Rel. Gain
AWQ_RTN vs RTN / opt_rtn 3/4 (75%) +2.07 pts +3.08%
AWQ_AR vs AR 2/2 (100%) +0.66 pts +1.02%
AWQ_AR2 vs AR2 2/2 (100%) +0.43 pts +0.62%
  • MXFP4 remains the clearest case for AWQ composition in these experiments. AWQ improves AVG for both AutoRound and AutoRound2 on both models.
  • AWQ_AR2 gives the highest MXFP4 AVG for both models, reaching 66.20 on Llama and 70.18 on Qwen.
  • AWQ_RTN improves over plain RTN on both models. This suggests AWQ smoothing is useful, while optimized RTN remains a strong low-cost MXFP4 baseline.
  • The main tradeoff is one-time runtime. AWQ_AR and AWQ_AR2 take roughly 1.8x-2.1x the corresponding native AutoRound runtime, while peak RAM/VRAM stays in the same range as the downstream AutoRound optimizer.

W4A16

Model Config GSM8K (%) HellaSwag (%) MMLU (%) PIQA (%) WinoGrande (%) AVG (%) Time cost (s) RAM VRAM
Llama-3.1-8B-I BF16 70.51 59.71 68.37 80.09 73.40 70.42 - - -
RTN 71.80 59.14 65.46 79.43 72.77 69.72 60.00 25.08GB 0.56GB
opt_rtn 67.63 58.99 65.67 78.94 73.95 69.04 162.30 28.31GB 7.90GB
AR 67.40 58.93 66.81 79.87 73.72 69.35 769.27 24.29GB 12.32GB
AR2 69.60 59.13 66.81 79.87 73.09 69.70 918.59 24.04GB 13.26GB
AWQ_RTN 70.58 59.09 65.89 79.87 72.85 69.66 1075.44 24.07GB 7.68GB
AWQ_AR 73.24 59.11 66.97 80.14 72.93 70.48 1813.51 24.07GB 13.32GB
AWQ_AR2 70.13 59.09 67.11 79.87 73.56 69.95 3533.26 24.07GB 13.17GB
Qwen3-8B BF16 87.41 57.16 72.92 76.71 67.72 72.38 - - -
RTN 86.43 56.18 70.84 75.52 67.17 71.23 97.00 23.10GB 0.50GB
opt_rtn 87.26 55.96 71.58 76.33 68.43 71.91 180.17 27.17GB 7.38GB
AR 88.02 56.03 72.13 76.22 67.72 72.02 779.03 21.73GB 11.41GB
AR2 86.96 55.95 72.38 76.71 68.27 72.05 1028.79 22.00GB 12.48GB
AWQ_RTN 86.13 56.04 71.91 75.63 67.56 71.45 1108.32 22.02GB 6.93GB
AWQ_AR 86.73 56.13 71.51 76.22 69.06 71.93 1957.19 21.94GB 12.07GB
AWQ_AR2 86.50 56.07 72.23 76.55 69.14 72.10 3673.00 22.03GB 12.79GB

W4A16 Observations

Comparison AVG Win Rate Max AVG Delta Gain Max AVG Rel. Gain
AWQ_RTN vs RTN / opt_rtn 2/4 (50%) +0.62 pts +0.90%
AWQ_AR vs AR 1/2 (50%) +1.13 pts +1.63%
AWQ_AR2 vs AR2 2/2 (100%) +0.25 pts +0.36%
  • W4A16 shows model-dependent behavior. The clearest gain is Llama AWQ_AR over AR, with +1.13 average points and +1.63% relative AVG gain.
  • AWQ_RTN is not a strong W4A16 default. It wins 2/4 AVG comparisons against RTN and opt_rtn.
  • AWQ_AR2 improves AVG on both models, but the gain is small compared with the added runtime.

INT8 / W8A8

Model Config GSM8K (%) HellaSwag (%) MMLU (%) PIQA (%) WinoGrande (%) AVG (%) Time cost (s) RAM VRAM
Llama-3.1-8B-I BF16 70.51 59.71 68.37 80.09 73.40 70.42 - - -
RTN 72.25 59.73 68.10 79.71 74.82 70.92 98.00 9.53GB 0.85GB
AR 68.99 59.63 67.87 79.71 74.11 70.06 988.81 23.87GB 13.38GB
AWQ_RTN 72.63 59.54 68.13 79.82 73.72 70.77 1001.96 22.98GB 7.55GB
AWQ_AR 70.05 59.65 68.20 79.87 74.11 70.38 1879.53 23.95GB 14.21GB
Qwen3-8B BF16 87.41 57.16 72.92 76.71 67.72 72.38 - - -
RTN 87.49 56.93 72.53 76.01 63.95 71.38 88.00 7.58GB 0.75GB
AR 88.10 56.74 72.46 76.06 68.11 72.29 905.88 21.83GB 12.37GB
AWQ_RTN 86.58 57.05 72.57 76.77 68.35 72.26 1046.69 20.93GB 6.89GB
AWQ_AR 86.81 56.77 72.58 76.61 68.11 72.18 2277.47 21.92GB 13.30GB

INT8 Observations

Comparison AVG Win Rate Max AVG Delta Gain Max AVG Rel. Gain
AWQ_RTN vs RTN 1/2 (50%) +0.88 pts +1.23%
AWQ_AR vs AR 1/2 (50%) +0.32 pts +0.46%
  • INT8/W8A8 is already a high-accuracy setting in these experiments. Most INT8 recipes are close to the BF16 reference, and some recipes slightly exceed BF16 AVG.
  • For AWQ-composed INT8, the AVG movement is small and model-dependent. AWQ_RTN improves over RTN on Qwen but is slightly below RTN on Llama; AWQ_AR improves over AR on Llama but is slightly below AR on Qwen.
  • Given the strong INT8 baseline and small AWQ deltas, AWQ is better positioned as an optional exploration path for INT8/W8A8 rather than a primary default.

Summary

AWQ is most useful for schemes where quantization is sensitive to activation-aware weight smoothing, such as MXFP4. It can improve scheme accuracy, especially when combined with AutoRound plus the algorithm extension (--algorithm auto_round_v2).

AWQ does not add persistent memory overhead to the quantized model and is orthogonal to terminal quantization algorithms, so it can be composed with RTN, AutoRound, or AutoRound2 without changing their core quantization logic.