PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
update awq detail data of fake eval (#2285)
Weiwei3942b57e9940
diff --git a/docs/awq_details.md b/docs/awq_details.mdindex a519e71d9..116f2e51c 100644--- a/docs/awq_details.md+++ b/docs/awq_details.md@@ -2,7 +2,7 @@This document summarizes an experimental comparison between native AutoRound flows and AWQ-composed flows. The goal is to evaluate whether applying AWQ smoothing before RTN or AutoRound improves accuracy for `W4A16`, `MXFP4`, and `INT8` (`W8A8`) schemes.-All results were measured with `--format fake` and evaluated on `mmlu,gsm8k,piqa,hellaswag,winogrande`. These five widely used LLM benchmarks cover complementary error modes: MMLU for factual knowledge and broad understanding, GSM8K for mathematical reasoning, PIQA for physical commonsense, HellaSwag for commonsense sentence completion, and Winogrande for pronoun and coreference resolution. Reporting the average score across this task set reduces sensitivity to single-task variance and provides a more stable view of quantization-induced accuracy changes. Unless noted otherwise, the remaining quantization settings follow the AutoRound defaults.+All table results are based on the AutoRound 0.15.0 release, measured with `--format fake`, and evaluated on `mmlu,gsm8k,piqa,hellaswag,winogrande`. These five widely used LLM benchmarks cover complementary error modes: MMLU for factual knowledge and broad understanding, GSM8K for mathematical reasoning, PIQA for physical commonsense, HellaSwag for commonsense sentence completion, and Winogrande for pronoun and coreference resolution. Reporting the average score across this task set reduces sensitivity to single-task variance and provides a more stable view of quantization-induced accuracy changes. Unless noted otherwise, the remaining quantization settings follow the AutoRound defaults.## AWQ Algorithm Characteristics@@ -12,7 +12,7 @@ AWQ changes the pre-quantization weight distribution, while RTN, AutoRound, andFrom the resource perspective, AWQ does not introduce a persistent RAM or VRAM burden in the quantized model. During calibration, `AWQ_RTN` stays around the same host RAM level as the other flows and uses much less VRAM than iterative AutoRound flows. When AWQ is followed by AutoRound or AutoRound2, the peak VRAM is mainly determined by the downstream AutoRound optimization rather than by AWQ itself.-The main added cost is timecost and the increase is acceptable when AWQ provides measurable accuracy gains, especially for MXFP4 and selected W4A16 cases.+The main added cost is calibration runtime. This is a one-time offline cost and is acceptable when AWQ provides measurable accuracy gains, especially for MXFP4 and selected W4A16 cases.The configurations in the tables map to the following CLI usage:@@ -48,14 +48,14 @@ auto-round --model <model> --scheme <scheme> --format fake \--tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"```-In the result tables, `AR2` denotes AutoRound with `--enable_alg_ext`, and `AWQ_AR2` denotes AWQ followed by AutoRound with `--enable_alg_ext`.+In the result tables, `AR2` denotes AutoRound with `--enable_alg_ext`, and `AWQ_AR2` denotes AWQ followed by AutoRound with `--enable_alg_ext`. Accuracy columns are reported as percentages, so a raw score such as `0.7058` is shown as `70.58`.++The AWQ rows use the default AWQ smoothing flow. They do not include AWQ weight clipping unless `--awq_apply_clip` is explicitly enabled.In the observation tables, `AVG Win Rate` counts how often the AWQ-composed method improves model-level AVG accuracy for that scheme, `Max AVG Delta Gain` is the largest absolute AVG improvement in percentage points, and `Max AVG Rel. Gain` is the largest relative AVG improvement. For `AWQ_RTN`, the comparison includes both `RTN` and `opt_rtn` when both baselines are available.## MXFP4-**Note: For Llama-3.1-Instruct, we recommend removing `@use_kernel_forward_from_hub("RMSNorm")` in [modeling_llama.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L52C1-L52C40); the tests below did not remove it, which can make the accuracy results inaccurate.**-<table border="1"><tr><td>Model</td>@@ -66,7 +66,7 @@ In the observation tables, `AVG Win Rate` counts how often the AWQ-composed meth<td>PIQA (%)</td><td>WinoGrande (%)</td><td>AVG (%)</td>- <td>Timecost</td>+ <td>Time cost (s)</td><td>RAM</td><td>VRAM</td></tr>@@ -85,88 +85,87 @@ In the observation tables, `AVG Win Rate` counts how often the AWQ-composed meth</tr><tr><td>RTN</td>- <td>59.06</td>- <td>58.88</td>- <td>64.63</td>- <td>78.94</td>- <td>72.77</td>- <td>66.86</td>- <td>72.30</td>- <td>25.52GB</td>- <td>0.83GB</td>+ <td>50.11</td>+ <td>55.54</td>+ <td>57.34</td>+ <td>76.01</td>+ <td>70.01</td>+ <td>61.80</td>+ <td>88.00</td>+ <td>9.83GB</td>+ <td>2.99GB</td></tr><tr><td>opt_rtn</td>- <td>65.20</td>- <td>58.91</td>- <td>65.04</td>- <td>79.05</td>- <td>73.48</td>- <td>68.34</td>- <td>62.00</td>- <td>25.65GB</td>- <td>1.65GB</td>- </tr>- <!-- TODO: MXFP4 Llama-3.1 AR data is anomalous (GSM8K=0.00, AVG=43.13); treat as preliminary and revisit for now. -->+ <td>52.08</td>+ <td>55.82</td>+ <td>57.86</td>+ <td>76.88</td>+ <td>71.11</td>+ <td>62.75</td>+ <td>100.00</td>+ <td>10.02GB</td>+ <td>3.84GB</td>+ </tr><tr><td>AR</td>- <td>0.00</td>- <td>31.20</td>- <td>48.50</td>- <td>75.24</td>- <td>60.69</td>- <td>43.13</td>- <td>959.21</td>- <td>25.39GB</td>- <td>16.30GB</td>+ <td>60.73</td>+ <td>55.29</td>+ <td>61.34</td>+ <td>77.53</td>+ <td>70.01</td>+ <td>64.98</td>+ <td>976.75</td>+ <td>24.33GB</td>+ <td>16.86GB</td></tr><tr><td>AR2</td>- <td>67.32</td>- <td>57.94</td>- <td>64.11</td>- <td>80.25</td>- <td>73.80</td>- <td>68.68</td>- <td>1104.91</td>+ <td>61.87</td>+ <td>56.59</td>+ <td>61.60</td>+ <td>77.97</td>+ <td>71.11</td>+ <td>65.83</td>+ <td>1151.81</td><td>24.38GB</td>- <td>17.78GB</td>+ <td>17.97GB</td></tr><tr><td>AWQ_RTN</td>- <td>66.26</td>- <td>58.36</td>- <td>66.12</td>- <td>79.00</td>- <td>73.95</td>- <td>68.74</td>- <td>1132.04</td>- <td>24.36GB</td>- <td>7.60GB</td>+ <td>50.27</td>+ <td>55.51</td>+ <td>58.55</td>+ <td>76.93</td>+ <td>70.56</td>+ <td>62.36</td>+ <td>993.51</td>+ <td>23.36GB</td>+ <td>7.91GB</td></tr><tr><td>AWQ_AR</td>- <td>69.75</td>- <td>57.74</td>- <td>65.71</td>- <td>79.27</td>- <td>72.69</td>- <td>69.03</td>- <td>2136.06</td>- <td>24.47GB</td>- <td>19.24GB</td>+ <td>60.05</td>+ <td>56.00</td>+ <td>61.96</td>+ <td>78.07</td>+ <td>72.14</td>+ <td>65.64</td>+ <td>2042.67</td>+ <td>24.35GB</td>+ <td>18.80GB</td></tr><tr><td><b>AWQ_AR2</b></td>- <td><b>71.27</b></td>- <td><b>58.29</b></td>- <td><b>66.07</b></td>- <td><b>80.25</b></td>- <td><b>73.88</b></td>- <td><b>69.95</b></td>- <td><b>2274.64</b></td>+ <td><b>62.02</b></td>+ <td><b>56.40</b></td>+ <td><b>61.84</b></td>+ <td><b>78.62</b></td>+ <td><b>72.14</b></td>+ <td><b>66.20</b></td>+ <td><b>2164.04</b></td><td><b>24.40GB</b></td>- <td><b>18.12GB</b></td>+ <td><b>17.94GB</b></td></tr><tr><td rowspan="8">Qwen3-8B</td>@@ -183,87 +182,87 @@ In the observation tables, `AVG Win Rate` counts how often the AWQ-composed meth</tr><tr><td>RTN</td>- <td>86.05</td>- <td>55.07</td>- <td>70.30</td>- <td>75.41</td>- <td>66.93</td>- <td>70.75</td>- <td>61.00</td>- <td>23.23GB</td>- <td>0.74GB</td>+ <td>81.27</td>+ <td>52.24</td>+ <td>66.13</td>+ <td>73.01</td>+ <td>63.14</td>+ <td>67.16</td>+ <td>119.00</td>+ <td>7.81GB</td>+ <td>2.66GB</td></tr><tr><td>opt_rtn</td>- <td>86.50</td>- <td>55.53</td>- <td>70.64</td>- <td>75.79</td>- <td>66.77</td>- <td>71.05</td>- <td>66.00</td>- <td>23.55GB</td>- <td>1.40GB</td>+ <td>83.55</td>+ <td>52.60</td>+ <td>66.10</td>+ <td>73.50</td>+ <td>64.40</td>+ <td>68.03</td>+ <td>125.00</td>+ <td>7.99GB</td>+ <td>3.42GB</td></tr><tr><td>AR</td>- <td>86.73</td>- <td>54.90</td>- <td>71.56</td>- <td>76.44</td>- <td>70.09</td>- <td>71.94</td>- <td>968.45</td>- <td>22.32GB</td>- <td>15.68GB</td>+ <td>80.82</td>+ <td>52.64</td>+ <td>67.98</td>+ <td>74.81</td>+ <td>66.30</td>+ <td>68.51</td>+ <td>999.52</td>+ <td>22.31GB</td>+ <td>15.43GB</td></tr><tr><td>AR2</td>- <td>86.58</td>- <td>55.07</td>- <td>71.86</td>- <td>76.61</td>- <td>69.77</td>- <td>71.98</td>- <td>1195.57</td>- <td>22.35GB</td>- <td>16.00GB</td>+ <td>83.70</td>+ <td>53.41</td>+ <td>68.51</td>+ <td>75.57</td>+ <td>67.56</td>+ <td>69.75</td>+ <td>1240.32</td>+ <td>22.34GB</td>+ <td>16.18GB</td></tr><tr><td>AWQ_RTN</td>- <td>85.90</td>- <td>55.79</td>- <td>71.54</td>- <td>76.93</td>- <td>67.80</td>- <td>71.59</td>- <td>1088.34</td>- <td>22.34GB</td>- <td>6.93GB</td>+ <td>82.79</td>+ <td>53.54</td>+ <td>67.48</td>+ <td>75.19</td>+ <td>67.17</td>+ <td>69.23</td>+ <td>1073.95</td>+ <td>21.33GB</td>+ <td>7.14GB</td></tr><tr><td>AWQ_AR</td>- <td>85.37</td>- <td>54.41</td>- <td>71.47</td>- <td>76.55</td>- <td>70.32</td>- <td>71.62</td>- <td>2346.84</td>- <td>22.45GB</td>- <td>16.75GB</td>+ <td>79.91</td>+ <td>53.07</td>+ <td>68.99</td>+ <td>75.35</td>+ <td>67.32</td>+ <td>68.93</td>+ <td>2095.77</td>+ <td>22.32GB</td>+ <td>16.72GB</td></tr><tr><td><b>AWQ_AR2</b></td>- <td><b>86.66</b></td>- <td><b>55.19</b></td>- <td><b>71.99</b></td>- <td><b>77.37</b></td>- <td><b>70.80</b></td>- <td><b>72.40</b></td>- <td><b>2266.50</b></td>+ <td><b>83.70</b></td>+ <td><b>53.89</b></td>+ <td><b>69.54</b></td>+ <td><b>75.57</b></td>+ <td><b>68.19</b></td>+ <td><b>70.18</b></td>+ <td><b>2244.87</b></td><td><b>22.37GB</b></td>- <td><b>16.16GB</b></td>+ <td><b>16.03GB</b></td></tr></table>@@ -271,16 +270,14 @@ In the observation tables, `AVG Win Rate` counts how often the AWQ-composed meth| Comparison | AVG Win Rate | Max AVG Delta Gain | Max AVG Rel. Gain || --- | ---: | ---: | ---: |-| `AWQ_RTN` vs `RTN` / `opt_rtn` | 4/4 (100%) | +1.88 pts | +2.81% |-| `AWQ_AR` vs `AR` | 1/2 (50%) | +25.90 pts | +60.05%* |-| `AWQ_AR2` vs `AR2` | 2/2 (100%) | +1.27 pts | +1.85% |--*The Llama `AWQ_AR` vs `AR` result is inflated by the unstable native `AR` baseline (`GSM8K=0.00`, `AVG=43.13`).+| `AWQ_RTN` vs `RTN` / `opt_rtn` | 3/4 (75%) | +2.07 pts | +3.08% |+| `AWQ_AR` vs `AR` | 2/2 (100%) | +0.66 pts | +1.02% |+| `AWQ_AR2` vs `AR2` | 2/2 (100%) | +0.43 pts | +0.62% |-- MXFP4 is the strongest case for AWQ composition. `AWQ_RTN` improves over both `RTN` and `opt_rtn` on both models, and `AWQ_AR2` improves over `AR2` on both models.-- `AWQ_AR2` gives the most consistent MXFP4 profile: AVG improves on both models, with up to +1.27 points and +1.85% relative AVG gain.-- `AWQ_AR` is not uniformly reliable. The large Llama gain is driven by an unstable native `AR` baseline, while Qwen `AWQ_AR` is below native `AR` on AVG.-- The tradeoff remains runtime and VRAM. `AWQ_RTN` costs around 18x the `RTN` runtime, while `AWQ_AR` / `AWQ_AR2` cost about 2x native `AR` / `AR2` on MXFP4.+- MXFP4 remains the clearest case for AWQ composition in these experiments. AWQ improves AVG for both AutoRound and AutoRound2 on both models.+- `AWQ_AR2` gives the highest MXFP4 AVG for both models, reaching 66.20 on Llama and 70.18 on Qwen.+- `AWQ_RTN` improves over plain `RTN` on both models. This suggests AWQ smoothing is useful, while optimized RTN remains a strong low-cost MXFP4 baseline.+- The main tradeoff is one-time runtime. `AWQ_AR` and `AWQ_AR2` take roughly 1.8x-2.1x the corresponding native AutoRound runtime, while peak RAM/VRAM stays in the same range as the downstream AutoRound optimizer.## W4A16@@ -294,7 +291,7 @@ In the observation tables, `AVG Win Rate` counts how often the AWQ-composed meth<td>PIQA (%)</td><td>WinoGrande (%)</td><td>AVG (%)</td>- <td>Timecost</td>+ <td>Time cost (s)</td><td>RAM</td><td>VRAM</td></tr>@@ -518,12 +515,12 @@ In the observation tables, `AVG Win Rate` counts how often the AWQ-composed meth<td>PIQA (%)</td><td>WinoGrande (%)</td><td>AVG (%)</td>- <td>Timecost</td>+ <td>Time cost (s)</td><td>RAM</td><td>VRAM</td></tr><tr>- <td rowspan="6">Llama-3.1-8B-I</td>+ <td rowspan="5">Llama-3.1-8B-I</td><td>BF16</td><td>70.51</td><td>59.71</td>@@ -537,66 +534,54 @@ In the observation tables, `AVG Win Rate` counts how often the AWQ-composed meth</tr><tr><td>RTN</td>- <td>71.49</td>- <td>59.64</td>- <td>68.37</td>- <td>80.30</td>- <td>74.27</td>- <td>70.81</td>- <td>58.00</td>- <td>25.00GB</td>- <td>0.56GB</td>+ <td><b>72.25</b></td>+ <td><b>59.73</b></td>+ <td><b>68.10</b></td>+ <td><b>79.71</b></td>+ <td><b>74.82</b></td>+ <td><b>70.92</b></td>+ <td><b>98.00</b></td>+ <td><b>9.53GB</b></td>+ <td><b>0.85GB</b></td></tr><tr><td>AR</td>- <td>69.90</td>- <td>59.62</td>- <td>68.36</td>- <td>80.36</td>- <td>73.64</td>- <td>70.38</td>- <td>836.48</td>- <td>24.90GB</td>- <td>13.63GB</td>+ <td>68.99</td>+ <td>59.63</td>+ <td>67.87</td>+ <td>79.71</td>+ <td>74.11</td>+ <td>70.06</td>+ <td>988.81</td>+ <td>23.87GB</td>+ <td>13.38GB</td></tr><tr><td>AWQ_RTN</td>- <td>73.24</td>- <td>59.66</td>- <td>68.30</td>- <td>80.09</td>- <td>73.32</td>- <td>70.92</td>- <td>1052.19</td>- <td>23.98GB</td>- <td>7.67GB</td>+ <td>72.63</td>+ <td>59.54</td>+ <td>68.13</td>+ <td>79.82</td>+ <td>73.72</td>+ <td>70.77</td>+ <td>1001.96</td>+ <td>22.98GB</td>+ <td>7.55GB</td></tr><tr><td>AWQ_AR</td>- <td>68.99</td>- <td>59.77</td>- <td>68.12</td>- <td>80.30</td>- <td>73.09</td><td>70.05</td>- <td>1228.53</td>- <td>24.10GB</td>- <td>14.29GB</td>- </tr>- <tr>- <td><b>RTN_smooth2048_clip</b></td>- <td><b>72.93</b></td>- <td><b>59.89</b></td>- <td><b>68.27</b></td>- <td><b>80.14</b></td>- <td><b>73.95</b></td>- <td><b>71.04</b></td>- <td><b>1668.43</b></td>- <td><b>24.50GB</b></td>- <td><b>18.46GB</b></td>+ <td>59.65</td>+ <td>68.20</td>+ <td>79.87</td>+ <td>74.11</td>+ <td>70.38</td>+ <td>1879.53</td>+ <td>23.95GB</td>+ <td>14.21GB</td></tr><tr>- <td rowspan="6">Qwen3-8B</td>+ <td rowspan="5">Qwen3-8B</td><td>BF16</td><td>87.41</td><td>57.16</td>@@ -610,63 +595,51 @@ In the observation tables, `AVG Win Rate` counts how often the AWQ-composed meth</tr><tr><td>RTN</td>- <td>87.79</td>- <td>57.09</td>- <td>72.90</td>- <td>76.77</td>- <td>67.80</td>- <td>72.47</td>- <td>84.00</td>- <td>23.09GB</td>- <td>0.50GB</td>- </tr>- <tr>- <td>AR</td>- <td>86.96</td>- <td>56.57</td>- <td>72.58</td>- <td>77.20</td>- <td>68.11</td>- <td>72.28</td>- <td>881.10</td>- <td>23.66GB</td>- <td>12.58GB</td>+ <td>87.49</td>+ <td>56.93</td>+ <td>72.53</td>+ <td>76.01</td>+ <td>63.95</td>+ <td>71.38</td>+ <td>88.00</td>+ <td>7.58GB</td>+ <td>0.75GB</td>+ </tr>+ <tr>+ <td><b>AR</b></td>+ <td><b>88.10</b></td>+ <td><b>56.74</b></td>+ <td><b>72.46</b></td>+ <td><b>76.06</b></td>+ <td><b>68.11</b></td>+ <td><b>72.29</b></td>+ <td><b>905.88</b></td>+ <td><b>21.83GB</b></td>+ <td><b>12.37GB</b></td></tr><tr><td>AWQ_RTN</td>- <td>86.96</td>- <td>57.22</td>- <td>72.85</td>- <td>76.39</td>- <td>68.27</td>- <td>72.34</td>- <td>1143.95</td>- <td>21.93GB</td>- <td>6.93GB</td>+ <td>86.58</td>+ <td>57.05</td>+ <td>72.57</td>+ <td>76.77</td>+ <td>68.35</td>+ <td>72.26</td>+ <td>1046.69</td>+ <td>20.93GB</td>+ <td>6.89GB</td></tr><tr><td>AWQ_AR</td>- <td>87.41</td>- <td>56.80</td>- <td>72.65</td>- <td>76.71</td>- <td>67.64</td>- <td>72.24</td>- <td>1324.51</td>- <td>22.85GB</td>- <td>13.40GB</td>- </tr>- <tr>- <td><b>RTN_smooth2048_clip</b></td>- <td><b>87.87</b></td>- <td><b>57.14</b></td>- <td><b>72.90</b></td>- <td><b>76.66</b></td>- <td><b>67.96</b></td>- <td><b>72.51</b></td>- <td><b>1796.55</b></td>- <td><b>22.47GB</b></td>- <td><b>15.77GB</b></td>+ <td>86.81</td>+ <td>56.77</td>+ <td>72.58</td>+ <td>76.61</td>+ <td>68.11</td>+ <td>72.18</td>+ <td>2277.47</td>+ <td>21.92GB</td>+ <td>13.30GB</td></tr></table>@@ -674,16 +647,15 @@ In the observation tables, `AVG Win Rate` counts how often the AWQ-composed meth| Comparison | AVG Win Rate | Max AVG Delta Gain | Max AVG Rel. Gain || --- | ---: | ---: | ---: |-| `AWQ_RTN` vs `RTN` | 1/2 (50%) | +0.11 pts | +0.16% |-| `AWQ_AR` vs `AR` | 0/2 (0%) | -0.04 pts | -0.06% |+| `AWQ_RTN` vs `RTN` | 1/2 (50%) | +0.88 pts | +1.23% |+| `AWQ_AR` vs `AR` | 1/2 (50%) | +0.32 pts | +0.46% |- INT8/W8A8 is already a high-accuracy setting in these experiments. Most INT8 recipes are close to the BF16 reference, and some recipes slightly exceed BF16 AVG.-- The experimental `RTN_smooth2048_clip` rows give the best INT8 AVG for both models in this table: 71.04 on Llama and 72.51 on Qwen.-- In `RTN_smooth2048_clip`, `smooth2048` denotes AWQ calibration with sequence length 2048 (`awq_seqlen=2048`), and `clip` denotes AWQ weight clipping (`apply_clip=True`) enabled during the AWQ search.-- For AWQ-composed INT8, the AVG movement is small. Llama `AWQ_RTN` improves over `RTN` by +0.11 points (+0.16% relative), while the other AWQ-composed comparisons are slightly below their native counterparts.-- Given the strong INT8 baseline, AWQ is better positioned as an optional exploration path for INT8/W8A8 rather than a primary default.+- For AWQ-composed INT8, the AVG movement is small and model-dependent. `AWQ_RTN` improves over `RTN` on Qwen but is slightly below `RTN` on Llama; `AWQ_AR` improves over `AR` on Llama but is slightly below `AR` on Qwen.+- Given the strong INT8 baseline and small AWQ deltas, AWQ is better positioned as an optional exploration path for INT8/W8A8 rather than a primary default.## SummaryAWQ is most useful for schemes where quantization is sensitive to activation-aware weight smoothing, such as MXFP4. It can improve scheme accuracy, especially when combined with AutoRound plus `--enable_alg_ext`.+AWQ does not add persistent memory overhead to the quantized model and is orthogonal to terminal quantization algorithms, so it can be composed with RTN, AutoRound, or AutoRound2 without changing their core quantization logic.diff --git a/docs/awq_details_CN.md b/docs/awq_details_CN.mdindex ea8207bbc..d4ca852cb 100644--- a/docs/awq_details_CN.md+++ b/docs/awq_details_CN.md@@ -2,7 +2,7 @@本文总结了原生 AutoRound 流程与 AWQ 组合流程的实验对比。目标是评估在 `W4A16`、`MXFP4` 和 `INT8`(`W8A8`)方案下,在 RTN 或 AutoRound 之前加入 AWQ smoothing 是否能够提升准确率。-所有结果均使用 `--format fake` 测得,并在 `mmlu,gsm8k,piqa,hellaswag,winogrande` 上评估。这五个常用 LLM 基准覆盖互补的误差类型:MMLU 侧重事实知识与综合理解,GSM8K 侧重数学推理,PIQA 侧重物理常识,HellaSwag 侧重常识句子补全,Winogrande 侧重代词与指代消解。报告该任务集合的平均分,可以降低单个任务波动对结论的影响,并更稳定地反映量化引入的精度变化。除非特别说明,其余量化参数均采用 AutoRound 默认设置。+所有表格结果均基于 AutoRound 0.15.0 release,使用 `--format fake` 测得,并在 `mmlu,gsm8k,piqa,hellaswag,winogrande` 上评估。这五个常用 LLM 基准覆盖互补的误差类型:MMLU 侧重事实知识与综合理解,GSM8K 侧重数学推理,PIQA 侧重物理常识,HellaSwag 侧重常识句子补全,Winogrande 侧重代词与指代消解。报告该任务集合的平均分,可以降低单个任务波动对结论的影响,并更稳定地反映量化引入的精度变化。除非特别说明,其余量化参数均采用 AutoRound 默认设置。## AWQ 算法特点@@ -48,14 +48,14 @@ auto-round --model <model> --scheme <scheme> --format fake \--tasks "mmlu,gsm8k,piqa,hellaswag,winogrande"```-表中 `AR2` 表示启用 `--enable_alg_ext` 的 AutoRound,`AWQ_AR2` 表示 AWQ 后接启用 `--enable_alg_ext` 的 AutoRound。准确率列以百分制展示,例如原始值 `0.7058` 写作 `70.58`。BF16 作为参考基线放在每个模型分组的第一行。每个模型在每个 scheme 下平均分最高的量化行使用粗体标出。原始表格中有一行 Llama MXFP4 标为 `WQ_AR2`,本文将其视为 `AWQ_AR2`。+表中 `AR2` 表示启用 `--enable_alg_ext` 的 AutoRound,`AWQ_AR2` 表示 AWQ 后接启用 `--enable_alg_ext` 的 AutoRound。准确率列以百分制展示,例如原始值 `0.7058` 写作 `70.58`。++AWQ 相关行使用默认 AWQ smoothing 流程。除非显式设置 `--awq_apply_clip`,否则不包含 AWQ weight clipping。观察表中,`AVG Win Rate` 表示在当前 scheme 下所有可比较模型中,AWQ 组合方法 AVG 更高的比例;`Max AVG Delta Gain` 表示最大的 AVG 绝对提升(百分点);`Max AVG Rel. Gain` 表示最大的 AVG 相对提升。对于 `AWQ_RTN`,当 `RTN` 和 `opt_rtn` 都存在时,会同时纳入这两个基线进行比较。## MXFP4-**注意:对于 Llama-3.1-Instruct,我们建议移除 [modeling_llama.py](https://github.com/huggingface/transformers/blob/main/src/transformers/models/llama/modeling_llama.py#L52C1-L52C40) 中的 `@use_kernel_forward_from_hub("RMSNorm")`;下方的测试并未移除该装饰器,会导致精度测试结果不准确。**-<table border="1"><tr><td>Model</td>@@ -66,7 +66,7 @@ auto-round --model <model> --scheme <scheme> --format fake \<td>PIQA (%)</td><td>WinoGrande (%)</td><td>AVG (%)</td>- <td>Timecost</td>+ <td>Time cost (s)</td><td>RAM</td><td>VRAM</td></tr>@@ -83,14 +83,13 @@ auto-round --model <model> --scheme <scheme> --format fake \<td>-</td><td>-</td></tr>- <tr><td>RTN</td><td>59.06</td><td>58.88</td><td>64.63</td><td>78.94</td><td>72.77</td><td>66.86</td><td>72.30</td><td>25.52GB</td><td>0.83GB</td></tr>- <tr><td>opt_rtn</td><td>65.20</td><td>58.91</td><td>65.04</td><td>79.05</td><td>73.48</td><td>68.34</td><td>62.00</td><td>25.65GB</td><td>1.65GB</td></tr>- <!-- TODO: MXFP4 Llama-3.1 的 AR 数据异常(GSM8K=0.00,AVG=43.13),暂时视为初步结果,后续复核。 -->- <tr><td>AR</td><td>0.00</td><td>31.20</td><td>48.50</td><td>75.24</td><td>60.69</td><td>43.13</td><td>959.21</td><td>25.39GB</td><td>16.30GB</td></tr>- <tr><td>AR2</td><td>67.32</td><td>57.94</td><td>64.11</td><td>80.25</td><td>73.80</td><td>68.68</td><td>1104.91</td><td>24.38GB</td><td>17.78GB</td></tr>- <tr><td>AWQ_RTN</td><td>66.26</td><td>58.36</td><td>66.12</td><td>79.00</td><td>73.95</td><td>68.74</td><td>1132.04</td><td>24.36GB</td><td>7.60GB</td></tr>- <tr><td>AWQ_AR</td><td>69.75</td><td>57.74</td><td>65.71</td><td>79.27</td><td>72.69</td><td>69.03</td><td>2136.06</td><td>24.47GB</td><td>19.24GB</td></tr>- <tr><td><b>AWQ_AR2</b></td><td><b>71.27</b></td><td><b>58.29</b></td><td><b>66.07</b></td><td><b>80.25</b></td><td><b>73.88</b></td><td><b>69.95</b></td><td><b>2274.64</b></td><td><b>24.40GB</b></td><td><b>18.12GB</b></td></tr>+ <tr><td>RTN</td><td>50.11</td><td>55.54</td><td>57.34</td><td>76.01</td><td>70.01</td><td>61.80</td><td>88.00</td><td>9.83GB</td><td>2.99GB</td></tr>+ <tr><td>opt_rtn</td><td>52.08</td><td>55.82</td><td>57.86</td><td>76.88</td><td>71.11</td><td>62.75</td><td>100.00</td><td>10.02GB</td><td>3.84GB</td></tr>+ <tr><td>AR</td><td>60.73</td><td>55.29</td><td>61.34</td><td>77.53</td><td>70.01</td><td>64.98</td><td>976.75</td><td>24.33GB</td><td>16.86GB</td></tr>+ <tr><td>AR2</td><td>61.87</td><td>56.59</td><td>61.60</td><td>77.97</td><td>71.11</td><td>65.83</td><td>1151.81</td><td>24.38GB</td><td>17.97GB</td></tr>+ <tr><td>AWQ_RTN</td><td>50.27</td><td>55.51</td><td>58.55</td><td>76.93</td><td>70.56</td><td>62.36</td><td>993.51</td><td>23.36GB</td><td>7.91GB</td></tr>+ <tr><td>AWQ_AR</td><td>60.05</td><td>56.00</td><td>61.96</td><td>78.07</td><td>72.14</td><td>65.64</td><td>2042.67</td><td>24.35GB</td><td>18.80GB</td></tr>+ <tr><td><b>AWQ_AR2</b></td><td><b>62.02</b></td><td><b>56.40</b></td><td><b>61.84</b></td><td><b>78.62</b></td><td><b>72.14</b></td><td><b>66.20</b></td><td><b>2164.04</b></td><td><b>24.40GB</b></td><td><b>17.94GB</b></td></tr><tr><td rowspan="8">Qwen3-8B</td><td>BF16</td>@@ -104,29 +103,27 @@ auto-round --model <model> --scheme <scheme> --format fake \<td>-</td><td>-</td></tr>- <tr><td>RTN</td><td>86.05</td><td>55.07</td><td>70.30</td><td>75.41</td><td>66.93</td><td>70.75</td><td>61.00</td><td>23.23GB</td><td>0.74GB</td></tr>- <tr><td>opt_rtn</td><td>86.50</td><td>55.53</td><td>70.64</td><td>75.79</td><td>66.77</td><td>71.05</td><td>66.00</td><td>23.55GB</td><td>1.40GB</td></tr>- <tr><td>AR</td><td>86.73</td><td>54.90</td><td>71.56</td><td>76.44</td><td>70.09</td><td>71.94</td><td>968.45</td><td>22.32GB</td><td>15.68GB</td></tr>- <tr><td>AR2</td><td>86.58</td><td>55.07</td><td>71.86</td><td>76.61</td><td>69.77</td><td>71.98</td><td>1195.57</td><td>22.35GB</td><td>16.00GB</td></tr>- <tr><td>AWQ_RTN</td><td>85.90</td><td>55.79</td><td>71.54</td><td>76.93</td><td>67.80</td><td>71.59</td><td>1088.34</td><td>22.34GB</td><td>6.93GB</td></tr>- <tr><td>AWQ_AR</td><td>85.37</td><td>54.41</td><td>71.47</td><td>76.55</td><td>70.32</td><td>71.62</td><td>2346.84</td><td>22.45GB</td><td>16.75GB</td></tr>- <tr><td><b>AWQ_AR2</b></td><td><b>86.66</b></td><td><b>55.19</b></td><td><b>71.99</b></td><td><b>77.37</b></td><td><b>70.80</b></td><td><b>72.40</b></td><td><b>2266.50</b></td><td><b>22.37GB</b></td><td><b>16.16GB</b></td></tr>+ <tr><td>RTN</td><td>81.27</td><td>52.24</td><td>66.13</td><td>73.01</td><td>63.14</td><td>67.16</td><td>119.00</td><td>7.81GB</td><td>2.66GB</td></tr>+ <tr><td>opt_rtn</td><td>83.55</td><td>52.60</td><td>66.10</td><td>73.50</td><td>64.40</td><td>68.03</td><td>125.00</td><td>7.99GB</td><td>3.42GB</td></tr>+ <tr><td>AR</td><td>80.82</td><td>52.64</td><td>67.98</td><td>74.81</td><td>66.30</td><td>68.51</td><td>999.52</td><td>22.31GB</td><td>15.43GB</td></tr>+ <tr><td>AR2</td><td>83.70</td><td>53.41</td><td>68.51</td><td>75.57</td><td>67.56</td><td>69.75</td><td>1240.32</td><td>22.34GB</td><td>16.18GB</td></tr>+ <tr><td>AWQ_RTN</td><td>82.79</td><td>53.54</td><td>67.48</td><td>75.19</td><td>67.17</td><td>69.23</td><td>1073.95</td><td>21.33GB</td><td>7.14GB</td></tr>+ <tr><td>AWQ_AR</td><td>79.91</td><td>53.07</td><td>68.99</td><td>75.35</td><td>67.32</td><td>68.93</td><td>2095.77</td><td>22.32GB</td><td>16.72GB</td></tr>+ <tr><td><b>AWQ_AR2</b></td><td><b>83.70</b></td><td><b>53.89</b></td><td><b>69.54</b></td><td><b>75.57</b></td><td><b>68.19</b></td><td><b>70.18</b></td><td><b>2244.87</b></td><td><b>22.37GB</b></td><td><b>16.03GB</b></td></tr></table>### MXFP4 观察| Comparison | AVG Win Rate | Max AVG Delta Gain | Max AVG Rel. Gain || --- | ---: | ---: | ---: |-| `AWQ_RTN` vs `RTN` / `opt_rtn` | 4/4 (100%) | +1.88 pts | +2.81% |-| `AWQ_AR` vs `AR` | 1/2 (50%) | +25.90 pts | +60.05%* |-| `AWQ_AR2` vs `AR2` | 2/2 (100%) | +1.27 pts | +1.85% |+| `AWQ_RTN` vs `RTN` / `opt_rtn` | 3/4 (75%) | +2.07 pts | +3.08% |+| `AWQ_AR` vs `AR` | 2/2 (100%) | +0.66 pts | +1.02% |+| `AWQ_AR2` vs `AR2` | 2/2 (100%) | +0.43 pts | +0.62% |-*Llama 的 `AWQ_AR` vs `AR` 结果受到原生 `AR` 基线异常影响(`GSM8K=0.00`,`AVG=43.13`),因此相对提升偏高。--- MXFP4 是 AWQ 组合收益最清晰的场景。`AWQ_RTN` 在两个模型上都高于 `RTN` 和 `opt_rtn`,`AWQ_AR2` 在两个模型上都高于 `AR2`。-- `AWQ_AR2` 是 MXFP4 中更稳定的组合:两个模型 AVG 均提升,最高带来 +1.27 分、+1.85% 的 AVG 相对提升。-- `AWQ_AR` 并不稳定。Llama 的大幅收益主要来自原生 `AR` 基线异常,而 Qwen 上 `AWQ_AR` 的 AVG 低于原生 `AR`。-- 代价仍然是耗时和显存。`AWQ_RTN` 约为 `RTN` 耗时的 18 倍;在 MXFP4 上,`AWQ_AR` / `AWQ_AR2` 约为原生 `AR` / `AR2` 的 2 倍。+- 在这些实验中,MXFP4 仍然是 AWQ 组合收益最清晰的场景。AWQ 后接 AutoRound 或 AutoRound2 时,两个模型的 AVG 都有提升。+- `AWQ_AR2` 在两个模型上都取得 MXFP4 最高 AVG:Llama 为 66.20,Qwen 为 70.18。+- `AWQ_RTN` 在两个模型上都高于原始 `RTN`。这说明 AWQ smoothing 有价值,同时优化版 RTN 仍是低成本 MXFP4 强基线。+- 主要代价是一次性的运行时间。`AWQ_AR` 和 `AWQ_AR2` 约为对应原生 AutoRound 流程的 1.8x-2.1x,峰值 RAM/VRAM 仍主要由后续 AutoRound 优化器决定。## W4A16@@ -140,7 +137,7 @@ auto-round --model <model> --scheme <scheme> --format fake \<td>PIQA (%)</td><td>WinoGrande (%)</td><td>AVG (%)</td>- <td>Timecost</td>+ <td>Time cost (s)</td><td>RAM</td><td>VRAM</td></tr>@@ -210,12 +207,12 @@ auto-round --model <model> --scheme <scheme> --format fake \<td>PIQA (%)</td><td>WinoGrande (%)</td><td>AVG (%)</td>- <td>Timecost</td>+ <td>Time cost (s)</td><td>RAM</td><td>VRAM</td></tr><tr>- <td rowspan="6">Llama-3.1-8B-I</td>+ <td rowspan="5">Llama-3.1-8B-I</td><td>BF16</td><td>70.51</td><td>59.71</td>@@ -227,13 +224,12 @@ auto-round --model <model> --scheme <scheme> --format fake \<td>-</td><td>-</td></tr>- <tr><td>RTN</td><td>71.49</td><td>59.64</td><td>68.37</td><td>80.30</td><td>74.27</td><td>70.81</td><td>58.00</td><td>25.00GB</td><td>0.56GB</td></tr>- <tr><td>AR</td><td>69.90</td><td>59.62</td><td>68.36</td><td>80.36</td><td>73.64</td><td>70.38</td><td>836.48</td><td>24.90GB</td><td>13.63GB</td></tr>- <tr><td>AWQ_RTN</td><td>73.24</td><td>59.66</td><td>68.30</td><td>80.09</td><td>73.32</td><td>70.92</td><td>1052.19</td><td>23.98GB</td><td>7.67GB</td></tr>- <tr><td>AWQ_AR</td><td>68.99</td><td>59.77</td><td>68.12</td><td>80.30</td><td>73.09</td><td>70.05</td><td>1228.53</td><td>24.10GB</td><td>14.29GB</td></tr>- <tr><td><b>RTN_smooth2048_clip</b></td><td><b>72.93</b></td><td><b>59.89</b></td><td><b>68.27</b></td><td><b>80.14</b></td><td><b>73.95</b></td><td><b>71.04</b></td><td><b>1668.43</b></td><td><b>24.50GB</b></td><td><b>18.46GB</b></td></tr>+ <tr><td><b>RTN</b></td><td><b>72.25</b></td><td><b>59.73</b></td><td><b>68.10</b></td><td><b>79.71</b></td><td><b>74.82</b></td><td><b>70.92</b></td><td><b>98.00</b></td><td><b>9.53GB</b></td><td><b>0.85GB</b></td></tr>+ <tr><td>AR</td><td>68.99</td><td>59.63</td><td>67.87</td><td>79.71</td><td>74.11</td><td>70.06</td><td>988.81</td><td>23.87GB</td><td>13.38GB</td></tr>+ <tr><td>AWQ_RTN</td><td>72.63</td><td>59.54</td><td>68.13</td><td>79.82</td><td>73.72</td><td>70.77</td><td>1001.96</td><td>22.98GB</td><td>7.55GB</td></tr>+ <tr><td>AWQ_AR</td><td>70.05</td><td>59.65</td><td>68.20</td><td>79.87</td><td>74.11</td><td>70.38</td><td>1879.53</td><td>23.95GB</td><td>14.21GB</td></tr><tr>- <td rowspan="6">Qwen3-8B</td>+ <td rowspan="5">Qwen3-8B</td><td>BF16</td><td>87.41</td><td>57.16</td>@@ -245,48 +241,25 @@ auto-round --model <model> --scheme <scheme> --format fake \<td>-</td><td>-</td></tr>- <tr><td>RTN</td><td>87.79</td><td>57.09</td><td>72.90</td><td>76.77</td><td>67.80</td><td>72.47</td><td>84.00</td><td>23.09GB</td><td>0.50GB</td></tr>- <tr><td>AR</td><td>86.96</td><td>56.57</td><td>72.58</td><td>77.20</td><td>68.11</td><td>72.28</td><td>881.10</td><td>23.66GB</td><td>12.58GB</td></tr>- <tr><td>AWQ_RTN</td><td>86.96</td><td>57.22</td><td>72.85</td><td>76.39</td><td>68.27</td><td>72.34</td><td>1143.95</td><td>21.93GB</td><td>6.93GB</td></tr>- <tr><td>AWQ_AR</td><td>87.41</td><td>56.80</td><td>72.65</td><td>76.71</td><td>67.64</td><td>72.24</td><td>1324.51</td><td>22.85GB</td><td>13.40GB</td></tr>- <tr><td><b>RTN_smooth2048_clip</b></td><td><b>87.87</b></td><td><b>57.14</b></td><td><b>72.90</b></td><td><b>76.66</b></td><td><b>67.96</b></td><td><b>72.51</b></td><td><b>1796.55</b></td><td><b>22.47GB</b></td><td><b>15.77GB</b></td></tr>+ <tr><td>RTN</td><td>87.49</td><td>56.93</td><td>72.53</td><td>76.01</td><td>63.95</td><td>71.38</td><td>88.00</td><td>7.58GB</td><td>0.75GB</td></tr>+ <tr><td><b>AR</b></td><td><b>88.10</b></td><td><b>56.74</b></td><td><b>72.46</b></td><td><b>76.06</b></td><td><b>68.11</b></td><td><b>72.29</b></td><td><b>905.88</b></td><td><b>21.83GB</b></td><td><b>12.37GB</b></td></tr>+ <tr><td>AWQ_RTN</td><td>86.58</td><td>57.05</td><td>72.57</td><td>76.77</td><td>68.35</td><td>72.26</td><td>1046.69</td><td>20.93GB</td><td>6.89GB</td></tr>+ <tr><td>AWQ_AR</td><td>86.81</td><td>56.77</td><td>72.58</td><td>76.61</td><td>68.11</td><td>72.18</td><td>2277.47</td><td>21.92GB</td><td>13.30GB</td></tr></table>### INT8 观察| Comparison | AVG Win Rate | Max AVG Delta Gain | Max AVG Rel. Gain || --- | ---: | ---: | ---: |-| `AWQ_RTN` vs `RTN` | 1/2 (50%) | +0.11 pts | +0.16% |-| `AWQ_AR` vs `AR` | 0/2 (0%) | -0.04 pts | -0.06% |+| `AWQ_RTN` vs `RTN` | 1/2 (50%) | +0.88 pts | +1.23% |+| `AWQ_AR` vs `AR` | 1/2 (50%) | +0.32 pts | +0.46% |- INT8/W8A8 在这些实验中本身已经是高精度设置。大多数 INT8 recipe 的结果接近 BF16 参考水平,部分 recipe 的 AVG 还略高于 BF16。-- 实验性的 `RTN_smooth2048_clip` 行在两个模型的 INT8 表中都取得最高 AVG:Llama 为 71.04,Qwen 为 72.51。-- 在 `RTN_smooth2048_clip` 中,`smooth2048` 表示 AWQ 校准使用序列长度 2048(`awq_seqlen=2048`),`clip` 表示在 AWQ 搜索中启用权重裁剪(`apply_clip=True`)。-- 对 AWQ 组合的 INT8 路径而言,AVG 变化幅度较小。Llama `AWQ_RTN` 相比 `RTN` 提升 +0.11 分(AVG 相对提升 +0.16%),其他 AWQ 组合结果则略低于对应的原生量化路径。-- 考虑到 INT8 的原生基线已经较强,AWQ 更适合作为 INT8/W8A8 的可选探索路径,而不是首选默认路径。+- 对 AWQ 组合的 INT8 路径而言,AVG 变化幅度较小且依赖模型。`AWQ_RTN` 在 Qwen 上高于 `RTN`,但在 Llama 上略低于 `RTN`;`AWQ_AR` 在 Llama 上高于 `AR`,但在 Qwen 上略低于 `AR`。+- 考虑到 INT8 的原生基线已经较强,且 AWQ 带来的 AVG 变化较小,AWQ 更适合作为 INT8/W8A8 的可选探索路径,而不是首选默认路径。## 总结-### 优势--- AWQ 可以提升 MXFP4 准确率,尤其是与启用 `--enable_alg_ext` 的 AutoRound 组合时。-- 在本次 Llama W4A16 实验中,AWQ 有明显收益:`AWQ_AR` 平均分达到 70.48,略高于 BF16 参考结果。-- AWQ 不会给最终量化模型增加持续性内存开销。在校准阶段,`AWQ_RTN` 的 RAM 与其他流程接近,VRAM 明显低于迭代式 AutoRound 流程。-- AWQ 与终端量化算法正交,可以与 RTN、AutoRound 或 AutoRound2 组合,而不改变这些算法本身的核心量化逻辑。-- AWQ 更适合量化误差对 activation-aware weight smoothing 敏感的 scheme,例如 MXFP4。--### 劣势--- AWQ 会增加一次性的校准时间。即使是 `AWQ_RTN`,这些 8B 模型也需要约 18 分钟,而原始 `RTN` 约 1 分钟;当准确率收益明确时,这个离线时间成本是可以接受的。-- AWQ 对 W4A16 的提升不稳定;Qwen W4A16 的收益很小或接近中性。-- 对 INT8/W8A8,AWQ 带来的额外 AVG 收益有限,因为原生 INT8 基线已经接近 BF16。-- `AWQ_AR` 与 `AWQ_AR2` 会继承迭代式 AutoRound 优化的 GPU 内存特征。这些行里的较高 VRAM 应理解为后续优化器的成本,而不是 AWQ 给最终模型增加的持续性内存开销。-- AWQ 的收益依赖模型和 scheme,因此更适合作为可选组合,而不是通用默认配置。--## 使用建议+AWQ 更适合量化误差对 activation-aware weight smoothing 敏感的 scheme,例如 MXFP4。它可以提升 scheme 准确率,尤其是与启用 `--enable_alg_ext` 的 AutoRound 组合时。-- 当目标是 MXFP4,或某个 W4A16 模型已验证存在收益时,可以将 AWQ 作为可选预处理算法使用。-- 当目标 scheme 显示出明确准确率收益时,可以把 AWQ 的额外校准时间视为可接受的离线成本。-- AWQ 内部 QDQ 默认使用原始 RTN 是合理的。这些结果没有显示出在 AWQ 校准期间使用优化版 RTN 有明确的准确率/成本收益。-- 对不含 AWQ 的终端 RTN,仍应保留优化版 RTN,尤其是 MXFP4,因为它能以较低成本提升准确率。-- 对 INT8/W8A8,除非具体模型显示出明确收益,否则优先使用原生 RTN。+AWQ 不会给最终量化模型增加持续性内存开销,并且与终端量化算法正交,因此可以与 RTN、AutoRound 或 AutoRound2 组合,而不改变这些算法本身的核心量化逻辑。