PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
update omniquant accuracy (#62)
pursure-Dbf97215b662c
diff --git a/docs/acc.md b/docs/acc.mdindex 1dd017818..4139f826b 100644--- a/docs/acc.md+++ b/docs/acc.md@@ -1,15 +1,15 @@## Comparison with other methods-To ensure a fair comparison as much as possible and alleviate overfitting in perplexity evaluation on Wikitext or C4, we utilized 512 samples from NeelNanda/pile-10k for all methods during calibration unless explicitly stated. For wikitext2/ptb-new/c4-new ppl, we follow the code of gptq and set the sequence length to 2048. For lm-eval wikitext ppl, we adopt lm-eval. The lm-eval-harness git id we used in the following is 008fc2a23245c40384f2312718433eeb1e0f87a9 and we evaluated on qdq fake models.+To ensure a fair comparison as much as possible and alleviate overfitting in perplexity evaluation on Wikitext or C4, we utilized 512 samples from NeelNanda/pile-10k for all methods during calibration unless explicitly stated. For wikitext2/ptb-new/c4-new ppl, we follow the code of gptq and set the sequence length to 2048. For lm-eval wikitext ppl, we adopt lm-eval. The lm-eval-harness git id we used in the following is 008fc2a23245c40384f2312718433eeb1e0f87a9 and we evaluated on qdq fake models.Due to memory constraints, we maintained the original sequence length of 512 for AWQ, while for GPTQ,Omniquant and our approach, a sequence length of 2048 is used. And HQQ is a data free method, no need to calibrate.-For GPTQ, we have enabled act-order and true-seqential, and also activated static group in scenarios where group_size!=-1. The notation GPTQ* indicates that we adjusted the random seed or data preprocessing to address issues related to the non-positive definite Hessian matrix or other issues.+For GPTQ, we have enabled act-order and true-seqential, and also activated static group in scenarios where group_size!=-1. The notation GPTQ* indicates that we adjusted the random seed or data preprocessing to address issues related to the non-positive definite Hessian matrix or other issues.-For Omniquant, we adhere to the official settings, which include running for 20 epochs and disabling 'let'. We conducted calibration tests using sample sizes of 512 and 128, as well as a sample size of 512 with a batch size of 4. Our findings show that using a sample size of 512 typically results in comparable or slight higher performance for models <=13B. Therefore, we present the results based on the sample size of 512. For 70B models, due the the NAN loss issue and to reduce the tuning cost, we adopted 128 samples for calibration.--With these configurations, the calibration costs for GPTQ, AWQ, and ours are similar, while HQQ is much faster and Omniquant is noticebal slower.+For Omniquant, we adhere to the official settings, which include running for 20 epochs and disabling 'let'. We conducted calibration tests using sample sizes of 512 and 128, as well as a sample size of 512 with a batch size of 4. Our findings show that using a sample size of 512 typically results in comparable or slight higher performance for models <=13B. Therefore, we present the results based on the sample size of 512. For 70B models, due the the NAN loss issue and to reduce the tuning cost, we adopted 128 samples for calibration.+For AutoRound, we used the default setting, iters 200, use_quant_input and enable_minmax_tuning, both the lr and minmax_lr are set to 1/iters,i.e. 5e-3.+With these configurations, the tuning costs for GPTQ, AWQ, and ours are similar, while HQQ is much faster and Omniquant is noticebal slower.</br>@@ -1853,7 +1853,7 @@ With these configurations, the calibration costs for GPTQ, AWQ, and ours are sim<td>59.44</td></tr><tr>- <td rowspan="6">V2-70B</td>+ <td rowspan="7">V2-70B</td><td>FP16</td><td>66.23</td><td>79.64</td>@@ -1928,6 +1928,21 @@ With these configurations, the calibration costs for GPTQ, AWQ, and ours are sim<td>53.67 </td><td>64.80 </td></tr>+ <tr>+ <td>Omniquant</td>+ <td>63.18 </td>+ <td>78.63 </td>+ <td>63.54 </td>+ <td>76.48 </td>+ <td>81.50 </td>+ <td>30.35 </td>+ <td>35.80 </td>+ <td>82.57 </td>+ <td>70.40 </td>+ <td>81.02 </td>+ <td>52.82 </td>+ <td>65.12 </td>+ </tr><tr><td>Ours</td><td>64.94</td>