PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
update W2g32 accuracy (#74)
pursure-D16d830b4a686
diff --git a/docs/acc.md b/docs/acc.mdindex 4139f826b..880228dce 100644--- a/docs/acc.md+++ b/docs/acc.md@@ -3016,9 +3016,7 @@ With these configurations, the tuning costs for GPTQ, AWQ, and ours are similar,</table>---## Other data+## Other data W4G128<table border="1"><tr><th>Model</th>@@ -3103,7 +3101,6 @@ With these configurations, the tuning costs for GPTQ, AWQ, and ours are similar,<td>-</td> <! lm-eval wikitext ppl --></tr>-<tr><td rowspan="3">mistralai/Mixtral-8x7B-v0.1 </td><th>BF16</th>@@ -3205,7 +3202,6 @@ With these configurations, the tuning costs for GPTQ, AWQ, and ours are similar,<td>11.37</td></tr>-</tr><th>Ours iters=1K,use_quant_input=False </th><td>61.47</td> <! acc avg -->@@ -3225,4 +3221,152 @@ With these configurations, the tuning costs for GPTQ, AWQ, and ours are similar,<td>14.37</td> <! c4_new ppl --><td>11.35</td> <! lm-eval wikitext ppl --></tr>+</br>++</table>++### Other data W2G32+<table border="1">+ <tr>+ <th>Model</th>+ <th>Method </th>+ <th>Acc AVG.</th>+ <th>MMLU</th>+ <th>Lamb.</th>+ <th>Hella.</th>+ <th>Wino.</th>+ <th>Piqa</th>+ <th>Truth.</th>+ <th>Open.</th>+ <th>Boolq</th>+ <th>RTE</th>+ <th>ARC-e</th>+ <th>ARC-c.</th>+ <th>wikitext2 ppl+ <th>ptb_new ppl</th>+ <th>c4_new ppl</th>+ <th>lm_eval wikitext ppl</th>++ </tr>+ <tr>+ <td rowspan="3">mistralai/Mistral-7B </td>+ <th>FP16</th>+ <td>63.30 </td>+ <td>61.35 </td>+ <td>75.68 </td>+ <td>61.27 </td>+ <td>74.03 </td>+ <td>80.79 </td>+ <td>28.03 </td>+ <td>32.80 </td>+ <td>83.67 </td>+ <td>67.51 </td>+ <td>80.81 </td>+ <td>50.34 </td>+ <td>5.25 </td>+ <td>35.00 </td>+ <td>8.38 </td>+ <td>-</td>+ </tr>++ </tr>+ <th>Ours iters=4K,minmax_lr=0.0005 </th>+ <td>57.16 </td>+ <td>50.28 </td>+ <td>67.03 </td>+ <td>55.37 </td>+ <td>68.11 </td>+ <td>77.53 </td>+ <td>26.44 </td>+ <td>26.00 </td>+ <td>80.58 </td>+ <td>58.12 </td>+ <td>75.63 </td>+ <td>43.69 </td>+ <td>7.07 </td>+ <td>51.88 </td>+ <td>10.67 </td>+ <td>-</td> <! lm-eval wikitext ppl -->+ </tr>++ </tr>+ <th>Ours iters=1K </th>+ <td>56.44 </td>+ <td>47.38 </td>+ <td>67.26 </td>+ <td>55.06 </td>+ <td>67.88 </td>+ <td>77.75 </td>+ <td>26.19 </td>+ <td>26.40 </td>+ <td>78.07 </td>+ <td>58.12 </td>+ <td>74.20 </td>+ <td>42.49 </td>+ <td>7.14 </td>+ <td>56.78 </td>+ <td>10.71 </td>+ <td>-</td> <! lm-eval wikitext ppl -->+ </tr>++ <tr>+ <td rowspan="3">Meta/LLaMA-2-13B </td>+ <th>fp16</th>+ <td>61.42 </td>+ <td>52.86 </td>+ <td>76.77 </td>+ <td>60.04 </td>+ <td>72.14 </td>+ <td>79.05 </td>+ <td>25.95 </td>+ <td>35.20 </td>+ <td>80.55 </td>+ <td>65.34 </td>+ <td>79.38 </td>+ <td>48.38 </td>+ <td>4.88 </td>+ <td>50.93 </td>+ <td>6.73 </td>+ <td>7.90 </td>++ </tr>+ <tr>+ <th>Ours iters=2K,minmax_lr=0.001</th>+ <td>57.53 </td>+ <td>44.42 </td>+ <td>71.63 </td>+ <td>55.23 </td>+ <td>68.03 </td>+ <td>76.66 </td>+ <td>24.48 </td>+ <td>32.00 </td>+ <td>76.91 </td>+ <td>65.70 </td>+ <td>76.09 </td>+ <td>41.64 </td>+ <td>6.27 </td>+ <td>75.40 </td>+ <td>8.70 </td>+ <td>11.22 </td>+ </tr>+ <tr>+ <th>Ours iters=1K,minmax_lr=0.002</th>+ <td>56.95 </td>+ <td>42.39 </td>+ <td>70.87 </td>+ <td>55.15 </td>+ <td>68.03 </td>+ <td>77.37 </td>+ <td>24.11 </td>+ <td>30.80 </td>+ <td>77.58 </td>+ <td>64.62 </td>+ <td>75.63 </td>+ <td>39.93 </td>+ <td>6.26 </td>+ <td>78.83 </td>+ <td>8.70 </td>+ <td>11.25 </td>+ </tr>+</table></table>