PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
Add acc data (#89)
pursure-D8a3da1444233
diff --git a/README.md b/README.mdindex 532d62140..dc4130f06 100644--- a/README.md+++ b/README.md@@ -170,12 +170,12 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [sym recipe](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh), [asym recipe ](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh), [example](./examples/language-modeling/) || baichuan-inc/Baichuan2-7B-Chat | [accuracy](./docs/baichuan2-7b-chat-acc.md), [recipe](./examples/language-modeling/scripts/baichuan2-7b-chat.sh), [example](./examples/language-modeling/) || 01-ai/Yi-6B-Chat | [accuracy](./docs/Yi-6B-Chat-acc.md), [recipe](./examples/language-modeling/scripts/Yi-6B-Chat.sh), [example](./examples/language-modeling/) |+| facebook/opt-2.7b | [accuracy](./docs/opt-2.7b-acc.md), [recipe](./examples/language-modeling/scripts/opt-2.7b.sh), [example](./examples/language-modeling/) |+| bigscience/bloom-3b | [accuracy](./docs/bloom-3B-acc.md), [recipe](./examples/language-modeling/scripts/bloom-3b.sh), [example](./examples/language-modeling/) |+| EleutherAI/gpt-j-6b | [accuracy](./docs/gpt-j-6B-acc.md), [recipe](./examples/language-modeling/scripts/gpt-j-6b.sh), [example](./examples/language-modeling/) || Salesforce/codegen25-7b-multi | [example](./examples/code-generation) |-| EleutherAI/gpt-j-6b | [example](./examples/language-modeling/) || huggyllama/llama-7b | [example](./examples/language-modeling/) |-| facebook/opt-6.7b | [example](./examples/language-modeling/) ||| mosaicml/mpt-7b | [example](./examples/language-modeling/) |-| bigscience/bloom-7b1 | [example](./examples/language-modeling/) || THUDM/chatglm3-6b | [example](./examples/language-modeling/) || MBZUAI/LaMini-GPT-124M | [example](./examples/language-modeling/) || EleutherAI/gpt-neo-125m | [example](./examples/language-modeling/) |diff --git a/docs/acc.md b/docs/acc.mdindex 880228dce..48ea5bbce 100644--- a/docs/acc.md+++ b/docs/acc.md@@ -3268,27 +3268,6 @@ With these configurations, the tuning costs for GPTQ, AWQ, and ours are similar,<td>8.38 </td><td>-</td></tr>-- </tr>- <th>Ours iters=4K,minmax_lr=0.0005 </th>- <td>57.16 </td>- <td>50.28 </td>- <td>67.03 </td>- <td>55.37 </td>- <td>68.11 </td>- <td>77.53 </td>- <td>26.44 </td>- <td>26.00 </td>- <td>80.58 </td>- <td>58.12 </td>- <td>75.63 </td>- <td>43.69 </td>- <td>7.07 </td>- <td>51.88 </td>- <td>10.67 </td>- <td>-</td> <! lm-eval wikitext ppl -->- </tr>-</tr><th>Ours iters=1K </th><td>56.44 </td>@@ -3308,10 +3287,29 @@ With these configurations, the tuning costs for GPTQ, AWQ, and ours are similar,<td>10.71 </td><td>-</td> <! lm-eval wikitext ppl --></tr>+ </tr>+ <th>Ours iters=4K,minmax_lr=0.0005 </th>+ <td>57.16 </td>+ <td>50.28 </td>+ <td>67.03 </td>+ <td>55.37 </td>+ <td>68.11 </td>+ <td>77.53 </td>+ <td>26.44 </td>+ <td>26.00 </td>+ <td>80.58 </td>+ <td>58.12 </td>+ <td>75.63 </td>+ <td>43.69 </td>+ <td>7.07 </td>+ <td>51.88 </td>+ <td>10.67 </td>+ <td>-</td> <! lm-eval wikitext ppl -->+ </tr><tr><td rowspan="3">Meta/LLaMA-2-13B </td>- <th>fp16</th>+ <th>FP16</th><td>61.42 </td><td>52.86 </td><td>76.77 </td>@@ -3330,6 +3328,25 @@ With these configurations, the tuning costs for GPTQ, AWQ, and ours are similar,<td>7.90 </td></tr>+<tr>+ <th>Ours iters=1K,minmax_lr=0.002</th>+ <td>56.95 </td>+ <td>42.39 </td>+ <td>70.87 </td>+ <td>55.15 </td>+ <td>68.03 </td>+ <td>77.37 </td>+ <td>24.11 </td>+ <td>30.80 </td>+ <td>77.58 </td>+ <td>64.62 </td>+ <td>75.63 </td>+ <td>39.93 </td>+ <td>6.26 </td>+ <td>78.83 </td>+ <td>8.70 </td>+ <td>11.25 </td>+ </tr><tr><th>Ours iters=2K,minmax_lr=0.001</th><td>57.53 </td>@@ -3349,24 +3366,65 @@ With these configurations, the tuning costs for GPTQ, AWQ, and ours are similar,<td>8.70 </td><td>11.22 </td></tr>- <tr>- <th>Ours iters=1K,minmax_lr=0.002</th>- <td>56.95 </td>- <td>42.39 </td>- <td>70.87 </td>- <td>55.15 </td>- <td>68.03 </td>- <td>77.37 </td>++ <tr>+ <td rowspan="3">Meta/LLaMA-2-7B </td>+ <th>FP16</th>+ <td>57.98 </td>+ <td>42.69 </td>+ <td>73.90 </td>+ <td>57.15 </td>+ <td>68.90 </td>+ <td>78.07 </td>+ <td>25.21 </td>+ <td>31.40 </td>+ <td>77.74 </td>+ <td>62.82 </td>+ <td>76.35 </td>+ <td>43.52 </td>+ <td>5.47 </td>+ <td>37.92 </td>+ <td>7.26 </td>+ <td>8.79 </td>+ </tr>+ </tr>+ <th>Ours iters=1K,minmax_lr=0.002 </th>+ <td>52.29 </td>+ <td>27.14 </td>+ <td>65.48 </td>+ <td>50.25 </td>+ <td>66.61 </td>+ <td>74.54 </td><td>24.11 </td>- <td>30.80 </td>- <td>77.58 </td>- <td>64.62 </td>- <td>75.63 </td>- <td>39.93 </td>- <td>6.26 </td>- <td>78.83 </td>- <td>8.70 </td>- <td>11.25 </td>+ <td>29.80 </td>+ <td>73.30 </td>+ <td>56.68 </td>+ <td>70.20 </td>+ <td>37.12 </td>+ <td>8.72 </td>+ <td>1692.95 </td>+ <td>10.06 </td>+ <td>12.80 </td>+ </tr></tr>+ <th>Ours iters=2K,minmax_lr=0.0005 </th>+ <td>52.32 </td>+ <td>28.26 </td>+ <td>64.16 </td>+ <td>50.66 </td>+ <td>64.80 </td>+ <td>75.14 </td>+ <td>23.87 </td>+ <td>30.20 </td>+ <td>71.74 </td>+ <td>57.76 </td>+ <td>71.13 </td>+ <td>37.80 </td>+ <td>8.54 </td>+ <td>0.00 </td>+ <td>10.14 </td>+ <td>0.00 </td>+ </tr>+</table></table>diff --git a/docs/bloom-3B-acc.md b/docs/bloom-3B-acc.mdnew file mode 100644index 000000000..17c451974--- /dev/null+++ b/docs/bloom-3B-acc.md@@ -0,0 +1,23 @@+Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d+##pip install auto-gptq[triton]+##pip install triton==2.2.0+```bash+lm_eval --model hf --model_args pretrained="./",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32+```++++| Metric | FP16 | INT4 |+| -------------- | ------ | ------ |+| Avg. | 0.4532 | 0.4514 |+| mmlu | 0.2592 | 0.2537 |+| lambada_openai | 0.5176 | 0.5135 |+| hellaswag | 0.4136 | 0.4093 |+| winogrande | 0.5864 | 0.5856 |+| piqa | 0.7062 | 0.7095 |+| truthfulqa_mc1 | 0.2326 | 0.2264 |+| openbookqa | 0.2160 | 0.2140 |+| boolq | 0.6156 | 0.6199 |+| rte | 0.5632 | 0.5632 |+| arc_easy | 0.5947 | 0.5888 |+| arc_challenge | 0.2799 | 0.2816 |diff --git a/docs/gpt-j-6B-acc.md b/docs/gpt-j-6B-acc.mdnew file mode 100644index 000000000..66679a92c--- /dev/null+++ b/docs/gpt-j-6B-acc.md@@ -0,0 +1,23 @@+Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d+##pip install auto-gptq[triton]+##pip install triton==2.2.0+```bash+lm_eval --model hf --model_args pretrained="./",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32+```++++| Metric | FP16 | INT4 |+| -------------- | ------ | ------ |+| Avg. | 0.5039 | 0.5034 |+| mmlu | 0.2694 | 0.2793 |+| lambada_openai | 0.6831 | 0.6790 |+| hellaswag | 0.4953 | 0.4902 |+| winogrande | 0.6409 | 0.6401 |+| piqa | 0.7541 | 0.7465 |+| truthfulqa_mc1 | 0.2020 | 0.2179 |+| openbookqa | 0.2900 | 0.2900 |+| boolq | 0.6544 | 0.6554 |+| rte | 0.5451 | 0.5271 |+| arc_easy | 0.6692 | 0.6734 |+| arc_challenge | 0.3396 | 0.3387 |diff --git a/docs/opt-2.7b-acc.md b/docs/opt-2.7b-acc.mdnew file mode 100644index 000000000..796934dca--- /dev/null+++ b/docs/opt-2.7b-acc.md@@ -0,0 +1,23 @@+Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d+##pip install auto-gptq[triton]+##pip install triton==2.2.0+```bash+lm_eval --model hf --model_args pretrained="./",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32+```++++| Metric | FP16 | INT4 |+| -------------- | ------ | ------ |+| Avg. | 0.4722 | 0.4757 |+| mmlu | 0.2568 | 0.2636 |+| lambada_openai | 0.6359 | 0.6487 |+| hellaswag | 0.4585 | 0.4519 |+| winogrande | 0.6077 | 0.5967 |+| piqa | 0.7367 | 0.7410 |+| truthfulqa_mc1 | 0.2240 | 0.2338 |+| openbookqa | 0.2500 | 0.2380 |+| boolq | 0.6046 | 0.6505 |+| rte | 0.5451 | 0.5379 |+| arc_easy | 0.6077 | 0.6035 |+| arc_challenge | 0.2679 | 0.2671 |diff --git a/examples/language-modeling/README.md b/examples/language-modeling/README.mdindex 2d2758e18..9d04437ee 100644--- a/examples/language-modeling/README.md+++ b/examples/language-modeling/README.md@@ -18,7 +18,8 @@ The transformers version required varies across different types of models. Here,| facebook/opt-6.7b | 4.28/4.30/4.34/4.36 || tiiuae/falcon-7b | 4.28/4.30/4.34/4.36 || mosaicml/mpt-7b | 4.28/4.30/4.34/4.36 |-| bigscience/bloom-7b1 | 4.28/4.30/4.34/4.36 |+| bigscience/bloom-3b | 4.28/4.30/4.34/4.36 |+| facebook/opt-2.7b | 4.28/4.30/4.34/4.36 || baichuan-inc/Baichuan2-7B-Chat | 4.36 || Qwen/Qwen-7B | 4.28/4.30/4.34/4.36 || THUDM/chatglm3-6b | 4.34/4.36 |diff --git a/examples/language-modeling/scripts/bloom-3b.sh b/examples/language-modeling/scripts/bloom-3b.shnew file mode 100644index 000000000..e4b30f5f6--- /dev/null+++ b/examples/language-modeling/scripts/bloom-3b.sh@@ -0,0 +1,9 @@+python3 main.py \+--model_name bigscience/bloom-3b \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--use_quant_input \+--deployment_device 'gpu' \+--output_dir "./tmp_autoround"diff --git a/examples/language-modeling/scripts/gpt-j-6b.sh b/examples/language-modeling/scripts/gpt-j-6b.shnew file mode 100644index 000000000..f80dddb06--- /dev/null+++ b/examples/language-modeling/scripts/gpt-j-6b.sh@@ -0,0 +1,8 @@+python3 main.py \+--model_name EleutherAI/gpt-j-6b \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--deployment_device 'gpu' \+--output_dir "./tmp_autoround"diff --git a/examples/language-modeling/scripts/opt-2.7b.sh b/examples/language-modeling/scripts/opt-2.7b.shnew file mode 100644index 000000000..8d6200cde--- /dev/null+++ b/examples/language-modeling/scripts/opt-2.7b.sh@@ -0,0 +1,9 @@+python3 main.py \+--model_name facebook/opt-2.7b \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--minmax_lr 0.002 \+--deployment_device 'gpu' \+--output_dir "./tmp_autoround"