Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

Add acc data (#89)

pursure-D8a3da1444233
diff --git a/README.md b/README.md
index 532d62140..dc4130f06 100644
--- a/README.md
+++ b/README.md
@@ -170,12 +170,12 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [sym recipe](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh), [asym recipe ](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh), [example](./examples/language-modeling/) |
| baichuan-inc/Baichuan2-7B-Chat | [accuracy](./docs/baichuan2-7b-chat-acc.md), [recipe](./examples/language-modeling/scripts/baichuan2-7b-chat.sh), [example](./examples/language-modeling/) |
| 01-ai/Yi-6B-Chat | [accuracy](./docs/Yi-6B-Chat-acc.md), [recipe](./examples/language-modeling/scripts/Yi-6B-Chat.sh), [example](./examples/language-modeling/) |
+| facebook/opt-2.7b | [accuracy](./docs/opt-2.7b-acc.md), [recipe](./examples/language-modeling/scripts/opt-2.7b.sh), [example](./examples/language-modeling/) |
+| bigscience/bloom-3b | [accuracy](./docs/bloom-3B-acc.md), [recipe](./examples/language-modeling/scripts/bloom-3b.sh), [example](./examples/language-modeling/) |
+| EleutherAI/gpt-j-6b | [accuracy](./docs/gpt-j-6B-acc.md), [recipe](./examples/language-modeling/scripts/gpt-j-6b.sh), [example](./examples/language-modeling/) |
| Salesforce/codegen25-7b-multi | [example](./examples/code-generation) |
-| EleutherAI/gpt-j-6b | [example](./examples/language-modeling/) |
| huggyllama/llama-7b | [example](./examples/language-modeling/) |
-| facebook/opt-6.7b | [example](./examples/language-modeling/) ||
| mosaicml/mpt-7b | [example](./examples/language-modeling/) |
-| bigscience/bloom-7b1 | [example](./examples/language-modeling/) |
| THUDM/chatglm3-6b | [example](./examples/language-modeling/) |
| MBZUAI/LaMini-GPT-124M | [example](./examples/language-modeling/) |
| EleutherAI/gpt-neo-125m | [example](./examples/language-modeling/) |
diff --git a/docs/acc.md b/docs/acc.md
index 880228dce..48ea5bbce 100644
--- a/docs/acc.md
+++ b/docs/acc.md
@@ -3268,27 +3268,6 @@ With these configurations, the tuning costs for GPTQ, AWQ, and ours are similar,
<td>8.38 </td>
<td>-</td>
</tr>
-
- </tr>
- <th>Ours iters=4K,minmax_lr=0.0005 </th>
- <td>57.16 </td>
- <td>50.28 </td>
- <td>67.03 </td>
- <td>55.37 </td>
- <td>68.11 </td>
- <td>77.53 </td>
- <td>26.44 </td>
- <td>26.00 </td>
- <td>80.58 </td>
- <td>58.12 </td>
- <td>75.63 </td>
- <td>43.69 </td>
- <td>7.07 </td>
- <td>51.88 </td>
- <td>10.67 </td>
- <td>-</td> <! lm-eval wikitext ppl -->
- </tr>
-
</tr>
<th>Ours iters=1K </th>
<td>56.44 </td>
@@ -3308,10 +3287,29 @@ With these configurations, the tuning costs for GPTQ, AWQ, and ours are similar,
<td>10.71 </td>
<td>-</td> <! lm-eval wikitext ppl -->
</tr>
+ </tr>
+ <th>Ours iters=4K,minmax_lr=0.0005 </th>
+ <td>57.16 </td>
+ <td>50.28 </td>
+ <td>67.03 </td>
+ <td>55.37 </td>
+ <td>68.11 </td>
+ <td>77.53 </td>
+ <td>26.44 </td>
+ <td>26.00 </td>
+ <td>80.58 </td>
+ <td>58.12 </td>
+ <td>75.63 </td>
+ <td>43.69 </td>
+ <td>7.07 </td>
+ <td>51.88 </td>
+ <td>10.67 </td>
+ <td>-</td> <! lm-eval wikitext ppl -->
+ </tr>
<tr>
<td rowspan="3">Meta/LLaMA-2-13B </td>
- <th>fp16</th>
+ <th>FP16</th>
<td>61.42 </td>
<td>52.86 </td>
<td>76.77 </td>
@@ -3330,6 +3328,25 @@ With these configurations, the tuning costs for GPTQ, AWQ, and ours are similar,
<td>7.90 </td>
</tr>
+<tr>
+ <th>Ours iters=1K,minmax_lr=0.002</th>
+ <td>56.95 </td>
+ <td>42.39 </td>
+ <td>70.87 </td>
+ <td>55.15 </td>
+ <td>68.03 </td>
+ <td>77.37 </td>
+ <td>24.11 </td>
+ <td>30.80 </td>
+ <td>77.58 </td>
+ <td>64.62 </td>
+ <td>75.63 </td>
+ <td>39.93 </td>
+ <td>6.26 </td>
+ <td>78.83 </td>
+ <td>8.70 </td>
+ <td>11.25 </td>
+ </tr>
<tr>
<th>Ours iters=2K,minmax_lr=0.001</th>
<td>57.53 </td>
@@ -3349,24 +3366,65 @@ With these configurations, the tuning costs for GPTQ, AWQ, and ours are similar,
<td>8.70 </td>
<td>11.22 </td>
</tr>
- <tr>
- <th>Ours iters=1K,minmax_lr=0.002</th>
- <td>56.95 </td>
- <td>42.39 </td>
- <td>70.87 </td>
- <td>55.15 </td>
- <td>68.03 </td>
- <td>77.37 </td>
+
+ <tr>
+ <td rowspan="3">Meta/LLaMA-2-7B </td>
+ <th>FP16</th>
+ <td>57.98 </td>
+ <td>42.69 </td>
+ <td>73.90 </td>
+ <td>57.15 </td>
+ <td>68.90 </td>
+ <td>78.07 </td>
+ <td>25.21 </td>
+ <td>31.40 </td>
+ <td>77.74 </td>
+ <td>62.82 </td>
+ <td>76.35 </td>
+ <td>43.52 </td>
+ <td>5.47 </td>
+ <td>37.92 </td>
+ <td>7.26 </td>
+ <td>8.79 </td>
+ </tr>
+ </tr>
+ <th>Ours iters=1K,minmax_lr=0.002 </th>
+ <td>52.29 </td>
+ <td>27.14 </td>
+ <td>65.48 </td>
+ <td>50.25 </td>
+ <td>66.61 </td>
+ <td>74.54 </td>
<td>24.11 </td>
- <td>30.80 </td>
- <td>77.58 </td>
- <td>64.62 </td>
- <td>75.63 </td>
- <td>39.93 </td>
- <td>6.26 </td>
- <td>78.83 </td>
- <td>8.70 </td>
- <td>11.25 </td>
+ <td>29.80 </td>
+ <td>73.30 </td>
+ <td>56.68 </td>
+ <td>70.20 </td>
+ <td>37.12 </td>
+ <td>8.72 </td>
+ <td>1692.95 </td>
+ <td>10.06 </td>
+ <td>12.80 </td>
+ </tr>
</tr>
+ <th>Ours iters=2K,minmax_lr=0.0005 </th>
+ <td>52.32 </td>
+ <td>28.26 </td>
+ <td>64.16 </td>
+ <td>50.66 </td>
+ <td>64.80 </td>
+ <td>75.14 </td>
+ <td>23.87 </td>
+ <td>30.20 </td>
+ <td>71.74 </td>
+ <td>57.76 </td>
+ <td>71.13 </td>
+ <td>37.80 </td>
+ <td>8.54 </td>
+ <td>0.00 </td>
+ <td>10.14 </td>
+ <td>0.00 </td>
+ </tr>
+
</table>
</table>
diff --git a/docs/bloom-3B-acc.md b/docs/bloom-3B-acc.md
new file mode 100644
index 000000000..17c451974
--- /dev/null
+++ b/docs/bloom-3B-acc.md
@@ -0,0 +1,23 @@
+Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d
+##pip install auto-gptq[triton]
+##pip install triton==2.2.0
+```bash
+lm_eval --model hf --model_args pretrained="./",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32
+```
+
+
+
+| Metric | FP16 | INT4 |
+| -------------- | ------ | ------ |
+| Avg. | 0.4532 | 0.4514 |
+| mmlu | 0.2592 | 0.2537 |
+| lambada_openai | 0.5176 | 0.5135 |
+| hellaswag | 0.4136 | 0.4093 |
+| winogrande | 0.5864 | 0.5856 |
+| piqa | 0.7062 | 0.7095 |
+| truthfulqa_mc1 | 0.2326 | 0.2264 |
+| openbookqa | 0.2160 | 0.2140 |
+| boolq | 0.6156 | 0.6199 |
+| rte | 0.5632 | 0.5632 |
+| arc_easy | 0.5947 | 0.5888 |
+| arc_challenge | 0.2799 | 0.2816 |
diff --git a/docs/gpt-j-6B-acc.md b/docs/gpt-j-6B-acc.md
new file mode 100644
index 000000000..66679a92c
--- /dev/null
+++ b/docs/gpt-j-6B-acc.md
@@ -0,0 +1,23 @@
+Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d
+##pip install auto-gptq[triton]
+##pip install triton==2.2.0
+```bash
+lm_eval --model hf --model_args pretrained="./",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32
+```
+
+
+
+| Metric | FP16 | INT4 |
+| -------------- | ------ | ------ |
+| Avg. | 0.5039 | 0.5034 |
+| mmlu | 0.2694 | 0.2793 |
+| lambada_openai | 0.6831 | 0.6790 |
+| hellaswag | 0.4953 | 0.4902 |
+| winogrande | 0.6409 | 0.6401 |
+| piqa | 0.7541 | 0.7465 |
+| truthfulqa_mc1 | 0.2020 | 0.2179 |
+| openbookqa | 0.2900 | 0.2900 |
+| boolq | 0.6544 | 0.6554 |
+| rte | 0.5451 | 0.5271 |
+| arc_easy | 0.6692 | 0.6734 |
+| arc_challenge | 0.3396 | 0.3387 |
diff --git a/docs/opt-2.7b-acc.md b/docs/opt-2.7b-acc.md
new file mode 100644
index 000000000..796934dca
--- /dev/null
+++ b/docs/opt-2.7b-acc.md
@@ -0,0 +1,23 @@
+Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d
+##pip install auto-gptq[triton]
+##pip install triton==2.2.0
+```bash
+lm_eval --model hf --model_args pretrained="./",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32
+```
+
+
+
+| Metric | FP16 | INT4 |
+| -------------- | ------ | ------ |
+| Avg. | 0.4722 | 0.4757 |
+| mmlu | 0.2568 | 0.2636 |
+| lambada_openai | 0.6359 | 0.6487 |
+| hellaswag | 0.4585 | 0.4519 |
+| winogrande | 0.6077 | 0.5967 |
+| piqa | 0.7367 | 0.7410 |
+| truthfulqa_mc1 | 0.2240 | 0.2338 |
+| openbookqa | 0.2500 | 0.2380 |
+| boolq | 0.6046 | 0.6505 |
+| rte | 0.5451 | 0.5379 |
+| arc_easy | 0.6077 | 0.6035 |
+| arc_challenge | 0.2679 | 0.2671 |
diff --git a/examples/language-modeling/README.md b/examples/language-modeling/README.md
index 2d2758e18..9d04437ee 100644
--- a/examples/language-modeling/README.md
+++ b/examples/language-modeling/README.md
@@ -18,7 +18,8 @@ The transformers version required varies across different types of models. Here,
| facebook/opt-6.7b | 4.28/4.30/4.34/4.36 |
| tiiuae/falcon-7b | 4.28/4.30/4.34/4.36 |
| mosaicml/mpt-7b | 4.28/4.30/4.34/4.36 |
-| bigscience/bloom-7b1 | 4.28/4.30/4.34/4.36 |
+| bigscience/bloom-3b | 4.28/4.30/4.34/4.36 |
+| facebook/opt-2.7b | 4.28/4.30/4.34/4.36 |
| baichuan-inc/Baichuan2-7B-Chat | 4.36 |
| Qwen/Qwen-7B | 4.28/4.30/4.34/4.36 |
| THUDM/chatglm3-6b | 4.34/4.36 |
diff --git a/examples/language-modeling/scripts/bloom-3b.sh b/examples/language-modeling/scripts/bloom-3b.sh
new file mode 100644
index 000000000..e4b30f5f6
--- /dev/null
+++ b/examples/language-modeling/scripts/bloom-3b.sh
@@ -0,0 +1,9 @@
+python3 main.py \
+--model_name bigscience/bloom-3b \
+--device 0 \
+--group_size 128 \
+--bits 4 \
+--iters 1000 \
+--use_quant_input \
+--deployment_device 'gpu' \
+--output_dir "./tmp_autoround"
diff --git a/examples/language-modeling/scripts/gpt-j-6b.sh b/examples/language-modeling/scripts/gpt-j-6b.sh
new file mode 100644
index 000000000..f80dddb06
--- /dev/null
+++ b/examples/language-modeling/scripts/gpt-j-6b.sh
@@ -0,0 +1,8 @@
+python3 main.py \
+--model_name EleutherAI/gpt-j-6b \
+--device 0 \
+--group_size 128 \
+--bits 4 \
+--iters 1000 \
+--deployment_device 'gpu' \
+--output_dir "./tmp_autoround"
diff --git a/examples/language-modeling/scripts/opt-2.7b.sh b/examples/language-modeling/scripts/opt-2.7b.sh
new file mode 100644
index 000000000..8d6200cde
--- /dev/null
+++ b/examples/language-modeling/scripts/opt-2.7b.sh
@@ -0,0 +1,9 @@
+python3 main.py \
+--model_name facebook/opt-2.7b \
+--device 0 \
+--group_size 128 \
+--bits 4 \
+--iters 1000 \
+--minmax_lr 0.002 \
+--deployment_device 'gpu' \
+--output_dir "./tmp_autoround"