Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

add Yi-6b-chat results (#85)

yintong-lud38a4a56ed73
diff --git a/README.md b/README.md
index 4ecb22260..532d62140 100644
--- a/README.md
+++ b/README.md
@@ -169,6 +169,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
| meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) |
| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [sym recipe](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh), [asym recipe ](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh), [example](./examples/language-modeling/) |
| baichuan-inc/Baichuan2-7B-Chat | [accuracy](./docs/baichuan2-7b-chat-acc.md), [recipe](./examples/language-modeling/scripts/baichuan2-7b-chat.sh), [example](./examples/language-modeling/) |
+| 01-ai/Yi-6B-Chat | [accuracy](./docs/Yi-6B-Chat-acc.md), [recipe](./examples/language-modeling/scripts/Yi-6B-Chat.sh), [example](./examples/language-modeling/) |
| Salesforce/codegen25-7b-multi | [example](./examples/code-generation) |
| EleutherAI/gpt-j-6b | [example](./examples/language-modeling/) |
| huggyllama/llama-7b | [example](./examples/language-modeling/) |
diff --git a/docs/Qwen1.5-7B-Chat-acc.md b/docs/Qwen1.5-7B-Chat-acc.md
index 9616ca077..2add074c7 100644
--- a/docs/Qwen1.5-7B-Chat-acc.md
+++ b/docs/Qwen1.5-7B-Chat-acc.md
@@ -1,15 +1,16 @@
Due to licensing restrictions, we are unable to release the model. Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d.
We used the following command for evaluation.
+For reference, the results of official AWQ-INT4 and GPTQ-INT4 release are listed.
~~~bash
lm_eval --model hf --model_args pretrained="./",autogptq=True,gptq_use_triton=True,trust_remote_code=True --device cuda:0 --tasks ceval-valid,cmmlu,mmlu,gsm8k --batch_size 16 --num_fewshot 0
~~~
-| Metric | BF16 | INT4 sym recipe | INT4 asym recipe |
-| -------------- | ------ |-----------------|------------------|
-| Avg. | 0.6231 | 0.6205 | 0.6186 |
-| ceval | 0.6887 | 0.6761 | 0.6820 |
-| cmmlu | 0.6959 | 0.6870 | 0.6884 |
-| mmlu | 0.6020 | 0.5974 | 0.5946 |
-| gsm8k | 0.5057 | 0.5216 | 0.5095 |
+| Metric | BF16 | [Qwen/Qwen1.5-7B-Chat-AWQ](https://huggingface.co/Qwen/Qwen1.5-7B-Chat-AWQ) | [Qwen/Qwen1.5-7B-Chat-GPTQ-Int4](https://huggingface.co/Qwen/Qwen1.5-7B-Chat-GPTQ-Int4) | INT4 sym recipe | INT4 asym recipe |
+| -------------- | ------ |-----------|--------------------------------|-----------------|------------------|
+| Avg. | 0.6231 | 0.6152 | 0.6070 | 0.6205 | 0.6186 |
+| ceval | 0.6887 | 0.6820 | 0.6679 | 0.6761 | 0.6820 |
+| cmmlu | 0.6959 | 0.6862 | 0.6831 | 0.6870 | 0.6884 |
+| mmlu | 0.6020 | 0.5944 | 0.5902 | 0.5974 | 0.5946 |
+| gsm8k | 0.5057 | 0.4981 | 0.4867 | 0.5216 | 0.5095 |
diff --git a/docs/Yi-6B-Chat-acc.md b/docs/Yi-6B-Chat-acc.md
new file mode 100644
index 000000000..f910d9195
--- /dev/null
+++ b/docs/Yi-6B-Chat-acc.md
@@ -0,0 +1,16 @@
+Due to licensing restrictions, we are unable to release the model. Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d.
+
+We used the following command for evaluation.
+For reference, the results of official AWQ-INT4 release are listed.
+
+~~~bash
+lm_eval --model hf --model_args pretrained="./",autogptq=True,gptq_use_triton=True,trust_remote_code=True --device cuda:0 --tasks ceval-valid,cmmlu,mmlu,gsm8k --batch_size 16 --num_fewshot 0
+~~~
+
+| Metric | BF16 |[01-ai/Yi-6B-Chat-4bits](https://huggingface.co/01-ai/Yi-6B-Chat-4bits)| INT4 |
+|--------|--------|----------------------|--------|
+| Avg. | 0.6043 | 0.5867 | 0.5939 |
+| mmlu | 0.6163 | 0.6133 | 0.6119 |
+| cmmlu | 0.7431 | 0.7312 | 0.7314 |
+| ceval | 0.7355 | 0.7155 | 0.7281 |
+| gsm8k | 0.3222 | 0.2866 | 0.3040 |
diff --git a/examples/language-modeling/scripts/Yi-6B-Chat.sh b/examples/language-modeling/scripts/Yi-6B-Chat.sh
new file mode 100644
index 000000000..514285b28
--- /dev/null
+++ b/examples/language-modeling/scripts/Yi-6B-Chat.sh
@@ -0,0 +1,9 @@
+## Due to licensing restrictions, we are unable to release the model.
+python3 main.py \
+--model_name 01-ai/Yi-6B-Chat \
+--group_size 128 \
+--bits 4 \
+--iters 1000 \
+--deployment_device 'gpu' \
+--minmax_lr 2e-3 \
+--use_quant_input