PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
add Yi-6b-chat results (#85)
yintong-lud38a4a56ed73
diff --git a/README.md b/README.mdindex 4ecb22260..532d62140 100644--- a/README.md+++ b/README.md@@ -169,6 +169,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))| meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) || Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [sym recipe](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh), [asym recipe ](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh), [example](./examples/language-modeling/) || baichuan-inc/Baichuan2-7B-Chat | [accuracy](./docs/baichuan2-7b-chat-acc.md), [recipe](./examples/language-modeling/scripts/baichuan2-7b-chat.sh), [example](./examples/language-modeling/) |+| 01-ai/Yi-6B-Chat | [accuracy](./docs/Yi-6B-Chat-acc.md), [recipe](./examples/language-modeling/scripts/Yi-6B-Chat.sh), [example](./examples/language-modeling/) || Salesforce/codegen25-7b-multi | [example](./examples/code-generation) || EleutherAI/gpt-j-6b | [example](./examples/language-modeling/) || huggyllama/llama-7b | [example](./examples/language-modeling/) |diff --git a/docs/Qwen1.5-7B-Chat-acc.md b/docs/Qwen1.5-7B-Chat-acc.mdindex 9616ca077..2add074c7 100644--- a/docs/Qwen1.5-7B-Chat-acc.md+++ b/docs/Qwen1.5-7B-Chat-acc.md@@ -1,15 +1,16 @@Due to licensing restrictions, we are unable to release the model. Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d.We used the following command for evaluation.+For reference, the results of official AWQ-INT4 and GPTQ-INT4 release are listed.~~~bashlm_eval --model hf --model_args pretrained="./",autogptq=True,gptq_use_triton=True,trust_remote_code=True --device cuda:0 --tasks ceval-valid,cmmlu,mmlu,gsm8k --batch_size 16 --num_fewshot 0~~~-| Metric | BF16 | INT4 sym recipe | INT4 asym recipe |-| -------------- | ------ |-----------------|------------------|-| Avg. | 0.6231 | 0.6205 | 0.6186 |-| ceval | 0.6887 | 0.6761 | 0.6820 |-| cmmlu | 0.6959 | 0.6870 | 0.6884 |-| mmlu | 0.6020 | 0.5974 | 0.5946 |-| gsm8k | 0.5057 | 0.5216 | 0.5095 |+| Metric | BF16 | [Qwen/Qwen1.5-7B-Chat-AWQ](https://huggingface.co/Qwen/Qwen1.5-7B-Chat-AWQ) | [Qwen/Qwen1.5-7B-Chat-GPTQ-Int4](https://huggingface.co/Qwen/Qwen1.5-7B-Chat-GPTQ-Int4) | INT4 sym recipe | INT4 asym recipe |+| -------------- | ------ |-----------|--------------------------------|-----------------|------------------|+| Avg. | 0.6231 | 0.6152 | 0.6070 | 0.6205 | 0.6186 |+| ceval | 0.6887 | 0.6820 | 0.6679 | 0.6761 | 0.6820 |+| cmmlu | 0.6959 | 0.6862 | 0.6831 | 0.6870 | 0.6884 |+| mmlu | 0.6020 | 0.5944 | 0.5902 | 0.5974 | 0.5946 |+| gsm8k | 0.5057 | 0.4981 | 0.4867 | 0.5216 | 0.5095 |diff --git a/docs/Yi-6B-Chat-acc.md b/docs/Yi-6B-Chat-acc.mdnew file mode 100644index 000000000..f910d9195--- /dev/null+++ b/docs/Yi-6B-Chat-acc.md@@ -0,0 +1,16 @@+Due to licensing restrictions, we are unable to release the model. Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d.++We used the following command for evaluation.+For reference, the results of official AWQ-INT4 release are listed.++~~~bash+lm_eval --model hf --model_args pretrained="./",autogptq=True,gptq_use_triton=True,trust_remote_code=True --device cuda:0 --tasks ceval-valid,cmmlu,mmlu,gsm8k --batch_size 16 --num_fewshot 0+~~~++| Metric | BF16 |[01-ai/Yi-6B-Chat-4bits](https://huggingface.co/01-ai/Yi-6B-Chat-4bits)| INT4 |+|--------|--------|----------------------|--------|+| Avg. | 0.6043 | 0.5867 | 0.5939 |+| mmlu | 0.6163 | 0.6133 | 0.6119 |+| cmmlu | 0.7431 | 0.7312 | 0.7314 |+| ceval | 0.7355 | 0.7155 | 0.7281 |+| gsm8k | 0.3222 | 0.2866 | 0.3040 |diff --git a/examples/language-modeling/scripts/Yi-6B-Chat.sh b/examples/language-modeling/scripts/Yi-6B-Chat.shnew file mode 100644index 000000000..514285b28--- /dev/null+++ b/examples/language-modeling/scripts/Yi-6B-Chat.sh@@ -0,0 +1,9 @@+## Due to licensing restrictions, we are unable to release the model.+python3 main.py \+--model_name 01-ai/Yi-6B-Chat \+--group_size 128 \+--bits 4 \+--iters 1000 \+--deployment_device 'gpu' \+--minmax_lr 2e-3 \+--use_quant_input