PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
qwen1.5 acc update (#67)
yintong-lu3cd0615eed50
diff --git a/README.md b/README.mdindex 9becb2c12..6585a24b0 100644--- a/README.md+++ b/README.md@@ -163,7 +163,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))| mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) || mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) || meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) |-| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [recipe1](./examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh), [recipe2](./examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.sh), [example](./examples/language-modeling/) |+| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [recipe](./examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh), [example](./examples/language-modeling/) || Salesforce/codegen25-7b-multi | [example](./examples/code-generation) || EleutherAI/gpt-j-6b | [example](./examples/language-modeling/) || huggyllama/llama-7b | [example](./examples/language-modeling/) |diff --git a/docs/Qwen1.5-7B-Chat-acc.md b/docs/Qwen1.5-7B-Chat-acc.mdindex 4f81fe21a..7e0f3a834 100644--- a/docs/Qwen1.5-7B-Chat-acc.md+++ b/docs/Qwen1.5-7B-Chat-acc.md@@ -2,12 +2,18 @@ Due to licensing restrictions, we are unable to release the model.Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d.+Please use the command below for evaluation.+'lm_eval --model hf \+ --model_args pretrained="gpu_model_path",autogptq=True,gptq_use_triton=True,trust_remote_code=True \+ --device cuda:0 --tasks ceval-valid,cmmlu,mmlu --batch_size 16 --num_fewshot 0'-| Metric | BF16 | INT4 recipe1 | INT4 recipe2 |-| -------------- | ------ | -----------------| ----------------|-| Avg. | 0.6220 | 0.6207 | 0.6185 |-| ceval | 0.6988 | 0.6855 | 0.6913 |-| cmmlu | 0.6879 | 0.6783 | 0.6776 |-| mmlu | 0.6016 | 0.5966 | 0.5970 |-| gsm8k | 0.4996 | 0.5224 | 0.5080 |+++| Metric | BF16 | INT4 recipe |+| -------------- | ------ | -----------------|+| Avg. | 0.6231 | 0.6205 |+| ceval | 0.6887 | 0.6761 |+| cmmlu | 0.6959 | 0.6870 |+| mmlu | 0.6020 | 0.5974 |+| gsm8k | 0.5057 | 0.5216 |diff --git a/examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh b/examples/language-modeling/scripts/Qwen1.5-7B-Chat.shindex 390cf1f8d..ce56a8382 100644--- a/examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh+++ b/examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh@@ -1,5 +1,5 @@## Due to licensing restrictions, we are unable to release the model.-#Recipe 1+# Recipepython3 main.py \--model_name Qwen/Qwen1.5-7B-Chat\--bits 4 \@@ -9,4 +9,3 @@ python3 main.py \--deployment_device gpu \--sym \--minmax_lr 0.01-diff --git a/examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.sh b/examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.shdeleted file mode 100644index 79e6dae10..000000000--- a/examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.sh+++ /dev/null@@ -1,11 +0,0 @@-## Due to licensing restrictions, we are unable to release the model.-# Recipe 2-python3 main.py \---model_name Qwen/Qwen1.5-7B-Chat\---bits 4 \---group_size 128 \---use_quant_input \---iters 1000 \---deployment_device gpu \---sym \---minmax_lr 0.002