Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

qwen1.5 acc update (#67)

yintong-lu3cd0615eed50
diff --git a/README.md b/README.md
index 9becb2c12..6585a24b0 100644
--- a/README.md
+++ b/README.md
@@ -163,7 +163,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
| mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) |
| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |
| meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) |
-| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [recipe1](./examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh), [recipe2](./examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.sh), [example](./examples/language-modeling/) |
+| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [recipe](./examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh), [example](./examples/language-modeling/) |
| Salesforce/codegen25-7b-multi | [example](./examples/code-generation) |
| EleutherAI/gpt-j-6b | [example](./examples/language-modeling/) |
| huggyllama/llama-7b | [example](./examples/language-modeling/) |
diff --git a/docs/Qwen1.5-7B-Chat-acc.md b/docs/Qwen1.5-7B-Chat-acc.md
index 4f81fe21a..7e0f3a834 100644
--- a/docs/Qwen1.5-7B-Chat-acc.md
+++ b/docs/Qwen1.5-7B-Chat-acc.md
@@ -2,12 +2,18 @@ Due to licensing restrictions, we are unable to release the model.
Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d.
+Please use the command below for evaluation.
+'lm_eval --model hf \
+ --model_args pretrained="gpu_model_path",autogptq=True,gptq_use_triton=True,trust_remote_code=True \
+ --device cuda:0 --tasks ceval-valid,cmmlu,mmlu --batch_size 16 --num_fewshot 0'
-| Metric | BF16 | INT4 recipe1 | INT4 recipe2 |
-| -------------- | ------ | -----------------| ----------------|
-| Avg. | 0.6220 | 0.6207 | 0.6185 |
-| ceval | 0.6988 | 0.6855 | 0.6913 |
-| cmmlu | 0.6879 | 0.6783 | 0.6776 |
-| mmlu | 0.6016 | 0.5966 | 0.5970 |
-| gsm8k | 0.4996 | 0.5224 | 0.5080 |
+
+
+| Metric | BF16 | INT4 recipe |
+| -------------- | ------ | -----------------|
+| Avg. | 0.6231 | 0.6205 |
+| ceval | 0.6887 | 0.6761 |
+| cmmlu | 0.6959 | 0.6870 |
+| mmlu | 0.6020 | 0.5974 |
+| gsm8k | 0.5057 | 0.5216 |
diff --git a/examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh b/examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh
index 390cf1f8d..ce56a8382 100644
--- a/examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh
+++ b/examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh
@@ -1,5 +1,5 @@
## Due to licensing restrictions, we are unable to release the model.
-#Recipe 1
+# Recipe
python3 main.py \
--model_name Qwen/Qwen1.5-7B-Chat\
--bits 4 \
@@ -9,4 +9,3 @@ python3 main.py \
--deployment_device gpu \
--sym \
--minmax_lr 0.01
-
diff --git a/examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.sh b/examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.sh
deleted file mode 100644
index 79e6dae10..000000000
--- a/examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.sh
+++ /dev/null
@@ -1,11 +0,0 @@
-## Due to licensing restrictions, we are unable to release the model.
-# Recipe 2
-python3 main.py \
---model_name Qwen/Qwen1.5-7B-Chat\
---bits 4 \
---group_size 128 \
---use_quant_input \
---iters 1000 \
---deployment_device gpu \
---sym \
---minmax_lr 0.002