Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

Qwen1.5-7b chat recipe (#36)

yintong-lu4fbd0884304b
diff --git a/README.md b/README.md
index 7d7e05605..8feff611f 100644
--- a/README.md
+++ b/README.md
@@ -155,6 +155,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) under review, [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |
| microsoft/phi-2 | [HF-int4-model](https://huggingface.co/Intel/phi-2-int4-inc) under review, [accuracy](./docs/phi-2-acc.md), [recipe](./examples/language-modeling/scripts/phi-2.sh), [example](./examples/language-modeling/) |
| meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) |
+| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [recipe1](./examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh), [recipe2](./examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.sh), [example](./examples/language-modeling/) |
| Salesforce/codegen25-7b-multi | [example](./examples/code-generation) |
| EleutherAI/gpt-j-6b | [example](./examples/language-modeling/) |
| huggyllama/llama-7b | [example](./examples/language-modeling/) |
diff --git a/docs/Qwen1.5-7B-Chat-acc.md b/docs/Qwen1.5-7B-Chat-acc.md
new file mode 100644
index 000000000..4f81fe21a
--- /dev/null
+++ b/docs/Qwen1.5-7B-Chat-acc.md
@@ -0,0 +1,13 @@
+Due to licensing restrictions, we are unable to release the model.
+
+Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d.
+
+
+
+| Metric | BF16 | INT4 recipe1 | INT4 recipe2 |
+| -------------- | ------ | -----------------| ----------------|
+| Avg. | 0.6220 | 0.6207 | 0.6185 |
+| ceval | 0.6988 | 0.6855 | 0.6913 |
+| cmmlu | 0.6879 | 0.6783 | 0.6776 |
+| mmlu | 0.6016 | 0.5966 | 0.5970 |
+| gsm8k | 0.4996 | 0.5224 | 0.5080 |
diff --git a/examples/language-modeling/eval/evaluation.py b/examples/language-modeling/eval/evaluation.py
index 9d4824b6b..cb3806a3d 100644
--- a/examples/language-modeling/eval/evaluation.py
+++ b/examples/language-modeling/eval/evaluation.py
@@ -351,6 +351,7 @@ def eval_model(model_path, tasks=["lambada_openai", "hellaswag", "winogrande", "
results = {}
model = None
lm = None
+
for tmp_tasks in tasks:
try:
num_fewshot = fewshots_dict[mark][tmp_tasks]
diff --git a/examples/language-modeling/eval/utils.py b/examples/language-modeling/eval/utils.py
index 2a3a2e4c5..37a43e8f5 100644
--- a/examples/language-modeling/eval/utils.py
+++ b/examples/language-modeling/eval/utils.py
@@ -213,3 +213,5 @@ def eval_ppl_same_with_gptq(model, testenc, dev):
if hasattr(model.config, "use_cache"):
model.config.use_cache = use_cache
return ppl.item()
+
+
diff --git a/examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh b/examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh
new file mode 100644
index 000000000..4161ee24c
--- /dev/null
+++ b/examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh
@@ -0,0 +1,16 @@
+## Due to licensing restrictions, we are unable to release the model.
+#Recipe 1
+python3 main.py \
+--model_name Qwen/Qwen1.5-7B-Chat\
+--bits 4 \
+--group_size 128 \
+--enable_minmax_tuning \
+--amp \
+--use_quant_input \
+--n_samples 512 \
+--iters 200 \
+--deployment_device gpu \
+--sym \
+--eval_bs 32 \
+--minmax_lr 0.01
+
diff --git a/examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.sh b/examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.sh
new file mode 100644
index 000000000..43fcb7b66
--- /dev/null
+++ b/examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.sh
@@ -0,0 +1,15 @@
+## Due to licensing restrictions, we are unable to release the model.
+# Recipe 2
+python3 main.py \
+--model_name Qwen/Qwen1.5-7B-Chat\
+--bits 4 \
+--group_size 128 \
+--enable_minmax_tuning \
+--amp \
+--use_quant_input \
+--n_samples 512 \
+--iters 1000 \
+--deployment_device gpu \
+--sym \
+--eval_bs 32 \
+--minmax_lr 0.002