PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
Qwen1.5-7b chat recipe (#36)
yintong-lu4fbd0884304b
diff --git a/README.md b/README.mdindex 7d7e05605..8feff611f 100644--- a/README.md+++ b/README.md@@ -155,6 +155,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) under review, [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) || microsoft/phi-2 | [HF-int4-model](https://huggingface.co/Intel/phi-2-int4-inc) under review, [accuracy](./docs/phi-2-acc.md), [recipe](./examples/language-modeling/scripts/phi-2.sh), [example](./examples/language-modeling/) || meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) |+| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [recipe1](./examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh), [recipe2](./examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.sh), [example](./examples/language-modeling/) || Salesforce/codegen25-7b-multi | [example](./examples/code-generation) || EleutherAI/gpt-j-6b | [example](./examples/language-modeling/) || huggyllama/llama-7b | [example](./examples/language-modeling/) |diff --git a/docs/Qwen1.5-7B-Chat-acc.md b/docs/Qwen1.5-7B-Chat-acc.mdnew file mode 100644index 000000000..4f81fe21a--- /dev/null+++ b/docs/Qwen1.5-7B-Chat-acc.md@@ -0,0 +1,13 @@+Due to licensing restrictions, we are unable to release the model.++Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d.++++| Metric | BF16 | INT4 recipe1 | INT4 recipe2 |+| -------------- | ------ | -----------------| ----------------|+| Avg. | 0.6220 | 0.6207 | 0.6185 |+| ceval | 0.6988 | 0.6855 | 0.6913 |+| cmmlu | 0.6879 | 0.6783 | 0.6776 |+| mmlu | 0.6016 | 0.5966 | 0.5970 |+| gsm8k | 0.4996 | 0.5224 | 0.5080 |diff --git a/examples/language-modeling/eval/evaluation.py b/examples/language-modeling/eval/evaluation.pyindex 9d4824b6b..cb3806a3d 100644--- a/examples/language-modeling/eval/evaluation.py+++ b/examples/language-modeling/eval/evaluation.py@@ -351,6 +351,7 @@ def eval_model(model_path, tasks=["lambada_openai", "hellaswag", "winogrande", "results = {}model = Nonelm = None+for tmp_tasks in tasks:try:num_fewshot = fewshots_dict[mark][tmp_tasks]diff --git a/examples/language-modeling/eval/utils.py b/examples/language-modeling/eval/utils.pyindex 2a3a2e4c5..37a43e8f5 100644--- a/examples/language-modeling/eval/utils.py+++ b/examples/language-modeling/eval/utils.py@@ -213,3 +213,5 @@ def eval_ppl_same_with_gptq(model, testenc, dev):if hasattr(model.config, "use_cache"):model.config.use_cache = use_cachereturn ppl.item()++diff --git a/examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh b/examples/language-modeling/scripts/Qwen1.5-7B-Chat.shnew file mode 100644index 000000000..4161ee24c--- /dev/null+++ b/examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh@@ -0,0 +1,16 @@+## Due to licensing restrictions, we are unable to release the model.+#Recipe 1+python3 main.py \+--model_name Qwen/Qwen1.5-7B-Chat\+--bits 4 \+--group_size 128 \+--enable_minmax_tuning \+--amp \+--use_quant_input \+--n_samples 512 \+--iters 200 \+--deployment_device gpu \+--sym \+--eval_bs 32 \+--minmax_lr 0.01+diff --git a/examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.sh b/examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.shnew file mode 100644index 000000000..43fcb7b66--- /dev/null+++ b/examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.sh@@ -0,0 +1,15 @@+## Due to licensing restrictions, we are unable to release the model.+# Recipe 2+python3 main.py \+--model_name Qwen/Qwen1.5-7B-Chat\+--bits 4 \+--group_size 128 \+--enable_minmax_tuning \+--amp \+--use_quant_input \+--n_samples 512 \+--iters 1000 \+--deployment_device gpu \+--sym \+--eval_bs 32 \+--minmax_lr 0.002