PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
Update llama3 acc (#84)
wenhuach21849cf9fedc0a
diff --git a/README.md b/README.mdindex 12a69fde7..4ecb22260 100644--- a/README.md+++ b/README.md@@ -165,6 +165,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))| google/gemma-7b-it | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-it-int4-inc) (under review), [accuracy](./docs/gemma-7b-it-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b-it.sh), [example](./examples/language-modeling/) || mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) || mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |+| meta-llama/Meta-Llama-3-8B-Instruct | [accuracy](./docs/Meta-Llama-3-8B-Instruct-acc.md), [recipe](./examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.sh), [example](./examples/language-modeling/) || meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) || Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [sym recipe](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh), [asym recipe ](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh), [example](./examples/language-modeling/) || baichuan-inc/Baichuan2-7B-Chat | [accuracy](./docs/baichuan2-7b-chat-acc.md), [recipe](./examples/language-modeling/scripts/baichuan2-7b-chat.sh), [example](./examples/language-modeling/) |diff --git a/docs/Meta-Llama-3-8B-Instruct-acc.md b/docs/Meta-Llama-3-8B-Instruct-acc.mdnew file mode 100644index 000000000..a724e16f3--- /dev/null+++ b/docs/Meta-Llama-3-8B-Instruct-acc.md@@ -0,0 +1,26 @@+Due to licensing restrictions, we are unable to release the model.++lm-eval 0.4.2 is used++For evaluating w4g128 without quantized lm-head,+```bash+lm_eval --model hf --model_args pretrained="./",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 16+```++For evaluating w4g128 with quantized lm-head, we opted to evaluate the qdq model instead,since we encountered an issue evaluating the model with lm-head quantized model. In our assessment, we found that its accuracy closely matches that of the real quantized model in most cases except for some small models like opt-125m.+++| Metric | **BF16** | w4g128 w/o lm-head | w4g128 with lm-head qdq |+| ---------------- | :------- |--------------------|---------------------------------|+| Avg. | 0.6352 | 0.6312 | 0.6303 |+| mmlu | 0.6386 | 0.6306 | 0.6318 |+| winogrande | 0.7143 | 0.7238 | 0.7269 |+| truthfulqa_mc1 | 0.3623 | 0.3537 | 0.3525 |+| rte | 0.6751 | 0.6859 | 0.6679 |+| piqa | 0.7867 | 0.7797 | 0.7802 |+| openbookqa | 0.3400 | 0.3300 | 0.3320 |+| lambada_openai | 0.7182 | 0.7200 | 0.7173 |+| hellaswag | 0.5769 | 0.5699 | 0.5701 |+| boolq | 0.8297 | 0.8309 | 0.8284 |+| arc_easy | 0.8152 | 0.8089 | 0.8106 |+| arc_challenge | 0.5299 | 0.5102 | 0.5154 |diff --git a/examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.sh b/examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.shnew file mode 100644index 000000000..e83f608d1--- /dev/null+++ b/examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.sh@@ -0,0 +1,26 @@+## quant lm head+python3 main.py \+--model_name meta-llama/Meta-Llama-3-8B-Instruct \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--use_quant_input \+--quant_lm_head \+--deployment_device 'gpu' \+--disable_low_gpu_mem_usage \+--output_dir "./tmp_autoround"++++## don't quant lm head+python3 main.py \+--model_name meta-llama/Meta-Llama-3-8B-Instruct \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--use_quant_input \+--deployment_device 'gpu' \+--disable_low_gpu_mem_usage \+--output_dir "./tmp_autoround"\ No newline at end of file