Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

Update llama3 acc (#84)

wenhuach21849cf9fedc0a
diff --git a/README.md b/README.md
index 12a69fde7..4ecb22260 100644
--- a/README.md
+++ b/README.md
@@ -165,6 +165,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
| google/gemma-7b-it | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-it-int4-inc) (under review), [accuracy](./docs/gemma-7b-it-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b-it.sh), [example](./examples/language-modeling/) |
| mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) |
| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |
+| meta-llama/Meta-Llama-3-8B-Instruct | [accuracy](./docs/Meta-Llama-3-8B-Instruct-acc.md), [recipe](./examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.sh), [example](./examples/language-modeling/) |
| meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) |
| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [sym recipe](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh), [asym recipe ](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh), [example](./examples/language-modeling/) |
| baichuan-inc/Baichuan2-7B-Chat | [accuracy](./docs/baichuan2-7b-chat-acc.md), [recipe](./examples/language-modeling/scripts/baichuan2-7b-chat.sh), [example](./examples/language-modeling/) |
diff --git a/docs/Meta-Llama-3-8B-Instruct-acc.md b/docs/Meta-Llama-3-8B-Instruct-acc.md
new file mode 100644
index 000000000..a724e16f3
--- /dev/null
+++ b/docs/Meta-Llama-3-8B-Instruct-acc.md
@@ -0,0 +1,26 @@
+Due to licensing restrictions, we are unable to release the model.
+
+lm-eval 0.4.2 is used
+
+For evaluating w4g128 without quantized lm-head,
+```bash
+lm_eval --model hf --model_args pretrained="./",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 16
+```
+
+For evaluating w4g128 with quantized lm-head, we opted to evaluate the qdq model instead,since we encountered an issue evaluating the model with lm-head quantized model. In our assessment, we found that its accuracy closely matches that of the real quantized model in most cases except for some small models like opt-125m.
+
+
+| Metric | **BF16** | w4g128 w/o lm-head | w4g128 with lm-head qdq |
+| ---------------- | :------- |--------------------|---------------------------------|
+| Avg. | 0.6352 | 0.6312 | 0.6303 |
+| mmlu | 0.6386 | 0.6306 | 0.6318 |
+| winogrande | 0.7143 | 0.7238 | 0.7269 |
+| truthfulqa_mc1 | 0.3623 | 0.3537 | 0.3525 |
+| rte | 0.6751 | 0.6859 | 0.6679 |
+| piqa | 0.7867 | 0.7797 | 0.7802 |
+| openbookqa | 0.3400 | 0.3300 | 0.3320 |
+| lambada_openai | 0.7182 | 0.7200 | 0.7173 |
+| hellaswag | 0.5769 | 0.5699 | 0.5701 |
+| boolq | 0.8297 | 0.8309 | 0.8284 |
+| arc_easy | 0.8152 | 0.8089 | 0.8106 |
+| arc_challenge | 0.5299 | 0.5102 | 0.5154 |
diff --git a/examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.sh b/examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.sh
new file mode 100644
index 000000000..e83f608d1
--- /dev/null
+++ b/examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.sh
@@ -0,0 +1,26 @@
+## quant lm head
+python3 main.py \
+--model_name meta-llama/Meta-Llama-3-8B-Instruct \
+--device 0 \
+--group_size 128 \
+--bits 4 \
+--iters 1000 \
+--use_quant_input \
+--quant_lm_head \
+--deployment_device 'gpu' \
+--disable_low_gpu_mem_usage \
+--output_dir "./tmp_autoround"
+
+
+
+## don't quant lm head
+python3 main.py \
+--model_name meta-llama/Meta-Llama-3-8B-Instruct \
+--device 0 \
+--group_size 128 \
+--bits 4 \
+--iters 1000 \
+--use_quant_input \
+--deployment_device 'gpu' \
+--disable_low_gpu_mem_usage \
+--output_dir "./tmp_autoround"
\ No newline at end of file