Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

add gemma-7b acc and shell info

wenhuach211f79e54f2651
diff --git a/README.md b/README.md
index d87810dbb..77893faa1 100644
--- a/README.md
+++ b/README.md
@@ -159,11 +159,13 @@ We fine-tuned the hyperparameters for each model with an iteration of 1K and suc
[Intel/Mistral-7B-v0.1-int4-inc](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc)
-[Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc](https://huggingface.co/Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc) coming soon
+[Intel/gemma-7b-int4-inc](https://huggingface.co/Intel/gemma-7b-int4-inc) internal review, [accuracy](./docs/gemma-7b-acc.md), [quantization shell](./examples/language-modeling/scripts/gemma-7b.sh)
-[Intel/Mixtral-8x7B-v0.1-int4-inc](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) coming soon
+[Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc](https://huggingface.co/Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc) coming soon, [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [quantization shell](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh)
-[Intel/phi-2-int4-inc](https://huggingface.co/Intel/phi-2-int4-inc) coming soon
+[Intel/Mixtral-8x7B-v0.1-int4-inc](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) coming soon,[accuracy](./docs/Mistral-7B-v0.1-acc.md), [quantization shell](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh)
+
+[Intel/phi-2-int4-inc](https://huggingface.co/Intel/phi-2-int4-inc) coming soon, [accuracy](./docs/phi-2-acc.md) [quantization shell](./examples/language-modeling/scripts/phi-2.sh)
### Itrex format
diff --git a/docs/gemma-7b-acc.md b/docs/gemma-7b-acc.md
new file mode 100644
index 000000000..ec39e9d30
--- /dev/null
+++ b/docs/gemma-7b-acc.md
@@ -0,0 +1,20 @@
+Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d, Install the latest [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) from source first
+
+```bash
+lm_eval --model hf --model_args pretrained="Intel/gemma-7b-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32
+```
+
+| Metric | FP16 | int4 |
+| -------------- | ------ | ------ |
+| Avg. | 0.6239 | 0.6307 |
+| mmlu | 0.6162 | 0.6147 |
+| lambada_openai | 0.6751 | 0.7204 |
+| hellaswag | 0.6047 | 0.5903 |
+| winogrande | 0.7324 | 0.7514 |
+| piqa | 0.7943 | 0.7949 |
+| truthfulqa_mc1 | 0.3097 | 0.3011 |
+| openbookqa | 0.3320 | 0.3400 |
+| boolq | 0.8278 | 0.8269 |
+| rte | 0.6534 | 0.7076 |
+| arc_easy | 0.8178 | 0.7959 |
+| arc_challenge | 0.4991 | 0.4940 |
\ No newline at end of file
diff --git a/examples/language-modeling/scripts/gemma-7b.sh b/examples/language-modeling/scripts/gemma-7b.sh
new file mode 100644
index 000000000..2999a3337
--- /dev/null
+++ b/examples/language-modeling/scripts/gemma-7b.sh
@@ -0,0 +1,15 @@
+python3 main.py \
+--model_name google/gemma-7b \
+--device 0 \
+--group_size 128 \
+--bits 4 \
+--iters 1000 \
+--enable_minmax_tuning \
+--use_quant_input \
+--minmax_lr 2e-3 \
+--low_gpu_mem_usage \
+--deployment_device 'gpu' \
+--scale_dtype 'fp32' \
+--eval_bs 32 \
+--output_dir "./tmp_autoround" \
+--amp
\ No newline at end of file