PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
add gemma-7b acc and shell info
wenhuach211f79e54f2651
diff --git a/README.md b/README.mdindex d87810dbb..77893faa1 100644--- a/README.md+++ b/README.md@@ -159,11 +159,13 @@ We fine-tuned the hyperparameters for each model with an iteration of 1K and suc[Intel/Mistral-7B-v0.1-int4-inc](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc)-[Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc](https://huggingface.co/Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc) coming soon+[Intel/gemma-7b-int4-inc](https://huggingface.co/Intel/gemma-7b-int4-inc) internal review, [accuracy](./docs/gemma-7b-acc.md), [quantization shell](./examples/language-modeling/scripts/gemma-7b.sh)-[Intel/Mixtral-8x7B-v0.1-int4-inc](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) coming soon+[Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc](https://huggingface.co/Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc) coming soon, [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [quantization shell](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh)-[Intel/phi-2-int4-inc](https://huggingface.co/Intel/phi-2-int4-inc) coming soon+[Intel/Mixtral-8x7B-v0.1-int4-inc](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) coming soon,[accuracy](./docs/Mistral-7B-v0.1-acc.md), [quantization shell](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh)++[Intel/phi-2-int4-inc](https://huggingface.co/Intel/phi-2-int4-inc) coming soon, [accuracy](./docs/phi-2-acc.md) [quantization shell](./examples/language-modeling/scripts/phi-2.sh)### Itrex formatdiff --git a/docs/gemma-7b-acc.md b/docs/gemma-7b-acc.mdnew file mode 100644index 000000000..ec39e9d30--- /dev/null+++ b/docs/gemma-7b-acc.md@@ -0,0 +1,20 @@+Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d, Install the latest [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) from source first++```bash+lm_eval --model hf --model_args pretrained="Intel/gemma-7b-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32+```++| Metric | FP16 | int4 |+| -------------- | ------ | ------ |+| Avg. | 0.6239 | 0.6307 |+| mmlu | 0.6162 | 0.6147 |+| lambada_openai | 0.6751 | 0.7204 |+| hellaswag | 0.6047 | 0.5903 |+| winogrande | 0.7324 | 0.7514 |+| piqa | 0.7943 | 0.7949 |+| truthfulqa_mc1 | 0.3097 | 0.3011 |+| openbookqa | 0.3320 | 0.3400 |+| boolq | 0.8278 | 0.8269 |+| rte | 0.6534 | 0.7076 |+| arc_easy | 0.8178 | 0.7959 |+| arc_challenge | 0.4991 | 0.4940 |\ No newline at end of filediff --git a/examples/language-modeling/scripts/gemma-7b.sh b/examples/language-modeling/scripts/gemma-7b.shnew file mode 100644index 000000000..2999a3337--- /dev/null+++ b/examples/language-modeling/scripts/gemma-7b.sh@@ -0,0 +1,15 @@+python3 main.py \+--model_name google/gemma-7b \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--enable_minmax_tuning \+--use_quant_input \+--minmax_lr 2e-3 \+--low_gpu_mem_usage \+--deployment_device 'gpu' \+--scale_dtype 'fp32' \+--eval_bs 32 \+--output_dir "./tmp_autoround" \+--amp\ No newline at end of file