PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
Add model recipe (#63)
wenhuach21f39638d983fc
diff --git a/README.md b/README.mdindex bb0b3fe6d..053e0dc2f 100644--- a/README.md+++ b/README.md@@ -149,40 +149,42 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))## Support List-| Model | Supported |-|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-| Intel/neural-chat-7b-v3-3 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc), [accuracy](./docs/neural-chat-7b-v3-3-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-3.sh), [example](./examples/language-modeling/) |-| Intel/neural-chat-7b-v3-1 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc), [accuracy](./docs/neural-chat-7b-v3-1-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-1.sh), [example](./examples/language-modeling/) |-| mistralai/Mistral-7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc), [accuracy](./docs/Mistral-7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-v0.1.sh), [example](./examples/language-modeling/) |-| microsoft/phi-2 | [HF-int4-model](https://huggingface.co/Intel/phi-2-int4-inc), [accuracy](./docs/phi-2-acc.md), [recipe](./examples/language-modeling/scripts/phi-2.sh), [example](./examples/language-modeling/)-| tiiuae/falcon-7b | [HF-int4-model](https://huggingface.co/Intel/falcon-7b-int4-inc), [accuracy](./docs/falcon-7b-acc.md), [recipe](./examples/language-modeling/scripts/falcon-7b.sh), [example](./examples/language-modeling/) |-| google/gemma-7b | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-int4-inc) (under review), [accuracy](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh), [example](./examples/language-modeling/) |-| google/gemma-7b-it | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-it-int4-inc) (under review), [accuracy](./docs/gemma-7b-it-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b-it.sh), [example](./examples/language-modeling/) | |- mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) |-| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |-| meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) |-| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [recipe1](./examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh), [recipe2](./examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.sh), [example](./examples/language-modeling/) |-| Salesforce/codegen25-7b-multi | [example](./examples/code-generation) |-| EleutherAI/gpt-j-6b | [example](./examples/language-modeling/) |-| huggyllama/llama-7b | [example](./examples/language-modeling/) |-| meta-llama/Llama-2-7b-hf | [example](./examples/language-modeling/) |-| facebook/opt-6.7b | [example](./examples/language-modeling/) ||-| mosaicml/mpt-7b | [example](./examples/language-modeling/) |-| bigscience/bloom-7b1 | [example](./examples/language-modeling/) |-| baichuan-inc/Baichuan-7B | [example](./examples/language-modeling/) |-| Qwen/Qwen-7B | [example](./examples/language-modeling/) |-| THUDM/chatglm3-6b | [example](./examples/language-modeling/) |-| MBZUAI/LaMini-GPT-124M | [example](./examples/language-modeling/) |-| EleutherAI/gpt-neo-125m | [example](./examples/language-modeling/) |-| databricks/dolly-v2-3b | [example](./examples/language-modeling/) |-| stabilityai/stablelm-base-alpha-3b | [example](./examples/language-modeling/)+| Model | Supported |+|--------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|+| Intel/neural-chat-7b-v3-3 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc), [accuracy](./docs/neural-chat-7b-v3-3-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-3.sh), [example](./examples/language-modeling/) |+| Intel/neural-chat-7b-v3-1 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc), [accuracy](./docs/neural-chat-7b-v3-1-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-1.sh), [example](./examples/language-modeling/) |+| mistralai/Mistral-7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc), [accuracy](./docs/Mistral-7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-v0.1.sh), [example](./examples/language-modeling/) |+| microsoft/phi-2 | [HF-int4-model](https://huggingface.co/Intel/phi-2-int4-inc), [accuracy](./docs/phi-2-acc.md), [recipe](./examples/language-modeling/scripts/phi-2.sh), [example](./examples/language-modeling/)+| tiiuae/falcon-7b | [HF-int4-model](https://huggingface.co/Intel/falcon-7b-int4-inc), [accuracy](./docs/falcon-7b-acc.md), [recipe](./examples/language-modeling/scripts/falcon-7b.sh), [example](./examples/language-modeling/) |+| google/gemma-2b | [HF-int4-model](https://huggingface.co/Intel/gemma-2b-int4-inc), [accuracy](./docs/gemma-2b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-2b.sh), [example](./examples/language-modeling/)+| mistralai/Mixtral-7B-Instruct-v0.2 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-Instruct-v0.2-int4-inc) (under review), [accuracy](./docs/Mistral-7B-Instruct-v0.2-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.sh), [example](./examples/language-modeling/) |+| google/gemma-7b | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-int4-inc) (under review), [accuracy](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh), [example](./examples/language-modeling/) |+| google/gemma-7b-it | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-it-int4-inc) (under review), [accuracy](./docs/gemma-7b-it-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b-it.sh), [example](./examples/language-modeling/) |+| mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) |+| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |+| meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) |+| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [recipe1](./examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh), [recipe2](./examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.sh), [example](./examples/language-modeling/) |+| Salesforce/codegen25-7b-multi | [example](./examples/code-generation) |+| EleutherAI/gpt-j-6b | [example](./examples/language-modeling/) |+| huggyllama/llama-7b | [example](./examples/language-modeling/) |+| meta-llama/Llama-2-7b-hf | [example](./examples/language-modeling/) |+| facebook/opt-6.7b | [example](./examples/language-modeling/) ||+| mosaicml/mpt-7b | [example](./examples/language-modeling/) |+| bigscience/bloom-7b1 | [example](./examples/language-modeling/) |+| baichuan-inc/Baichuan-7B | [example](./examples/language-modeling/) |+| Qwen/Qwen-7B | [example](./examples/language-modeling/) |+| THUDM/chatglm3-6b | [example](./examples/language-modeling/) |+| MBZUAI/LaMini-GPT-124M | [example](./examples/language-modeling/) |+| EleutherAI/gpt-neo-125m | [example](./examples/language-modeling/) |+| databricks/dolly-v2-3b | [example](./examples/language-modeling/) |+| stabilityai/stablelm-base-alpha-3b | [example](./examples/language-modeling/)## Comparison with other methods-We provide a [comprehensive analysis](docs/acc.md) with other methods in our accuracy data section. Notably, our approach has outperformed GPTQ with a score of 30/32 and AWQ with a score of 27/32 across llamv1/llamav2/mistral-7b on W4G-1, W4G128, W3G128, W2G128. And the tuning costs are comparable.+We provide a [comprehensive analysis](docs/acc.md) with other methods in our accuracy data section. In summary, our approach achieved superior performance compared to GPTQ, scoring 30/32, AWQ with 27/32, HQQ with 15/16, and OmniQuant with a perfect score of 16/16 across llamv1/llamav2/mistral-7b on W4G-1, W4G128, W3G128, and W2G128, based on the average accuracies of 11 zero-shot tasks.. And the tuning costs are comparable.## Tips1 Consider increasing tuning steps to achieve better results, albeit with increased tuning time.diff --git a/docs/Mistral-7B-Instruct-v0.2-acc.md b/docs/Mistral-7B-Instruct-v0.2-acc.mdnew file mode 100644index 000000000..c14d2f73a--- /dev/null+++ b/docs/Mistral-7B-Instruct-v0.2-acc.md@@ -0,0 +1,23 @@+Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d+##pip install auto-gptq[triton]+##pip install triton==2.2.0+```bash+lm_eval --model hf --model_args pretrained="Intel/Mistral-7B-Instruct-v0.2-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32+```++++| Metric | BF16 | INT4 |+| -------------- | ------ | ------ |+| Avg. | 0.6647 | 0.6621 |+| mmlu | 0.5906 | 0.5872 |+| lambada_openai | 0.7141 | 0.7141 |+| hellaswag | 0.6602 | 0.6557 |+| winogrande | 0.7395 | 0.7364 |+| piqa | 0.8052 | 0.8047 |+| truthfulqa_mc1 | 0.5251 | 0.5153 |+| openbookqa | 0.3600 | 0.3420 |+| boolq | 0.8535 | 0.8541 |+| rte | 0.7040 | 0.7148 |+| arc_easy | 0.8161 | 0.8165 |+| arc_challenge | 0.5435 | 0.5435 |diff --git a/docs/Mistral-7B-v0.1-acc.md b/docs/Mistral-7B-v0.1-acc.mdindex d260fa41d..58ddfd1d9 100644--- a/docs/Mistral-7B-v0.1-acc.md+++ b/docs/Mistral-7B-v0.1-acc.md@@ -6,8 +6,8 @@ lm_eval --model hf --model_args pretrained="Intel/Mistral-7B-v0.1-int4-inc",auto-| Metric | FP16 | INT4 |-| -------------- | ------ | ------ |+| Metric | BF16 | INT4 |+| -------------- |--------| ------ || Avg. | 0.6306 | 0.6308 || mmlu | 0.5961 | 0.5880 || lambada_openai | 0.7561 | 0.7551 |diff --git a/docs/Mixtral-8x7B-Instruct-v0.1-acc.md b/docs/Mixtral-8x7B-Instruct-v0.1-acc.mdindex d7b10b775..2dd4d62ea 100644--- a/docs/Mixtral-8x7B-Instruct-v0.1-acc.md+++ b/docs/Mixtral-8x7B-Instruct-v0.1-acc.md@@ -6,8 +6,8 @@ Download the model from hf(coming soon) or follow examples/language-modeling/scrlm_eval --model hf --model_args pretrained="Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32```-| Metric | FP16 | INT4 |-| -------------- | ------ | ------ |+| Metric | BF16 | INT4 |+| -------------- |--------| ------ || Avg. | 0.7000 | 0.6977 || mmlu | 0.6885 | 0.6824 || lambada_openai | 0.7718 | 0.7790 |diff --git a/docs/Mixtral-8x7B-v0.1-acc.md b/docs/Mixtral-8x7B-v0.1-acc.mdindex 78f26e6d7..d9f9632d3 100644--- a/docs/Mixtral-8x7B-v0.1-acc.md+++ b/docs/Mixtral-8x7B-v0.1-acc.md@@ -7,8 +7,8 @@ Download the model from hf(coming soon) or follow examples/language-modeling/scrlm_eval --model hf --model_args pretrained="Intel/Mixtral-8x7B-v0.1-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32~~~-| Metric | FP16 | INT4 |-| -------------- | ------ | ------ |+| Metric | BF16 | INT4 |+| -------------- |--------| ------ || Avg. | 0.6698 | 0.6633 || mmlu | 0.6802 | 0.6693 || lambada_openai | 0.7827 | 0.7825 |diff --git a/docs/gemma-2b-acc.md b/docs/gemma-2b-acc.mdnew file mode 100644index 000000000..0b5c5e918--- /dev/null+++ b/docs/gemma-2b-acc.md@@ -0,0 +1,27 @@+### Evaluate the model++Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id we used is 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d++pip install auto-gptq[triton]+pip install triton==2.2.0++Please note that there is a discrepancy between the baseline result and the official data, which is a known issue within the official model card community.++```bash+lm_eval --model hf --model_args pretrained="Intel/gemma-2b-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 16+```++| Metric | FP16 | INT4 |+| -------------- | ------ |--------|+| Avg. | 0.5383 | 0.5338 |+| mmlu | 0.3337 | 0.3276 |+| lambada_openai | 0.6398 | 0.6319 |+| hellaswag | 0.5271 | 0.5161 |+| winogrande | 0.6472 | 0.6472 |+| piqa | 0.7699 | 0.7622 |+| truthfulqa_mc1 | 0.2203 | 0.2191 |+| openbookqa | 0.3020 | 0.2980 |+| boolq | 0.6939 | 0.6939 |+| rte | 0.6426 | 0.6498 |+| arc_easy | 0.7424 | 0.7348 |+| arc_challenge | 0.4019 | 0.3908 |diff --git a/docs/gemma-7b-acc.md b/docs/gemma-7b-acc.mdindex 1ec091ce7..9f03be4dd 100644--- a/docs/gemma-7b-acc.md+++ b/docs/gemma-7b-acc.md@@ -1,5 +1,7 @@Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d, Install the latest [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) from source first+Please note that there is a discrepancy between the baseline result and the official data, which is a known issue within the official model card community.+```bashlm_eval --model hf --model_args pretrained="Intel/gemma-7b-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32```diff --git a/docs/gemma-7b-it-acc.md b/docs/gemma-7b-it-acc.mdindex ca0afd07d..1d1f1b52f 100644--- a/docs/gemma-7b-it-acc.md+++ b/docs/gemma-7b-it-acc.md@@ -1,5 +1,7 @@Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d, Install the latest [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) from source first+Please note that there is a discrepancy between the baseline result and the official data, which is a known issue within the official model card community.+```bashlm_eval --model hf --model_args pretrained="Intel/gemma-7b-it-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32```diff --git a/examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.sh b/examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.shnew file mode 100644index 000000000..baa55cf31--- /dev/null+++ b/examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.sh@@ -0,0 +1,9 @@+python3 main.py \+--model_name mistralai/Mistral-7B-Instruct-v0.2 \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--use_quant_input \+--deployment_device 'gpu' \+--output_dir "./tmp_autoround"\ No newline at end of filediff --git a/examples/language-modeling/scripts/gemma-2b.sh b/examples/language-modeling/scripts/gemma-2b.shnew file mode 100644index 000000000..5ef8716a1--- /dev/null+++ b/examples/language-modeling/scripts/gemma-2b.sh@@ -0,0 +1,12 @@+git clone https://github.com/intel/auto-round+cd auto-round/examples/language-modeling+pip install -r requirements.txt+python3 main.py \+--model_name google/gemma-2b \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 400 \+--use_quant_input \+--deployment_device 'gpu' \+--output_dir "./tmp_autoround"\ No newline at end of file