Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

Add model recipe (#63)

wenhuach21f39638d983fc
diff --git a/README.md b/README.md
index bb0b3fe6d..053e0dc2f 100644
--- a/README.md
+++ b/README.md
@@ -149,40 +149,42 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
## Support List
-| Model | Supported |
-|--------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Intel/neural-chat-7b-v3-3 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc), [accuracy](./docs/neural-chat-7b-v3-3-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-3.sh), [example](./examples/language-modeling/) |
-| Intel/neural-chat-7b-v3-1 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc), [accuracy](./docs/neural-chat-7b-v3-1-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-1.sh), [example](./examples/language-modeling/) |
-| mistralai/Mistral-7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc), [accuracy](./docs/Mistral-7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-v0.1.sh), [example](./examples/language-modeling/) |
-| microsoft/phi-2 | [HF-int4-model](https://huggingface.co/Intel/phi-2-int4-inc), [accuracy](./docs/phi-2-acc.md), [recipe](./examples/language-modeling/scripts/phi-2.sh), [example](./examples/language-modeling/)
-| tiiuae/falcon-7b | [HF-int4-model](https://huggingface.co/Intel/falcon-7b-int4-inc), [accuracy](./docs/falcon-7b-acc.md), [recipe](./examples/language-modeling/scripts/falcon-7b.sh), [example](./examples/language-modeling/) |
-| google/gemma-7b | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-int4-inc) (under review), [accuracy](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh), [example](./examples/language-modeling/) |
-| google/gemma-7b-it | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-it-int4-inc) (under review), [accuracy](./docs/gemma-7b-it-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b-it.sh), [example](./examples/language-modeling/) | |
- mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) |
-| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |
-| meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) |
-| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [recipe1](./examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh), [recipe2](./examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.sh), [example](./examples/language-modeling/) |
-| Salesforce/codegen25-7b-multi | [example](./examples/code-generation) |
-| EleutherAI/gpt-j-6b | [example](./examples/language-modeling/) |
-| huggyllama/llama-7b | [example](./examples/language-modeling/) |
-| meta-llama/Llama-2-7b-hf | [example](./examples/language-modeling/) |
-| facebook/opt-6.7b | [example](./examples/language-modeling/) ||
-| mosaicml/mpt-7b | [example](./examples/language-modeling/) |
-| bigscience/bloom-7b1 | [example](./examples/language-modeling/) |
-| baichuan-inc/Baichuan-7B | [example](./examples/language-modeling/) |
-| Qwen/Qwen-7B | [example](./examples/language-modeling/) |
-| THUDM/chatglm3-6b | [example](./examples/language-modeling/) |
-| MBZUAI/LaMini-GPT-124M | [example](./examples/language-modeling/) |
-| EleutherAI/gpt-neo-125m | [example](./examples/language-modeling/) |
-| databricks/dolly-v2-3b | [example](./examples/language-modeling/) |
-| stabilityai/stablelm-base-alpha-3b | [example](./examples/language-modeling/)
+| Model | Supported |
+|--------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Intel/neural-chat-7b-v3-3 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc), [accuracy](./docs/neural-chat-7b-v3-3-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-3.sh), [example](./examples/language-modeling/) |
+| Intel/neural-chat-7b-v3-1 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc), [accuracy](./docs/neural-chat-7b-v3-1-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-1.sh), [example](./examples/language-modeling/) |
+| mistralai/Mistral-7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc), [accuracy](./docs/Mistral-7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-v0.1.sh), [example](./examples/language-modeling/) |
+| microsoft/phi-2 | [HF-int4-model](https://huggingface.co/Intel/phi-2-int4-inc), [accuracy](./docs/phi-2-acc.md), [recipe](./examples/language-modeling/scripts/phi-2.sh), [example](./examples/language-modeling/)
+| tiiuae/falcon-7b | [HF-int4-model](https://huggingface.co/Intel/falcon-7b-int4-inc), [accuracy](./docs/falcon-7b-acc.md), [recipe](./examples/language-modeling/scripts/falcon-7b.sh), [example](./examples/language-modeling/) |
+| google/gemma-2b | [HF-int4-model](https://huggingface.co/Intel/gemma-2b-int4-inc), [accuracy](./docs/gemma-2b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-2b.sh), [example](./examples/language-modeling/)
+| mistralai/Mixtral-7B-Instruct-v0.2 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-Instruct-v0.2-int4-inc) (under review), [accuracy](./docs/Mistral-7B-Instruct-v0.2-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.sh), [example](./examples/language-modeling/) |
+| google/gemma-7b | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-int4-inc) (under review), [accuracy](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh), [example](./examples/language-modeling/) |
+| google/gemma-7b-it | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-it-int4-inc) (under review), [accuracy](./docs/gemma-7b-it-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b-it.sh), [example](./examples/language-modeling/) |
+| mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) |
+| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |
+| meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) |
+| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [recipe1](./examples/language-modeling/scripts/Qwen1.5-7B-Chat.sh), [recipe2](./examples/language-modeling/scripts/Qwen1.5-7B-Chat_2.sh), [example](./examples/language-modeling/) |
+| Salesforce/codegen25-7b-multi | [example](./examples/code-generation) |
+| EleutherAI/gpt-j-6b | [example](./examples/language-modeling/) |
+| huggyllama/llama-7b | [example](./examples/language-modeling/) |
+| meta-llama/Llama-2-7b-hf | [example](./examples/language-modeling/) |
+| facebook/opt-6.7b | [example](./examples/language-modeling/) ||
+| mosaicml/mpt-7b | [example](./examples/language-modeling/) |
+| bigscience/bloom-7b1 | [example](./examples/language-modeling/) |
+| baichuan-inc/Baichuan-7B | [example](./examples/language-modeling/) |
+| Qwen/Qwen-7B | [example](./examples/language-modeling/) |
+| THUDM/chatglm3-6b | [example](./examples/language-modeling/) |
+| MBZUAI/LaMini-GPT-124M | [example](./examples/language-modeling/) |
+| EleutherAI/gpt-neo-125m | [example](./examples/language-modeling/) |
+| databricks/dolly-v2-3b | [example](./examples/language-modeling/) |
+| stabilityai/stablelm-base-alpha-3b | [example](./examples/language-modeling/)
## Comparison with other methods
-We provide a [comprehensive analysis](docs/acc.md) with other methods in our accuracy data section. Notably, our approach has outperformed GPTQ with a score of 30/32 and AWQ with a score of 27/32 across llamv1/llamav2/mistral-7b on W4G-1, W4G128, W3G128, W2G128. And the tuning costs are comparable.
+We provide a [comprehensive analysis](docs/acc.md) with other methods in our accuracy data section. In summary, our approach achieved superior performance compared to GPTQ, scoring 30/32, AWQ with 27/32, HQQ with 15/16, and OmniQuant with a perfect score of 16/16 across llamv1/llamav2/mistral-7b on W4G-1, W4G128, W3G128, and W2G128, based on the average accuracies of 11 zero-shot tasks.. And the tuning costs are comparable.
## Tips
1 Consider increasing tuning steps to achieve better results, albeit with increased tuning time.
diff --git a/docs/Mistral-7B-Instruct-v0.2-acc.md b/docs/Mistral-7B-Instruct-v0.2-acc.md
new file mode 100644
index 000000000..c14d2f73a
--- /dev/null
+++ b/docs/Mistral-7B-Instruct-v0.2-acc.md
@@ -0,0 +1,23 @@
+Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d
+##pip install auto-gptq[triton]
+##pip install triton==2.2.0
+```bash
+lm_eval --model hf --model_args pretrained="Intel/Mistral-7B-Instruct-v0.2-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32
+```
+
+
+
+| Metric | BF16 | INT4 |
+| -------------- | ------ | ------ |
+| Avg. | 0.6647 | 0.6621 |
+| mmlu | 0.5906 | 0.5872 |
+| lambada_openai | 0.7141 | 0.7141 |
+| hellaswag | 0.6602 | 0.6557 |
+| winogrande | 0.7395 | 0.7364 |
+| piqa | 0.8052 | 0.8047 |
+| truthfulqa_mc1 | 0.5251 | 0.5153 |
+| openbookqa | 0.3600 | 0.3420 |
+| boolq | 0.8535 | 0.8541 |
+| rte | 0.7040 | 0.7148 |
+| arc_easy | 0.8161 | 0.8165 |
+| arc_challenge | 0.5435 | 0.5435 |
diff --git a/docs/Mistral-7B-v0.1-acc.md b/docs/Mistral-7B-v0.1-acc.md
index d260fa41d..58ddfd1d9 100644
--- a/docs/Mistral-7B-v0.1-acc.md
+++ b/docs/Mistral-7B-v0.1-acc.md
@@ -6,8 +6,8 @@ lm_eval --model hf --model_args pretrained="Intel/Mistral-7B-v0.1-int4-inc",auto
-| Metric | FP16 | INT4 |
-| -------------- | ------ | ------ |
+| Metric | BF16 | INT4 |
+| -------------- |--------| ------ |
| Avg. | 0.6306 | 0.6308 |
| mmlu | 0.5961 | 0.5880 |
| lambada_openai | 0.7561 | 0.7551 |
diff --git a/docs/Mixtral-8x7B-Instruct-v0.1-acc.md b/docs/Mixtral-8x7B-Instruct-v0.1-acc.md
index d7b10b775..2dd4d62ea 100644
--- a/docs/Mixtral-8x7B-Instruct-v0.1-acc.md
+++ b/docs/Mixtral-8x7B-Instruct-v0.1-acc.md
@@ -6,8 +6,8 @@ Download the model from hf(coming soon) or follow examples/language-modeling/scr
lm_eval --model hf --model_args pretrained="Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32
```
-| Metric | FP16 | INT4 |
-| -------------- | ------ | ------ |
+| Metric | BF16 | INT4 |
+| -------------- |--------| ------ |
| Avg. | 0.7000 | 0.6977 |
| mmlu | 0.6885 | 0.6824 |
| lambada_openai | 0.7718 | 0.7790 |
diff --git a/docs/Mixtral-8x7B-v0.1-acc.md b/docs/Mixtral-8x7B-v0.1-acc.md
index 78f26e6d7..d9f9632d3 100644
--- a/docs/Mixtral-8x7B-v0.1-acc.md
+++ b/docs/Mixtral-8x7B-v0.1-acc.md
@@ -7,8 +7,8 @@ Download the model from hf(coming soon) or follow examples/language-modeling/scr
lm_eval --model hf --model_args pretrained="Intel/Mixtral-8x7B-v0.1-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32
~~~
-| Metric | FP16 | INT4 |
-| -------------- | ------ | ------ |
+| Metric | BF16 | INT4 |
+| -------------- |--------| ------ |
| Avg. | 0.6698 | 0.6633 |
| mmlu | 0.6802 | 0.6693 |
| lambada_openai | 0.7827 | 0.7825 |
diff --git a/docs/gemma-2b-acc.md b/docs/gemma-2b-acc.md
new file mode 100644
index 000000000..0b5c5e918
--- /dev/null
+++ b/docs/gemma-2b-acc.md
@@ -0,0 +1,27 @@
+### Evaluate the model
+
+Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id we used is 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d
+
+pip install auto-gptq[triton]
+pip install triton==2.2.0
+
+Please note that there is a discrepancy between the baseline result and the official data, which is a known issue within the official model card community.
+
+```bash
+lm_eval --model hf --model_args pretrained="Intel/gemma-2b-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 16
+```
+
+| Metric | FP16 | INT4 |
+| -------------- | ------ |--------|
+| Avg. | 0.5383 | 0.5338 |
+| mmlu | 0.3337 | 0.3276 |
+| lambada_openai | 0.6398 | 0.6319 |
+| hellaswag | 0.5271 | 0.5161 |
+| winogrande | 0.6472 | 0.6472 |
+| piqa | 0.7699 | 0.7622 |
+| truthfulqa_mc1 | 0.2203 | 0.2191 |
+| openbookqa | 0.3020 | 0.2980 |
+| boolq | 0.6939 | 0.6939 |
+| rte | 0.6426 | 0.6498 |
+| arc_easy | 0.7424 | 0.7348 |
+| arc_challenge | 0.4019 | 0.3908 |
diff --git a/docs/gemma-7b-acc.md b/docs/gemma-7b-acc.md
index 1ec091ce7..9f03be4dd 100644
--- a/docs/gemma-7b-acc.md
+++ b/docs/gemma-7b-acc.md
@@ -1,5 +1,7 @@
Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d, Install the latest [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) from source first
+Please note that there is a discrepancy between the baseline result and the official data, which is a known issue within the official model card community.
+
```bash
lm_eval --model hf --model_args pretrained="Intel/gemma-7b-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32
```
diff --git a/docs/gemma-7b-it-acc.md b/docs/gemma-7b-it-acc.md
index ca0afd07d..1d1f1b52f 100644
--- a/docs/gemma-7b-it-acc.md
+++ b/docs/gemma-7b-it-acc.md
@@ -1,5 +1,7 @@
Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d, Install the latest [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) from source first
+Please note that there is a discrepancy between the baseline result and the official data, which is a known issue within the official model card community.
+
```bash
lm_eval --model hf --model_args pretrained="Intel/gemma-7b-it-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32
```
diff --git a/examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.sh b/examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.sh
new file mode 100644
index 000000000..baa55cf31
--- /dev/null
+++ b/examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.sh
@@ -0,0 +1,9 @@
+python3 main.py \
+--model_name mistralai/Mistral-7B-Instruct-v0.2 \
+--device 0 \
+--group_size 128 \
+--bits 4 \
+--iters 1000 \
+--use_quant_input \
+--deployment_device 'gpu' \
+--output_dir "./tmp_autoround"
\ No newline at end of file
diff --git a/examples/language-modeling/scripts/gemma-2b.sh b/examples/language-modeling/scripts/gemma-2b.sh
new file mode 100644
index 000000000..5ef8716a1
--- /dev/null
+++ b/examples/language-modeling/scripts/gemma-2b.sh
@@ -0,0 +1,12 @@
+git clone https://github.com/intel/auto-round
+cd auto-round/examples/language-modeling
+pip install -r requirements.txt
+python3 main.py \
+--model_name google/gemma-2b \
+--device 0 \
+--group_size 128 \
+--bits 4 \
+--iters 400 \
+--use_quant_input \
+--deployment_device 'gpu' \
+--output_dir "./tmp_autoround"
\ No newline at end of file