PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
Update falcon recipe (#128)
wenhuach21416ec7e98b58
diff --git a/README.md b/README.mdindex 926bf5db5..a5d64c3b4 100644--- a/README.md+++ b/README.md@@ -25,9 +25,11 @@ image presents an overview of AutoRound.<div align="left">## What's New+* [2024/05] Check out our updated paper on [arxiv](https://arxiv.org/pdf/2309.05516v4)* [2024/05] AutoRound supports lm-head quantization, saving 0.7G for LLaMA3-8B at W4G128.-* [2024/05] AutoRound performs well in [low_bit_open_llm_leaderboard](https://huggingface.co/spaces/Intel/low_bit_open_llm_leaderboard)+* [2024/05] AutoRound performs well+ in [low_bit_open_llm_leaderboard](https://huggingface.co/spaces/Intel/low_bit_open_llm_leaderboard)## Prerequisites@@ -51,7 +53,10 @@ pip install auto-round## Model quantization### Gaudi2/ CPU/ GPU-We found a significant accuracy discrepancy with the qdq model using the AutoGPTQ GPU backend with asymmetric quantization in some scenarios. Please switch to symmetric quantization to alleviate this issue.++We found a significant accuracy discrepancy with the qdq model using the AutoGPTQ GPU backend with asymmetric+quantization in some scenarios. Please switch to symmetric quantization to alleviate this issue.+```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizer@@ -110,7 +115,9 @@ autoround.save_quantized(output_dir)- `low_gpu_mem_usage (bool)`: Whether to save GPU memory at the cost of ~20% more tuning time (default is True).-- `dataset Union[str, list, tuple, torch.utils.data.DataLoader]`: The dataset name for tuning (default is "NeelNanda/pile-10k"). Local json file and combination of datasets have been supported, e.g. "./tmp.json,NeelNanda/pile-10k:train, mbpp:train+validation+test"+- `dataset Union[str, list, tuple, torch.utils.data.DataLoader]`: The dataset name for tuning (default is "+ NeelNanda/pile-10k"). Local json file and combination of datasets have been supported, e.g. "+ ./tmp.json,NeelNanda/pile-10k:train, mbpp:train+validation+test"- `weight_config (dict)`: Configuration for weight quantization (default is an empty dictionary), mainly for mixed bitsor mixed precision.@@ -142,6 +149,7 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizer+##from auto_round.export import AutoHfQuantizer ## uncomment it for models with quantized lm-headquantized_model_path = "./tmp_autoround"model = AutoModelForCausalLM.from_pretrained(quantized_model_path, device_map="auto", trust_remote_code=True)@@ -153,33 +161,34 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))## Support List-| Model | Supported |-|--------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-| Intel/neural-chat-7b-v3-3 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc), [accuracy](./docs/neural-chat-7b-v3-3-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-3.sh), [example](./examples/language-modeling/) |-| Intel/neural-chat-7b-v3-1 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc), [accuracy](./docs/neural-chat-7b-v3-1-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-1.sh), [example](./examples/language-modeling/) |+| Model | Supported |+|--------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|+| Intel/neural-chat-7b-v3-3 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc), [accuracy](./docs/neural-chat-7b-v3-3-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-3.sh), [example](./examples/language-modeling/) |+| Intel/neural-chat-7b-v3-1 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc), [accuracy](./docs/neural-chat-7b-v3-1-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-1.sh), [example](./examples/language-modeling/) || mistralai/Mistral-7B-v0.1 | [HF-int4-model-lmhead](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc-lmhead),[HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc), [accuracy](./docs/Mistral-7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-v0.1.sh), [example](./examples/language-modeling/) |-| microsoft/phi-2 | [HF-int4-sym-model](https://huggingface.co/Intel/phi-2-int4-inc), [accuracy](./docs/phi-2-acc.md), [recipe](./examples/language-modeling/scripts/phi-2.sh), [example](./examples/language-modeling/)-| google/gemma-2b | [HF-int4-model](https://huggingface.co/Intel/gemma-2b-int4-inc), [accuracy](./docs/gemma-2b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-2b.sh), [example](./examples/language-modeling/)-| mistralai/Mistral-7B-Instruct-v0.2 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-Instruct-v0.2-int4-inc) (under review), [accuracy](./docs/Mistral-7B-Instruct-v0.2-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.sh), [example](./examples/language-modeling/) |-| google/gemma-7b | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-int4-inc) (under review), [accuracy](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh), [example](./examples/language-modeling/) |-| mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) |-| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |-| meta-llama/Meta-Llama-3-8B-Instruct | [accuracy](./docs/Meta-Llama-3-8B-Instruct-acc.md), [recipe](./examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.sh), [example](./examples/language-modeling/) |-| meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) |-| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [sym recipe](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh), [asym recipe ](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh), [example](./examples/language-modeling/) |-| baichuan-inc/Baichuan2-7B-Chat | [accuracy](./docs/baichuan2-7b-chat-acc.md), [recipe](./examples/language-modeling/scripts/baichuan2-7b-chat.sh), [example](./examples/language-modeling/) |-| 01-ai/Yi-6B-Chat | [accuracy](./docs/Yi-6B-Chat-acc.md), [recipe](./examples/language-modeling/scripts/Yi-6B-Chat.sh), [example](./examples/language-modeling/) |-| facebook/opt-2.7b | [accuracy](./docs/opt-2.7b-acc.md), [recipe](./examples/language-modeling/scripts/opt-2.7b.sh), [example](./examples/language-modeling/) |-| bigscience/bloom-3b | [accuracy](./docs/bloom-3B-acc.md), [recipe](./examples/language-modeling/scripts/bloom-3b.sh), [example](./examples/language-modeling/) |-| EleutherAI/gpt-j-6b | [accuracy](./docs/gpt-j-6B-acc.md), [recipe](./examples/language-modeling/scripts/gpt-j-6b.sh), [example](./examples/language-modeling/) |-| Salesforce/codegen25-7b-multi | [example](./examples/language-modeling/) |-| huggyllama/llama-7b | [example](./examples/language-modeling/) |-| mosaicml/mpt-7b | [example](./examples/language-modeling/) |-| THUDM/chatglm3-6b | [example](./examples/language-modeling/) |-| MBZUAI/LaMini-GPT-124M | [example](./examples/language-modeling/) |-| EleutherAI/gpt-neo-125m | [example](./examples/language-modeling/) |-| databricks/dolly-v2-3b | [example](./examples/language-modeling/) |-| stabilityai/stablelm-base-alpha-3b | [example](./examples/language-modeling/)+| microsoft/phi-2 | [HF-int4-sym-model](https://huggingface.co/Intel/phi-2-int4-inc), [accuracy](./docs/phi-2-acc.md), [recipe](./examples/language-modeling/scripts/phi-2.sh), [example](./examples/language-modeling/)+| google/gemma-2b | [HF-int4-model](https://huggingface.co/Intel/gemma-2b-int4-inc), [accuracy](./docs/gemma-2b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-2b.sh), [example](./examples/language-modeling/)+| tiiuae/falcon-7b | [HF-int4-model-G64](https://huggingface.co/Intel/falcon-7b-int4-inc), [accuracy](./docs/falcon-7b-acc.md), [recipe](./examples/language-modeling/scripts/falcon-7b.sh), [example](./examples/language-modeling/) |+| mistralai/Mistral-7B-Instruct-v0.2 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-Instruct-v0.2-int4-inc) (under review), [accuracy](./docs/Mistral-7B-Instruct-v0.2-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.sh), [example](./examples/language-modeling/) |+| google/gemma-7b | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-int4-inc) (under review), [accuracy](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh), [example](./examples/language-modeling/) |+| mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) |+| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |+| meta-llama/Meta-Llama-3-8B-Instruct | [accuracy](./docs/Meta-Llama-3-8B-Instruct-acc.md), [recipe](./examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.sh), [example](./examples/language-modeling/) |+| meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) |+| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [sym recipe](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh), [asym recipe ](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh), [example](./examples/language-modeling/) |+| baichuan-inc/Baichuan2-7B-Chat | [accuracy](./docs/baichuan2-7b-chat-acc.md), [recipe](./examples/language-modeling/scripts/baichuan2-7b-chat.sh), [example](./examples/language-modeling/) |+| 01-ai/Yi-6B-Chat | [accuracy](./docs/Yi-6B-Chat-acc.md), [recipe](./examples/language-modeling/scripts/Yi-6B-Chat.sh), [example](./examples/language-modeling/) |+| facebook/opt-2.7b | [accuracy](./docs/opt-2.7b-acc.md), [recipe](./examples/language-modeling/scripts/opt-2.7b.sh), [example](./examples/language-modeling/) |+| bigscience/bloom-3b | [accuracy](./docs/bloom-3B-acc.md), [recipe](./examples/language-modeling/scripts/bloom-3b.sh), [example](./examples/language-modeling/) |+| EleutherAI/gpt-j-6b | [accuracy](./docs/gpt-j-6B-acc.md), [recipe](./examples/language-modeling/scripts/gpt-j-6b.sh), [example](./examples/language-modeling/) |+| Salesforce/codegen25-7b-multi | [example](./examples/language-modeling/) |+| huggyllama/llama-7b | [example](./examples/language-modeling/) |+| mosaicml/mpt-7b | [example](./examples/language-modeling/) |+| THUDM/chatglm3-6b | [example](./examples/language-modeling/) |+| MBZUAI/LaMini-GPT-124M | [example](./examples/language-modeling/) |+| EleutherAI/gpt-neo-125m | [example](./examples/language-modeling/) |+| databricks/dolly-v2-3b | [example](./examples/language-modeling/) |+| stabilityai/stablelm-base-alpha-3b | [example](./examples/language-modeling/)## Comparison with other methodsdiff --git a/docs/falcon-7b-acc.md b/docs/falcon-7b-acc.mdindex f3a459bdf..98bbf5cc9 100644--- a/docs/falcon-7b-acc.md+++ b/docs/falcon-7b-acc.md@@ -1,22 +1,22 @@-Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d,+We generate the model with group_size 64 as there is an issue when evaluating with group_size 128.+Evaluate the model+pip3 install lm-eval==0.4.2-pip install auto-gptq[triton]+```bash+lm_eval --model hf --model_args pretrained="Intel/falcon-7b-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,mmlu --batch_size 16+```-pip install triton==2.2.0+| Metric | BF16 | int4 |+| -------------- | ------ | ------ |+| Avg. | 0.5462 | 0.5454 |+| mmlu | 0.2546 | 0.2562 |+| lambada_openai | 0.7450 | 0.7485 |+| hellaswag | 0.5773 | 0.5719 |+| winogrande | 0.6740 | 0.6835 |+| piqa | 0.7943 | 0.7905 |+| truthfulqa_mc1 | 0.2228 | 0.2166 |+| openbookqa | 0.3080 | 0.3100 |+| boolq | 0.7361 | 0.7431 |+| arc_easy | 0.7475 | 0.7424 |+| arc_challenge | 0.4027 | 0.3908 |-Since we encountered an issue evaluating this model with lm-eval, we opted to evaluate the qdq model instead. In our assessment, we found that its accuracy closely matches that of the real quantized model in most cases except for some small models like opt-125m. The batch size 32 is used.--| Metric | FP16 | int4 qdq |-| -------------- | ------ | -------- |-| Avg. | 0.5521 | 0.5507 |-| mmlu | 0.2495 | 0.2427 |-| lambada_openai | 0.7452 | 0.7487 |-| hellaswag | 0.5771 | 0.5731 |-| winogrande | 0.6725 | 0.6756 |-| piqa | 0.7949 | 0.7943 |-| truthfulqa_mc1 | 0.2252 | 0.2142 |-| openbookqa | 0.3060 | 0.3060 |-| boolq | 0.7364 | 0.7382 |-| rte | 0.6173 | 0.6245 |-| arc_easy | 0.7479 | 0.7433 |-| arc_challenge | 0.4019 | 0.3968 |diff --git a/examples/language-modeling/scripts/falcon-7b.sh b/examples/language-modeling/scripts/falcon-7b.shindex 20beca6c0..07bbae4d4 100644--- a/examples/language-modeling/scripts/falcon-7b.sh+++ b/examples/language-modeling/scripts/falcon-7b.sh@@ -1,9 +1,10 @@python3 main.py \--model_name tiiuae/falcon-7b \--device 0 \---group_size 128 \+--group_size 64 \--bits 4 \--iters 1000 \--deployment_device 'gpu' \--output_dir "./tmp_autoround" \---disable_quanted_input\ No newline at end of file+--disable_quanted_input \+--disable_low_gpu_mem_usage\ No newline at end of file