PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
add support matrix in readme (#26)
wenhuach217caadac77b58
diff --git a/README.md b/README.mdindex ca793d669..960c7dcec 100644--- a/README.md+++ b/README.md@@ -59,7 +59,8 @@ bits, group_size, scheme = 4, 128, "asym"# need to load model first, then importfrom auto_round import AutoRound-autoround = AutoRound(model, tokenizer, bits=bits, group_size=group_size, scheme=scheme, device="hpu", amp=True)+tuning_device = "hpu"+autoround = AutoRound(model, tokenizer, bits=bits, group_size=group_size, scheme=scheme, device=tuning_device)autoround.quantize()```@@ -111,8 +112,40 @@ autoround.quantize()- `device`: The device to be used for tuning (default is "cuda:0").+- `scale_dtype (str)`: The data type of quantization scale to be used (default is "float32"), different kernels have different choices.+</details>++## Support List++| Model | Supported |+|--------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|+| Intel/neural-chat-7b-v3-3 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc), [accuracy](./docs/neural-chat-7b-v3-3-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-3.sh), [example](./examples/language-modeling/) |+| Intel/neural-chat-7b-v3-1 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc), [accuracy](./docs/neural-chat-7b-v3-1-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-1.sh), [example](./examples/language-modeling/) |+| mistralai/Mistral-7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc), [accuracy](./docs/Mistral-7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-v0.1.sh), [example](./examples/language-modeling/) |+| google/gemma-7b | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-int4-inc) under review, [accuracy](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh), [example](./examples/language-modeling/) |+| google/gemma-7b-it | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-it-int4-inc) under review, [accuracy](./docs/gemma-7b-it-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b-it.sh), [example](./examples/language-modeling/) | |+ mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) under review, [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) |+| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) under review, [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |+| microsoft/phi-2 | [HF-int4-model](https://huggingface.co/Intel/phi-2-int4-inc) under review, [accuracy](./docs/phi-2-acc.md), [recipe](./examples/language-modeling/scripts/phi-2.sh), [example](./examples/language-modeling/) |+| Salesforce/codegen25-7b-multi | [example](./examples/code-generation) |+| EleutherAI/gpt-j-6b | [example](./examples/language-modeling/) |+| huggyllama/llama-7b | [example](./examples/language-modeling/) |+| meta-llama/Llama-2-7b-hf | [example](./examples/language-modeling/) |+| facebook/opt-6.7b | [example](./examples/language-modeling/) |+| tiiuae/falcon-7b | [example](./examples/language-modeling/) |+| mosaicml/mpt-7b | [example](./examples/language-modeling/) |+| bigscience/bloom-7b1 | [example](./examples/language-modeling/) |+| baichuan-inc/Baichuan-7B | [example](./examples/language-modeling/) |+| Qwen/Qwen-7B | [example](./examples/language-modeling/) |+| THUDM/chatglm3-6b | [example](./examples/language-modeling/) |+| MBZUAI/LaMini-GPT-124M | [example](./examples/language-modeling/) |+| EleutherAI/gpt-neo-125m | [example](./examples/language-modeling/) |+| databricks/dolly-v2-3b | [example](./examples/language-modeling/) |+| stabilityai/stablelm-base-alpha-3b | [example](./examples/language-modeling/) |++## Model inferencePlease run the tuning code first@@ -150,27 +183,36 @@ text = "There is a girl who likes adventure,"inputs = tokenizer(text, return_tensors="pt").to(model.device)print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))```-## Huggingface Model cards-We fine-tuned the hyperparameters for each model with an iteration of 1K and successfully achieved near-lossless quantized models in the majority of scenarios. Some of these models has been uploaded to the Huggingface Hub.--### AutoGPTQ format-[Intel/neural-chat-7b-v3-3-int4-inc](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc)--[Intel/neural-chat-7b-v3-1-int4-inc](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc)--[Intel/Mistral-7B-v0.1-int4-inc](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc)--[Intel/gemma-7b-int4-inc](https://huggingface.co/Intel/gemma-7b-int4-inc) internal review, [accuracy](./docs/gemma-7b-acc.md), [quantization shell](./examples/language-modeling/scripts/gemma-7b.sh)--[Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc](https://huggingface.co/Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc) coming soon, [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [quantization shell](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh)-[Intel/Mixtral-8x7B-v0.1-int4-inc](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) coming soon,[accuracy](./docs/Mistral-7B-v0.1-acc.md), [quantization shell](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh)--[Intel/phi-2-int4-inc](https://huggingface.co/Intel/phi-2-int4-inc) coming soon, [accuracy](./docs/phi-2-acc.md) [quantization shell](./examples/language-modeling/scripts/phi-2.sh)-### Itrex format+## Support List++| Model | Supported |+|--------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|+| Intel/neural-chat-7b-v3-3 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc), [acc](./docs/neural-chat-7b-v3-3-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-3.sh), [example](./examples/language-modeling/) |+| Intel/neural-chat-7b-v3-1 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc), [acc](./docs/neural-chat-7b-v3-1-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-1.sh), [example](./examples/language-modeling/) |+| mistralai/Mistral-7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc), [acc](./docs/Mistral-7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-v0.1.sh), [example](./examples/language-modeling/) |+| google/gemma-7b | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-int4-inc) in review, [acc](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh), [example](./examples/language-modeling/) |+| google/gemma-7b-it | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-it-int4-inc) in review, [acc](./docs/gemma-7b-it-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b-it.sh), [example](./examples/language-modeling/) | |+ mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) in review, [acc](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) |+| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) in review, [acc](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |+| microsoft/phi-2 | [HF-int4-model](https://huggingface.co/Intel/phi-2-int4-inc) in review, [acc](./docs/phi-2-acc.md), [recipe](./examples/language-modeling/scripts/phi-2.sh), [example](./examples/language-modeling/) |+| Salesforce/codegen25-7b-multi | [example](./examples/code-generation) |+| EleutherAI/gpt-j-6b | [example](./examples/language-modeling/) |+| huggyllama/llama-7b | [example](./examples/language-modeling/) |+| meta-llama/Llama-2-7b-hf | [example](./examples/language-modeling/) |+| facebook/opt-6.7b | [example](./examples/language-modeling/) |+| tiiuae/falcon-7b | [example](./examples/language-modeling/) |+| mosaicml/mpt-7b | [example](./examples/language-modeling/) |+| bigscience/bloom-7b1 | [example](./examples/language-modeling/) |+| baichuan-inc/Baichuan-7B | [example](./examples/language-modeling/) |+| Qwen/Qwen-7B | [example](./examples/language-modeling/) |+| THUDM/chatglm3-6b | [example](./examples/language-modeling/) |+| MBZUAI/LaMini-GPT-124M | [example](./examples/language-modeling/) |+| EleutherAI/gpt-neo-125m | [example](./examples/language-modeling/) |+| databricks/dolly-v2-3b | [example](./examples/language-modeling/) |+| stabilityai/stablelm-base-alpha-3b | [example](./examples/language-modeling/) |-Please stay tuned## Comparison with other methodsdiff --git a/auto_round/autoround.py b/auto_round/autoround.pyindex 429403f67..fb492f926 100644--- a/auto_round/autoround.py+++ b/auto_round/autoround.py@@ -504,6 +504,7 @@ class AutoRound(object):not_use_best_mse (bool): Whether to use mean squared error (default is False).dynamic_max_gap (int): The dynamic maximum gap (default is -1).data_type (str): The data type to be used (default is "int").+ scale_dtype (str): The data type of quantization scale to be used (default is "float32")**kwargs: Additional keyword arguments.Returns:@@ -541,7 +542,7 @@ class AutoRound(object):not_use_best_mse: bool = False,dynamic_max_gap: int = -1,data_type: str = "int", ##only support data_type- scale_dtype="fp16",+ scale_dtype: str = "fp32",**kwargs,):from .calib_dataset import CALIB_DATASETS@@ -575,9 +576,9 @@ class AutoRound(object):self.n_blocks = n_blocksself.device = device- if scale_dtype == "fp16":+ if scale_dtype == "fp16" or scale_dtype == "float16":self.scale_dtype = torch.float16- elif scale_dtype == "bf16":+ elif scale_dtype == "bf16" or scale_dtype == "bfloat16":self.scale_dtype = torch.bfloat16else:self.scale_dtype = torch.float32diff --git a/docs/gemma-7b-it-acc.md b/docs/gemma-7b-it-acc.mdnew file mode 100644index 000000000..ca0afd07d--- /dev/null+++ b/docs/gemma-7b-it-acc.md@@ -0,0 +1,20 @@+Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d, Install the latest [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) from source first++```bash+lm_eval --model hf --model_args pretrained="Intel/gemma-7b-it-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32+```++| Metric | FP16 | int4 |+| -------------- | ------ | ------ |+| Avg. | 0.6022 | 0.6017 |+| mmlu | 0.5029 | 0.4993 |+| lambada_openai | 0.6035 | 0.6286 |+| hellaswag | 0.5620 | 0.5564 |+| winogrande | 0.6796 | 0.6788 |+| piqa | 0.7709 | 0.7731 |+| truthfulqa_mc1 | 0.3048 | 0.3035 |+| openbookqa | 0.3740 | 0.3700 |+| boolq | 0.8138 | 0.8144 |+| rte | 0.7870 | 0.7870 |+| arc_easy | 0.7525 | 0.7508 |+| arc_challenge | 0.4727 | 0.4573 |diff --git a/examples/language-modeling/scripts/gemma-7b-it.sh b/examples/language-modeling/scripts/gemma-7b-it.shnew file mode 100644index 000000000..222387add--- /dev/null+++ b/examples/language-modeling/scripts/gemma-7b-it.sh@@ -0,0 +1,15 @@+python3 main.py \+--model_name google/gemma-7b-it \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--enable_minmax_tuning \+--use_quant_input \+--minmax_lr 2e-3 \+--low_gpu_mem_usage \+--deployment_device 'gpu' \+--scale_dtype 'fp32' \+--eval_bs 32 \+--output_dir "./tmp_autoround" \+--amp\ No newline at end of file