Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

add support matrix in readme (#26)

wenhuach217caadac77b58
diff --git a/README.md b/README.md
index ca793d669..960c7dcec 100644
--- a/README.md
+++ b/README.md
@@ -59,7 +59,8 @@ bits, group_size, scheme = 4, 128, "asym"
# need to load model first, then import
from auto_round import AutoRound
-autoround = AutoRound(model, tokenizer, bits=bits, group_size=group_size, scheme=scheme, device="hpu", amp=True)
+tuning_device = "hpu"
+autoround = AutoRound(model, tokenizer, bits=bits, group_size=group_size, scheme=scheme, device=tuning_device)
autoround.quantize()
```
@@ -111,8 +112,40 @@ autoround.quantize()
- `device`: The device to be used for tuning (default is "cuda:0").
+- `scale_dtype (str)`: The data type of quantization scale to be used (default is "float32"), different kernels have different choices.
+
</details>
+
+## Support List
+
+| Model | Supported |
+|--------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Intel/neural-chat-7b-v3-3 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc), [accuracy](./docs/neural-chat-7b-v3-3-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-3.sh), [example](./examples/language-modeling/) |
+| Intel/neural-chat-7b-v3-1 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc), [accuracy](./docs/neural-chat-7b-v3-1-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-1.sh), [example](./examples/language-modeling/) |
+| mistralai/Mistral-7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc), [accuracy](./docs/Mistral-7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-v0.1.sh), [example](./examples/language-modeling/) |
+| google/gemma-7b | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-int4-inc) under review, [accuracy](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh), [example](./examples/language-modeling/) |
+| google/gemma-7b-it | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-it-int4-inc) under review, [accuracy](./docs/gemma-7b-it-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b-it.sh), [example](./examples/language-modeling/) | |
+ mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) under review, [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) |
+| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) under review, [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |
+| microsoft/phi-2 | [HF-int4-model](https://huggingface.co/Intel/phi-2-int4-inc) under review, [accuracy](./docs/phi-2-acc.md), [recipe](./examples/language-modeling/scripts/phi-2.sh), [example](./examples/language-modeling/) |
+| Salesforce/codegen25-7b-multi | [example](./examples/code-generation) |
+| EleutherAI/gpt-j-6b | [example](./examples/language-modeling/) |
+| huggyllama/llama-7b | [example](./examples/language-modeling/) |
+| meta-llama/Llama-2-7b-hf | [example](./examples/language-modeling/) |
+| facebook/opt-6.7b | [example](./examples/language-modeling/) |
+| tiiuae/falcon-7b | [example](./examples/language-modeling/) |
+| mosaicml/mpt-7b | [example](./examples/language-modeling/) |
+| bigscience/bloom-7b1 | [example](./examples/language-modeling/) |
+| baichuan-inc/Baichuan-7B | [example](./examples/language-modeling/) |
+| Qwen/Qwen-7B | [example](./examples/language-modeling/) |
+| THUDM/chatglm3-6b | [example](./examples/language-modeling/) |
+| MBZUAI/LaMini-GPT-124M | [example](./examples/language-modeling/) |
+| EleutherAI/gpt-neo-125m | [example](./examples/language-modeling/) |
+| databricks/dolly-v2-3b | [example](./examples/language-modeling/) |
+| stabilityai/stablelm-base-alpha-3b | [example](./examples/language-modeling/) |
+
+
## Model inference
Please run the tuning code first
@@ -150,27 +183,36 @@ text = "There is a girl who likes adventure,"
inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
```
-## Huggingface Model cards
-We fine-tuned the hyperparameters for each model with an iteration of 1K and successfully achieved near-lossless quantized models in the majority of scenarios. Some of these models has been uploaded to the Huggingface Hub.
-
-### AutoGPTQ format
-[Intel/neural-chat-7b-v3-3-int4-inc](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc)
-
-[Intel/neural-chat-7b-v3-1-int4-inc](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc)
-
-[Intel/Mistral-7B-v0.1-int4-inc](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc)
-
-[Intel/gemma-7b-int4-inc](https://huggingface.co/Intel/gemma-7b-int4-inc) internal review, [accuracy](./docs/gemma-7b-acc.md), [quantization shell](./examples/language-modeling/scripts/gemma-7b.sh)
-
-[Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc](https://huggingface.co/Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc) coming soon, [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [quantization shell](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh)
-[Intel/Mixtral-8x7B-v0.1-int4-inc](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) coming soon,[accuracy](./docs/Mistral-7B-v0.1-acc.md), [quantization shell](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh)
-
-[Intel/phi-2-int4-inc](https://huggingface.co/Intel/phi-2-int4-inc) coming soon, [accuracy](./docs/phi-2-acc.md) [quantization shell](./examples/language-modeling/scripts/phi-2.sh)
-### Itrex format
+## Support List
+
+| Model | Supported |
+|--------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Intel/neural-chat-7b-v3-3 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc), [acc](./docs/neural-chat-7b-v3-3-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-3.sh), [example](./examples/language-modeling/) |
+| Intel/neural-chat-7b-v3-1 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc), [acc](./docs/neural-chat-7b-v3-1-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-1.sh), [example](./examples/language-modeling/) |
+| mistralai/Mistral-7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc), [acc](./docs/Mistral-7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-v0.1.sh), [example](./examples/language-modeling/) |
+| google/gemma-7b | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-int4-inc) in review, [acc](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh), [example](./examples/language-modeling/) |
+| google/gemma-7b-it | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-it-int4-inc) in review, [acc](./docs/gemma-7b-it-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b-it.sh), [example](./examples/language-modeling/) | |
+ mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) in review, [acc](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) |
+| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) in review, [acc](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |
+| microsoft/phi-2 | [HF-int4-model](https://huggingface.co/Intel/phi-2-int4-inc) in review, [acc](./docs/phi-2-acc.md), [recipe](./examples/language-modeling/scripts/phi-2.sh), [example](./examples/language-modeling/) |
+| Salesforce/codegen25-7b-multi | [example](./examples/code-generation) |
+| EleutherAI/gpt-j-6b | [example](./examples/language-modeling/) |
+| huggyllama/llama-7b | [example](./examples/language-modeling/) |
+| meta-llama/Llama-2-7b-hf | [example](./examples/language-modeling/) |
+| facebook/opt-6.7b | [example](./examples/language-modeling/) |
+| tiiuae/falcon-7b | [example](./examples/language-modeling/) |
+| mosaicml/mpt-7b | [example](./examples/language-modeling/) |
+| bigscience/bloom-7b1 | [example](./examples/language-modeling/) |
+| baichuan-inc/Baichuan-7B | [example](./examples/language-modeling/) |
+| Qwen/Qwen-7B | [example](./examples/language-modeling/) |
+| THUDM/chatglm3-6b | [example](./examples/language-modeling/) |
+| MBZUAI/LaMini-GPT-124M | [example](./examples/language-modeling/) |
+| EleutherAI/gpt-neo-125m | [example](./examples/language-modeling/) |
+| databricks/dolly-v2-3b | [example](./examples/language-modeling/) |
+| stabilityai/stablelm-base-alpha-3b | [example](./examples/language-modeling/) |
-Please stay tuned
## Comparison with other methods
diff --git a/auto_round/autoround.py b/auto_round/autoround.py
index 429403f67..fb492f926 100644
--- a/auto_round/autoround.py
+++ b/auto_round/autoround.py
@@ -504,6 +504,7 @@ class AutoRound(object):
not_use_best_mse (bool): Whether to use mean squared error (default is False).
dynamic_max_gap (int): The dynamic maximum gap (default is -1).
data_type (str): The data type to be used (default is "int").
+ scale_dtype (str): The data type of quantization scale to be used (default is "float32")
**kwargs: Additional keyword arguments.
Returns:
@@ -541,7 +542,7 @@ class AutoRound(object):
not_use_best_mse: bool = False,
dynamic_max_gap: int = -1,
data_type: str = "int", ##only support data_type
- scale_dtype="fp16",
+ scale_dtype: str = "fp32",
**kwargs,
):
from .calib_dataset import CALIB_DATASETS
@@ -575,9 +576,9 @@ class AutoRound(object):
self.n_blocks = n_blocks
self.device = device
- if scale_dtype == "fp16":
+ if scale_dtype == "fp16" or scale_dtype == "float16":
self.scale_dtype = torch.float16
- elif scale_dtype == "bf16":
+ elif scale_dtype == "bf16" or scale_dtype == "bfloat16":
self.scale_dtype = torch.bfloat16
else:
self.scale_dtype = torch.float32
diff --git a/docs/gemma-7b-it-acc.md b/docs/gemma-7b-it-acc.md
new file mode 100644
index 000000000..ca0afd07d
--- /dev/null
+++ b/docs/gemma-7b-it-acc.md
@@ -0,0 +1,20 @@
+Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d, Install the latest [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) from source first
+
+```bash
+lm_eval --model hf --model_args pretrained="Intel/gemma-7b-it-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32
+```
+
+| Metric | FP16 | int4 |
+| -------------- | ------ | ------ |
+| Avg. | 0.6022 | 0.6017 |
+| mmlu | 0.5029 | 0.4993 |
+| lambada_openai | 0.6035 | 0.6286 |
+| hellaswag | 0.5620 | 0.5564 |
+| winogrande | 0.6796 | 0.6788 |
+| piqa | 0.7709 | 0.7731 |
+| truthfulqa_mc1 | 0.3048 | 0.3035 |
+| openbookqa | 0.3740 | 0.3700 |
+| boolq | 0.8138 | 0.8144 |
+| rte | 0.7870 | 0.7870 |
+| arc_easy | 0.7525 | 0.7508 |
+| arc_challenge | 0.4727 | 0.4573 |
diff --git a/examples/language-modeling/scripts/gemma-7b-it.sh b/examples/language-modeling/scripts/gemma-7b-it.sh
new file mode 100644
index 000000000..222387add
--- /dev/null
+++ b/examples/language-modeling/scripts/gemma-7b-it.sh
@@ -0,0 +1,15 @@
+python3 main.py \
+--model_name google/gemma-7b-it \
+--device 0 \
+--group_size 128 \
+--bits 4 \
+--iters 1000 \
+--enable_minmax_tuning \
+--use_quant_input \
+--minmax_lr 2e-3 \
+--low_gpu_mem_usage \
+--deployment_device 'gpu' \
+--scale_dtype 'fp32' \
+--eval_bs 32 \
+--output_dir "./tmp_autoround" \
+--amp
\ No newline at end of file