Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

Add baichuan-7b chat recipe (#73)

wenhuach21b68eaf7254de
diff --git a/README.md b/README.md
index 7adab513d..12a69fde7 100644
--- a/README.md
+++ b/README.md
@@ -10,9 +10,13 @@ AutoRound
---
<div align="left">
-AutoRound is an advanced weight-only quantization algorithm for low-bits LLM inference. It's tailored for a wide range of models and consistently delivers noticeable improvements, often significantly outperforming SignRound with the cost of more tuning time for quantization.
+AutoRound is an advanced weight-only quantization algorithm for low-bits LLM inference. It's tailored for a wide range
+of models and consistently delivers noticeable improvements, often significantly outperforming SignRound with the cost
+of more tuning time for quantization.
-Our method adopts sign gradient descent to fine-tune rounding values and minmax values of weights in just 200 steps, which competes impressively against recent methods without introducing any additional inference overhead. The below image presents an overview of AutoRound.
+Our method adopts sign gradient descent to fine-tune rounding values and minmax values of weights in just 200 steps,
+which competes impressively against recent methods without introducing any additional inference overhead. The below
+image presents an overview of AutoRound.
<div align="center">
@@ -21,18 +25,24 @@ Our method adopts sign gradient descent to fine-tune rounding values and minmax
<div align="left">
## Prerequisites
+
- Python 3.9 or higher
## Installation
+
### Build from Source
+
```bash
pip install -r requirements.txt
python setup.py install
```
+
### Install from pypi
+
```bash
pip install auto-round
```
+
## Model quantization
### Gaudi2/ CPU/ GPU
@@ -53,63 +63,67 @@ autoround.quantize()
output_dir = "./tmp_autoround"
autoround.save_quantized(output_dir)
```
+
<details>
<summary>Detailed Hyperparameters</summary>
- `model`: The PyTorch model to be quantized.
-
+
- `tokenizer`: An optional tokenizer for processing input data. If none is provided, a dataloader must be supplied.
-
+
- `bits (int)`: Number of bits for quantization (default is 4).
-
+
- `group_size (int)`: Size of the quantization group (default is 128).
- `sym (bool)`: Whether to use symmetric quantization.
-
-- `use_quant_input (bool)`: Whether to use the output of the previous quantized block as the input for the current block (default is True).
-
+
+- `use_quant_input (bool)`: Whether to use the output of the previous quantized block as the input for the current
+ block (default is True).
+
- `enable_minmax_tuning (bool)`: Whether to enable weight min-max tuning (default is True).
-
+
- `iters (int)`: Number of tuning iterations (default is 200).
-
+
- `lr (float)`: The learning rate for rounding value (default is None, it will be set to 1.0/iters automatically).
-
+
- `minmax_lr (float)`: The learning rate for min-max tuning (default is None, it will be set to lr automatically).
-
+
- `n_samples (int)`: Number of samples for tuning (default is 512).
-
+
- `seqlen (int)`: Data length of the sequence for tuning (default is 2048).
-
+
- `batch_size (int)`: Batch size for training (default is 8).
-- `scale_dtype (str)`: The data type of quantization scale to be used (default is "float32"), different kernels have different choices.
-
+- `scale_dtype (str)`: The data type of quantization scale to be used (default is "float32"), different kernels have
+ different choices.
+
- `amp (bool)`: Whether to use automatic mixed precision (default is True).
-
+
- `n_blocks (int)`: Packing several blocks as one for tuning together (default is 1).
-
+
- `gradient_accumulate_steps (int)`: Number of gradient accumulation steps (default is 1).
-
+
- `low_gpu_mem_usage (bool)`: Whether to save GPU memory at the cost of a little tuning time (default is True).
-
+
- `dataset (str)`: The default dataset name for tuning (default is "NeelNanda/pile-10k").
-
+
- `dataset_split (str)`: The split of the dataset to be used for tuning (default is "train").
-
+
- `dataloader`: The dataloader for tuning data.
-
-- `weight_config (dict)`: Configuration for weight quantization (default is an empty dictionary), mainly for mixed bits or mixed precision.
-
+
+- `weight_config (dict)`: Configuration for weight quantization (default is an empty dictionary), mainly for mixed bits
+ or mixed precision.
+
- `device`: The device to be used for tuning. The default is set to 'auto', allowing for automatic detection.
</details>
## Model inference
-Please run the quantization code first.
-
+Please run the quantization code first.
### CPU
+
```python
##Install the latest https://github.com/intel/intel-extension-for-transformers from source first.
from intel_extension_for_transformers.transformers import AutoModelForCausalLM
@@ -123,8 +137,8 @@ inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
```
-
### GPU
+
```python
from transformers import AutoModelForCausalLM, AutoTokenizer
@@ -136,57 +150,55 @@ inputs = tokenizer(text, return_tensors="pt").to(model.device)
print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))
```
-
-
-
## Support List
-| Model | Supported |
-|--------------------------------------|--------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
-| Intel/neural-chat-7b-v3-3 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc), [accuracy](./docs/neural-chat-7b-v3-3-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-3.sh), [example](./examples/language-modeling/) |
-| Intel/neural-chat-7b-v3-1 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc), [accuracy](./docs/neural-chat-7b-v3-1-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-1.sh), [example](./examples/language-modeling/) |
-| mistralai/Mistral-7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc), [accuracy](./docs/Mistral-7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-v0.1.sh), [example](./examples/language-modeling/) |
-| microsoft/phi-2 | [HF-int4-model](https://huggingface.co/Intel/phi-2-int4-inc), [accuracy](./docs/phi-2-acc.md), [recipe](./examples/language-modeling/scripts/phi-2.sh), [example](./examples/language-modeling/)
-| tiiuae/falcon-7b | [HF-int4-model](https://huggingface.co/Intel/falcon-7b-int4-inc), [accuracy](./docs/falcon-7b-acc.md), [recipe](./examples/language-modeling/scripts/falcon-7b.sh), [example](./examples/language-modeling/) |
-| google/gemma-2b | [HF-int4-model](https://huggingface.co/Intel/gemma-2b-int4-inc), [accuracy](./docs/gemma-2b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-2b.sh), [example](./examples/language-modeling/)
+| Model | Supported |
+|--------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|
+| Intel/neural-chat-7b-v3-3 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc), [accuracy](./docs/neural-chat-7b-v3-3-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-3.sh), [example](./examples/language-modeling/) |
+| Intel/neural-chat-7b-v3-1 | [HF-int4-model](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc), [accuracy](./docs/neural-chat-7b-v3-1-acc.md), [recipe](./examples/language-modeling/scripts/neural-chat-7b-v3-1.sh), [example](./examples/language-modeling/) |
+| mistralai/Mistral-7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc), [accuracy](./docs/Mistral-7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-v0.1.sh), [example](./examples/language-modeling/) |
+| microsoft/phi-2 | [HF-int4-model](https://huggingface.co/Intel/phi-2-int4-inc), [accuracy](./docs/phi-2-acc.md), [recipe](./examples/language-modeling/scripts/phi-2.sh), [example](./examples/language-modeling/)
+| tiiuae/falcon-7b | [HF-int4-model](https://huggingface.co/Intel/falcon-7b-int4-inc), [accuracy](./docs/falcon-7b-acc.md), [recipe](./examples/language-modeling/scripts/falcon-7b.sh), [example](./examples/language-modeling/) |
+| google/gemma-2b | [HF-int4-model](https://huggingface.co/Intel/gemma-2b-int4-inc), [accuracy](./docs/gemma-2b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-2b.sh), [example](./examples/language-modeling/)
| mistralai/Mixtral-7B-Instruct-v0.2 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-Instruct-v0.2-int4-inc) (under review), [accuracy](./docs/Mistral-7B-Instruct-v0.2-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.sh), [example](./examples/language-modeling/) |
-| google/gemma-7b | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-int4-inc) (under review), [accuracy](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh), [example](./examples/language-modeling/) |
-| google/gemma-7b-it | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-it-int4-inc) (under review), [accuracy](./docs/gemma-7b-it-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b-it.sh), [example](./examples/language-modeling/) |
-| mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) |
-| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |
-| meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) |
-| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [sym recipe](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh), [asym recipe ](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh), [example](./examples/language-modeling/) |
-| Salesforce/codegen25-7b-multi | [example](./examples/code-generation) |
-| EleutherAI/gpt-j-6b | [example](./examples/language-modeling/) |
-| huggyllama/llama-7b | [example](./examples/language-modeling/) |
-| meta-llama/Llama-2-7b-hf | [example](./examples/language-modeling/) |
-| facebook/opt-6.7b | [example](./examples/language-modeling/) ||
-| mosaicml/mpt-7b | [example](./examples/language-modeling/) |
-| bigscience/bloom-7b1 | [example](./examples/language-modeling/) |
-| baichuan-inc/Baichuan-7B | [example](./examples/language-modeling/) |
-| Qwen/Qwen-7B | [example](./examples/language-modeling/) |
-| THUDM/chatglm3-6b | [example](./examples/language-modeling/) |
-| MBZUAI/LaMini-GPT-124M | [example](./examples/language-modeling/) |
-| EleutherAI/gpt-neo-125m | [example](./examples/language-modeling/) |
-| databricks/dolly-v2-3b | [example](./examples/language-modeling/) |
-| stabilityai/stablelm-base-alpha-3b | [example](./examples/language-modeling/)
-
-
-
+| google/gemma-7b | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-int4-inc) (under review), [accuracy](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh), [example](./examples/language-modeling/) |
+| google/gemma-7b-it | [HF-int4-model](https://huggingface.co/Intel/gemma-7b-it-int4-inc) (under review), [accuracy](./docs/gemma-7b-it-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b-it.sh), [example](./examples/language-modeling/) |
+| mistralai/Mixtral-8x7B-Instruct-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh), [example](./examples/language-modeling/) |
+| mistralai/Mixtral-8x7B-v0.1 | [HF-int4-model](https://huggingface.co/Intel/Mixtral-8x7B-v0.1-int4-inc) (under review), [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh), [example](./examples/language-modeling/) |
+| meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh), [example](./examples/language-modeling/) |
+| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [sym recipe](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh), [asym recipe ](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh), [example](./examples/language-modeling/) |
+| baichuan-inc/Baichuan2-7B-Chat | [accuracy](./docs/baichuan2-7b-chat-acc.md), [recipe](./examples/language-modeling/scripts/baichuan2-7b-chat.sh), [example](./examples/language-modeling/) |
+| Salesforce/codegen25-7b-multi | [example](./examples/code-generation) |
+| EleutherAI/gpt-j-6b | [example](./examples/language-modeling/) |
+| huggyllama/llama-7b | [example](./examples/language-modeling/) |
+| facebook/opt-6.7b | [example](./examples/language-modeling/) ||
+| mosaicml/mpt-7b | [example](./examples/language-modeling/) |
+| bigscience/bloom-7b1 | [example](./examples/language-modeling/) |
+| THUDM/chatglm3-6b | [example](./examples/language-modeling/) |
+| MBZUAI/LaMini-GPT-124M | [example](./examples/language-modeling/) |
+| EleutherAI/gpt-neo-125m | [example](./examples/language-modeling/) |
+| databricks/dolly-v2-3b | [example](./examples/language-modeling/) |
+| stabilityai/stablelm-base-alpha-3b | [example](./examples/language-modeling/)
## Comparison with other methods
-We provide a [comprehensive analysis](docs/acc.md) with other methods in our accuracy data section. In summary, our approach achieved superior performance compared to GPTQ, scoring 30/32, AWQ with 27/32, HQQ with 15/16, and OmniQuant with a perfect score of 16/16 across llamv1/llamav2/mistral-7b on W4G-1, W4G128, W3G128, and W2G128, based on the average accuracies of 11 zero-shot tasks. And the tuning costs are comparable.
+We provide a [comprehensive analysis](docs/acc.md) with other methods in our accuracy data section. In summary, our
+approach achieved superior performance compared to GPTQ, scoring 30/32, AWQ with 27/32, HQQ with 15/16, and OmniQuant
+with a perfect score of 16/16 across llamv1/llamav2/mistral-7b on W4G-1, W4G128, W3G128, and W2G128, based on the
+average accuracies of 11 zero-shot tasks.
## Tips
-1 Consider increasing tuning steps to achieve better results, albeit with increased tuning time.
+
+1 Consider increasing tuning steps to achieve better results, albeit with increased tuning time.
2 Setting 'use_quant_input' to False has been observed to occasionally yield improved results.
3 Setting 'minmax_lr' to 2.0/iters has been observed to occasionally yield improved results.
## Reference
+
If you find SignRound useful for your research, please cite our paper:
+
```bash
@article{cheng2023optimize,
title={Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs},
diff --git a/docs/Qwen1.5-7B-Chat-acc.md b/docs/Qwen1.5-7B-Chat-acc.md
index a07463a04..9616ca077 100644
--- a/docs/Qwen1.5-7B-Chat-acc.md
+++ b/docs/Qwen1.5-7B-Chat-acc.md
@@ -1,16 +1,11 @@
-Due to licensing restrictions, we are unable to release the model.
-
-Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d.
+Due to licensing restrictions, we are unable to release the model. Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d.
We used the following command for evaluation.
~~~bash
-'lm_eval --model hf \
- --model_args pretrained="gpu_model_path",autogptq=True,gptq_use_triton=True,trust_remote_code=True \
- --device cuda:0 --tasks ceval-valid,cmmlu,mmlu,gsm8k --batch_size 16 --num_fewshot 0'
+lm_eval --model hf --model_args pretrained="./",autogptq=True,gptq_use_triton=True,trust_remote_code=True --device cuda:0 --tasks ceval-valid,cmmlu,mmlu,gsm8k --batch_size 16 --num_fewshot 0
~~~
-
| Metric | BF16 | INT4 sym recipe | INT4 asym recipe |
| -------------- | ------ |-----------------|------------------|
| Avg. | 0.6231 | 0.6205 | 0.6186 |
diff --git a/docs/baichuan2-7b-chat-acc.md b/docs/baichuan2-7b-chat-acc.md
new file mode 100644
index 000000000..6acbd9448
--- /dev/null
+++ b/docs/baichuan2-7b-chat-acc.md
@@ -0,0 +1,15 @@
+Due to licensing restrictions, we are unable to release the model. Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d.
+
+We used the following command for evaluation.
+
+~~~bash
+lm_eval --model hf --model_args pretrained="./",autogptq=True,gptq_use_triton=True,trust_remote_code=True --device cuda:0 --tasks ceval-valid,cmmlu,mmlu,gsm8k --batch_size 16 --num_fewshot 0
+~~~
+
+| Metric | BF16 | INT4 |
+|--------|--------|--------|
+| Avg. | 0.4504 | 0.4470 |
+| mmlu | 0.5096 | 0.5053 |
+| cmmlu | 0.5486 | 0.5426 |
+| ceval | 0.5394 | 0.5223 |
+| gsm8k | 0.2039 | 0.2176 |
diff --git a/examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh b/examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh
index 1823f3d28..7b90de63d 100644
--- a/examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh
+++ b/examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh
@@ -1,5 +1,6 @@
+## Due to licensing restrictions, we are unable to release the model.
python3 main.py \
---model_name Qwen/Qwen1.5-7B-Chat\
+--model_name Qwen/Qwen1.5-7B-Chat \
--group_size 128 \
--bits 4 \
--iters 1000 \
diff --git a/examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh b/examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh
index ce56a8382..cbe434bf2 100644
--- a/examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh
+++ b/examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh
@@ -1,7 +1,6 @@
-## Due to licensing restrictions, we are unable to release the model.
-# Recipe
+## Due to licensing restrictions, we are unable to release the model.
python3 main.py \
---model_name Qwen/Qwen1.5-7B-Chat\
+--model_name Qwen/Qwen1.5-7B-Chat \
--bits 4 \
--group_size 128 \
--use_quant_input \
diff --git a/examples/language-modeling/scripts/baichuan2-7b-chat.sh b/examples/language-modeling/scripts/baichuan2-7b-chat.sh
new file mode 100644
index 000000000..203fe2b67
--- /dev/null
+++ b/examples/language-modeling/scripts/baichuan2-7b-chat.sh
@@ -0,0 +1,9 @@
+## Due to licensing restrictions, we are unable to release the model.
+python3 main.py \
+--model_name baichuan-inc/Baichuan2-7B-Chat \
+--group_size 128 \
+--bits 4 \
+--iters 1000 \
+--deployment_device 'gpu' \
+--minmax_lr 2e-3 \
+--use_quant_input