Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

update readme

wenhuach21ffbeae1f4b2e
diff --git a/README.md b/README.md
index 87d4852dc..7318094d9 100644
--- a/README.md
+++ b/README.md
@@ -6,26 +6,6 @@ AutoRound is an advanced weight-only quantization algorithm, based on SignRound.
## Prerequisites
- Python 3.9 or higher
-
-- The transformers version required varies across different types of models. Here, the transformers version used for running models during experiments is provided as a reference.
- | Model | Transformers version |
- | :----: | :----: |
- | EleutherAI/gpt-j-6b | 4.28/4.30/4.34/4.36 |
- | huggyllama/llama-7b | 4.28/4.30/4.34/4.36 |
- | meta-llama/Llama-2-7b-hf | 4.30/4.34/4.36 |
- | facebook/opt-6.7b | 4.28/4.30/4.34/4.36 |
- | tiiuae/falcon-7b | 4.28/4.30/4.34/4.36 |
- | mosaicml/mpt-7b | 4.28/4.30/4.34/4.36 |
- | bigscience/bloom-7b1 | 4.28/4.30/4.34/4.36 |
- | baichuan-inc/Baichuan-7B | 4.28/4.30 |
- | Qwen/Qwen-7B | 4.28/4.30/4.34/4.36 |
- | THUDM/chatglm3-6b | 4.34/4.36 |
- | mistralai/Mistral-7B-v0.1 | 4.34/4.36 |
-
-Please note that all experiments in the SignRound+ technical report were conducted using transformers version 4.34.1.
-
-
-
## Installation
Install the necessary dependencies with the following command:
```bash
@@ -76,32 +56,6 @@ fake_qdq_model,weight_config = autoround.quantize() ##scale,zp info are saved in
- `device`: The device to be used for tuning (default is "cuda:0").
-
-### Examples
-cd to examples folder, install lm-eval to run the evaluation
-```bash
-pip install -r requirements.txt
-```
-
-- **Default Settings:**
-```bash
-CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --amp --bits 4 --group_size -1 --enable_minmax_tuning --use_quant_input
-```
-- **Reduced GPU Memory Usage and Adjusted Training Batch Size:**
-```bash
-CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --amp --bits 4 --group_size -1 --low_gpu_mem_usage --train_bs 1 --gradient_accumulate_steps 8
-```
-- **Utilizing the AdamW Optimizer:**
-Include the flag `--adam`. Note that AdamW is less effective than Sign gradient descent in many scenarios we tested.
-
-- **Running the Original SignRound:**
-```bash
-CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --amp --bits 4 --group_size -1 --iters 400 --lr 0.0025 --minmax_lr 0.0025
-```
- `--enable_minmax_tuning` is strongly recommended
-
-
-
## Tips
Consider increasing tuning steps and adjusting the learning rate based on a scaling law to achieve better results, albeit with increased tuning time. For instance, at step 800, a learning rate of 0.00125 could be employed.
@@ -113,12 +67,9 @@ Random issues in tuning Qwen models. ChatGlm-V1 is not supported
-
-
## Validated Models
-
| W4G128 | MMLU | Lamb. | Hella. | Wino. | Piqa | Truth. | Open. | Boolq | RTE | ARC-e | ARC-c. | AVG. |
|-------------------------------------------------|-------|-------|--------|-------|-------|--------|-------|-------|-------|-------|--------|-------|
| mistralai/Mixtral-8x7B-v0.1 BF16 | 69.83 | 78.44 | 64.89 | 76.40 | 82.43 | 34.15 | 35.40 | 84.98 | 71.12 | 84.22 | 56.91 | 67.16 |
@@ -137,38 +88,49 @@ For a fair comparison, we utilized 512 samples from Pile-10k for all methods dur
![](./figs/W3G128.png)
![](./figs/W2G128.png)
-Mistral-7b done
-
-LLaMAV1 done
-
-LLaMAv2 done
-
-PI2 done
-
-mixstral-7Bx8 done
-
-LaMini-GPT-124M done
-
-QWEN1-8B done,but has random issue
-
-OPT-125M done
-
-Bloom-560 smoke test done
-
-falcon-7b smoke test done
-
-gpt-leo-125m smoke test done
-
-stablelm-base-alpha-3b smoke test done
+### Modes passed smoke test
+LaMini-GPT-124M; QWEN1-8B; OPT-125M; Bloom-560m;falcon-7b;gpt-leo-125m;stablelm-base-alpha-3b;dolly-v2-3b;mpt-7b;gpt-j-6b;chatglm2-6b
-dolly-v2-3b smoke test done
-mpt-7b smoke test done
+### Examples
+cd to examples folder, install lm-eval to run the evaluation
+```bash
+pip install -r requirements.txt
+```
-gpt-j-6b smoke test done
+- **Default Settings:**
+```bash
+CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --amp --bits 4 --group_size -1 --enable_minmax_tuning --use_quant_input
+```
+- **Reduced GPU Memory Usage and Adjusted Training Batch Size:**
+```bash
+CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --amp --bits 4 --group_size -1 --low_gpu_mem_usage --train_bs 1 --gradient_accumulate_steps 8
+```
+- **Utilizing the AdamW Optimizer:**
+Include the flag `--adam`. Note that AdamW is less effective than Sign gradient descent in many scenarios we tested.
-chatglm2-6b smoke test done
+- **Running the Original SignRound:**
+```bash
+CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --amp --bits 4 --group_size -1 --iters 400 --lr 0.0025 --minmax_lr 0.0025
+```
+ `--enable_minmax_tuning` is strongly recommended
+- The transformers version required varies across different types of models. Here, the transformers version used for running models during experiments is provided as a reference.
+-
+ | Model | Transformers version |
+ | :----: | :----: |
+ | EleutherAI/gpt-j-6b | 4.28/4.30/4.34/4.36 |
+ | huggyllama/llama-7b | 4.28/4.30/4.34/4.36 |
+ | meta-llama/Llama-2-7b-hf | 4.30/4.34/4.36 |
+ | facebook/opt-6.7b | 4.28/4.30/4.34/4.36 |
+ | tiiuae/falcon-7b | 4.28/4.30/4.34/4.36 |
+ | mosaicml/mpt-7b | 4.28/4.30/4.34/4.36 |
+ | bigscience/bloom-7b1 | 4.28/4.30/4.34/4.36 |
+ | baichuan-inc/Baichuan-7B | 4.28/4.30 |
+ | Qwen/Qwen-7B | 4.28/4.30/4.34/4.36 |
+ | THUDM/chatglm3-6b | 4.34/4.36 |
+ | mistralai/Mistral-7B-v0.1 | 4.34/4.36 |
+
## Reference