PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
update readme
wenhuach21ffbeae1f4b2e
diff --git a/README.md b/README.mdindex 87d4852dc..7318094d9 100644--- a/README.md+++ b/README.md@@ -6,26 +6,6 @@ AutoRound is an advanced weight-only quantization algorithm, based on SignRound.## Prerequisites- Python 3.9 or higher--- The transformers version required varies across different types of models. Here, the transformers version used for running models during experiments is provided as a reference.- | Model | Transformers version |- | :----: | :----: |- | EleutherAI/gpt-j-6b | 4.28/4.30/4.34/4.36 |- | huggyllama/llama-7b | 4.28/4.30/4.34/4.36 |- | meta-llama/Llama-2-7b-hf | 4.30/4.34/4.36 |- | facebook/opt-6.7b | 4.28/4.30/4.34/4.36 |- | tiiuae/falcon-7b | 4.28/4.30/4.34/4.36 |- | mosaicml/mpt-7b | 4.28/4.30/4.34/4.36 |- | bigscience/bloom-7b1 | 4.28/4.30/4.34/4.36 |- | baichuan-inc/Baichuan-7B | 4.28/4.30 |- | Qwen/Qwen-7B | 4.28/4.30/4.34/4.36 |- | THUDM/chatglm3-6b | 4.34/4.36 |- | mistralai/Mistral-7B-v0.1 | 4.34/4.36 |--Please note that all experiments in the SignRound+ technical report were conducted using transformers version 4.34.1.---## InstallationInstall the necessary dependencies with the following command:```bash@@ -76,32 +56,6 @@ fake_qdq_model,weight_config = autoround.quantize() ##scale,zp info are saved in- `device`: The device to be used for tuning (default is "cuda:0").--### Examples-cd to examples folder, install lm-eval to run the evaluation-```bash-pip install -r requirements.txt-```--- **Default Settings:**-```bash-CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --amp --bits 4 --group_size -1 --enable_minmax_tuning --use_quant_input-```-- **Reduced GPU Memory Usage and Adjusted Training Batch Size:**-```bash-CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --amp --bits 4 --group_size -1 --low_gpu_mem_usage --train_bs 1 --gradient_accumulate_steps 8-```-- **Utilizing the AdamW Optimizer:**-Include the flag `--adam`. Note that AdamW is less effective than Sign gradient descent in many scenarios we tested.--- **Running the Original SignRound:**-```bash-CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --amp --bits 4 --group_size -1 --iters 400 --lr 0.0025 --minmax_lr 0.0025-```- `--enable_minmax_tuning` is strongly recommended---## TipsConsider increasing tuning steps and adjusting the learning rate based on a scaling law to achieve better results, albeit with increased tuning time. For instance, at step 800, a learning rate of 0.00125 could be employed.@@ -113,12 +67,9 @@ Random issues in tuning Qwen models. ChatGlm-V1 is not supported--## Validated Models-| W4G128 | MMLU | Lamb. | Hella. | Wino. | Piqa | Truth. | Open. | Boolq | RTE | ARC-e | ARC-c. | AVG. ||-------------------------------------------------|-------|-------|--------|-------|-------|--------|-------|-------|-------|-------|--------|-------|| mistralai/Mixtral-8x7B-v0.1 BF16 | 69.83 | 78.44 | 64.89 | 76.40 | 82.43 | 34.15 | 35.40 | 84.98 | 71.12 | 84.22 | 56.91 | 67.16 |@@ -137,38 +88,49 @@ For a fair comparison, we utilized 512 samples from Pile-10k for all methods dur-Mistral-7b done--LLaMAV1 done--LLaMAv2 done--PI2 done--mixstral-7Bx8 done--LaMini-GPT-124M done--QWEN1-8B done,but has random issue--OPT-125M done--Bloom-560 smoke test done--falcon-7b smoke test done--gpt-leo-125m smoke test done--stablelm-base-alpha-3b smoke test done+### Modes passed smoke test+LaMini-GPT-124M; QWEN1-8B; OPT-125M; Bloom-560m;falcon-7b;gpt-leo-125m;stablelm-base-alpha-3b;dolly-v2-3b;mpt-7b;gpt-j-6b;chatglm2-6b-dolly-v2-3b smoke test done-mpt-7b smoke test done+### Examples+cd to examples folder, install lm-eval to run the evaluation+```bash+pip install -r requirements.txt+```-gpt-j-6b smoke test done+- **Default Settings:**+```bash+CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --amp --bits 4 --group_size -1 --enable_minmax_tuning --use_quant_input+```+- **Reduced GPU Memory Usage and Adjusted Training Batch Size:**+```bash+CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --amp --bits 4 --group_size -1 --low_gpu_mem_usage --train_bs 1 --gradient_accumulate_steps 8+```+- **Utilizing the AdamW Optimizer:**+Include the flag `--adam`. Note that AdamW is less effective than Sign gradient descent in many scenarios we tested.-chatglm2-6b smoke test done+- **Running the Original SignRound:**+```bash+CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --amp --bits 4 --group_size -1 --iters 400 --lr 0.0025 --minmax_lr 0.0025+```+ `--enable_minmax_tuning` is strongly recommended+- The transformers version required varies across different types of models. Here, the transformers version used for running models during experiments is provided as a reference.+-+ | Model | Transformers version |+ | :----: | :----: |+ | EleutherAI/gpt-j-6b | 4.28/4.30/4.34/4.36 |+ | huggyllama/llama-7b | 4.28/4.30/4.34/4.36 |+ | meta-llama/Llama-2-7b-hf | 4.30/4.34/4.36 |+ | facebook/opt-6.7b | 4.28/4.30/4.34/4.36 |+ | tiiuae/falcon-7b | 4.28/4.30/4.34/4.36 |+ | mosaicml/mpt-7b | 4.28/4.30/4.34/4.36 |+ | bigscience/bloom-7b1 | 4.28/4.30/4.34/4.36 |+ | baichuan-inc/Baichuan-7B | 4.28/4.30 |+ | Qwen/Qwen-7B | 4.28/4.30/4.34/4.36 |+ | THUDM/chatglm3-6b | 4.34/4.36 |+ | mistralai/Mistral-7B-v0.1 | 4.34/4.36 |+## Reference