PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
first commit
wenhuach212883387964ca
diff --git a/autoround/README.md b/autoround/README.mdnew file mode 100644index 000000000..2e3c1f650--- /dev/null+++ b/autoround/README.md@@ -0,0 +1,71 @@++# AutoRound: Advanced Weight-Only Quantization Algorithm for a Broad Range of Models++AutoRound is an advanced weight-only quantization algorithm, based on SignRound. It's tailored for a wide range of models and consistently delivers noticeable improvements, often significantly outperforming SignRound. However, it comes at the cost of approximately 2.5 times the tuning runtime.++## Prerequisites+- Python 3.9 or higher+++- The transformers version required varies across different types of models. Here, the transformers version used for running models during experiments is provided as a reference.+ | Model | Transformers version |+ | :----: | :----: |+ | EleutherAI/gpt-j-6b | 4.28/4.30/4.34/4.36 |+ | huggyllama/llama-7b | 4.28/4.30/4.34/4.36 |+ | meta-llama/Llama-2-7b-hf | 4.30/4.34/4.36 |+ | facebook/opt-6.7b | 4.28/4.30/4.34/4.36 |+ | tiiuae/falcon-7b | 4.28/4.30/4.34/4.36 |+ | mosaicml/mpt-7b | 4.28/4.30/4.34/4.36 |+ | bigscience/bloom-7b1 | 4.28/4.30/4.34/4.36 |+ | baichuan-inc/Baichuan-7B | 4.28/4.30 |+ | Qwen/Qwen-7B | 4.28/4.30/4.34/4.36 |+ | THUDM/chatglm3-6b | 4.34/4.36 |+ | mistralai/Mistral-7B-v0.1 | 4.34/4.36 |++ Please note that all experiments in the SignRound+ technical report were conducted using transformers version 4.34.1.++++## Installation+Install the necessary dependencies with the following command:+```bash+pip install -r requirements.txt+```++## Usage+- **Default Settings:**+```bash+CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --amp --num_bits 4 --group_size -1 --enable_minmax_tuning --use_quant_input+```+- **Reduced GPU Memory Usage and Adjusted Training Batch Size:**+```bash+CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --amp --num_bits 4 --group_size -1 --low_gpu_mem_usage --train_bs 1 --gradient_accumulate_steps 8+```+- **Utilizing the AdamW Optimizer:**+Include the flag `--adam`. Note that AdamW may be slightly less effective than Sign gradient descent in certain scenarios.++- **Running the Original SignRound:**+```bash+CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --amp --num_bits 4 --group_size -1 --iters 400 --lr 0.0025 --minmax_lr 0.0025+```+It's recommended to use `--enable_minmax_tuning`.++## Tips+Consider increasing tuning steps and adjusting the learning rate based on a scaling law to achieve better results, albeit with increased tuning time. For instance, at step 800, a learning rate of 0.00125 could be employed.++## Known Issues+Auto Rounding may encounter random issues with Qwen models.++ChatGlm-V1 is not supported++## Reference+If you find SignRound useful for your research, please cite our paper:+```bash+@article{cheng2023optimize,+ title={Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs},+ author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao},+ journal={arXiv preprint arXiv:2309.05516},+ year={2023}+}+```+