PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
upload_auto-round-light results (#454)
WeiweiZhang1eec34cd6bc9e
diff --git a/README.md b/README.mdindex 0f1baf66f..200a4110f 100644--- a/README.md+++ b/README.md@@ -1,5 +1,6 @@<div align="center">+AutoRound===========================<h3> Advanced Quantization Algorithm for LLMs</h3>@@ -29,6 +30,8 @@ and [fbaldassarri](https://huggingface.co/fbaldassarri).<div align="left">++## What's New* [2024/03] The INT2-mixed R1 model (~200GB) retains 97.9% accuracy. Check@@ -36,9 +39,9 @@ and [fbaldassarri](https://huggingface.co/fbaldassarri).* [2024/01] We provide experimental support for GGUF q4_0 and q4_1 formats.* [2024/11] We provide experimental support for VLM quantization, please check outthe [README](./auto_round/mllm/README.md)-## Installation+### Install from pypi```bash@@ -67,6 +70,7 @@ pip install auto-round-lib```</details>+<br>## Model Quantization@@ -87,7 +91,7 @@ auto-round \--output_dir ./tmp_autoround```-We provide two recipes for best accuracy and fast running speed with low memory. Details as below.+We offer two configurations, `auto-round-best` and `auto-round-light`, designed for optimal accuracy and improved speed, respectively. Details are as follows.<details><summary>Other Recipes</summary>@@ -102,15 +106,52 @@ auto-round-best \``````bash+auto-round-light \+## light accuracy, 2-3X speedup, slight accuracy drop at W4 and larger accuracy drop at W2+ --model facebook/opt-125m \+ --bits 4 \+ --group_size 128 \+ --disable_eval+ ```++ <!-- ```bash## fast and low memory, 2-3X speedup, slight accuracy drop at W4G128auto-round-fast \--model facebook/opt-125m \--bits 4 \--group_size 128 \--disable_eval- ```+ ``` --></details>+<br>++#### Auto-Round Recipe Results+In conclusion, we recommend using auto-round for INT4 and auto-round-best for INT2. However, you may adjust the configuration to suit your specific requirements and available resources.++- Average Accuracy of 13 tasks(W4G128) and Time Cost(enable_torch_compile) Results++ | Model | Qwen2.5-0.5B-Instruct | Falcon3-3B | Qwen2.5-7B-Instruct | Falcon3-10B | Qwen2.5-72B-Instruct |+ |---------|-----------------------|----------------------|---------------------|----------------------|-----------------------|+ | 16bits | 0.5541 | 0.6614 | 0.6470 | 0.6151 | 0.7229 |+ | Best | **0.5675**(7m) | **0.6638**(23m) | 0.6426(58m) | **0.6092**(81m) | 0.7242(575m) |+ | Default | 0.5659(2m) | 0.6496(6m) | 0.6441(13m) | 0.6080(18m) | **0.7252**(118m) |+ | Light | 0.5564(2m) | 0.6433(3m) | **0.6453**(5m) | 0.6063(6m) | 0.7243(37m) |++ <details>+ <summary>W2G64 quantization Results</summary>++ - Average Accuracy of 13 tasks(W2G64) and Time Cost(enable_torch_compile) Results++ | Model | Qwen2.5-0.5B-Instruct | Falcon3-3B | Qwen2.5-7B-Instruct | Falcon3-10B | Qwen2.5-72B-Instruct |+ |---------|-----------------------|----------------------|---------------------|----------------------|-----------------------|+ | 16bits | 0.5541 | 0.6614 | 0.6470 | 0.6151 | 0.7229 |+ | Best | **0.3794**(6m) | **0.5272**(24m) | **0.6097**(56m)| **0.6066**(79m) | **0.7201**(564m) |+ | Default | 0.3762(2m) | 0.5232(6m) | 0.6004(13m) | 0.5972(18m) | 0.7143(122m) |+ | Light | 0.3540(2m) | 0.5041(3m) | 0.5532(5m) | 0.5716(7m) | 0.7089(38m) |+ </details>++<br>### API Usage (Gaudi2/CPU/GPU)@@ -189,9 +230,15 @@ autoround.save_quantized(output_dir, format='auto_round', inplace=True)- `device`: The device to be used for tuning. The default is set to 'auto', allowing for automatic detection.</details>+<br>+### API Usage for VLMs++<details>+ <summary>Click to expand</summary>+**This feature is experimental and may be subject to changes**, including potential bug fixes, API modifications, oradjustments to default hype-parameters@@ -220,9 +267,11 @@ autoround.quantize()output_dir = "./tmp_autoround"autoround.save_quantized(output_dir, format='auto_round', inplace=True)```+</details>-#### Export Formats+<br>+### Export Formats**AutoRound Format**: This format is well-suited for CPU, HPU devices, 2 bits, as well as mixed-precisioninference. **[2,4] bits are supported**. However, it has not yet gained widespread community adoption.@@ -232,11 +281,13 @@ asymmetric kernel has issues** that can cause considerable accuracy drops, partimodels.**AutoAWQ Format**: This format is well-suited for asymmetric 4-bit quantization on CUDA devices and is widely-adopted within the community, **only 4-bits quantization is supported**.+adopted within the community, **only 4-bits quantization is supported**.**GGUF** Format: This format is well-suited for CPU devices and is widely adopted by the community, **only q4_0 andq4_1 (W4G32) is supported in our repo**.+<br>+### Quantization CostsTesting was conducted on the Nvidia A100 80G using the nightly version of PyTorch 2.6.0.dev20241029+cu124. Please note@@ -293,8 +344,10 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))```<br>++#### Evaluation<details>- <summary>Evaluation</summary>+ <summary>Click to expand</summary>```bashauto-round --model saved_quantized_model \@@ -304,6 +357,7 @@ auto-round --model saved_quantized_model \```</details>+<br>### AutoGPTQ/AutoAWQ format@@ -323,10 +377,15 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))AutoRound supports basically all the major large language models.+<details>+ <summary>Supported Models List</summary>+Please note that an asterisk (*) indicates third-party quantized models, which may lack accuracy data and use adifferent recipe. We greatly appreciate their efforts and encourage more users to share their models, as we cannotrelease most of the models ourselves.++Model | Supported ||-------------------------------------------|------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|| nvidia/Llama-3.1-Nemotron-70B-Instruct-HF | [model-opea-int4-sym-autoround](https://huggingface.co/OPEA/Llama-3.1-Nemotron-70B-Instruct-HF-int4-sym-inc), [model-opea-int4-sym-autogptq](https://huggingface.co/OPEA/Llama-3.1-Nemotron-70B-Instruct-HF-int4-sym-inc), |@@ -369,7 +428,12 @@ release most of the models ourselves.| 01-ai/Yi-6B-Chat | [outdated-recipe](./docs/Yi-6B-Chat-asym-recipe.md) || facebook/opt-2.7b | [outdated-recipe](./docs/opt-2.7b-asym-recipe.md) || bigscience/bloom-3b | [outdated-recipe](./docs/bloom-3B-asym-recipe.md) |-| EleutherAI/gpt-j-6b | [outdated-recipe](./docs/gpt-j-6B-asym-recipe.md) |+| EleutherAI/gpt-j-6b | [outdated-recipe](./docs/gpt-j-6B-asym-recipe.md) |++</details>+++<br>## Integration@@ -381,6 +445,8 @@ AutoRound has been integrated into multiple repositories.[pytorch/ao](https://github.com/pytorch/ao)+<br>+## ReferenceIf you find AutoRound useful for your research, please cite our paper:@@ -396,3 +462,4 @@ If you find AutoRound useful for your research, please cite our paper:+diff --git a/auto_round/__main__.py b/auto_round/__main__.pyindex 246b66fa1..6b4649545 100644--- a/auto_round/__main__.py+++ b/auto_round/__main__.py@@ -42,6 +42,11 @@ def run_best():from auto_round.script.llm import setup_best_parser, tuneargs = setup_best_parser()tune(args)++def run_light():+ from auto_round.script.llm import setup_light_parser, tune+ args = setup_light_parser()+ tune(args)def run_fast():from auto_round.script.llm import setup_fast_parser, tune@@ -78,3 +83,4 @@ def switch():if __name__ == '__main__':switch()+diff --git a/auto_round/script/llm.py b/auto_round/script/llm.pyindex 85c4ad735..9765fff0b 100644--- a/auto_round/script/llm.py+++ b/auto_round/script/llm.py@@ -254,6 +254,29 @@ def setup_best_parser():return args+def setup_light_parser():+ parser = BasicArgumentParser()++ parser.add_argument("--group_size", default=128, type=int, help="group size")++ parser.add_argument("--batch_size", "--train_bs", "--bs", default=8, type=int, help="train batch size")++ parser.add_argument("--iters", "--iter", default=50, type=int, help="iterations to tune each block")++ parser.add_argument(+ "--seqlen", "--seq_len", default=2048, type=int, help="sequence length of the calibration samples")++ parser.add_argument("--nsamples", "--nsample", default=128, type=int, help="number of samples")++ parser.add_argument(+ "--lr", default=5e-3, type=float, help="learning rate, if None, it will be set to 1.0/iters automatically")++ args = parser.parse_args()+ args.low_gpu_mem_usage = True++ return args++def setup_fast_parser():parser = BasicArgumentParser()@@ -616,8 +639,7 @@ def tune(args):def _eval_init(tasks, model_path, device, disable_trust_remote_code=False):set_cuda_visible_devices(device)device_str, parallelism = get_device_and_parallelism(device)- ##model_args = f'pretrained={model_path},trust_remote_code={not disable_trust_remote_code},add_bos_token=True'- model_args = f'pretrained={model_path},trust_remote_code={not disable_trust_remote_code}'+ model_args = f'pretrained={model_path},trust_remote_code={not disable_trust_remote_code}' #,add_bos_token={True}if parallelism:model_args += ",parallelize=True"if isinstance(tasks, str):@@ -690,3 +712,4 @@ def eval_task_by_task(model, device, tasks, batch_size=None, max_batch_size=64,for key in res_keys:res_all[key].update(res[key])print(make_table(res_all))+diff --git a/test/test_calib_dataset.py b/test/test_calib_dataset.pyindex cfa368bbc..9c2469912 100644--- a/test/test_calib_dataset.py+++ b/test/test_calib_dataset.py@@ -89,13 +89,13 @@ class TestLocalCalibDataset(unittest.TestCase):)autoround.quantize()- def test_pile_val_backup_dataset(self):- dataset = "swift/pile-val-backup"- bits, group_size, sym = 4, 128, True- autoround = AutoRound(- self.model, self.tokenizer, bits=bits, group_size=group_size, sym=sym, iters=2, seqlen=128, dataset=dataset- )- autoround.quantize()+ # def test_pile_val_backup_dataset(self):+ # dataset = "swift/pile-val-backup"+ # bits, group_size, sym = 4, 128, True+ # autoround = AutoRound(+ # self.model, self.tokenizer, bits=bits, group_size=group_size, sym=sym, iters=2, seqlen=128, dataset=dataset+ # )+ # autoround.quantize()@classmethoddef tearDownClass(self):@@ -106,3 +106,4 @@ class TestLocalCalibDataset(unittest.TestCase):if __name__ == "__main__":unittest.main()+