PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
delete llm example and refine readme (#354)
Wenhua Cheng715cd8a21891
diff --git a/README.md b/README.mdindex 46f0cbfbe..55929419b 100644--- a/README.md+++ b/README.md@@ -26,13 +26,17 @@ more accuracy data and recipes across various models.<div align="left">## What's New-* [2024/11] We provide experimental support for VLLM quantization, please check out the [README](./auto_round/mllm/README.md)-* [2024/11] We provide some tips and tricks for LLM&VLM quantization, please check out [this blog](https://medium.com/@NeuralCompressor/10-tips-for-quantizing-llms-and-vlms-with-autoround-923e733879a7)++* [2024/11] We provide experimental support for VLLM quantization, please check out+ the [README](./auto_round/mllm/README.md)+* [2024/11] We provide some tips and tricks for LLM&VLM quantization, please check+ out [this blog](https://medium.com/@NeuralCompressor/10-tips-for-quantizing-llms-and-vlms-with-autoround-923e733879a7)* [2024/10] AutoRound has been integrated to [torch/ao](https://github.com/pytorch/ao), check outtheir [release note](https://github.com/pytorch/ao/releases/tag/v0.6.1)* [2024/10] Important update: We now support full-range symmetric quantization and have made it the defaultconfiguration. This configuration is typically better or comparable to asymmetric quantization and significantly- outperforms other symmetric variants, especially at low bit-widths like 2-bit, check out [some accuracy data](./docs/full_range_sym.md).+ outperforms other symmetric variants, especially at low bit-widths like 2-bit, check+ out [some accuracy data](./docs/full_range_sym.md).* [2024/08] AutoRound format supports Intel Gaudi2 devices. Please referto [Intel/Qwen2-7B-int4-inc](https://huggingface.co/Intel/Qwen2-7B-int4-inc).* [2024/08] AutoRound introduces several experimental features, including fast tuning of norm/bias parameters (for 2-bit@@ -40,7 +44,6 @@ more accuracy data and recipes across various models.## Installation-### Install from pypi```bash@@ -54,7 +57,6 @@ pip install auto-round[cpu]pip install auto-round[hpu]```-<details><summary>Build from Source</summary>@@ -70,15 +72,17 @@ pip install auto-round[hpu]# HPUpip install -vvv --no-build-isolation -e .[hpu]```+</details>## Model Quantization### Basic Usage (Gaudi2/CPU/GPU)- A user guide detailing the full list of supported arguments is provided by calling ```auto-round -h``` on the terminal.- Set the format you want in `format` and-multiple formats exporting has been supported. Please check out [step-by-step-instruction](./docs/step_by_step.md) for more details about calibration dataset or evaluation.+A user guide detailing the full list of supported arguments is provided by calling ```auto-round -h``` on the terminal.+Set the format you want in `format` and+multiple formats exporting has been supported. Please check out [step-by-step-instruction](./docs/step_by_step.md) for+more details about calibration dataset or evaluation.```bashauto-round \@@ -219,6 +223,37 @@ autoround.save_quantized(output_dir, format='auto_round', inplace=True)</details>+### API Usage for VLMs++**This feature is experimental and may be subject to changes**, including potential bug fixes, API modifications, or+adjustments to default hype-parameters++By default, AutoRoundMLLM only quantizes the text module of VLMs and uses `NeelNanda/pile-10k` for calibration. To+quantize the entire model, you can enable `quant_nontext_module` by setting it to True, though support for this feature+is limited. For more information, please refer to the AutoRoundMLLM [readme](./auto_round/mllm/README.md).++```python+from auto_round import AutoRoundMLLM+from transformers import Qwen2VLForConditionalGeneration, AutoProcessor, AutoTokenizer++## load the model+model_name = "Qwen/Qwen2-VL-2B-Instruct"+model = Qwen2VLForConditionalGeneration.from_pretrained(+ model_name, trust_remote_code=True)+tokenizer = AutoTokenizer.from_pretrained(model_name)+processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)++## quantize the model+bits, group_size, sym = 4, 128, True+autoround = AutoRoundMLLM(model, tokenizer, processor,+ bits=bits, group_size=group_size, sym=sym)+autoround.quantize()++# save the quantized model, set format='auto_gptq' to use AutoGPTQ format+output_dir = "./tmp_autoround"+autoround.save_quantized(output_dir, format='auto_round', inplace=True)+```+### Quantization CostsTesting was conducted on the Nvidia A100 80G using the nightly version of PyTorch 2.6.0.dev20241029+cu124. Please note@@ -247,7 +282,7 @@ Please run the quantization code first### AutoRound format-**CPU**: **auto_round version >0.3.1**, pip install intel-extension-for-pytorch(much higher speed on Intel CPU) or pip+**CPU**: pip install intel-extension-for-pytorch(much higher speed on Intel CPU) or pipinstall intel-extension-for-transformers,**HPU**: docker image with Gaudi Software Stack is recommended. More details can be found@@ -310,49 +345,47 @@ Please note that an asterisk (*) indicates third-party quantized models, which mdifferent recipe. We greatly appreciate their efforts and encourage more users to share their models, as we cannotrelease most of the models ourselves.- Model | Supported |-|----------------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|-| THUDM/cogvlm2-llama3-chinese-chat-19B | [recipe](./docs/cogvlm2-llama3-chat-19B-sym.md) |-| Qwen/Qwen2-VL-Instruct | [recipe](./docs/Qwen2-VL-7B-Instruct-sym.md) |+ Model | Supported |+|--------------------------------|---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------|+| THUDM/cogvlm2-llama3-chat-19B | [recipe](./docs/cogvlm2-llama3-chat-19B-sym.md) |+| Qwen/Qwen2-VL-Instruct | [recipe](./docs/Qwen2-VL-7B-Instruct-sym.md) || meta-llama/Llama-3.2-11B-Vision | [recipe](./docs/Llama-3.2-11B-Vision-Instruct-sym.md) || microsoft/Phi-3.5-vision-instruct | [recipe](./docs/Phi-3.5-vision-instruct-sym.md) |-| liuhaotian/llava-v1.5-7b | [recipe](./docs/llava-v1.5-7b-sym.md) |-| Qwen/Qwen2.5-7B-Instruct | [model-kaitchup-autogptq-int4*](https://beta-index.hf-mirror.com/kaitchup/Qwen2.5-7B-Instruct-AutoRound-GPTQ-asym-4bit), [recipe](./docs/Qwen2.5-7B-Instruct-sym.md) |-| Qwen/Qwen2.5-14B-Instruct | [recipe](./docs/Qwen2.5-14B-Instruct-sym.md) |-| Qwen/Qwen2.5-32B-Instruct | [recipe](./docs/Qwen2.5-32B-Instruct-sym.md) |+| liuhaotian/llava-v1.5-7b | [recipe](./docs/llava-v1.5-7b-sym.md) |+| Qwen/Qwen2.5-7B-Instruct | [model-kaitchup-autogptq-int4*](https://beta-index.hf-mirror.com/kaitchup/Qwen2.5-7B-Instruct-AutoRound-GPTQ-asym-4bit), [recipe](./docs/Qwen2.5-7B-Instruct-sym.md) |+| Qwen/Qwen2.5-14B-Instruct | [recipe](./docs/Qwen2.5-14B-Instruct-sym.md) |+| Qwen/Qwen2.5-32B-Instruct | [recipe](./docs/Qwen2.5-32B-Instruct-sym.md) || Qwen/Qwen2.5-Coder-32B-Instruct | [model-kaitchup-autogptq-int4*](https://beta-index.hf-mirror.com/kaitchup/Qwen2.5-Coder-32B-Instruct-AutoRound-GPTQ-4bit) |-| Qwen/Qwen2.5-72B-Instruct | [model-kaitchup-autogptq-int4*](https://beta-index.hf-mirror.com/kaitchup/Qwen2.5-72B-Instruct-AutoRound-GPTQ-4bit), [model-kaitchup-autogptq-int2*](https://beta-index.hf-mirror.com/kaitchup/Qwen2.5-72B-Instruct-AutoRound-GPTQ-2bit), [recipe](./docs/Qwen2.5-72B-Instruct-sym.md) |+| Qwen/Qwen2.5-72B-Instruct | [model-kaitchup-autogptq-int4*](https://beta-index.hf-mirror.com/kaitchup/Qwen2.5-72B-Instruct-AutoRound-GPTQ-4bit), [model-kaitchup-autogptq-int2*](https://beta-index.hf-mirror.com/kaitchup/Qwen2.5-72B-Instruct-AutoRound-GPTQ-2bit), [recipe](./docs/Qwen2.5-72B-Instruct-sym.md) || meta-llama/Meta-Llama-3.1-70B-Instruct | [recipe](https://huggingface.co/Intel/Meta-Llama-3.1-70B-Instruct-int4-inc) |-| meta-llama/Meta-Llama-3.1-8B-Instruct | [model-kaitchup-autogptq-int4*](https://huggingface.co/kaitchup/Meta-Llama-3.1-8B-Instruct-autoround-gptq-4bit-asym), [model-kaitchup-autogptq-sym-int4*](https://huggingface.co/kaitchup/Meta-Llama-3.1-8B-Instruct-autoround-gptq-4bit-sym), [recipe](https://huggingface.co/Intel/Meta-Llama-3.1-8B-Instruct-int4-inc) |-| meta-llama/Meta-Llama-3.1-8B | [model-kaitchup-autogptq-sym-int4*](https://huggingface.co/kaitchup/Meta-Llama-3.1-8B-autoround-gptq-4bit-sym) |-| Qwen/Qwen-VL | [accuracy](./examples/multimodal-modeling/Qwen-VL/README.md), [recipe](./examples/multimodal-modeling/Qwen-VL/run_autoround.sh)-| Qwen/Qwen2-7B | [model-autoround-sym-int4](https://huggingface.co/Intel/Qwen2-7B-int4-inc), [model-autogptq-sym-int4](https://huggingface.co/Intel/Qwen2-7B-int4-inc) |-| THUDM/glm-4-9b-chat | [recipe](./docs/glm-4-9b-chat-recipe.md) |-| Qwen/Qwen2-57B-A14B-Instruct | [model-autoround-sym-int4](https://huggingface.co/Intel/Qwen2-57B-A14B-Instruct-int4-inc),[model-autogptq-sym-int4](https://huggingface.co/Intel/Qwen2-57B-A14B-Instruct-int4-inc) |-| 01-ai/Yi-1.5-9B | [model-LnL-AI-autogptq-int4*](https://huggingface.co/LnL-AI/Yi-1.5-9B-4bit-gptq-autoround) |-| 01-ai/Yi-1.5-9B-Chat | [model-LnL-AI-autogptq-int4*](https://huggingface.co/LnL-AI/Yi-1.5-9B-Chat-4bit-gptq-autoround) |-| Intel/neural-chat-7b-v3-3 | [model-autogptq-int4](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc) |-| Intel/neural-chat-7b-v3-1 | [model-autogptq-int4](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc) |-| TinyLlama-1.1B-intermediate | [model-LnL-AI-autogptq-int4*](https://huggingface.co/LnL-AI/TinyLlama-1.1B-intermediate-step-1341k-3T-autoround-lm_head-symFalse) |-| mistralai/Mistral-7B-v0.1 | [model-autogptq-lmhead-int4](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc-lmhead), [model-autogptq-int4](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) |-| google/gemma-2b | [model-autogptq-int4](https://huggingface.co/Intel/gemma-2b-int4-inc) |-| tiiuae/falcon-7b | [model-autogptq-int4-G64](https://huggingface.co/Intel/falcon-7b-int4-inc) |-| sapienzanlp/modello-italia-9b | [model-fbaldassarri-autogptq-int4*](https://huggingface.co/fbaldassarri/modello-italia-9b-autoround-w4g128-cpu) |-| microsoft/phi-2 | [model-autoround-sym-int4](https://huggingface.co/Intel/phi-2-int4-inc) [model-autogptq-sym-int4](https://huggingface.co/Intel/phi-2-int4-inc) |-| microsoft/Phi-3.5-mini-instruct | [model-kaitchup-autogptq-sym-int4*](https://huggingface.co/kaitchup/Phi-3.5-Mini-instruct-AutoRound-4bit) |-| microsoft/Phi-3-vision-128k-instruct | [recipe](./examples/multimodal-modeling/Phi-3-vision/run_autoround.sh)-| mistralai/Mistral-7B-Instruct-v0.2 | [accuracy](./docs/Mistral-7B-Instruct-v0.2-acc.md), [recipe](./examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.sh) |-| mistralai/Mixtral-8x7B-Instruct-v0.1 | [accuracy](./docs/Mixtral-8x7B-Instruct-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh) |-| mistralai/Mixtral-8x7B-v0.1 | [accuracy](./docs/Mixtral-8x7B-v0.1-acc.md), [recipe](./examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh) |-| meta-llama/Meta-Llama-3-8B-Instruct | [accuracy](./docs/Meta-Llama-3-8B-Instruct-acc.md), [recipe](./examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.sh) |-| google/gemma-7b | [accuracy](./docs/gemma-7b-acc.md), [recipe](./examples/language-modeling/scripts/gemma-7b.sh) |-| meta-llama/Llama-2-7b-chat-hf | [accuracy](./docs/Llama-2-7b-chat-hf-acc.md), [recipe](./examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh) |-| Qwen/Qwen1.5-7B-Chat | [accuracy](./docs/Qwen1.5-7B-Chat-acc.md), [sym recipe](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh), [asym recipe ](./examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh) |-| baichuan-inc/Baichuan2-7B-Chat | [accuracy](./docs/baichuan2-7b-chat-acc.md), [recipe](./examples/language-modeling/scripts/baichuan2-7b-chat.sh) |-| 01-ai/Yi-6B-Chat | [accuracy](./docs/Yi-6B-Chat-acc.md), [recipe](./examples/language-modeling/scripts/Yi-6B-Chat.sh) |-| facebook/opt-2.7b | [accuracy](./docs/opt-2.7b-acc.md), [recipe](./examples/language-modeling/scripts/opt-2.7b.sh) |-| bigscience/bloom-3b | [accuracy](./docs/bloom-3B-acc.md), [recipe](./examples/language-modeling/scripts/bloom-3b.sh) |-| EleutherAI/gpt-j-6b | [accuracy](./docs/gpt-j-6B-acc.md), [recipe](./examples/language-modeling/scripts/gpt-j-6b.sh) |+| meta-llama/Meta-Llama-3.1-8B-Instruct | [model-kaitchup-autogptq-int4*](https://huggingface.co/kaitchup/Meta-Llama-3.1-8B-Instruct-autoround-gptq-4bit-asym), [model-kaitchup-autogptq-sym-int4*](https://huggingface.co/kaitchup/Meta-Llama-3.1-8B-Instruct-autoround-gptq-4bit-sym), [recipe](https://huggingface.co/Intel/Meta-Llama-3.1-8B-Instruct-int4-inc) |+| meta-llama/Meta-Llama-3.1-8B | [model-kaitchup-autogptq-sym-int4*](https://huggingface.co/kaitchup/Meta-Llama-3.1-8B-autoround-gptq-4bit-sym) |+| Qwen/Qwen-VL | [accuracy](./examples/multimodal-modeling/Qwen-VL/README.md), [recipe](./examples/multimodal-modeling/Qwen-VL/run_autoround.sh)+| Qwen/Qwen2-7B | [model-autoround-sym-int4](https://huggingface.co/Intel/Qwen2-7B-int4-inc), [model-autogptq-sym-int4](https://huggingface.co/Intel/Qwen2-7B-int4-inc) |+| THUDM/glm-4-9b-chat | [recipe](./docs/glm-4-9b-chat-recipe.md) |+| Qwen/Qwen2-57B-A14B-Instruct | [model-autoround-sym-int4](https://huggingface.co/Intel/Qwen2-57B-A14B-Instruct-int4-inc),[model-autogptq-sym-int4](https://huggingface.co/Intel/Qwen2-57B-A14B-Instruct-int4-inc) |+| 01-ai/Yi-1.5-9B | [model-LnL-AI-autogptq-int4*](https://huggingface.co/LnL-AI/Yi-1.5-9B-4bit-gptq-autoround) |+| 01-ai/Yi-1.5-9B-Chat | [model-LnL-AI-autogptq-int4*](https://huggingface.co/LnL-AI/Yi-1.5-9B-Chat-4bit-gptq-autoround) |+| Intel/neural-chat-7b-v3-3 | [model-autogptq-int4](https://huggingface.co/Intel/neural-chat-7b-v3-3-int4-inc) |+| Intel/neural-chat-7b-v3-1 | [model-autogptq-int4](https://huggingface.co/Intel/neural-chat-7b-v3-1-int4-inc) |+| TinyLlama-1.1B-intermediate | [model-LnL-AI-autogptq-int4*](https://huggingface.co/LnL-AI/TinyLlama-1.1B-intermediate-step-1341k-3T-autoround-lm_head-symFalse) |+| mistralai/Mistral-7B-v0.1 | [model-autogptq-lmhead-int4](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc-lmhead), [model-autogptq-int4](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) |+| google/gemma-2b | [model-autogptq-int4](https://huggingface.co/Intel/gemma-2b-int4-inc) |+| tiiuae/falcon-7b | [model-autogptq-int4-G64](https://huggingface.co/Intel/falcon-7b-int4-inc) |+| sapienzanlp/modello-italia-9b | [model-fbaldassarri-autogptq-int4*](https://huggingface.co/fbaldassarri/modello-italia-9b-autoround-w4g128-cpu) |+| microsoft/phi-2 | [model-autoround-sym-int4](https://huggingface.co/Intel/phi-2-int4-inc) [model-autogptq-sym-int4](https://huggingface.co/Intel/phi-2-int4-inc) |+| microsoft/Phi-3.5-mini-instruct | [model-kaitchup-autogptq-sym-int4*](https://huggingface.co/kaitchup/Phi-3.5-Mini-instruct-AutoRound-4bit) |+| mistralai/Mistral-7B-Instruct-v0.2 | [outdated-recipe](./docs/Mistral-7B-Instruct-v0.2-asym-recipe.md) |+| mistralai/Mixtral-8x7B-Instruct-v0.1 | [outdated-recipe](./docs/Mixtral-8x7B-Instruct-v0.1-asym-recipe.md) |+| mistralai/Mixtral-8x7B-v0.1 | [outdated-recipe](./docs/Mixtral-8x7B-v0.1-asym-acc.md) |+| meta-llama/Meta-Llama-3-8B-Instruct | [outdated-recipe](./docs/Meta-Llama-3-8B-Instruct-asym-recipe.md) |+| google/gemma-7b | [outdated-recipe](./docs/gemma-7b-asym-recipe.md) |+| meta-llama/Llama-2-7b-chat-hf | [outdated-recipe](./docs/Llama-2-7b-chat-hf-asym-recipe.md) |+| baichuan-inc/Baichuan2-7B-Chat | [outdated-recipe](./docs/baichuan2-7b-cha-asym-recipe.md) |+| 01-ai/Yi-6B-Chat | [outdated-recipe](./docs/Yi-6B-Chat-asym-recipe.md) |+| facebook/opt-2.7b | [outdated-recipe](./docs/opt-2.7b-asym-recipe.md) |+| bigscience/bloom-3b | [outdated-recipe](./docs/bloom-3B-asym-recipe.md) |+| EleutherAI/gpt-j-6b | [outdated-recipe](./docs/gpt-j-6B-asym-recipe.md) |## Integrationdiff --git a/auto_round/mllm/README.md b/auto_round/mllm/README.mdindex dc7ff69b9..fb206c532 100644--- a/auto_round/mllm/README.md+++ b/auto_round/mllm/README.md@@ -1,63 +1,69 @@# AutoRound for MLLMs+This feature is experimental and may be subject to changes, including potential bug fixes, API modifications, or+adjustments to default parameters+## Quantization### API Usage (Gaudi2/CPU/GPU) Recommended-AutoRound uses the text module of MLLM (LLM component) as the main quantization target. with NeelNanda/pile-10k as the default calibration dataset.++By default, AutoRoundMLLM only quantizes the text module of VLMs and uses `NeelNanda/pile-10k` for calibration. To+quantize the entire model, you can enable `quant_nontext_module` by setting it to True, though support for this feature+is limited.```pythonfrom auto_round import AutoRoundMLLM- from transformers import Qwen2VLForConditionalGeneration, AutoProcessor, AutoTokenizer- ## load the model- model_name = "Qwen/Qwen2-VL-2B-Instruct"- tokenizer = AutoTokenizer.from_pretrained(model_name)- processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=trust_remote_code)- model = Qwen2VLForConditionalGeneration.from_pretrained(- model_name, trust_remote_code=True)-- ## quantize the model- bits, group_size = 4, 128- autoround = AutoRoundMLLM(model, tokenizer, processor, bits=bits, group_size=group_size)- autoround.quantize()-- # save the quantized model, set format='auto_gptq' to use AutoGPTQ format- output_dir = "./tmp_autoround"- autoround.save_quantized(output_dir, format='auto_round', inplace=True)+from transformers import Qwen2VLForConditionalGeneration, AutoProcessor, AutoTokenizer++## load the model+model_name = "Qwen/Qwen2-VL-2B-Instruct"+model = Qwen2VLForConditionalGeneration.from_pretrained(+ model_name, trust_remote_code=True)+tokenizer = AutoTokenizer.from_pretrained(model_name)+processor = AutoProcessor.from_pretrained(model_name, trust_remote_code=True)++## quantize the model+bits, group_size, sym = 4, 128, True+autoround = AutoRoundMLLM(model, tokenizer, processor, bits=bits, group_size=group_size, sym=sym)+autoround.quantize()++# save the quantized model, set format='auto_gptq' to use AutoGPTQ format+output_dir = "./tmp_autoround"+autoround.save_quantized(output_dir, format='auto_round', inplace=True)```-- `dataset`: the dataset for quantization training. current support NeelNanda/pile-10k,llava_conv_58k,llava_instruct_80k and llava_instruct_150k. It can be a custom one. Please note that the effectiveness of the Llava calibration dataset has only been validated on five models so far.+- `dataset`: the dataset for quantization training. Currently only support NeelNanda/pile-10k, llava_conv_58k,+ llava_instruct_80k and llava_instruct_150k. Please note that the feasibility of the Llava calibration dataset has only been validated on five models so far.-- `quant_nontext_module`: whether to quantize non-text module, e.g. vision component.+- `quant_nontext_module`: whether to quantize non-text module, e.g. vision component.-- `extra_data_dir`:dataset dir for storing images/audio/videos, default to None. Can be a dir path or multiple dir path with format as 'image=path_to_image,video=path_to_video,audio=path_to_audio' By default, it will search in the relative path, and if not find, will automatic download.+for more hyperparameters introduction, please+refer [Homepage Detailed Hyperparameters](../../README.md#api-usage-gaudi2cpugpu)-for more hyperparameters introduction, please refer [Homepage Detailed Hyperparameters](../../README.md#api-usage-gaudi2cpugpu)+### Basic Usage-<details>-<summary style="font-size:17px;">Basic Usage (Gaudi2/CPU/GPU)</summary>- A user guide detailing the full list of supported arguments is provided by calling ```auto-round-mllm -h``` on the terminal. Alternatively, you can use ```auto_round_mllm``` instead of ```auto-round-mllm```. Set the format you want in `format` and- multiple formats exporting has been supported.+A user guide detailing the full list of supported arguments is provided by calling ```auto-round-mllm -h``` on the+terminal. Set the format you want in `format` and+multiple formats exporting has been supported. **Only five model families are supported now.```bash- # experimental feature, default hyperparameters may be changed later- auto—round-mllm \- --model Qwen/Qwen2-VL-2B-Instruct \- --bits 4 \- --group_size 128 \- --format "auto_round" \- --output_dir ./tmp_autoround+auto-round-mllm \+ --model Qwen/Qwen2-VL-2B-Instruct \+ --bits 4 \+ --group_size 128 \+ --format "auto_round" \+ --output_dir ./tmp_autoround```--</details>--<details><summary style="font-size:17px;">Calibration Dataset</summary>-For mllm, we used **text-only** calibration dataset (NeelNanda/pile-10k) as our default. If the model type does not support plain text calibration(e.g. Llama-3.2-vision), it will also automatically switch to llava dataset and adjust the hyperparameters.+For mllm, we used **text-only** calibration dataset (NeelNanda/pile-10k) as our default. If the model type does not+support plain text calibration(e.g. Llama-3.2-vision), it will also automatically switch to llava dataset and adjust the+hyperparameters.-Through argument --dataset(text file), user can use other datasets such as "liuhaotian/llava_conv_58k" "liuhaotian/llava_instruct_80k", "liuhaotian/llava_instruct_150k" or a file path to use local file.+Through argument --dataset(text file), user can use other datasets such as "liuhaotian/llava_conv_58k" "+liuhaotian/llava_instruct_80k", "liuhaotian/llava_instruct_150k" or a file path to use local file.</details>@@ -68,9 +74,12 @@ Through argument --dataset(text file), user can use other datasets such as "liuh### Support Matrix-For typical VLLMs, we assume that the default quantization, which excludes quantizing the visual component, is supported. The design of vision components in MLLM model APIs is not standardized, and some models do not support the quantization of non-text modules.+For typical VLLMs, we assume that the default quantization, which excludes quantizing the visual component, is+supported. The design of vision components in MLLM model APIs is not standardized, and some models do not support the+quantization of non-text modules.-Currently, the quantization of vision components is supported for Llama-3.2-11B-Vision, Phi-3.5-Vision-Instruct, and Llava-v1.5-7B.+Currently, the quantization of vision components is supported for Llama-3.2-11B-Vision, Phi-3.5-Vision-Instruct, and+Llava-v1.5-7B.| Model | Eval Lib | calibration dataset | quant nontext module ||--------------|-----------|---------------------|----------------------|@@ -80,53 +89,62 @@ Currently, the quantization of vision components is supported for Llama-3.2-11B-| Llava-v1.5 | lmms_eval | pile/llava | - || CogVLM2 | lmms_eval | pile/llava | ✔ |--### New Models Support+#### Template-For autoround MLLMs, using Template to customize different operations for different models. User can add a custom chat template through json file as below.++For autoround MLLMs, using Template to customize different operations for different models. User can add a custom chat+template through json file as below.+```json{- "model_type": "qwen2_vl",- "format_user": "<|im_start|>user\n{{content}}<|im_end|>\n",- "format_assistant": "<|im_start|>assistant\n{{content}}<|im_end|>\n",- "format_system": "<|im_start|>system\n{{content}}<|im_end|>\n",- "format_observation": "<|im_start|>tool\n{{content}}<|im_end|>\n<|im_start|>assistant\n",- "format_separator": "\n",- "default_system": "You are a helpful assistant.",- "replace_tokens": ["<image>", "<|vision_start|><|image_pad|><|vision_end|>"],- "extra_encode": "True",- "processor": "qwen2_vl"+ "model_type": "qwen2_vl",+ "format_user": "<|im_start|>user\n{{content}}<|im_end|>\n",+ "format_assistant": "<|im_start|>assistant\n{{content}}<|im_end|>\n",+ "format_system": "<|im_start|>system\n{{content}}<|im_end|>\n",+ "format_observation": "<|im_start|>tool\n{{content}}<|im_end|>\n<|im_start|>assistant\n",+ "format_separator": "\n",+ "default_system": "You are a helpful assistant.",+ "replace_tokens": [+ "<image>",+ "<|vision_start|><|image_pad|><|vision_end|>"+ ],+ "extra_encode": "True",+ "processor": "qwen2_vl"}```-The special token ```{{content}}``` is a placeholder to tell the preprocessor where to fill in the corresponding dialogue content.++The special token ```{{content}}``` is a placeholder to tell the preprocessor where to fill in the corresponding+dialogue content.```format_*```: Add specific token to chat content depends on different role names.For example, the input conversations:<br>- ```[{'role': 'user', 'value': '<image>\nWhat are the colors of the bus in the image?'}, {'role': 'assistant', 'value': 'The bus in the image is white and red.'}]```+```[{'role': 'user', 'value': '<image>\nWhat are the colors of the bus in the image?'}, {'role': 'assistant', 'value': 'The bus in the image is white and red.'}]```Using the above template, the input will be converted to the specified format required by Qwen2-vl as below: <br>- ```'<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>\nWhat are the colors of the bus in the image?<|im_end|>\n<|im_start|>assistant\nThe bus in the image is white and red.<|im_end|>\n<|im_start|>user\nWhat feature can be seen on the back of the bus?<|im_end|>\n<|im_start|>assistant\nThe back of the bus features an advertisement.<|im_end|>\n<|im_start|>user\nIs the bus driving down the street or pulled off to the side?<|im_end|>\n<|im_start|>assistant\nThe bus is driving down the street, which is crowded with people and other vehicles.<|im_end|>\n'```.+```'<|im_start|>system\nYou are a helpful assistant.<|im_end|>\n<|im_start|>user\n<|vision_start|><|image_pad|><|vision_end|>\nWhat are the colors of the bus in the image?<|im_end|>\n<|im_start|>assistant\nThe bus in the image is white and red.<|im_end|>\n<|im_start|>user\nWhat feature can be seen on the back of the bus?<|im_end|>\n<|im_start|>assistant\nThe back of the bus features an advertisement.<|im_end|>\n<|im_start|>user\nIs the bus driving down the street or pulled off to the side?<|im_end|>\n<|im_start|>assistant\nThe bus is driving down the street, which is crowded with people and other vehicles.<|im_end|>\n'```.#### Processor-Processor is callback interface for calling different processors, such as texts or images processors, for MLLMs. User can define own processor and use registration function to declare. For more information, please refer to the relevant code in ```auto_round/mllm/processor.py```.--</details>+Processor is callback interface for calling different processors, such as texts or images processors, for MLLMs. User+can define own processor and use registration function to declare. For more information, please refer to the relevant+code in ```auto_round/mllm/processor.py```.+</details>## Inference+For the AutoRound format, please add the following code at the beginning of the original model's inference code.```python-from auto_round import AutoRoundConfig ## must import for auto-round format+from auto_round import AutoRoundConfig ## must import for auto-round format```For more details on quantization, inference, evaluation, and environment, see the following recipe:- [Qwen2-VL-7B-Instruct](../../docs/Qwen2-VL-7B-Instruct-sym.md)-- [Llama-3.2-11B-Vision](../../docs/Llama-3.2-11B-Vision-Instruct-sym.md)+- [Llama-3.2-11B-Vision](../../docs/Llama-3.2-11B-Vision-Instruct-sym.md)- [Phi-3.5-vision-instruct](../../docs/Phi-3.5-vision-instruct-sym.md)- [llava-v1.5-7b](../../docs/llava-v1.5-7b-sym.md)- [cogvlm2-llama3-chat-19B](../../docs/cogvlm2-llama3-chat-19B-sym.md)diff --git a/auto_round/version.py b/auto_round/version.pyindex b43b23350..b2c7f0198 100644--- a/auto_round/version.py+++ b/auto_round/version.py@@ -14,4 +14,4 @@"""Intel® auto-round: An open-source Python librarysupporting popular model weight only compression based on signround."""-__version__ = "0.4.1"+__version__ = "0.4.2.dev"diff --git a/docs/Llama-2-7b-chat-hf-acc.md b/docs/Llama-2-7b-chat-hf-asym-recipe.mdsimilarity index 73%rename from docs/Llama-2-7b-chat-hf-acc.mdrename to docs/Llama-2-7b-chat-hf-asym-recipe.mdindex 4b8740661..9cb68e375 100644--- a/docs/Llama-2-7b-chat-hf-acc.md+++ b/docs/Llama-2-7b-chat-hf-asym-recipe.md@@ -1,3 +1,20 @@+ **This recipe is outdated, we recommend using symmetric quantization.** You can remove --asym from the command.++A sample command to generate an INT4 model.+```bash+auto-round \+--model meta-llama/Llama-2-7b-chat-hf \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--nsamples 512 \+--asym \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"+```++Due to licensing restrictions, we are unable to release the model.Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d.@@ -19,3 +36,5 @@ Since we encountered an issue evaluating this model with lm-eval, we opted to ev| rte | 0.6968 | 0.7292 || arc_easy | 0.7382 | 0.7336 || arc_challenge | 0.4420 | 0.4352 |++diff --git a/docs/Llama-3.2-11B-Vision-Instruct-sym.md b/docs/Llama-3.2-11B-Vision-Instruct-sym.mdindex c41a853a6..a027d98fd 100644--- a/docs/Llama-3.2-11B-Vision-Instruct-sym.md+++ b/docs/Llama-3.2-11B-Vision-Instruct-sym.md@@ -7,7 +7,7 @@ This model is an int4 model with group_size 128 and symmetric quantization of [m### RequirementsPlease use Transformers version 4.45.0 or later-+AutoRound version >= 0.4.1### INT4 Inference```python@@ -91,7 +91,7 @@ messages = [```## Evaluation the model-pip3 install git+https://github.com/open-compass/VLMEvalKit.git@7de2dcb+pip3 install git+https://github.com/open-compass/VLMEvalKit.git@7de2dcb. The evaluation process may encounter errors that require changing model backend or evaluation code. Detailed instructions will be provided in a future update.```bashauto-round-mllm --eval --model Intel/Llama-3.2-11B-Vision-Instruct-inc-private --tasks MMBench_DEV_EN_V11,ScienceQA_VAL,TextVQA_VAL,POPE --output_dir "./eval_result"```@@ -107,7 +107,7 @@ auto-round-mllm --eval --model Intel/Llama-3.2-11B-Vision-Instruct-inc-private -Here is the sample command to reproduce the model.```bashpip install auto-round-auto-round-mllm+auto-round-mllm \--model meta-llama/Llama-3.2-11B-Vision-Instruct \--device 0 \--group_size 128 \diff --git a/docs/Meta-Llama-3-8B-Instruct-acc.md b/docs/Meta-Llama-3-8B-Instruct-asym-recipe.mdsimilarity index 65%rename from docs/Meta-Llama-3-8B-Instruct-acc.mdrename to docs/Meta-Llama-3-8B-Instruct-asym-recipe.mdindex 448c4e473..a8831fa67 100644--- a/docs/Meta-Llama-3-8B-Instruct-acc.md+++ b/docs/Meta-Llama-3-8B-Instruct-asym-recipe.md@@ -1,16 +1,34 @@-Due to licensing restrictions, we are unable to release the model. lm-eval 0.4.2 is used+**This recipe is outdated, we recommend using symmetric quantization.** You can remove --asym from the command.-For evaluating w4g128 without quantized lm-head,+A sample command to generate an INT4 model.```bash-lm_eval --model hf --model_args pretrained="./",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 16+auto-round \+--model meta-llama/Meta-Llama-3-8B-Instruct \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--nsamples 512 \+--asym \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"```-for evaluation with quantized lm-head+quant lm-head```bash-git clone https://github.com/intel/auto-round-cd auto-round/examples/language-modeling-python3 eval_042/evaluation.py --model_name "./" --eval_bs 16+auto-round \+--model meta-llama/Meta-Llama-3-8B-Instruct \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--nsamples 512 \+--asym \+--quant_lm_head \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"```+lm-eval 0.4.2 is used| Metric | **BF16** | w4g128 w/o lm-head | w4g128 with lm-head || ---------------- | :------- |--------------------|-----------------------------|@@ -26,3 +44,5 @@ python3 eval_042/evaluation.py --model_name "./" --eval_bs 16| boolq | 0.8297 | 0.8309 | 0.8266 || arc_easy | 0.8152 | 0.8089 | 0.8123 || arc_challenge | 0.5299 | 0.5102 | 0.5111 |++diff --git a/docs/Mistral-7B-Instruct-v0.2-acc.md b/docs/Mistral-7B-Instruct-v0.2-asym-recipe.mdsimilarity index 52%rename from docs/Mistral-7B-Instruct-v0.2-acc.mdrename to docs/Mistral-7B-Instruct-v0.2-asym-recipe.mdindex c14d2f73a..eb9e9a8ec 100644--- a/docs/Mistral-7B-Instruct-v0.2-acc.md+++ b/docs/Mistral-7B-Instruct-v0.2-asym-recipe.md@@ -1,12 +1,19 @@-Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d-##pip install auto-gptq[triton]-##pip install triton==2.2.0+**This recipe is outdated, we recommend using symmetric quantization.** You can remove --asym from the command.++A sample command to generate an INT4 model.```bash-lm_eval --model hf --model_args pretrained="Intel/Mistral-7B-Instruct-v0.2-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32+auto-round \+--model mistralai/Mistral-7B-Instruct-v0.2 \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--nsamples 512 \+--asym \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"```--| Metric | BF16 | INT4 || -------------- | ------ | ------ || Avg. | 0.6647 | 0.6621 |@@ -21,3 +28,4 @@ lm_eval --model hf --model_args pretrained="Intel/Mistral-7B-Instruct-v0.2-int4-| rte | 0.7040 | 0.7148 || arc_easy | 0.8161 | 0.8165 || arc_challenge | 0.5435 | 0.5435 |+diff --git a/docs/Mistral-7B-v0.1-acc.md b/docs/Mistral-7B-v0.1-asym-recipe.mdsimilarity index 70%rename from docs/Mistral-7B-v0.1-acc.mdrename to docs/Mistral-7B-v0.1-asym-recipe.mdindex 8f2a93c6a..c75f6ea7d 100644--- a/docs/Mistral-7B-v0.1-acc.md+++ b/docs/Mistral-7B-v0.1-asym-recipe.md@@ -1,11 +1,37 @@-Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used 0.4.2+ **This recipe is outdated, we recommend using symmetric quantization.** You can remove --asym from the command.++A sample command to generate an INT4 model.```bash-lm_eval --model hf --model_args pretrained="Intel/Mistral-7B-v0.1-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32+auto-round \+--model mistralai/Mistral-7B-v0.1 \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--nsamples 512 \+--asym \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"```+quant_lm_head+```bash+auto-round \+--model mistralai/Mistral-7B-v0.1 \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--nsamples 512 \+--asym \+--quant_lm_head \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"+```+lm-eval 0.4.2 is used| Metric | BF16 | [INT4-lmhead](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc-lmhead) | [INT4](https://huggingface.co/Intel/Mistral-7B-v0.1-int4-inc) || -------------- | ------ |-----------------| ------------------------------------------------------------ |diff --git a/docs/Mixtral-8x7B-Instruct-v0.1-acc.md b/docs/Mixtral-8x7B-Instruct-v0.1-asym-recipe.mdsimilarity index 63%rename from docs/Mixtral-8x7B-Instruct-v0.1-acc.mdrename to docs/Mixtral-8x7B-Instruct-v0.1-asym-recipe.mdindex 2dd4d62ea..651ffeca0 100644--- a/docs/Mixtral-8x7B-Instruct-v0.1-acc.md+++ b/docs/Mixtral-8x7B-Instruct-v0.1-asym-recipe.md@@ -1,11 +1,21 @@-Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id f3b7917091afba325af3980a35d8a6dcba03dc3f is used--Download the model from hf(coming soon) or follow examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh to generate the model+ **This recipe is outdated, we recommend using symmetric quantization.** You can remove --asym from the command.+A sample command to generate an INT4 model.```bash-lm_eval --model hf --model_args pretrained="Intel/Mixtral-8x7B-Instruct-v0.1-int4-inc",autogptq=True,gptq_use_triton=True --device cuda:0 --tasks lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,rte,arc_easy,arc_challenge,mmlu --batch_size 32+auto-round \+--model mistralai/Mixtral-8x7B-Instruct-v0.1 \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--nsamples 512 \+--asym \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"```+Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id f3b7917091afba325af3980a35d8a6dcba03dc3f is used+| Metric | BF16 | INT4 || -------------- |--------| ------ || Avg. | 0.7000 | 0.6977 |@@ -20,3 +30,4 @@ lm_eval --model hf --model_args pretrained="Intel/Mixtral-8x7B-Instruct-v0.1-int| rte | 0.7184 | 0.7004 || arc_easy | 0.8699 | 0.8712 || arc_challenge | 0.6220 | 0.6229 |+diff --git a/docs/Mixtral-8x7B-v0.1-acc.md b/docs/Mixtral-8x7B-v0.1-asym-acc.mdsimilarity index 74%rename from docs/Mixtral-8x7B-v0.1-acc.mdrename to docs/Mixtral-8x7B-v0.1-asym-acc.mdindex d9f9632d3..82967a3c3 100644--- a/docs/Mixtral-8x7B-v0.1-acc.md+++ b/docs/Mixtral-8x7B-v0.1-asym-acc.md@@ -1,4 +1,20 @@-### Evaluate the model+ **This recipe is outdated, we recommend using symmetric quantization.** You can remove --asym from the command.++A sample command to generate an INT4 model.+```bash+auto-round \+--model mistralai/Mixtral-8x7B-v0.1 \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--nsamples 512 \+--asym \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"+```++Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id f3b7917091afba325af3980a35d8a6dcba03dc3fDownload the model from hf(coming soon) or follow examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh to generate the modeldiff --git a/docs/Phi-3.5-vision-instruct-sym.md b/docs/Phi-3.5-vision-instruct-sym.mdindex 3141f00c8..ec2001c67 100644--- a/docs/Phi-3.5-vision-instruct-sym.md+++ b/docs/Phi-3.5-vision-instruct-sym.md@@ -103,9 +103,9 @@ content = "This image represents which company?"## Evaluation the model-pip3 install git+https://github.com/open-compass/VLMEvalKit.git@7de2dcb+pip3 install git+https://github.com/open-compass/VLMEvalKit.git@7de2dcb. The evaluation process may encounter errors that require changing model backend or evaluation code. Detailed instructions will be provided in a future update```bash-auto-round-mllm --eval --model Intel/Qwen2-VL-7B-Instruct-inc-private --tasks MMBench_DEV_EN_V11,ScienceQA_VAL,TextVQA_VAL,POPE --output_dir "./eval_result"+auto-round-mllm --eval --model Intel/Phi-3.5-vision-instruct-inc-private --tasks MMBench_DEV_EN_V11,ScienceQA_VAL,TextVQA_VAL,POPE --output_dir "./eval_result"```|Metric |16bits|Pile Calib INT4 | Llava Calib INT4 ||-------------------|:------|:------|:------|@@ -119,7 +119,7 @@ auto-round-mllm --eval --model Intel/Qwen2-VL-7B-Instruct-inc-private --tasks MMHere is the sample command to reproduce the model.```bashpip install auto-round-auto-round-mllm+auto-round-mllm \--model microsoft/Phi-3.5-vision-instruct \--device 0 \--group_size 128 \diff --git a/docs/Qwen2-VL-7B-Instruct-sym.md b/docs/Qwen2-VL-7B-Instruct-sym.mdindex 0ac51b53d..88d3e76d1 100644--- a/docs/Qwen2-VL-7B-Instruct-sym.md+++ b/docs/Qwen2-VL-7B-Instruct-sym.md@@ -109,7 +109,7 @@ messages = [```## Evaluation the model-pip3 install git+https://github.com/open-compass/VLMEvalKit.git@7de2dcb+pip3 install git+https://github.com/open-compass/VLMEvalKit.git@7de2dcb. The evaluation process may encounter errors that require changing model backend or evaluation code. Detailed instructions will be provided in a future update.```bashauto-round-mllm --eval --model Intel/Qwen2-VL-7B-Instruct-inc-private --tasks MMBench_DEV_EN_V11,ScienceQA_VAL,TextVQA_VAL,POPE --output_dir "./eval_result"```@@ -125,7 +125,7 @@ auto-round-mllm --eval --model Intel/Qwen2-VL-7B-Instruct-inc-private --tasks MMHere is the sample command to reproduce the model.```bashpip install auto_round-auto-round-mllm+auto-round-mllm \--model Qwen/Qwen2-VL-7B-Instruct \--device 0 \--group_size 128 \diff --git a/docs/Yi-6B-Chat-acc.md b/docs/Yi-6B-Chat-asym-recipe.mdsimilarity index 73%rename from docs/Yi-6B-Chat-acc.mdrename to docs/Yi-6B-Chat-asym-recipe.mdindex f910d9195..d62c36c6b 100644--- a/docs/Yi-6B-Chat-acc.md+++ b/docs/Yi-6B-Chat-asym-recipe.md@@ -1,3 +1,19 @@+ **This recipe is outdated, we recommend using symmetric quantization.** You can remove --asym from the command.+```bash+auto-round \+--model 01-ai/Yi-6B-Chat \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--nsamples 512 \+--asym \+--minmax_lr 2e-3 \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"+```++Due to licensing restrictions, we are unable to release the model. Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d.We used the following command for evaluation.diff --git a/docs/baichuan2-7b-chat-acc.md b/docs/baichuan2-7b-cha-asym-recipe.mdsimilarity index 65%rename from docs/baichuan2-7b-chat-acc.mdrename to docs/baichuan2-7b-cha-asym-recipe.mdindex 6acbd9448..32b3b8d99 100644--- a/docs/baichuan2-7b-chat-acc.md+++ b/docs/baichuan2-7b-cha-asym-recipe.md@@ -1,3 +1,20 @@+**This recipe is outdated, we recommend using symmetric quantization.** You can remove --asym from the command.+```bash+auto-round \+--model baichuan-inc/Baichuan2-7B-Chat \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--nsamples 512 \+--asym \+--minmax_lr 2e-3 \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"+```+++Due to licensing restrictions, we are unable to release the model. Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d.We used the following command for evaluation.diff --git a/docs/bloom-3B-acc.md b/docs/bloom-3B-asym-recipe.mdsimilarity index 75%rename from docs/bloom-3B-acc.mdrename to docs/bloom-3B-asym-recipe.mdindex 17c451974..4f2096f98 100644--- a/docs/bloom-3B-acc.md+++ b/docs/bloom-3B-asym-recipe.md@@ -1,3 +1,18 @@+**This recipe is outdated, we recommend using symmetric quantization.** You can remove --asym from the command.+```bash+auto-round \+--model bigscience/bloom-3b \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--nsamples 512 \+--asym \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"+```++Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d##pip install auto-gptq[triton]##pip install triton==2.2.0diff --git a/docs/cogvlm2-llama3-chat-19B-sym.md b/docs/cogvlm2-llama3-chat-19B-sym.mdindex b58b3e875..257be35f0 100644--- a/docs/cogvlm2-llama3-chat-19B-sym.md+++ b/docs/cogvlm2-llama3-chat-19B-sym.md@@ -80,17 +80,11 @@ content = "这张图片代表哪家公司?"-## Evaluation the model-pip3 install lmms_eval-```bash-auto-round-mllm --lmms --model Intel/cogvlm2-llama3-chat-19B-inc-private --tasks pope,textvqa_val,scienceqa,mmbench_en --output_dir "./eval_result" --device cuda:0-```-### Generate the modelHere is the sample command to reproduce the model.```bashpip install auto-round-auto-round-mllm+auto-round-mllm \--model THUDM/cogvlm2-llama3-chat-19B \--device 0 \--group_size 128 \diff --git a/docs/falcon-7b-acc.md b/docs/falcon-7b-asym-recipe.mdsimilarity index 73%rename from docs/falcon-7b-acc.mdrename to docs/falcon-7b-asym-recipe.mdindex 98bbf5cc9..d0e101eb5 100644--- a/docs/falcon-7b-acc.md+++ b/docs/falcon-7b-asym-recipe.md@@ -1,3 +1,18 @@+**This recipe is outdated, we recommend using symmetric quantization.** You can remove --asym from the command.+```bash+auto-round \+--model tiiuae/falcon-7b \+--device 0 \+--group_size 64 \+--bits 4 \+--iters 1000 \+--nsamples 512 \+--asym \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"+```++We generate the model with group_size 64 as there is an issue when evaluating with group_size 128.Evaluate the modelpip3 install lm-eval==0.4.2diff --git a/docs/gemma-2b-acc.md b/docs/gemma-2b-asym-recipe.mdsimilarity index 82%rename from docs/gemma-2b-acc.mdrename to docs/gemma-2b-asym-recipe.mdindex 7ba60de03..f1c16c899 100644--- a/docs/gemma-2b-acc.md+++ b/docs/gemma-2b-asym-recipe.md@@ -1,4 +1,19 @@-### Evaluate the model+**This recipe is outdated, we recommend using symmetric quantization.** You can remove --asym from the command.+```bash+auto-round \+--model google/gemma-2b \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 400 \+--model_dtype "float16" \+--nsamples 512 \+--asym \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"+```++Evaluate the modelInstall [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, lm-eval 0.4.2 is useddiff --git a/docs/gemma-7b-acc.md b/docs/gemma-7b-asym-recipe.mdsimilarity index 80%rename from docs/gemma-7b-acc.mdrename to docs/gemma-7b-asym-recipe.mdindex aefcb8b9d..de0194ccb 100644--- a/docs/gemma-7b-acc.md+++ b/docs/gemma-7b-asym-recipe.md@@ -1,3 +1,22 @@+ **This recipe is outdated, we recommend using symmetric quantization.** You can remove --asym from the command.++A sample command to generate an INT4 model.+```bash+auto-round \+--model google/gemma-7b \+--device 0 \+--group_size 128 \+--bits 4 \+--minmax_lr 2e-3 \+--model_dtype "float16" \+--iters 1000 \+--nsamples 512 \+--asym \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"+```++pip install lm-eval==0.4.2pip install auto-gptqdiff --git a/docs/gemma-7b-it-acc.md b/docs/gemma-7b-it-asym-recipe.mdsimilarity index 74%rename from docs/gemma-7b-it-acc.mdrename to docs/gemma-7b-it-asym-recipe.mdindex 1cd2395f2..08b3ef0c6 100644--- a/docs/gemma-7b-it-acc.md+++ b/docs/gemma-7b-it-asym-recipe.md@@ -1,3 +1,21 @@+ **This recipe is outdated, we recommend using symmetric quantization.** You can remove --asym from the command.++A sample command to generate an INT4 model.+```bash+auto-round \+--model google/gemma-7b-it \+--device 0 \+--group_size 128 \+--bits 4 \+--minmax_lr 2e-3 \+--model_dtype "float16" \+--iters 1000 \+--nsamples 512 \+--asym \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"+```+Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, and the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d, Install the latest [AutoGPTQ](https://github.com/AutoGPTQ/AutoGPTQ) from source firstPlease note that there is a discrepancy between the baseline result and the official data, which is a known issue within the official model card community.diff --git a/docs/gpt-j-6B-acc.md b/docs/gpt-j-6B-asym-recipe.mdsimilarity index 72%rename from docs/gpt-j-6B-acc.mdrename to docs/gpt-j-6B-asym-recipe.mdindex 66679a92c..6b950a344 100644--- a/docs/gpt-j-6B-acc.md+++ b/docs/gpt-j-6B-asym-recipe.md@@ -1,3 +1,20 @@+ **This recipe is outdated, we recommend using symmetric quantization.** You can remove --asym from the command.++A sample command to generate an INT4 model.+```bash+auto-round \+--model EleutherAI/gpt-j-6b \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--nsamples 512 \+--asym \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"+```++Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d##pip install auto-gptq[triton]##pip install triton==2.2.0diff --git a/docs/llava-v1.5-7b-sym.md b/docs/llava-v1.5-7b-sym.mdindex 633a65e08..b5c6058d3 100644--- a/docs/llava-v1.5-7b-sym.md+++ b/docs/llava-v1.5-7b-sym.md@@ -80,7 +80,7 @@ messages = [{"from": "human", "value": "How many people and animals are there in```## Evaluation the model-pip3 install lmms_eval+pip3 install lmms_eval. The evaluation process may encounter errors that require changing model backend or evaluation code. Detailed instructions will be provided in a future update```bashauto-round-mllm --lmms --model Intel/llava-v1.5-7b-inc-private --tasks pope,textvqa_val,scienceqa,mmbench_en --output_dir "./eval_result" --device cuda:0```@@ -96,7 +96,7 @@ auto-round-mllm --lmms --model Intel/llava-v1.5-7b-inc-private --tasks pope,textHere is the sample command to reproduce the model.```bashpip install auto-round-auto-round-mllm+auto-round-mllm \--model liuhaotian/llava-v1.5-7b \--device 0 \--group_size 128 \diff --git a/docs/neural-chat-7b-v3-1-acc.md b/docs/neural-chat-7b-v3-1-asym-recipe.mdsimilarity index 70%rename from docs/neural-chat-7b-v3-1-acc.mdrename to docs/neural-chat-7b-v3-1-asym-recipe.mdindex f4df2353e..f7e2288a6 100644--- a/docs/neural-chat-7b-v3-1-acc.md+++ b/docs/neural-chat-7b-v3-1-asym-recipe.md@@ -1,3 +1,21 @@+ **This recipe is outdated, we recommend using symmetric quantization.** You can remove --asym from the command.++A sample command to generate an INT4 model.+```bash+auto-round \+--model intel/neural-chat-7b-v3-1 \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--nsamples 512 \+--minmax_lr 2e-3 \+--asym \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"+```++Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id f3b7917091afba325af3980a35d8a6dcba03dc3f~~~bashdiff --git a/docs/neural-chat-7b-v3-3-acc.md b/docs/neural-chat-7b-v3-3-asym-recipe.mdsimilarity index 70%rename from docs/neural-chat-7b-v3-3-acc.mdrename to docs/neural-chat-7b-v3-3-asym-recipe.mdindex 985e23ccc..7d1334057 100644--- a/docs/neural-chat-7b-v3-3-acc.md+++ b/docs/neural-chat-7b-v3-3-asym-recipe.md@@ -1,3 +1,22 @@+ **This recipe is outdated, we recommend using symmetric quantization.** You can remove --asym from the command.++A sample command to generate an INT4 model.+```bash+auto-round \+--model intel/neural-chat-7b-v3-3 \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--nsamples 512 \+--minmax_lr 2e-3 \+--asym \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"+```+++Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id f3b7917091afba325af3980a35d8a6dcba03dc3f~~~bashdiff --git a/docs/opt-2.7b-acc.md b/docs/opt-2.7b-asym-recipe.mdsimilarity index 71%rename from docs/opt-2.7b-acc.mdrename to docs/opt-2.7b-asym-recipe.mdindex 796934dca..2ec433660 100644--- a/docs/opt-2.7b-acc.md+++ b/docs/opt-2.7b-asym-recipe.md@@ -1,3 +1,21 @@+ **This recipe is outdated, we recommend using symmetric quantization.** You can remove --asym from the command.++A sample command to generate an INT4 model.+```bash+auto-round \+--model facebook/opt-2.7b \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--nsamples 512 \+--minmax_lr 2e-3 \+--asym \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"+```++Install [lm-eval-harness](https://github.com/EleutherAI/lm-evaluation-harness.git) from source, we used the git id 96d185fa6232a5ab685ba7c43e45d1dbb3bb906d##pip install auto-gptq[triton]##pip install triton==2.2.0diff --git a/docs/phi-2-acc.md b/docs/phi-2-old-sym-recipe.mdsimilarity index 70%rename from docs/phi-2-acc.mdrename to docs/phi-2-old-sym-recipe.mdindex 0ccdfeb57..868493d49 100644--- a/docs/phi-2-acc.md+++ b/docs/phi-2-old-sym-recipe.md@@ -1,3 +1,19 @@+ **This recipe is outdated, we recommend using the latest full range symmetric quantization.** You can remove --asym from the command.++A sample command to generate an INT4 model.+```bash+auto-round \+--model facebook/opt-2.7b \+--device 0 \+--group_size 128 \+--bits 4 \+--iters 1000 \+--nsamples 512 \+--format 'auto_gptq,auto_round' \+--output_dir "./tmp_autoround"+```++pip install lm-eval==0.4.2Due to the significant accuracy drop with the asymmetric kernel for this model, we opted to use symmetric quantization.diff --git a/examples/language-modeling/README.md b/examples/language-modeling/README.mddeleted file mode 100644index 9ca72f561..000000000--- a/examples/language-modeling/README.md+++ /dev/null@@ -1,198 +0,0 @@-Step-by-Step-============--This document presents step-by-step instructions for auto-round.--# Prerequisite---## 1. Prepare Calibration Dataset--### Default Dataset-The [NeelNanda/pile-10k](https://huggingface.co/datasets/NeelNanda/pile-10k) in huggingface is adopted as the default calibration data and will be downloaded automatically from the datasets Hub. To customize a dataset, please kindly follow our dataset code.-See more about loading [huggingface dataset](https://huggingface.co/docs/datasets/main/en/quickstart)--### Customized Dataset-- Option 1: Pass a local json file path to dataset argument-- Option 2: Register your dataset following the [code](../../auto_round/calib_dataset.py) and pass the new dataset and split args to initialize AutoRound object, e.g. autoround=Autoround(dataset="NeelNanda/pile-10k:train", ...)-- Option 3: pass list of string or list of input_ids to dataset.-~~~python-def customized_data():- ##Important Notice!!! Autoround will drop data < args.seqlen and truncate data to args.seqlen- data = ["AutoRound is an advanced weight-only quantization algorithm for low-bits LLM inference" * 240]- data.append("AutoRound is an advanced weight-only quantization algorithm for low-bits LLM inference")- return data---def customized_data_with_tokenizer(tokenizer, seqlen=2048):- ##Import notice!!! Autoround will drop data < args.seqlen- data = ["AutoRound is an advanced weight-only quantization algorithm for low-bits LLM inference" * 240]- data.append("AutoRound is an advanced weight-only quantization algorithm for low-bits LLM inference")- tokens = []- for d in data:- token = tokenizer(d, truncation=True, max_length=seqlen, return_tensors="pt").data- tokens.append(token)- return tokens-~~~--We support combination of different datasets and parametrization of calibration datasets by using "--dataset ./tmp.json:concat,NeelNanda/pile-10k:split=train+val:num=256,mbpp:concat=True:num=128:apply_template". Both local calibration file and huggingface dataset are supported. Through parametrization, users could specify splits of a dataset by setting "split=split1+split2". A concatenation option could enable users to merge calibration samples, a process commonly used to enhance calibration reliability. An 'apply_template' option would enable users to apply chat_template to calibration data before tokenization and is widely used by instruct-models in generation. Please note that samples shorter than args.seqlen will be dropped when concatenation option is not enabled.-Please use ',' to split datasets, ':' to split parameters of a dataset and '+' to add values for one targeted parameter.---<br />--## 2. Run Examples-Enter into the examples folder and install lm-eval to run the evaluation-```bash-pip install -r requirements.txt-```--- **Default Settings:**-```bash-CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --bits 4 --group_size 128-```-- **Reduced GPU Memory Usage:**-enable low_gpu_mem_usage(more tuning cost)--set "--train_bs 1 --gradient_accumulate_steps 8" (more tuning cost)--reduce the train bs to 4(potential accuracy drop)--reduce the seqlen to 512 (potential accuracy drop)--or combine them--- **Reduced CPU Memory Usage:**-- - set "--low_cpu_mem_mode 1" to use block-wise mode, load the weights from disk of each block when tuning and release the memory of the block after tuning. (more tuning cost)-- - set "--low_cpu_mem_mode 2" to use layer-wise mode, load the weights of each layer from disk when tuning, minimum memory consumption and also slowest running speed.---- **Speedup the tuning:**--reduce the train bs to 4(little accuracy drop)--reduce the seqlen to 512(potential large accuracy drop)--or combine them--- **Enable quantized lm-head:**--Currently only support in Intel xpu and AutoRound format,however, we found the fake tuning could improve the accuracy is some scenarios.-```bash-CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --bits 4 --group_size 128 --quant_lm_head-```--- **Enable marlin kernel:**--please install auto-gptq from source--```bash-CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --sym --bits 4 --group_size 128 --format "auto_round:gptq:marlin"-```--for AutoGPTQ format, enable `--sym` in the quantization and set the format to "auto_gptq" and then follow the API in AutoGPTQ--- **Utilize the AdamW Optimizer:**--Include the flag `--adam`. Note that AdamW is less effective than sign gradient descent in many scenarios we tested.---- **Code generation LLM:**--We utilized mbpp for calibration, but your own training dataset is highly recommended. Please note that samples with seqlen < args.seqlen will be dropped in current version.-```bash-CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name Salesforce/codegen25-7b-multi --bits 4 --group_size 128 --dataset "mbpp" --seqlen 128 "-```-- **Running on Intel Gaudi2**-```bash-bash run_autoround_on_gaudi.sh-```----## 3. Evaluation-The example supports evaluation for various tasks in lm_eval. Moreover, it facilitates separate evaluation through the 'evaluation.py' script, which extends support to three additional tasks (ptb, c4, and wikitext2) beyond the capabilities of the official lm_eval. Additionally, evaluation results will be neatly organized into an Excel file for ease of demonstration.--For large models, GPU memory may be insufficient. Enable multi-GPU evaluation by setting 'CUDA_VISIBLE_DEVICES'.--Due to the large size of the model, the quantization and evaluation processes may be time-consuming. To provide flexibility in the process, two options are offered:--- You can set up multi-GPU cards for the quantization example, which will only use the first card for quantization and then evaluate with all GPU cards.-```bash-CUDA_VISIBLE_DEVICES=1,2 python3 main.py --model_name facebook/opt-125m --amp --bits 4 --group_size -1 --deployment_device fake,cpu --output_dir /save_model_path/-```--- Enable 'disable_eval' for the quantization example, save the qdq model by setting 'deployment_device=fake', and then set up multi-GPU cards for the evaluation script.-```bash-CUDA_VISIBLE_DEVICES=0 python3 main.py --model_name facebook/opt-125m --amp --bits 4 --group_size -1 --disable_eval --deployment_device fake --output_dir /save_model_path/--CUDA_VISIBLE_DEVICES=1,2 python3 eval/evaluation.py --model_name /save_model_path/ --eval_bs 8 --tasks mmlu,lambada_openai,ptb --excel_path /result_excel/save_path/-```--- User could also perform evaluation on Intel Gaudi-2 using the following script.-```bash-python3 eval/evaluation.py --model_name /saved_model_path/ --tasks mmlu --device hpu-```--You can also utilize the official lm_eval [link](https://github.com/EleutherAI/lm-evaluation-harness/tree/main?tab=readme-ov-file#basic-usage).--## 4. Known Issues-* Random quantization results in tuning some models-* ChatGlm-V1 is not supported---## 5. Environment--PyTorch 1.8 or higher version is needed-The transformers version required varies across different types of models. Here, the transformers version used for running models during experiments is provided as a reference.--| Model | Transformers version |-| :----: | :----: |-| EleutherAI/gpt-j-6b | 4.28/4.30/4.34/4.36/4.40 |-| huggyllama/llama-7b | 4.28/4.30/4.34/4.36/4.40 |-| meta-llama/Llama-2-7b-hf | 4.30/4.34/4.36/4.40 |-| facebook/opt-6.7b | 4.28/4.30/4.34/4.36/4.40 |-| tiiuae/falcon-7b | 4.28/4.30/4.34/4.36/4.40 |-| mosaicml/mpt-7b | 4.28/4.30/4.34/4.36 |-| mosaicml/mpt-7b-chat | 4.34 |-| bigscience/bloom-7b1 | 4.28/4.30/4.34/4.36/4.40 |-| baichuan-inc/Baichuan2-7B-Chat | 4.36/4.40 |-| Qwen/Qwen-7B | 4.28/4.30/4.34/4.36/4.40 |-| Qwen/Qwen1.5-7B-Chat | 4.38/4.40/4.40 |-| THUDM/chatglm3-6b | 4.34/4.36/4.40 |-| mistralai/Mistral-7B-v0.1 | 4.34/4.36/4.40 |-| mistralai/Mixtral-8x7B-Instruct-v0.1 | 4.40 |-| MBZUAI/LaMini-GPT-124M | 4.34/4.36/4.40 |-| EleutherAI/gpt-neo-125m | 4.34/4.40 |-| databricks/dolly-v2-3b | 4.34/4.40 |-| stabilityai/stablelm-base-alpha-3b | 4.34/4.40 |-| Intel/neural-chat-7b-v3 | 4.34/4.36/4.40 |-| rinna/bilingual-gpt-neox-4b | 4.36/4.40 |-| microsoft/phi-2 | 4.36/4.40 |-| google/gemma-7b | 4.38/4.40 |-| Salesforce/codegen25-7b-multi | 4.33.2 |------## Reference-If you find SignRound useful for your research, please cite our paper:-```bash-@article{cheng2023optimize,- title={Optimize Weight Rounding via Signed Gradient Descent for the Quantization of LLMs},- author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao},- journal={arXiv preprint arXiv:2309.05516},- year={2023}-}-```--------diff --git a/examples/language-modeling/eval/__init__.py b/examples/language-modeling/eval/__init__.pydeleted file mode 100644index e69de29bb..000000000diff --git a/examples/language-modeling/eval/evaluation.py b/examples/language-modeling/eval/evaluation.pydeleted file mode 100644index ab1aa24a6..000000000--- a/examples/language-modeling/eval/evaluation.py+++ /dev/null@@ -1,131 +0,0 @@-# Copyright (c) 2024 Intel Corporation-#-# Licensed under the Apache License, Version 2.0 (the "License");-# you may not use this file except in compliance with the License.-# You may obtain a copy of the License at-#-# http://www.apache.org/licenses/LICENSE-2.0-#-# Unless required by applicable law or agreed to in writing, software-# distributed under the License is distributed on an "AS IS" BASIS,-# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.-# See the License for the specific language governing permissions and-# limitations under the License.--import logging-import random-import time-from typing import TYPE_CHECKING, List, Optional, Union--import lm_eval-from lm_eval import simple_evaluate as lm_simple_evaluate---def simple_evaluate(- model,- model_args: Optional[Union[str, dict]] = None,- user_model = None,- batch_size: Optional[int] = None,- max_batch_size: Optional[int] = None,- device: Optional[str] = None,- **kwargs):-- try:- from auto_round import AutoRoundConfig- except:- from auto_round.auto_quantizer import AutoHfQuantizer-- if model_args is None:- model_args = ""-- if isinstance(model_args, dict):- lm = lm_eval.api.registry.get_model(model).create_from_arg_obj(- model_args,- {- "batch_size": batch_size,- "max_batch_size": max_batch_size,- "device": device,- },- )-- else:- lm = lm_eval.api.registry.get_model(model).create_from_arg_string(- model_args,- {- "batch_size": batch_size,- "max_batch_size": max_batch_size,- "device": device,- },- )- if user_model is not None:- lm._model = user_model- return lm_simple_evaluate(- model=lm,- model_args=model_args,- batch_size=batch_size,- max_batch_size=max_batch_size,- device=device,- **kwargs)----if __name__ == "__main__":-- import sys-- sys.path.insert(0, '../../')- import time- import argparse-- parser = argparse.ArgumentParser()- parser.add_argument(- "--model_name", default="facebook/opt-125m/"- )- parser.add_argument(- "--eval_bs", default=1,- )- parser.add_argument(- "--trust_remote_code", action='store_true',- help="Whether to enable trust_remote_code"- )- parser.add_argument(- "--device", default="cuda:0",- help="PyTorch device (e.g. cpu/cuda:0/hpu) for evaluation."- )- parser.add_argument("--tasks",- default="lambada_openai,hellaswag,winogrande,piqa,mmlu,truthfulqa_mc1," \- "openbookqa,boolq,rte,arc_easy,arc_challenge",- help="lm-eval tasks for lm_eval version 0.4.2")-- args = parser.parse_args()- s = time.time()- from transformers import AutoConfig-- config = AutoConfig.from_pretrained(args.model_name, trust_remote_code=args.trust_remote_code)-- if hasattr(config, "quantization_config"):- quantization_config = config.quantization_config- if "quant_method" in quantization_config and "auto-round" in quantization_config["quant_method"]:- from auto_round.auto_quantizer import AutoHfQuantizer- elif "quant_method" in quantization_config and quantization_config["quant_method"] == "gptq":- if args.device == "hpu":- from auto_round.auto_quantizer import AutoHfQuantizer-- test_tasks = args.tasks- if isinstance(test_tasks, str):- test_tasks = test_tasks.split(',')- model_name = args.model_name.rstrip('/')- from lm_eval.utils import make_table-- model_args = f"pretrained={args.model_name}"- model_args += ",dtype=float16"- if args.trust_remote_code:- model_args += f",trust_remote_code=True"- result = simple_evaluate(model="hf",- model_args=model_args,- tasks=test_tasks,- device=args.device,- batch_size=args.eval_bs)- print(make_table(result))-- print("cost time: ", time.time() - s)diff --git a/examples/language-modeling/eval_legacy/__init__.py b/examples/language-modeling/eval_legacy/__init__.pydeleted file mode 100644index 35dde853f..000000000--- a/examples/language-modeling/eval_legacy/__init__.py+++ /dev/null@@ -1,5 +0,0 @@-if __name__ == "__main__":- import sys- sys.path.insert(0, './')-from .evaluation import eval_model-from .evaluation_legacy import eval_model_legacydiff --git a/examples/language-modeling/eval_legacy/evaluation.py b/examples/language-modeling/eval_legacy/evaluation.pydeleted file mode 100644index 458986dc8..000000000--- a/examples/language-modeling/eval_legacy/evaluation.py+++ /dev/null@@ -1,525 +0,0 @@--import os-os.environ["TOKENIZERS_PARALLELISM"] = "false"-import os.path-import torch-import logging-import pprint-import re-import shutil-import transformers-from typing import TYPE_CHECKING, Optional, Union-import time--if __name__ == "__main__":- import sys- sys.path.insert(0, './')--EXT_TASKS = ['wikitext2', 'ptb', 'c4', 'ptb-new', 'c4-new']-fewshots_dict = {}-fewshots_dict['paper'] = {- "lambada_openai": [0],- "hellaswag": [0],- "winogrande": [0],- "piqa": [0],- "mmlu": [0],- "wikitext": [0],- "truthfulqa_mc1": [0],- "truthfulqa_mc2": [0],- "openbookqa": [0],- "boolq": [0],- "rte": [0],- "arc_easy": [0],- "arc_challenge": [0],- "gsm8k": [0],- "ceval-valid": [0],- "cmmlu": [0],-}-fewshots_dict['leadboard'] = {- "hellaswag": [10],- "winogrande": [5],- "arc_easy": [25],- "arc_challenge": [25],- "mmlu": [5],- "drop": [3],- "gsm8k": [5],-}-fewshots_dict['all'] = {- "lambada_openai": [0],- "hellaswag": [0, 10],- "winogrande": [0, 5],- "piqa": [0],- "coqa": [], ## coqa is not enabled in llamav1 models- "truthfulqa_mc1": [0],- "truthfulqa_mc2": [0],- "openbookqa": [0],- "boolq": [0],- "rte": [0],- "arc_easy": [0, 25],- "arc_challenge": [0, 25],- "mmlu": [0, 5],- "wikitext": [0],- "drop": [3],- "gsm8k": [5]-}---def simple_evaluate(- model,- model_args: Optional[Union[str, dict, None]] = None,- tasks=None,- num_fewshot: Optional[int] = None,- batch_size: Optional[int] = None,- max_batch_size: Optional[int] = None,- device: Optional[str] = None,- use_cache: Optional[str] = None,- cache_requests: bool = False,- rewrite_requests_cache: bool = False,- delete_requests_cache: bool = False,- limit: Optional[Union[int, float]] = None,- bootstrap_iters: int = 100000,- check_integrity: bool = False,- decontamination_ngrams_path=None,- write_out: bool = False,- log_samples: bool = True,- gen_kwargs: str = None,- task_manager=None,- verbosity: str = "INFO",- predict_only: bool = False,- random_seed: int = 1234,- numpy_random_seed: int = 1234,- torch_random_seed: int = 1234,- lm=None-):- """Instantiate and evaluate a model on a list of tasks.-- :param model: Union[str, LM]- Name of model, transformers.PreTrainedModel object or LM object, see lm_eval.models.get_model- :param model_args: Optional[str, dict]- String or dict arguments for each model class, see LM.create_from_arg_string and LM.create_from_arg_object.- Ignored if `model` argument is a LM object.- :param tasks: list[Union[str, dict, Task]]- List of task names or Task objects. Task objects will be taken to have name task.EVAL_HARNESS_NAME if defined and type(task).__name__ otherwise.- :param num_fewshot: int- Number of examples in few-shot context- :param batch_size: int or str, optional- Batch size for model- :param max_batch_size: int, optional- Maximal batch size to try with automatic batch size detection- :param device: str, optional- PyTorch device (e.g. "cpu" or "cuda:0") for running models- :param use_cache: str, optional- A path to a sqlite db file for caching model responses. `None` if not caching.- :param cache_requests: bool, optional- Speed up evaluation by caching the building of dataset requests. `None` if not caching.- :param rewrite_requests_cache: bool, optional- Rewrites all of the request cache if set to `True`. `None` if not desired.- :param delete_requests_cache: bool, optional- Deletes all of the request cache if set to `True`. `None` if not desired.- :param limit: int or float, optional- Limit the number of examples per task (only use this for testing), If <1, limit is a percentage of the total number of examples.- :param bootstrap_iters:- Number of iterations for bootstrap statistics- :param check_integrity: bool- Whether to run the relevant part of the test suite for the tasks- :param write_out: bool- If True, write out an example document and model input for checking task integrity- :param log_samples: bool- If True, write out all model outputs and documents for per-sample measurement and post-hoc analysis- :param gen_kwargs: str- String arguments for model generation- Ignored for all tasks with loglikelihood output_type- :param predict_only: bool- If true only model outputs will be generated and returned. Metrics will not be evaluated- :param random_seed: int- Random seed for python's random module. If set to None, the seed will not be set.- :param numpy_random_seed: int- Random seed for numpy. If set to None, the seed will not be set.- :param torch_random_seed: int- Random seed for torch. If set to None, the seed will not be set.-- :return- Dictionary of results- """- from lm_eval.tasks import TaskManager- import random- import numpy as np- import transformers- import lm_eval.api- import lm_eval.tasks- import lm_eval.models- import lm_eval.api.metrics- import lm_eval.api.registry- from lm_eval.evaluator import evaluate- from lm_eval.logging_utils import add_env_info, get_git_commit_hash- from lm_eval.tasks import get_task_dict-- from lm_eval.utils import (- eval_logger,- positional_deprecated,- run_task_tests,- simple_parse_args_string,- )-- eval_logger.setLevel(getattr(logging, f"{verbosity}"))-- if delete_requests_cache:- eval_logger.info("Deleting requests cache...")- delete_cache()-- if random_seed is not None:- # See https://github.com/EleutherAI/lm-evaluation-harness/pull/1412- eval_logger.info(f"Setting random seed to {random_seed}")- random.seed(random_seed)-- if numpy_random_seed is not None:- eval_logger.info(f"Setting numpy seed to {numpy_random_seed}")- np.random.seed(numpy_random_seed)-- if torch_random_seed is not None:- eval_logger.info(f"Setting torch manual seed to {torch_random_seed}")- torch.manual_seed(torch_random_seed)-- if tasks is None:- tasks = []- assert (- tasks != []- ), "No tasks specified, or no tasks found. Please verify the task names."-- if lm == None:- if gen_kwargs is not None:- gen_kwargs = simple_parse_args_string(gen_kwargs)- eval_logger.warning(- "generation_kwargs specified through cli, these settings will update set parameters in yaml tasks. Ensure 'do_sample=True' for non-greedy decoding!"- )- if gen_kwargs == "":- gen_kwargs = None-- if isinstance(model, str):- if model_args is None:- model_args = ""-- elif isinstance(model_args, dict):- lm = lm_eval.api.registry.get_model(model).create_from_arg_obj(- model_args,- {- "batch_size": batch_size,- "max_batch_size": max_batch_size,- "device": device,- },- )-- else:- lm = lm_eval.api.registry.get_model(model).create_from_arg_string(- model_args,- {- "batch_size": batch_size,- "max_batch_size": max_batch_size,- "device": device,- },- )- elif isinstance(model, transformers.PreTrainedModel):- lm = lm_eval.api.registry.get_model("hf")(- pretrained=model,- batch_size=batch_size,- max_batch_size=max_batch_size,- )- use_cache = None- else:- assert isinstance(model, lm_eval.api.model.LM)- lm = model-- if use_cache is not None:- print(f"Using cache at {use_cache + '_rank' + str(lm.rank) + '.db'}")- lm = lm_eval.api.model.CachingLM(- lm,- "lm_cache/"- + (model if isinstance(model, str) else model.model.config._name_or_path)- + "_"- + model_args.replace("=", "-").replace(",", "_").replace("/", "-")- + ".db",- )-- if task_manager is None:- task_manager = TaskManager(verbosity)-- eval_logger.info(- "get_task_dict has been updated to accept an optional argument, `task_manager`"- "Read more here:https://github.com/EleutherAI/lm-evaluation-harness/blob/main/docs/interface.md#external-library-usage"- )- task_dict = get_task_dict(tasks, task_manager)- for task_name in task_dict.keys():- task_obj = task_dict[task_name]- if isinstance(task_obj, tuple):- _, task_obj = task_obj- if task_obj is None:- continue-- if task_obj.get_config("output_type") == "generate_until":- if gen_kwargs is not None:- task_obj.set_config(- key="generation_kwargs", value=gen_kwargs, update=True- )-- if predict_only:- log_samples = True- eval_logger.info(- f"Processing {task_name} in output-only mode. Metrics will not be calculated!"- )- # we have to change the class properties post-hoc. This is pretty hacky.- task_obj.override_metric(metric_name="bypass")-- if num_fewshot is not None:- if (default_num_fewshot := task_obj.get_config("num_fewshot")) == 0:- eval_logger.info(- f"num_fewshot has been set to 0 for {task_name} in its config. Manual configuration will be ignored."- )- else:- eval_logger.warning(- f"Overwriting default num_fewshot of {task_name} from {default_num_fewshot} to {num_fewshot}"- )- task_obj.set_config(key="num_fewshot", value=num_fewshot)-- if check_integrity:- run_task_tests(task_list=tasks)-- results = evaluate(- lm=lm,- task_dict=task_dict,- limit=limit,- cache_requests=cache_requests,- rewrite_requests_cache=rewrite_requests_cache,- bootstrap_iters=bootstrap_iters,- decontamination_ngrams_path=decontamination_ngrams_path,- write_out=write_out,- log_samples=log_samples,- verbosity=verbosity,- )-- if lm.rank == 0:- if isinstance(model, str):- model_name = model- elif hasattr(model, "config") and hasattr(model.config, "_name_or_path"):- model_name = model.config._name_or_path- else:- model_name = type(model).__name__-- # add info about the model and few shot config- results["config"] = {- "model": model_name,- "model_args": model_args,- "batch_size": batch_size,- "batch_sizes": (- list(lm.batch_sizes.values()) if hasattr(lm, "batch_sizes") else []- ),- "device": device,- "use_cache": use_cache,- "limit": limit,- "bootstrap_iters": bootstrap_iters,- "gen_kwargs": gen_kwargs,- }- results["git_hash"] = get_git_commit_hash()- add_env_info(results) # additional environment info to results- return results, lm- else:- return None---def eval_model(model_path, tasks=["lambada_openai", "hellaswag", "winogrande", "piqa"],- eval_bs=32, use_accelerate=True, dtype=None, limit=None, trust_remote_code=True,- device="cuda:0", seed=0, nsamples=128, mark="paper", excel_file="tmp.xlsx"):- print("evaluation with official lm-eval", flush=True)- try:- import lm_eval.api- import lm_eval.tasks- import lm_eval.models- import lm_eval.api.metrics- import lm_eval.api.registry- from lm_eval.evaluator import evaluate-- except:- raise ImportError("""follow requirements to install dependencies.""")-- org_s = time.time()- if dtype == None:- from .utils import convert_dtype_torch2str_hf- from transformers import AutoConfig- config = AutoConfig.from_pretrained(model_path, trust_remote_code=trust_remote_code)- if hasattr(config, "torch_dtype"):- dtype = convert_dtype_torch2str_hf(config.torch_dtype)- else:- dtype = "float16"- print(f"Using {dtype} as evaluation data type.")-- external_tasks = []- if isinstance(tasks, str):- tasks = tasks.split(',')- for each in EXT_TASKS:- if each in tasks:- external_tasks.append(each)- tasks.remove(each)-- results = {}- model = None- lm = None-- for tmp_tasks in tasks:- try:- num_fewshot = fewshots_dict[mark][tmp_tasks]- print(f'********* {tmp_tasks} evaluate ************')- task_s = time.time()- for shot in num_fewshot:- model_type = "hf"- model_args = f'pretrained={model_path},tokenizer={model_path},dtype={dtype},trust_remote_code={trust_remote_code}'- if 'gpu' in model_path:- model_args = f'pretrained={model_path},tokenizer={model_path},dtype={dtype},autogptq=True,gptq_use_triton=True,trust_remote_code={trust_remote_code}'- if use_accelerate: # bool(re.search("chatglm", model_path.lower()))- model_args += f',parallelize=True'-- if "wikitext" in tmp_tasks:- tmp_eval_bs = 1- else:- tmp_eval_bs = eval_bs- tmp_results, lm = simple_evaluate(model=model_type, model_args=model_args, tasks=tmp_tasks,- num_fewshot=shot, limit=limit, batch_size=tmp_eval_bs,- max_batch_size=tmp_eval_bs, lm=lm, device=str(device))- if 'mmlu' in tmp_tasks and 'cmmlu' not in tmp_tasks:- sub_name = f'hendrycksTest-* {shot}-shot'- else:- sub_name = f'{tmp_tasks} {shot}-shot'- print(f'{sub_name}: ')- pprint.pprint(tmp_results["results"])- print(f"\n{sub_name} cost time: {time.time() - task_s}\n")- results[sub_name] = {}- if 'mmlu' in tmp_tasks and 'cmmlu' not in tmp_tasks:- for cata in ['humanities', 'other', 'stem', 'sociology']:- results[sub_name][cata] = tmp_results['results'][f'mmlu_{cata}']- results[sub_name]['avg'] = tmp_results['results'][f'mmlu']- else:- results[sub_name] = tmp_results['results']- except Exception as e:- print(f'********* {tmp_tasks} ERROR ************')- print(str(e))- continue-- tokenizer = transformers.AutoTokenizer.from_pretrained(model_path, use_fast=False, trust_remote_code=True)- model = lm.model- # for external tasks- # maybe adjust for specific model- # if hasattr(lm.model.config, "max_position_embeddings"):- # lm.model.seqlen = lm.model.config.max_position_embeddings- # else:- # ## for llama-1, opt- # lm.model.seqlen = 2048-- # if "opt" in model_name:- # seqlen = model.config.max_position_embeddings- # model.seqlen = model.config.max_position_embeddings- # else:- # seqlen = 2048- # model.seqlen = seqlen-- model.seqlen = 2048- from .utils import get_loaders, eval_ppl_same_with_gptq- for dataset in external_tasks:- try:- dataloader, testloader = get_loaders(- dataset, nsamples=nsamples, seed=seed,- tokenizer=tokenizer, seqlen=model.seqlen- )- ppl = eval_ppl_same_with_gptq(model, testloader, device)- print(dataset, ppl)-- results.update({dataset: ppl})- except Exception as e:- print(str(e))- continue-- print(results, flush=True)- print("cost time: ", time.time() - org_s)- import pickle- from collections import OrderedDict- new_dict = OrderedDict()- new_dict["model"] = "tmp"- new_dict["paper-avg"] = 0- new_dict["leaderboard-avg"] = 0- new_dict["wikitext2"] = 0- new_dict["ptb-new"] = 0- new_dict["c4-new"] = 0- new_dict["wikitext 0-shot_word_perplexity"] = 0- new_dict["wikitext 0-shot_byte_perplexity"] = 0- new_dict["wikitext 0-shot_bits_per_byte"] = 0-- # Special handling of mmlu for compatibility with the old excel results sequence- search_str = 'hendrycksTest'- mmlu_matching_keys = [key for key in results.keys() if search_str.lower() in key.lower()]- for key in mmlu_matching_keys:- data = results.pop(key)- for sub_key in data.keys():- for sub_sub_key in data[sub_key].keys():- new_key = key + "-" + sub_key + "-" + sub_sub_key- if "std" in new_key or "alias" in new_key:- continue- new_key = new_key.split(",")[0]- new_dict[new_key] = data[sub_key][sub_sub_key]- new_key = new_key + "_norm"- new_dict[new_key] = data[sub_key][sub_sub_key]-- for key in results.keys():- if key == "model" or key == "paper-avg" or key == "leaderboard-avg":- continue- data = results[key]- if not isinstance(data, dict):- new_dict[key] = results[key]- continue- for sub_key in data.keys():- for sub_sub_key in data[sub_key].keys():- new_key = key + "_" + sub_sub_key- if "std" in new_key or "alias" in new_key:- continue- new_key = new_key.split(",")[0]- new_dict[new_key] = data[sub_key][sub_sub_key]--- import pandas as pd- df = pd.DataFrame(data=new_dict, index=[0])- df.to_excel(excel_file)---if __name__ == "__main__":- import time- import argparse- parser = argparse.ArgumentParser()- parser.add_argument(- "--model_name", default="facebook/opt-125m"- )- parser.add_argument(- "--eval_bs", default=1,- )- parser.add_argument(- "--tasks", default=['wikitext2', 'ptb-new', 'c4-new', 'lambada_openai', 'hellaswag', 'winogrande', 'piqa',- "mmlu", "wikitext", "truthfulqa_mc1", "truthfulqa_mc2", "openbookqa", "boolq", "rte",- "arc_easy", "arc_challenge"]- )- parser.add_argument(- "--excel_path", default=None,- help="The path to save eval results with excel format."- )-- args = parser.parse_args()- s = time.time()-- test_tasks = args.tasks- if isinstance(test_tasks, str):- test_tasks = test_tasks.split(',')- model_name = args.model_name.rstrip('/')- if args.excel_path is None:- excel_name = model_name.split('/')[-1] + ".xlsx"- else:- excel_name = args.excel_path- eval_model(model_path=args.model_name,- tasks=test_tasks,- eval_bs=args.eval_bs, limit=None, excel_file=excel_name)-- print("cost time: ", time.time() - s)-diff --git a/examples/language-modeling/eval_legacy/evaluation_legacy.py b/examples/language-modeling/eval_legacy/evaluation_legacy.pydeleted file mode 100644index 5aecf3c24..000000000--- a/examples/language-modeling/eval_legacy/evaluation_legacy.py+++ /dev/null@@ -1,373 +0,0 @@--import os-os.environ["TOKENIZERS_PARALLELISM"] = "false"-import os.path-import torch--import pprint-import re-import shutil-import transformers-if __name__ == "__main__":- import sys- sys.path.insert(0, './')-from eval_legacy.parse_results import result_parser-import time-EXT_TASKS = ['wikitext2', 'ptb', 'c4', 'ptb-new', 'c4-new']-fewshots_dict = {}-fewshots_dict['paper'] = {- "lambada_openai": [0],- "hellaswag": [0],- "winogrande": [0],- "piqa": [0],- "hendrycksTest-*": [0],- "wikitext": [0],- "truthfulqa_mc": [0],- "openbookqa": [0],- "boolq": [0],- "rte": [0],- "arc_easy": [0],- "arc_challenge": [0],-}-fewshots_dict['leadboard'] = {- "hellaswag": [10],- "winogrande": [5],- "arc_easy": [25],- "arc_challenge": [25],- "hendrycksTest-*": [5],- "drop": [3],- "gsm8k": [5],-}-fewshots_dict['all'] = {- "lambada_openai": [0],- "hellaswag": [0, 10],- "winogrande": [0, 5],- "piqa": [0],- "coqa": [], ## coqa is not enabled in llamav1 models- "truthfulqa_mc": [0],- "openbookqa": [0],- "boolq": [0],- "rte": [0],- "arc_easy": [0, 25],- "arc_challenge": [0, 25],- "hendrycksTest-*": [0, 5],- "wikitext": [0],- "drop": [3],- "gsm8k": [5]-}---def simple_evaluate(- model,- model_args=None,- tasks=[],- num_fewshot=0,- batch_size=None,- max_batch_size=None,- device=None,- no_cache=True,- limit=None,- bootstrap_iters=100000,- description_dict=None,- check_integrity=False,- decontamination_ngrams_path=None,- write_out=False,- output_base_path=None,- lm=None-):- """Instantiate and evaluate a model on a list of tasks.-- :param model: Union[str, LM]- Name of model, transformers.PreTrainedModel object, or LM object, see lm_eval.models.get_model- :param model_args: Optional[str]- String arguments for each model class, see LM.create_from_arg_string.- Ignored if `model` argument is a LM object.- :param tasks: list[Union[str, Task]]- List of task names or Task objects. Task objects will be taken to have name task.EVAL_HARNESS_NAME if defined and type(task).__name__ otherwise.- :param num_fewshot: int- Number of examples in few-shot context- :param batch_size: int or str, optional- Batch size for model- :param max_batch_size: int, optional- Maximal batch size to try with automatic batch size detection- :param device: str, optional- PyTorch device (e.g. "cpu" or "cuda:0") for running models- :param no_cache: bool- Whether or not to cache- :param limit: int or float, optional- Limit the number of examples per task (only use this for testing), If <1, limit is a percentage of the total number of examples.- :param bootstrap_iters:- Number of iterations for bootstrap statistics- :param description_dict: dict[str, str]- Dictionary of custom task descriptions of the form: `task_name: description`- :param check_integrity: bool- Whether to run the relevant part of the test suite for the tasks- :param write_out: bool- If True, write details about prompts and logits to json for all tasks- :param output_base_path: str, optional- Directory to which detailed eval info will be written. Defaults to present working dir.- :return- Dictionary of results- """- import collections- import itertools- import random-- import lm_eval.metrics- import lm_eval.models- import lm_eval.tasks- import lm_eval.base- from lm_eval.utils import positional_deprecated, run_task_tests- from lm_eval.models.gpt2 import HFLM-- import numpy as np- import transformers- from lm_eval.evaluator import evaluate- random.seed(1234)- np.random.seed(1234)-- assert tasks != [], "No tasks specified"- if lm == None:- if isinstance(model, str):- if model_args is None:- model_args = ""- lm = lm_eval.models.get_model(model).create_from_arg_string(- model_args,- {- "batch_size": batch_size,- "max_batch_size": max_batch_size,- "device": device,- },- )- elif isinstance(model, transformers.PreTrainedModel):- lm = lm_eval.models.get_model("hf-causal")(- pretrained=model,- batch_size=batch_size,- max_batch_size=max_batch_size,- )- no_cache = True- else:- assert isinstance(model, lm_eval.base.LM)- lm = model-- if not no_cache:- lm = lm_eval.base.CachingLM(- lm,- "lm_cache/"- + (model if isinstance(model, str) else model.model.config._name_or_path)- + "_"- + model_args.replace("=", "-").replace(",", "_").replace("/", "-")- + ".db",- )-- task_dict = lm_eval.tasks.get_task_dict(tasks)-- if check_integrity:- run_task_tests(task_list=tasks)-- results = evaluate(- lm=lm,- task_dict=task_dict,- num_fewshot=num_fewshot,- limit=limit,- bootstrap_iters=bootstrap_iters,- description_dict=description_dict,- decontamination_ngrams_path=decontamination_ngrams_path,- write_out=write_out,- output_base_path=output_base_path,- )-- # add info about the model and few shot config- model_name = None- if isinstance(model, str):- model_name = model- elif isinstance(model, transformers.PreTrainedModel):- model_name = "pretrained=" + model.config._name_or_path- results["config"] = {- "model": model_name,- "model_args": model_args,- "num_fewshot": num_fewshot,- "batch_size": batch_size,- "batch_sizes": list(lm.batch_sizes.values())- if hasattr(lm, "batch_sizes")- else [],- "device": device,- "no_cache": no_cache,- "limit": limit,- "bootstrap_iters": bootstrap_iters,- "description_dict": description_dict,- }-- return results, lm---def eval_model_legacy(model_path=None, tasks=["lambada_openai", "hellaswag", "winogrande", "piqa"],- eval_bs=32, use_accelerate=True, dtype=None, limit=None, trust_remote_code=True,- device="cuda:0", seed=0, nsamples=128, mark="paper", excel_file="tmp.xlsx"):- print("evaluation with official lm-eval", flush=True)- try:- import lm_eval- from lm_eval import evaluator- from lm_eval.tasks import ALL_TASKS, get_task_dict- import pandas as pd- except:- raise ImportError("""follow requirements to install dependencies.""")-- org_s = time.time()- if dtype == None:- from .utils import convert_dtype_torch2str_hf- from transformers import AutoConfig- config = AutoConfig.from_pretrained(model_path, trust_remote_code=trust_remote_code)- if hasattr(config, "torch_dtype"):- dtype = convert_dtype_torch2str_hf(config.torch_dtype)- else:- dtype = "float16"- print(f"Using {dtype} as evaluation data type.")--- external_tasks = []- if isinstance(tasks, str):- tasks = tasks.split(',')- for each in EXT_TASKS:- if each in tasks:- external_tasks.append(each)- tasks.remove(each)-- results = {}- model = None- lm = None- for tmp_tasks in tasks:- try:- num_fewshot = fewshots_dict[mark][tmp_tasks]- task_names = lm_eval.utils.pattern_match([tmp_tasks], ALL_TASKS)- print(f'********* {tmp_tasks} evaluate ************')- task_s = time.time()- for shot in num_fewshot:- if bool(re.search("chatglm", model_path.lower())):- model_args = f'pretrained={model_path},tokenizer={model_path},dtype={dtype},trust_remote_code={trust_remote_code}'- model_type = "hf-causal"- else:- model_args = f'pretrained={model_path},tokenizer={model_path},dtype={dtype},use_accelerate={use_accelerate},trust_remote_code={trust_remote_code}'- model_type = "hf-causal-experimental"-- if "wikitext" in task_names:- tmp_eval_bs = 1- else:- tmp_eval_bs = eval_bs- tmp_results, lm = simple_evaluate(model=model_type, model_args=model_args, tasks=task_names,- num_fewshot=shot, limit=limit, batch_size=tmp_eval_bs,- max_batch_size=tmp_eval_bs, lm=lm, device=str(device))- sub_name = f'{tmp_tasks} {shot}-shot'- print(f'{sub_name}: ')- pprint.pprint(tmp_results["results"])- print(f"\n{sub_name} cost time: {time.time() - task_s}\n")- results[sub_name] = tmp_results- except Exception as e:- print(f'********* {tmp_tasks} ERROR ************')- print(str(e))- continue-- tokenizer = transformers.AutoTokenizer.from_pretrained(model_path, use_fast=False, trust_remote_code=True)- model = lm.model- # for external tasks- # maybe adjust for specific model- # if hasattr(lm.model.config, "max_position_embeddings"):- # lm.model.seqlen = lm.model.config.max_position_embeddings- # else:- # ## for llama-1, opt- # lm.model.seqlen = 2048-- # if "opt" in model_name:- # seqlen = model.config.max_position_embeddings- # model.seqlen = model.config.max_position_embeddings- # else:- # seqlen = 2048- # model.seqlen = seqlen-- model.seqlen = 2048- from .utils import get_loaders, eval_ppl_same_with_gptq- for dataset in external_tasks:- try:- dataloader, testloader = get_loaders(- dataset, nsamples=nsamples, seed=seed,- tokenizer=tokenizer, seqlen=model.seqlen- )- ppl = eval_ppl_same_with_gptq(model, testloader, device)- print(dataset, ppl)-- results.update({dataset: ppl})- except Exception as e:- print(str(e))- continue-- print(results, flush=True)- new_results = result_parser(results)- print("cost time: ", time.time() - org_s)- import pickle- from collections import OrderedDict- new_dict = OrderedDict()- new_dict["model"] = "tmp"- new_dict["paper-avg"] = 0- new_dict["leaderboard-avg"] = 0- new_dict["wikitext2"] = 0- new_dict["ptb-new"] = 0- new_dict["c4-new"] = 0- new_dict["wikitext 0-shot_word_perplexity"] = 0- new_dict["wikitext 0-shot_byte_perplexity"] = 0- new_dict["wikitext 0-shot_bits_per_byte"] = 0- for key in new_results.keys():- if key == "model" or key == "paper-avg" or key == "leaderboard-avg":- continue- data = new_results[key]- if not isinstance(data, dict):- new_dict[key] = new_results[key]- continue- for sub_key in data.keys():- for sub_sub_key in data[sub_key].keys():- if "hendry" in key:- new_key = key + "-" + sub_key + "-" + sub_sub_key- else:- new_key = key + "_" + sub_sub_key- if "std" in new_key:- continue- # if "norm" in new_key:- # continue- new_dict[new_key] = data[sub_key][sub_sub_key]-- df = pd.DataFrame(data=new_dict, index=[0])- df.to_excel(excel_file)---if __name__ == "__main__":- import time- import argparse- parser = argparse.ArgumentParser()- parser.add_argument(- "--model_name", default="facebook/opt-125m"- )- parser.add_argument(- "--eval_bs", default=1,- )-- args = parser.parse_args()- s = time.time()-- test_tasks = ['wikitext2', 'ptb-new', 'c4-new', 'lambada_openai', 'hellaswag', 'winogrande', 'piqa',- "hendrycksTest-*", "wikitext", "truthfulqa_mc", "openbookqa", "boolq", "rte", "arc_easy",- "arc_challenge"]- seen = set()- tmp_tasks = test_tasks- test_tasks = [x for x in tmp_tasks if not (x in seen or seen.add(x))]- model_name = args.model_name.rstrip('/')- excel_name = model_name.split('/')[-1] + ".xlsx"- eval_model_legacy(model_path=args.model_name,- tasks=test_tasks,- eval_bs=args.eval_bs, limit=None, excel_file=excel_name)-- print("cost time: ", time.time() - s)------diff --git a/examples/language-modeling/eval_legacy/parse_results.py b/examples/language-modeling/eval_legacy/parse_results.pydeleted file mode 100644index a1f237392..000000000--- a/examples/language-modeling/eval_legacy/parse_results.py+++ /dev/null@@ -1,140 +0,0 @@--import sys-import json-import pprint--# input_file = sys.argv[1]--# with open(input_file) as f:-# data = json.load(f)--# data = data["results"]--task_domain = {- "abstract_algebra": ["STEM"],- "anatomy": ["Other"],- "astronomy": ["STEM"],- "business_ethics": ["Other"],- "clinical_knowledge": ["Other"],- "college_biology": ["STEM"],- "college_chemistry": ["STEM"],- "college_computer_science": ["STEM"],- "college_mathematics": ["STEM"],- "college_medicine": ["Other"],- "college_physics": ["STEM"],- "computer_security": ["STEM"],- "conceptual_physics": ["STEM"],- "econometrics": ["Social"],- "electrical_engineering": ["STEM"],- "elementary_mathematics": ["STEM"],- "formal_logic": ["Hums"],- "global_facts": ["Other"],- "high_school_biology": ["STEM"],- "high_school_chemistry": ["STEM"],- "high_school_computer_science": ["STEM"],- "high_school_european_history": ["Hums"],- "high_school_geography": ["Social"],- "high_school_government_and_politics": ["Social"],- "high_school_macroeconomics": ["Social"],- "high_school_mathematics": ["STEM"],- "high_school_microeconomics": ["Social"],- "high_school_physics": ["STEM"],- "high_school_psychology": ["Social"],- "high_school_statistics": ["STEM"],- "high_school_us_history": ["Hums"],- "high_school_world_history": ["Hums"],- "human_aging": ["Other"],- "human_sexuality": ["Social"],- "international_law": ["Hums"],- "jurisprudence": ["Hums"],- "logical_fallacies": ["Hums"],- "machine_learning": ["STEM"],- "management": ["Other"],- "marketing": ["Other"],- "medical_genetics": ["Other"],- "miscellaneous": ["Other"],- "moral_disputes": ["Hums"],- "moral_scenarios": ["Hums"],- "nutrition": ["Other"],- "philosophy": ["Hums"],- "prehistory": ["Hums"],- "professional_accounting": ["Other"],- "professional_law": ["Hums"],- "professional_medicine": ["Other"],- "professional_psychology": ["Social"],- "public_relations": ["Social"],- "security_studies": ["Social"],- "sociology": ["Social"],- "us_foreign_policy": ["Social"],- "virology": ["Other"],- "world_religions": ["Hums"],-}---def result_parser(data):- mmlu_results = {}- other_results = {}- for task in data.keys():- acc_dict = {"Hums": [0, 0., 0.], "STEM": [0, 0., 0.], "Social": [0, 0., 0.], "Other": [0, 0., 0.],- "Avg.": [0, 0., 0.]}- if isinstance(data[task], float):- other_results[task] = data[task]- else:- results = data[task]['results']- if "hendrycksTest" in task:- for sub_task in results.keys():- sub_name = sub_task.split("hendrycksTest-")[1]- domain = task_domain[sub_name][0]- domain_list = acc_dict[domain]- domain_list[0] += 1- domain_list[1] += results[sub_task]["acc"]- domain_list[2] += results[sub_task]["acc_norm"]- acc_dict['Avg.'][0] += 1- acc_dict['Avg.'][1] += results[sub_task]["acc"]- acc_dict['Avg.'][2] += results[sub_task]["acc_norm"]-- mmlu_results[task] = {}- for key in acc_dict.keys():- mmlu_results[task][key] = {}- mmlu_results[task][key]['acc'] = acc_dict[key][1] / acc_dict[key][0]- mmlu_results[task][key]['acc_norm'] = acc_dict[key][2] / acc_dict[key][0]- else:- other_results[task] = results-- # other_results = {}- # for key in results.keys():- # if 'hendrycksTest' in key:- # sub_name = key.split("hendrycksTest-")[1]- # domain=task_domain[sub_name][0]- # domain_list = acc_dict[domain]- # domain_list[0] += 1- # domain_list[1] += results[key]["acc"]- # domain_list[2] += results[key]["acc_norm"]- # acc_dict['Avg.'][0] += 1- # acc_dict['Avg.'][1] += results[key]["acc"]- # acc_dict['Avg.'][2] += results[key]["acc_norm"]- # else:- # other_results[key] = results[key]-- # mmlu_results = {}- # for key in acc_dict.keys():- # mmlu_results[key] = {}- # mmlu_results[key]['acc'] = acc_dict[key][1]/acc_dict[key][0]- # mmlu_results[key]['acc_norm'] = acc_dict[key][2]/acc_dict[key][0]-- print("***** MMLU RESULTS *****")- pprint.pprint(mmlu_results)- print()-- print("***** OTHER RESULTS *****")- pprint.pprint(other_results)- print()- new_results = mmlu_results- new_results.update(other_results)- return new_results------diff --git a/examples/language-modeling/eval_legacy/utils.py b/examples/language-modeling/eval_legacy/utils.pydeleted file mode 100644index 9514c811c..000000000--- a/examples/language-modeling/eval_legacy/utils.py+++ /dev/null@@ -1,298 +0,0 @@--import numpy as np-import torch-import torch.nn as nn--import datasets-def set_seed(seed):- np.random.seed(seed)- torch.random.manual_seed(seed)---def get_wikitext2(nsamples, seed, seqlen, tokenizer):- from datasets import load_dataset- traindata = load_dataset('wikitext', 'wikitext-2-raw-v1', split='train')- testdata = load_dataset('wikitext', 'wikitext-2-raw-v1', split='test')-- trainenc = tokenizer("\n\n".join(traindata['text']), return_tensors='pt')- testenc = tokenizer("\n\n".join(testdata['text']), return_tensors='pt')-- import random- random.seed(seed)- trainloader = []- for _ in range(nsamples):- i = random.randint(0, trainenc.input_ids.shape[1] - seqlen - 1)- j = i + seqlen- inp = trainenc.input_ids[:, i:j]- tar = inp.clone()- tar[:, :-1] = -100- trainloader.append((inp, tar))- return trainloader, testenc--def get_ptb(nsamples, seed, seqlen, tokenizer):- from datasets import load_dataset- traindata = load_dataset('ptb_text_only', 'penn_treebank', split='train')- valdata = load_dataset('ptb_text_only', 'penn_treebank', split='validation')-- trainenc = tokenizer("\n\n".join(traindata['sentence']), return_tensors='pt')- testenc = tokenizer("\n\n".join(valdata['sentence']), return_tensors='pt')-- import random- random.seed(seed)- trainloader = []- for _ in range(nsamples):- i = random.randint(0, trainenc.input_ids.shape[1] - seqlen - 1)- j = i + seqlen- inp = trainenc.input_ids[:, i:j]- tar = inp.clone()- tar[:, :-1] = -100- trainloader.append((inp, tar))- return trainloader, testenc--def get_c4(nsamples, seed, seqlen, tokenizer):- from datasets import load_dataset- from packaging import version- if version.parse(datasets.__version__)>=version.parse("2.16.1"):- traindata = load_dataset(- 'allenai/c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train'- )- valdata = load_dataset(- 'allenai/c4', data_files={'validation': 'en/c4-validation.00000-of-00008.json.gz'}, split='validation'- )- else:- traindata = load_dataset(- 'allenai/c4', 'allenai--c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train'- )- valdata = load_dataset(- 'allenai/c4', 'allenai--c4', data_files={'validation': 'en/c4-validation.00000-of-00008.json.gz'}, split='validation'- )-- import random- random.seed(seed)- trainloader = []- for _ in range(nsamples):- while True:- i = random.randint(0, len(traindata) - 1)- trainenc = tokenizer(traindata[i]['text'], return_tensors='pt')- if trainenc.input_ids.shape[1] >= seqlen:- break- i = random.randint(0, trainenc.input_ids.shape[1] - seqlen - 1)- j = i + seqlen- inp = trainenc.input_ids[:, i:j]- tar = inp.clone()- tar[:, :-1] = -100- trainloader.append((inp, tar))-- import random- random.seed(0)- valenc = []- for _ in range(256):- while True:- i = random.randint(0, len(valdata) - 1)- tmp = tokenizer(valdata[i]['text'], return_tensors='pt')- if tmp.input_ids.shape[1] >= seqlen:- break- i = random.randint(0, tmp.input_ids.shape[1] - seqlen - 1)- j = i + seqlen- valenc.append(tmp.input_ids[:, i:j])- valenc = torch.hstack(valenc)- class TokenizerWrapper:- def __init__(self, input_ids):- self.input_ids = input_ids- valenc = TokenizerWrapper(valenc)-- return trainloader, valenc--def get_ptb_new(nsamples, seed, seqlen, tokenizer):- from datasets import load_dataset- traindata = load_dataset('ptb_text_only', 'penn_treebank', split='train')- testdata = load_dataset('ptb_text_only', 'penn_treebank', split='test')-- trainenc = tokenizer(" ".join(traindata['sentence']), return_tensors='pt')- testenc = tokenizer(" ".join(testdata['sentence']), return_tensors='pt')-- import random- random.seed(seed)- trainloader = []- for _ in range(nsamples):- i = random.randint(0, trainenc.input_ids.shape[1] - seqlen - 1)- j = i + seqlen- inp = trainenc.input_ids[:, i:j]- tar = inp.clone()- tar[:, :-1] = -100- trainloader.append((inp, tar))- return trainloader, testenc--def get_c4_new(nsamples, seed, seqlen, tokenizer):-- from datasets import load_dataset- from packaging import version- if version.parse(datasets.__version__)>=version.parse("2.16.1"):- traindata = load_dataset(- 'allenai/c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train'- )- valdata = load_dataset(- 'allenai/c4', data_files={'validation': 'en/c4-validation.00000-of-00008.json.gz'}, split='validation'- )- else:- traindata = load_dataset(- 'allenai/c4', 'allenai--c4', data_files={'train': 'en/c4-train.00000-of-01024.json.gz'}, split='train'- )- valdata = load_dataset(- 'allenai/c4', 'allenai--c4', data_files={'validation': 'en/c4-validation.00000-of-00008.json.gz'}, split='validation'- )-- import random- random.seed(seed)- trainloader = []- for _ in range(nsamples):- while True:- i = random.randint(0, len(traindata) - 1)- trainenc = tokenizer(traindata[i]['text'], return_tensors='pt')- if trainenc.input_ids.shape[1] >= seqlen:- break- i = random.randint(0, trainenc.input_ids.shape[1] - seqlen - 1)- j = i + seqlen- inp = trainenc.input_ids[:, i:j]- tar = inp.clone()- tar[:, :-1] = -100- trainloader.append((inp, tar))-- valenc = tokenizer(' '.join(valdata[:1100]['text']), return_tensors='pt')- valenc = valenc.input_ids[:, :(256 * seqlen)]-- class TokenizerWrapper:- def __init__(self, input_ids):- self.input_ids = input_ids- valenc = TokenizerWrapper(valenc)-- return trainloader, valenc---def get_loaders(- name, nsamples=128, seed=0, seqlen=2048, tokenizer=''-):- if 'wikitext2' in name:- return get_wikitext2(nsamples, seed, seqlen, tokenizer)- if 'ptb' in name:- if 'new' in name:- return get_ptb_new(nsamples, seed, seqlen, tokenizer)- return get_ptb(nsamples, seed, seqlen, tokenizer)- if 'c4' in name:- if 'new' in name:- return get_c4_new(nsamples, seed, seqlen, tokenizer)- return get_c4(nsamples, seed, seqlen, tokenizer)--[email protected]_grad()-def eval_ppl_same_with_gptq(model, testenc, dev):- print('Evaluating ...', flush=True)-- testenc = testenc.input_ids- nsamples = testenc.numel() // model.seqlen- use_cache = None- if hasattr(model.config, "use_cache"):- use_cache = model.config.use_cache- model.config.use_cache = False-- testenc = testenc.to(dev)- nlls = []- for i in range(nsamples):- batch = testenc[:, (i * model.seqlen):((i + 1) * model.seqlen)].to(dev)- lm_logits = model(batch).logits- # to main gpu for multi-gpus- shift_logits = lm_logits[:, :-1, :].contiguous().to(dev)- shift_labels = testenc[- :, (i * model.seqlen):((i + 1) * model.seqlen)- ][:, 1:].to(dev)- loss_fct = nn.CrossEntropyLoss()- loss = loss_fct(shift_logits.view(-1, shift_logits.size(-1)), shift_labels.view(-1))- neg_log_likelihood = loss.float() * model.seqlen- nlls.append(neg_log_likelihood)- ppl = torch.exp(torch.stack(nlls).sum() / (nsamples * model.seqlen))- if hasattr(model.config, "use_cache"):- model.config.use_cache = use_cache- return ppl.item()---def convert_dtype_str2torch(str_dtype):- """Converts a string dtype to its corresponding PyTorch dtype.-- Args:- str_dtype (str): The string representation of the dtype.-- Returns:- torch.dtype: The PyTorch dtype.-- Raises:- AssertionError: If the input str_dtype is unsupported.- """- if isinstance(str_dtype, torch.dtype) or str_dtype is None:- return str_dtype- if str_dtype == "int8":- return torch.int8- elif str_dtype == "fp32" or str_dtype == "auto":- return torch.float- elif str_dtype == "fp16":- return torch.float16- elif str_dtype == "bf16":- return torch.bfloat16- else:- assert False, "Unsupported str dtype {} to torch dtype".format(str_dtype)---def convert_dtype_torch2str(dtype):- """Converts a PyTorch dtype to its corresponding string representation.-- Args:- dtype: PyTorch dtype or str. The dtype to convert.-- Returns:- str: The string representation of the dtype.-- Raises:- AssertionError: If the input dtype is unsupported.- """- if isinstance(dtype, str) or dtype is None:- return dtype- if dtype == torch.int8:- return "int8"- elif dtype == torch.float:- return "fp32"- elif dtype == torch.float16:- return "fp16"- elif dtype == torch.bfloat16:- return "bf16"- elif isinstance(dtype, str) and dtype in ["int8", "fp32", "fp16", "bf16"]:- return dtype- else:- assert False, "Unsupported pytorch dtype {} to str dtype".format(dtype)---def convert_dtype_torch2str_hf(dtype):- """Converts a PyTorch dtype to its corresponding huggingface string dtype, e.g. torch.float32 -> 'float32'.-- Args:- dtype: PyTorch dtype or str. The dtype to convert.-- Returns:- str: The string representation of the dtype.-- Raises:- AssertionError: If the input str_dtype is unsupported.- """- if dtype is None:- return dtype- if isinstance(dtype, str):- if "float" not in dtype and "int" not in dtype:- dtype = convert_dtype_str2torch(dtype)- else:- return dtype- str_dtype = str(dtype)- if "." not in str_dtype:- assert False, "Unsupported pytorch dtype {} to huggingface str dtype".format(dtype)- str_dtype = str_dtype.split(".")[1]- return str_dtype---diff --git a/examples/language-modeling/main.py b/examples/language-modeling/main.pydeleted file mode 100644index 88c4bf320..000000000--- a/examples/language-modeling/main.py+++ /dev/null@@ -1,442 +0,0 @@-import os-import re-import sys-import argparse-import subprocess-from packaging import version--sys.path.insert(0, '../..')-parser = argparse.ArgumentParser()--if __name__ == '__main__':-- parser.add_argument(- "--model_name", "--model", "--model_name_or_path", default="facebook/opt-125m"- )-- parser.add_argument("--bits", default=4, type=int,- help="number of bits")-- parser.add_argument("--group_size", default=128, type=int,- help="group size")-- parser.add_argument("--batch_size", "--bs", "--train_bs", default=8, type=int,- help="train batch size")-- parser.add_argument("--eval_bs", default=None, type=int,- help="eval batch size")-- parser.add_argument("--device", "--devices", default="auto", type=str,- help="The device to be used for tuning. The default is set to auto/None,"- "allowing for automatic detection. Currently, device settings support CPU, GPU, and HPU.")-- parser.add_argument("--asym", action='store_true',- help=" asym quantization")-- parser.add_argument("--iters", "--iter", default=200, type=int,- help=" iters")-- parser.add_argument("--dataset", default="NeelNanda/pile-10k", type=str,- help="The dataset for quantization training. It can be a custom one.")-- parser.add_argument("--enable_quanted_input", action='store_true',- help="enable_quanted_input is deprecated.")-- parser.add_argument("--lr", default=None, type=float,- help="learning rate, if None, it will be set to 1.0/iters automatically")-- parser.add_argument("--minmax_lr", default=None, type=float,- help="minmax learning rate, if None,it will beset to be the same with lr")-- parser.add_argument("--seed", default=42, type=int,- help="seed")-- parser.add_argument("--amp", action='store_true',- help="amp is deprecated ")-- parser.add_argument("--adam", action='store_true',- help="adam")-- parser.add_argument("--seqlen", "--seq_len", default=2048, type=int,- help="sequence length")-- parser.add_argument("--gradient_accumulate_steps", default=1, type=int, help="gradient accumulate steps")-- parser.add_argument("--nblocks", default=1, type=int, help="num of blocks to tune together")-- parser.add_argument("--nsamples", default=128, type=int,- help="number of samples")-- parser.add_argument("--low_gpu_mem_usage", action='store_true',- help="offload intermediate features to cpu")-- parser.add_argument("--enable_minmax_tuning", action='store_true',- help="enable_minmax_tuning is deprecated")-- parser.add_argument("--format", default="auto_round", type=str,- help="the format to save the quantized model"- )-- parser.add_argument("--data_type", "--dtype", default='int',- help="data type for tuning, 'int', 'mx_fp' and etc.")-- parser.add_argument("--scale_dtype", default='fp16',- help="which scale data type to use for quantization, 'fp16', 'fp32' or 'bf16'.")-- parser.add_argument("--tasks",- default="lambada_openai,hellaswag,winogrande,piqa,mmlu,wikitext,truthfulqa_mc1," \- "truthfulqa_mc2,openbookqa,boolq,rte,arc_easy,arc_challenge",- help="lm-eval tasks for lm_eval version 0.4")-- parser.add_argument("--output_dir", default="./tmp_autoround", type=str,- help="the directory to save quantized model")-- parser.add_argument("--disable_eval", action='store_true',- help="whether to do lmeval evaluation.")-- parser.add_argument("--disable_amp", action='store_true',- help="disable amp")-- parser.add_argument("--disable_minmax_tuning", action='store_true',- help="whether disable enable weight minmax tuning")-- parser.add_argument("--enable_norm_bias_tuning", action='store_true',- help="whether enable norm bias tuning")-- parser.add_argument("--disable_trust_remote_code", action='store_true',- help="Whether to disable trust_remote_code")-- parser.add_argument("--disable_quanted_input", action='store_true',- help="whether to disuse the output of quantized block to tune the next block")-- parser.add_argument("--quant_lm_head", action='store_true',- help="quant_lm_head")-- parser.add_argument("--low_cpu_mem_mode", default=0, type=int,- help="Choose which low cpu memory mode to use. Can significantly reduce cpu memory footprint but cost more time."- "1 means choose block-wise mode, load the weights of each block from disk when tuning and release the memory of the block after tuning."- "2 means choose layer-wise mode, load the weights of each layer from disk when tuning, minimum memory consumption and also slowest running speed."- "others means not use low cpu memory. Default to 0, not use low cpu memory.")-- parser.add_argument("--low_cpu_mem_tmp_dir", default=None, type=str,- help="temp work space to store the temporary files when using low cpu memory mode. Will remove after tuning.")-- parser.add_argument("--model_dtype", default=None, type=str,- help="force to convert the dtype, some backends supports fp16 dtype better")-- parser.add_argument("--act_bits", default=16, type=int,- help="activation bits")-- parser.add_argument("--fp_layers", default="", type=str,- help="List of Layers to maintain original data type")-- parser.add_argument("--enable_torch_compile", default=None, type=bool,- help="whether to enable torch compile")-- args = parser.parse_args()-- print(- "Warning, examples/language-modeling/main.py is deprecated, please use auto-round cmd line instead. The file will be deleted in the V0.4.2 release ")-- if args.enable_minmax_tuning:- print(- "enable_minmax_tuning is deprecated, it has been set to the default, use disable_minmax_tuning to turn it off")- if args.amp:- print(- "amp is deprecated, it has been set to the default, use disable_amp to turn it off")- if args.enable_quanted_input:- print(- "enable_quanted_input is deprecated. It has been set to the default; use disable_quanted_input to turn it off")-- if args.act_bits <= 8:- print(- "Warning, activation quantization is an experiment feature")-- if args.format is None:- args.format = "auto_round"- supported_formats = ["auto_round", "auto_gptq", "auto_awq", "auto_round:auto_gptq", "auto_round:auto_awq",- "auto_gptq:marlin", "itrex", "iterx_xpu", "fake"]- formats = args.format.replace(' ', '').split(",")- for format in formats:- if format not in supported_formats:- raise ValueError(f"{format} is not supported, we only support {supported_formats}")- devices = args.device.replace(" ", "").split(',')- use_auto_mapping = False- if all(s.isdigit() for s in devices):- if "CUDA_VISIBLE_DEVICES" in os.environ:- current_visible_devices = os.environ["CUDA_VISIBLE_DEVICES"]- current_visible_devices = current_visible_devices.split(',')- indices = [int(device) for device in devices]- try:- pick_device = [current_visible_devices[i] for i in indices]- except:- raise ValueError(- "Invalid '--device' value: It must be smaller than the number of available devices. "- "For example, with CUDA_VISIBLE_DEVICES=4,5, --device 0,1 is valid, but --device 4,5 is not supported.")- visible_devices = ','.join(pick_device)- os.environ["CUDA_VISIBLE_DEVICES"] = visible_devices- else:- os.environ["CUDA_VISIBLE_DEVICES"] = args.device- args.device = ",".join(map(str, range(len(devices))))- devices = args.device.replace(" ", "").split(',')- if len(devices) > 1:- use_auto_mapping = True-- import torch- import transformers-- os.environ["CUBLAS_WORKSPACE_CONFIG"] = ":4096:8"- torch.use_deterministic_algorithms(True, warn_only=True)- from transformers import AutoModelForCausalLM, AutoTokenizer, AutoModel-- import logging- import warnings- import numexpr-- warnings.filterwarnings('ignore', category=DeprecationWarning)- warnings.filterwarnings('ignore', category=FutureWarning)- os.environ['TF_CPP_MIN_LOG_LEVEL'] = '3'- dataset_logger = logging.getLogger("datasets")- dataset_logger.disabled = True- numexpr_logger = logging.getLogger("numexpr")- numexpr_logger.disabled = True-- os.environ["TOKENIZERS_PARALLELISM"] = "false"-- tasks = args.tasks- use_eval_legacy = False-- model_name = args.model_name- if model_name[-1] == "/":- model_name = model_name[:-1]- print(model_name, flush=True)-- from auto_round.utils import detect_device-- device_str = detect_device(devices[0])-- torch_dtype = "auto"- if "hpu" in device_str:- torch_dtype = torch.bfloat16- torch_device = torch.device(device_str)-- is_glm = bool(re.search("chatglm", model_name.lower()))- low_cpu_mem_usage = False- model_cls = AutoModel if is_glm else AutoModelForCausalLM- if args.low_cpu_mem_tmp_dir is None:- args.low_cpu_mem_tmp_dir = os.path.join(args.output_dir, "low_cpu_mem_tmp")- if args.low_cpu_mem_mode == 2:- from auto_round.low_cpu_mem.utils import load_model_with_hooks-- model = load_model_with_hooks(- model_name,- model_cls,- device=device_str,- clean_weight=True,- saved_path=args.low_cpu_mem_tmp_dir,- torch_dtype=torch_dtype,- trust_remote_code=not args.disable_trust_remote_code- )- elif args.low_cpu_mem_mode == 1:- from auto_round.low_cpu_mem.utils import load_empty_model-- low_cpu_mem_usage = True- model = load_empty_model(- model_name,- model_cls,- device=device_str,- saved_path=args.low_cpu_mem_tmp_dir,- torch_dtype=torch_dtype,- trust_remote_code=not args.disable_trust_remote_code- )- else:-- model = model_cls.from_pretrained(- model_name, low_cpu_mem_usage=True, torch_dtype=torch_dtype,- trust_remote_code=not args.disable_trust_remote_code, device_map="auto" if use_auto_mapping else None- )-- from auto_round import (AutoRound,- AutoRoundAdam)-- model = model.eval()-- seqlen = args.seqlen-- if args.model_dtype != None:- try:- if args.model_dtype == "float16" or args.model_dtype == "fp16":- model = model.to(torch.float16)- elif args.model_dtype == "bfloat16" or args.model_dtype == "bfp16" or args.model_dtype == "bf16":- model = model.to(torch.bfloat16)- elif args.model_dtype == "float32" or args.model_dtype == "fp32":- model = model.to(torch.float32)- except:- print("please use more device to fit the device or just use one device")- exit(-1)-- tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=not args.disable_trust_remote_code)-- if hasattr(tokenizer, "model_max_length"):- if tokenizer.model_max_length < seqlen:- print(f"change sequence length to {tokenizer.model_max_length} due to the limitation of model_max_length",- flush=True)- seqlen = min(seqlen, tokenizer.model_max_length)- args.seqlen = seqlen-- excel_name = f"{model_name}_{args.bits}_{args.group_size}"- if (hasattr(model, 'config') and (model.dtype is torch.bfloat16 or model.config.torch_dtype is torch.bfloat16)):- dtype = 'bfloat16'- else:- if "cpu" not in device_str:- dtype = 'float16'- else:- dtype = 'float32'-- excel_name = f"{model_name}_{args.bits}_{args.group_size}"- if "bloom" in model_name:- args.low_gpu_mem_usage = False-- round = AutoRound- if args.adam:- round = AutoRoundAdam-- layer_config = {}- if args.fp_layers != "":- fp_layers = args.fp_layers.replace(" ", "").split(",")- for n, m in model.named_modules():- if not isinstance(m, (torch.nn.Linear, transformers.modeling_utils.Conv1D)):- continue- for fp_layer in fp_layers:- if fp_layer in n:- layer_config[n] = {"bits": 16}- print(- f"{n} will not be quantized.")- if len(layer_config) > 0:- for format in formats:- if "auto_round" not in format and "fake" not in format:- ##TODO gptq, awq could support some mixed precision config- print(f"mixed precision exporting does not support {format} currently")-- for n, m in model.named_modules():- if isinstance(m, torch.nn.Linear) or isinstance(m, transformers.modeling_utils.Conv1D):- if m.weight.shape[0] % 32 != 0 or m.weight.shape[1] % 32 != 0:- layer_config[n] = {"bits": 16}- print(- f"{n} will not be quantized due to its shape not being divisible by 32, resulting in an exporting issue to autogptq")-- lm_head_layer_name = "lm_head"- for n, _ in model.named_modules():- lm_head_layer_name = n- if args.quant_lm_head:- from transformers import AutoConfig-- config = AutoConfig.from_pretrained(model_name, trust_remote_code=not args.disable_trust_remote_code)- if config.tie_word_embeddings and hasattr(model, "_tied_weights_keys"):- tied_keys = model._tied_weights_keys- for item in tied_keys:- if lm_head_layer_name in item: ##TODO extend to encoder-decoder layer, seq classification model- args.quant_lm_head = False- print(- f"warning, disable quant_lm_head as quantizing lm_head with tied weights has not been "- f"supported currently")- break- if args.quant_lm_head:- layer_config[lm_head_layer_name] = {"bits": args.bits}- for format in formats:- if "auto_round" not in format:- auto_round_formats = [s for s in supported_formats if s.startswith("auto_round")]- raise ValueError(- f"{format} is not supported for lm-head quantization, please change to {auto_round_formats}")-- autoround = round(model, tokenizer, args.bits, args.group_size, sym=not args.asym, batch_size=args.batch_size,- dataset=args.dataset, seqlen=seqlen, nblocks=args.nblocks, iters=args.iters, lr=args.lr,- minmax_lr=args.minmax_lr, enable_quanted_input=not args.disable_quanted_input, device=device_str,- amp=not args.disable_amp, nsamples=args.nsamples,- low_gpu_mem_usage=args.low_gpu_mem_usage,- seed=args.seed, gradient_accumulate_steps=args.gradient_accumulate_steps,- scale_dtype=args.scale_dtype, layer_config=layer_config,- enable_minmax_tuning=not args.disable_minmax_tuning, act_bits=args.act_bits,- low_cpu_mem_usage=low_cpu_mem_usage, data_type=args.data_type,- enable_norm_bias_tuning=args.enable_norm_bias_tuning,- enable_torch_compile=args.enable_torch_compile)- model, _ = autoround.quantize()- model_name = args.model_name.rstrip("/")- if args.low_cpu_mem_mode == 1 or args.low_cpu_mem_mode == 2:- import shutil-- shutil.rmtree(args.low_cpu_mem_tmp_dir, ignore_errors=True)-- model.eval()- if "cpu" not in device_str:- torch.cuda.empty_cache()-- if model_name.split('/')[-1].strip(".") == "":- export_dir = os.path.join(args.output_dir, f"w{args.bits}g{args.group_size}")- else:- export_dir = os.path.join(args.output_dir, model_name.split('/')[-1] + f"-w{args.bits}g{args.group_size}")-- eval_folder = None-- format_list = args.format.replace(' ', '').split(',')- inplace = False if len(format_list) > 1 else True- for format_ in format_list:- save_format_ = format_.replace(":", "-")- save_format_ = save_format_.replace("_", "-")- eval_folder = f'{export_dir}-{save_format_}'- autoround.save_quantized(eval_folder, format=format_, inplace=inplace)-- if args.disable_eval:- exit()-- from packaging.version import Version- from auto_round.utils import get_library_version-- lm_eval_version = get_library_version("lm_eval")- lm_eval_version = Version(lm_eval_version)- if lm_eval_version == Version("0.3.0"):- use_eval_legacy = True- from eval_legacy import eval_model_legacy as eval_model- else:- use_eval_legacy = False- from eval_legacy import eval_model-- # evaluation- if use_eval_legacy:- print("Using the legacy lm_eval(0.3.0)")- else:- print(f"Using the lm_eval version {lm_eval_version}")-- if isinstance(tasks, str):- tasks = tasks.split(',')-- if lm_eval_version >= Version("0.4.2"):- if args.eval_bs is None:- args.eval_bs = "auto"- from eval.evaluation import simple_evaluate-- model_args = f"pretrained={eval_folder}"- model_args = model_args + f",trust_remote_code={not args.disable_trust_remote_code}"- if use_auto_mapping:- model_args += ",parallelize=True"- user_model = None- if args.act_bits <= 8:- if hasattr(model, "hf_device_map") and len(model.hf_device_map) > 1:- from accelerate.big_modeling import dispatch_model-- dispatch_model(model, model.hf_device_map)- user_model = model- else:- user_model = model.to(device_str)- if args.eval_bs == "auto":- args.eval_bs = 16- from auto_round.eval.evaluation import simple_evaluate_user_model-- res = simple_evaluate_user_model(user_model, tokenizer, tasks=tasks, batch_size=args.eval_bs)- else:- res = simple_evaluate(model="hf", model_args=model_args,- tasks=tasks,- batch_size=args.eval_bs)- from lm_eval.utils import make_table-- print(make_table(res))diff --git a/examples/language-modeling/requirements.txt b/examples/language-modeling/requirements.txtdeleted file mode 100644index 13edc765f..000000000--- a/examples/language-modeling/requirements.txt+++ /dev/null@@ -1,24 +0,0 @@-transformers-torch-lm-eval==0.4.4-##for lm-eval 0.4.1 we usually used-##git+https://github.com/EleutherAI/lm-evaluation-harness.git@96d185fa6232a5ab685ba7c43e45d1dbb3bb906d-# For the paper results use the old lm_eval (0.3.0)-# git+https://github.com/EleutherAI/lm-evaluation-harness.git@008fc2a23245c40384f2312718433eeb1e0f87a9-tiktoken-transformers_stream_generator-peft-sentencepiece-einops-accelerate-datasets-protobuf-auto-gptq-openpyxl-wandb-py-cpuinfo-numpy < 2.0-threadpoolctl-numexpr-bitsandbytes # for baichuan Series-diff --git a/examples/language-modeling/run_autoround.sh b/examples/language-modeling/run_autoround.shdeleted file mode 100644index 2f875ff51..000000000--- a/examples/language-modeling/run_autoround.sh+++ /dev/null@@ -1,16 +0,0 @@-#!/bin/bash-set -x-device=0-model_name="Intel/neural-chat-7b-v3-3"--CUDA_VISIBLE_DEVICES=$device \-python3 main.py \---model_name $model_name \---bits 4 \---group_size 128 \---iters 200 \---format 'auto_round,auto_gptq' \---eval_bs 16 \---output_dir "./tmp_autoround"--diff --git a/examples/language-modeling/run_autoround_on_gaudi.sh b/examples/language-modeling/run_autoround_on_gaudi.shdeleted file mode 100644index 6f0cf5cf5..000000000--- a/examples/language-modeling/run_autoround_on_gaudi.sh+++ /dev/null@@ -1,13 +0,0 @@-#!/bin/bash-set -x-model_name="Intel/neural-chat-7b-v3-3"--python3 main.py \- --model_name $model_name \- --group_size 128 \- --bits 4 \- --device 0 \- --format "auto_round" \- --output_dir "./tmp_autoround"--diff --git a/examples/language-modeling/run_xpu.sh b/examples/language-modeling/run_xpu.shdeleted file mode 100644index f52157251..000000000--- a/examples/language-modeling/run_xpu.sh+++ /dev/null@@ -1,13 +0,0 @@-#!/bin/bash-set -x-device=0-model_name="facebook/opt-125m"--CUDA_VISIBLE_DEVICES=$device \-python3 main.py \---model_name $model_name \---bits 4 \---group_size 32 \---format "auto_gptq" \---output_dir "./tmp_autoround"-diff --git a/examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh b/examples/language-modeling/scripts/Llama-2-7b-chat-hf.shdeleted file mode 100644index 94781f0cf..000000000--- a/examples/language-modeling/scripts/Llama-2-7b-chat-hf.sh+++ /dev/null@@ -1,11 +0,0 @@-## Due to licensing restrictions, we are unable to release the model. To run the int4 model on GPU, please install transformers==4.32.0, currently the 4.38.1 version has some issues.-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-python3 main.py \---model_name meta-llama/Llama-2-7b-chat-hf \---device 0 \---group_size 128 \---bits 4 \---iters 1000 \---nsamples 512 \---format 'auto_round,auto_gptq' \---output_dir "./tmp_autoround"diff --git a/examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.sh b/examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.shdeleted file mode 100644index 57a6ce378..000000000--- a/examples/language-modeling/scripts/Meta-Llama-3-8B-Instruct.sh+++ /dev/null@@ -1,25 +0,0 @@-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-## quant lm head-python3 main.py \---model_name meta-llama/Meta-Llama-3-8B-Instruct \---device 0 \---group_size 128 \---bits 4 \---iters 1000 \---quant_lm_head \---nsamples 512 \---format 'auto_round,auto_gptq' \---output_dir "./tmp_autoround"----## don't quant lm head-python3 main.py \---model_name meta-llama/Meta-Llama-3-8B-Instruct \---device 0 \---group_size 128 \---bits 4 \---iters 1000 \---format 'auto_round,auto_gptq' \---nsamples 512 \---output_dir "./tmp_autoround"\ No newline at end of filediff --git a/examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.sh b/examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.shdeleted file mode 100644index aeece3d50..000000000--- a/examples/language-modeling/scripts/Mistral-7B-Instruct-v0.2.sh+++ /dev/null@@ -1,10 +0,0 @@-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-python3 main.py \---model_name mistralai/Mistral-7B-Instruct-v0.2 \---device 0 \---group_size 128 \---bits 4 \---iters 1000 \---nsamples 512 \---format 'auto_round,auto_gptq' \---output_dir "./tmp_autoround"\ No newline at end of filediff --git a/examples/language-modeling/scripts/Mistral-7B-v0.1.sh b/examples/language-modeling/scripts/Mistral-7B-v0.1.shdeleted file mode 100644index 4c3f01367..000000000--- a/examples/language-modeling/scripts/Mistral-7B-v0.1.sh+++ /dev/null@@ -1,10 +0,0 @@-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-python3 main.py \---model_name mistralai/Mistral-7B-v0.1 \---device 0 \---group_size 128 \---bits 4 \---iters 1000 \---nsamples 512 \---format 'auto_round,auto_gptq' \---output_dir "./tmp_autoround"\ No newline at end of filediff --git a/examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh b/examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.shdeleted file mode 100644index 719840058..000000000--- a/examples/language-modeling/scripts/Mixtral-8x7B-Instruct-v0.1.sh+++ /dev/null@@ -1,10 +0,0 @@-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-python3 main.py \---model_name mistralai/Mixtral-8x7B-Instruct-v0.1 \---device 0 \---group_size 128 \---bits 4 \---iters 1000 \---nsamples 512 \---format 'auto_round,auto_gptq' \---output_dir "./tmp_autoround"\ No newline at end of filediff --git a/examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh b/examples/language-modeling/scripts/Mixtral-8x7B-v0.1.shdeleted file mode 100644index 2557eff5d..000000000--- a/examples/language-modeling/scripts/Mixtral-8x7B-v0.1.sh+++ /dev/null@@ -1,11 +0,0 @@-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-python3 main.py \---model_name mistralai/Mixtral-8x7B-v0.1 \---device 0 \---group_size 128 \---bits 4 \---iters 1000 \---nsamples 512 \---format 'auto_round,auto_gptq' \---output_dir "./tmp_autoround" \---disable_quanted_input\ No newline at end of filediff --git a/examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh b/examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.shdeleted file mode 100644index ebfca329f..000000000--- a/examples/language-modeling/scripts/Qwen1.5-7B-Chat-asym.sh+++ /dev/null@@ -1,11 +0,0 @@-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-## Due to licensing restrictions, we are unable to release the model.-python3 main.py \---model_name Qwen/Qwen1.5-7B-Chat \---group_size 128 \---bits 4 \---iters 1000 \---format 'auto_round,auto_gptq' \---minmax_lr 2e-3\---nsamples 512-diff --git a/examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh b/examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.shdeleted file mode 100644index 2da700826..000000000--- a/examples/language-modeling/scripts/Qwen1.5-7B-Chat-sym.sh+++ /dev/null@@ -1,11 +0,0 @@-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-## Due to licensing restrictions, we are unable to release the model.-python3 main.py \---model_name Qwen/Qwen1.5-7B-Chat \---bits 4 \---group_size 128 \---iters 200 \---format 'auto_round,auto_gptq' \---nsamples 512 \---minmax_lr 0.01-diff --git a/examples/language-modeling/scripts/Yi-6B-Chat.sh b/examples/language-modeling/scripts/Yi-6B-Chat.shdeleted file mode 100644index f66b0efb7..000000000--- a/examples/language-modeling/scripts/Yi-6B-Chat.sh+++ /dev/null@@ -1,11 +0,0 @@-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-## Due to licensing restrictions, we are unable to release the model.-python3 main.py \---model_name 01-ai/Yi-6B-Chat \---group_size 128 \---bits 4 \---iters 1000 \---format 'auto_round,auto_gptq' \---minmax_lr 2e-3--diff --git a/examples/language-modeling/scripts/baichuan2-7b-chat.sh b/examples/language-modeling/scripts/baichuan2-7b-chat.shdeleted file mode 100644index 7425aeca6..000000000--- a/examples/language-modeling/scripts/baichuan2-7b-chat.sh+++ /dev/null@@ -1,10 +0,0 @@-## Due to licensing restrictions, we are unable to release the model.-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-python3 main.py \---model_name baichuan-inc/Baichuan2-7B-Chat \---group_size 128 \---bits 4 \---iters 1000 \---nsamples 512 \---format 'auto_round,auto_gptq' \---minmax_lr 2e-3diff --git a/examples/language-modeling/scripts/bloom-3b.sh b/examples/language-modeling/scripts/bloom-3b.shdeleted file mode 100644index 64e20cf66..000000000--- a/examples/language-modeling/scripts/bloom-3b.sh+++ /dev/null@@ -1,10 +0,0 @@-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-python3 main.py \---model_name bigscience/bloom-3b \---device 0 \---group_size 128 \---bits 4 \---iters 1000 \---nsamples 512 \---format 'auto_round,auto_gptq' \---output_dir "./tmp_autoround"diff --git a/examples/language-modeling/scripts/falcon-7b.sh b/examples/language-modeling/scripts/falcon-7b.shdeleted file mode 100644index b99bdc0c0..000000000--- a/examples/language-modeling/scripts/falcon-7b.sh+++ /dev/null@@ -1,11 +0,0 @@-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-python3 main.py \---model_name tiiuae/falcon-7b \---device 0 \---group_size 64 \---bits 4 \---iters 1000 \---nsamples 512 \---format 'auto_round,auto_gptq' \---output_dir "./tmp_autoround" \---disable_quanted_input \\ No newline at end of filediff --git a/examples/language-modeling/scripts/gemma-2b.sh b/examples/language-modeling/scripts/gemma-2b.shdeleted file mode 100644index 40de71621..000000000--- a/examples/language-modeling/scripts/gemma-2b.sh+++ /dev/null@@ -1,11 +0,0 @@-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-python3 main.py \---model_name google/gemma-2b \---device 0 \---group_size 128 \---bits 4 \---iters 400 \---model_dtype "float16" \---nsamples 512 \---format 'auto_round,auto_gptq' \---output_dir "./tmp_autoround"\ No newline at end of filediff --git a/examples/language-modeling/scripts/gemma-7b-it.sh b/examples/language-modeling/scripts/gemma-7b-it.shdeleted file mode 100644index a480ebbd3..000000000--- a/examples/language-modeling/scripts/gemma-7b-it.sh+++ /dev/null@@ -1,12 +0,0 @@-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-python3 main.py \---model_name google/gemma-7b-it \---device 0 \---group_size 128 \---bits 4 \---iters 1000 \---minmax_lr 2e-3 \---model_dtype "float16" \---format 'auto_round,auto_gptq' \---nsamples 512 \---output_dir "./tmp_autoround"\ No newline at end of filediff --git a/examples/language-modeling/scripts/gemma-7b.sh b/examples/language-modeling/scripts/gemma-7b.shdeleted file mode 100644index 42efb57e9..000000000--- a/examples/language-modeling/scripts/gemma-7b.sh+++ /dev/null@@ -1,12 +0,0 @@-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-python3 main.py \---model_name google/gemma-7b \---device 0 \---group_size 128 \---bits 4 \---iters 1000 \---nsamples 512 \---minmax_lr 2e-3 \---format 'auto_round,auto_gptq' \---model_dtype "float16" \---output_dir "./tmp_autoround"\ No newline at end of filediff --git a/examples/language-modeling/scripts/gpt-j-6b.sh b/examples/language-modeling/scripts/gpt-j-6b.shdeleted file mode 100644index 0d954ba28..000000000--- a/examples/language-modeling/scripts/gpt-j-6b.sh+++ /dev/null@@ -1,11 +0,0 @@-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-python3 main.py \---model_name EleutherAI/gpt-j-6b \---device 0 \---group_size 128 \---bits 4 \---iters 1000 \---disable_quanted_input \---nsamples 512 \---format 'auto_round,auto_gptq' \---output_dir "./tmp_autoround"diff --git a/examples/language-modeling/scripts/neural-chat-7b-v3-1.sh b/examples/language-modeling/scripts/neural-chat-7b-v3-1.shdeleted file mode 100644index 5d749ea2a..000000000--- a/examples/language-modeling/scripts/neural-chat-7b-v3-1.sh+++ /dev/null@@ -1,13 +0,0 @@-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-## enter to language-modeling fold-python3 main.py \---model_name /models/neural-chat-7b-v3-1 \---device 0 \---group_size 128 \---bits 4 \---iters 1000 \---nsamples 512 \---minmax_lr 2e-3 \---format 'auto_round,auto_gptq' \---output_dir "./tmp_autoround" \---disable_quanted_input\ No newline at end of filediff --git a/examples/language-modeling/scripts/neural-chat-7b-v3-3.sh b/examples/language-modeling/scripts/neural-chat-7b-v3-3.shdeleted file mode 100644index 0906841d0..000000000--- a/examples/language-modeling/scripts/neural-chat-7b-v3-3.sh+++ /dev/null@@ -1,12 +0,0 @@-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-python3 main.py \---model_name Intel/neural-chat-7b-v3-3 \---device 0 \---group_size 128 \---bits 4 \---iters 1000 \---minmax_lr 2e-3 \---nsamples 512 \---format 'auto_round,auto_gptq' \---output_dir "./tmp_autoround" \---disable_quanted_input\ No newline at end of filediff --git a/examples/language-modeling/scripts/opt-2.7b.sh b/examples/language-modeling/scripts/opt-2.7b.shdeleted file mode 100644index 9d8cf6708..000000000--- a/examples/language-modeling/scripts/opt-2.7b.sh+++ /dev/null@@ -1,12 +0,0 @@-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-python3 main.py \---model_name facebook/opt-2.7b \---device 0 \---group_size 128 \---bits 4 \---iters 1000 \---minmax_lr 0.002 \---nsamples 512 \---disable_quanted_input \---format 'auto_round,auto_gptq' \---output_dir "./tmp_autoround"diff --git a/examples/language-modeling/scripts/phi-2.sh b/examples/language-modeling/scripts/phi-2.shdeleted file mode 100644index 5421373f9..000000000--- a/examples/language-modeling/scripts/phi-2.sh+++ /dev/null@@ -1,11 +0,0 @@-## This recipe is outdated, we recommend using the latest recipe for best accuracy in homepage-python3 main.py \---model_name microsoft/phi-2 \---device 0 \---group_size 128 \---bits 4 \---iters 1000 \---sym \---nsamples 512 \---format 'auto_round,auto_gptq' \---output_dir "./tmp_autoround"\ No newline at end of file