Skip to content

PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度

add qwen2.5 recipe and refine readme (#338)

WeiweiZhang1e1d820feaa1f
diff --git a/README.md b/README.md
index 59fe9e2ba..f7af87f5e 100644
--- a/README.md
+++ b/README.md
@@ -26,14 +26,13 @@ more accuracy data and recipes across various models.
<div align="left">
## What's New
-* [2024/11] We provide some tips and tricks for LLM&VLM quantization, please check out [this file](./docs/tips_and_tricks.md).
+* [2024/11] We provide experimental support for VLLM quantization, please check out [MLLM README](./auto_round/mllm/README.md)
+* [2024/11] We provide some tips and tricks for LLM&VLM quantization, please check out [this file](./docs/tips_and_tricks.md)
* [2024/10] AutoRound has been integrated to [torch/ao](https://github.com/pytorch/ao), check out
their [release note](https://github.com/pytorch/ao/releases/tag/v0.6.1)
* [2024/10] Important update: We now support full-range symmetric quantization and have made it the default
configuration. This configuration is typically better or comparable to asymmetric quantization and significantly
outperforms other symmetric variants, especially at low bit-widths like 2-bit, check out [some accuracy data](./docs/full_range_sym.md).
-* [2024/09] AutoRound format supports several LVM models, check out the
- examples [Qwen2-Vl](./examples/multimodal-modeling/Qwen-VL),[Phi-3-vision](./examples/multimodal-modeling/Phi-3-vision), [Llava](./examples/multimodal-modeling/Llava)
* [2024/08] AutoRound format supports Intel Gaudi2 devices. Please refer
to [Intel/Qwen2-7B-int4-inc](https://huggingface.co/Intel/Qwen2-7B-int4-inc).
* [2024/08] AutoRound introduces several experimental features, including fast tuning of norm/bias parameters (for 2-bit
@@ -317,7 +316,11 @@ release most of the models ourselves.
| meta-llama/Llama-3.2-11B-Vision | [recipe](./docs/Llama-3.2-11B-Vision-Instruct_sym.md) |
| microsoft/Phi-3.5-vision-instruct | [recipe](./docs/Phi-3.5-vision-instruct_sym.md) |
| liuhaotian/llava-v1.5-7b | [recipe](./docs/llava-v1.5-7b_sym.md) |
-| Qwen/Qwen2.5-7B-Instruct | [recipe](./docs/Qwen2.5-7B-Instruct_sym.md) |
+| Qwen/Qwen2.5-7B-Instruct |[model-kaitchup-autogptq-int4*](https://beta-index.hf-mirror.com/kaitchup/Qwen2.5-7B-Instruct-AutoRound-GPTQ-asym-4bit), [recipe](./docs/Qwen2.5-7B-Instruct_sym.md) |
+| Qwen/Qwen2.5-14B-Instruct |[recipe](./docs/Qwen2.5-14B-Instruct_sym.md) |
+| Qwen/Qwen2.5-32B-Instruct |[recipe](./docs/Qwen2.5-32B-Instruct_sym.md) |
+| Qwen/Qwen2.5-Coder-32B-Instruct |[model-kaitchup-autogptq-int4*](https://beta-index.hf-mirror.com/kaitchup/Qwen2.5-Coder-32B-Instruct-AutoRound-GPTQ-4bit) |
+| Qwen/Qwen2.5-72B-Instruct |[model-kaitchup-autogptq-int4*](https://beta-index.hf-mirror.com/kaitchup/Qwen2.5-72B-Instruct-AutoRound-GPTQ-4bit), [model-kaitchup-autogptq-int2*](https://beta-index.hf-mirror.com/kaitchup/Qwen2.5-72B-Instruct-AutoRound-GPTQ-2bit), [recipe](./docs/Qwen2.5-72B-Instruct_sym.md) |
| meta-llama/Meta-Llama-3.1-70B-Instruct | [recipe](https://huggingface.co/Intel/Meta-Llama-3.1-70B-Instruct-int4-inc) |
| meta-llama/Meta-Llama-3.1-8B-Instruct | [model-kaitchup-autogptq-int4*](https://huggingface.co/kaitchup/Meta-Llama-3.1-8B-Instruct-autoround-gptq-4bit-asym), [model-kaitchup-autogptq-sym-int4*](https://huggingface.co/kaitchup/Meta-Llama-3.1-8B-Instruct-autoround-gptq-4bit-sym), [recipe](https://huggingface.co/Intel/Meta-Llama-3.1-8B-Instruct-int4-inc) |
| meta-llama/Meta-Llama-3.1-8B | [model-kaitchup-autogptq-sym-int4*](https://huggingface.co/kaitchup/Meta-Llama-3.1-8B-autoround-gptq-4bit-sym) |
@@ -373,3 +376,4 @@ If you find AutoRound useful for your research, please cite our paper:
}
```
+
diff --git a/auto_round/mllm/README.md b/auto_round/mllm/README.md
index c2a05df35..799cef644 100644
--- a/auto_round/mllm/README.md
+++ b/auto_round/mllm/README.md
@@ -25,6 +25,14 @@ AutoRound uses the text module of MLLM (LLM component) as the main quantization
autoround.save_quantized(output_dir, format='auto_round', inplace=True)
```
+- `dataset`: the dataset for quantization training. current support NeelNanda/pile-10k,llava_conv_58k,llava_instruct_80k and llava_instruct_150k. It can be a custom one. Please note that the effectiveness of the Llava calibration dataset has only been validated on five models so far.
+
+- `quant_nontext_module`: whether to quantize non-text module, e.g. vision component.
+
+- `extra_data_dir`:dataset dir for storing images/audio/videos, default to None. Can be a dir path or multiple dir path with format as 'image=path_to_image,video=path_to_video,audio=path_to_audio' By default, it will search in the relative path, and if not find, will automatic download.
+
+for more hyperparameters introduction, please refer [Homepage Detailed Hyperparameters](../../README.md#api-usage-gaudi2cpugpu)
+
<details>
<summary style="font-size:17px;">Basic Usage (Gaudi2/CPU/GPU)</summary>
A user guide detailing the full list of supported arguments is provided by calling ```auto-round-mllm -h``` on the terminal. Alternatively, you can use ```auto_round_mllm``` instead of ```auto-round-mllm```. Set the format you want in `format` and
@@ -40,11 +48,6 @@ AutoRound uses the text module of MLLM (LLM component) as the main quantization
--output_dir ./tmp_autoround
```
-- `dataset`: the dataset for quantization training. current support NeelNanda/pile-10k,llava_conv_58k,llava_instruct_80k. It can be a custom one.
-
-- `quant_nontext_module`: whether to quantize non-text module, e.g. vision component.
-
-- `extra_data_dir`:dataset dir for storing images/audio/videos, default to None. Can be a dir path or multiple dir path with format as 'image=path_to_image,video=path_to_video,audio=path_to_audio' By default, it will search in the relative path, and if not find, will automatic download.
</details>
@@ -56,19 +59,6 @@ For mllm, we used **text-only** calibration dataset (NeelNanda/pile-10k) as our
Through argument --dataset(text file), user can use other datasets such as "liuhaotian/llava_conv_58k" "liuhaotian/llava_instruct_80k", "liuhaotian/llava_instruct_150k" or a file path to use local file.
-
-### Support List
-
-The llava calibration dataset supports the five existing MLLMs.
-
-|Model |Eval Lib |calibration dataset|Feasibility of quantification|
-|---------------|-----------|-------------------|--------------------|
-|Qwen/Qwen2-VL-Instruct |vlmeval |llava |✔ |
-|meta-llama/Llama-3.2-11B-Vision |vlmeval/lmms_eval |llava |✔ |
-|microsoft/Phi-3.5-vision-instruct |vlmeval |llava |✔ |
-|liuhaotian/llava-v1.5-7b |lmms_eval |llava |✔ |
-|THUDM/cogvlm2-llama3-chat-19B |lmms_eval |llava |✔ |
-
</details>
@@ -78,15 +68,17 @@ The llava calibration dataset supports the five existing MLLMs.
### Support Matrix
-The design of the MLLM model API is not uniform, and some models do not support the quantization nontext module. Quantization of the vision components of Llama-3.2-11B-Vision, Phi-3.5-vision-instruct and llava-v1.5-7b is currently supported.
+For typical VLLMs, we assume that the default quantization, which excludes quantizing the visual component, is supported. The design of vision components in MLLM model APIs is not standardized, and some models do not support the quantization of non-text modules.
-|Model |Eval Lib |quant nontext module|
-|---------------|-----------|-------------------|
-|Qwen/Qwen2-VL-Instruct |vlmeval |- |
-|meta-llama/Llama-3.2-11B-Vision |lmms_eval |✔ |
-|microsoft/Phi-3.5-vision-instruct |vlmeval |✔ |
-|liuhaotian/llava-v1.5-7b |lmms_eval |- |
-|THUDM/cogvlm2-llama3-chat-19B |lmms_eval |✔ |
+Currently, the quantization of vision components is supported for Llama-3.2-11B-Vision, Phi-3.5-Vision-Instruct, and Llava-v1.5-7B.
+
+| Model | Eval Lib | calibration dataset | quant nontext module |
+|--------------|-----------|---------------------|----------------------|
+| Qwen2-VL | vlmeval | pile/llava | - |
+| Llama-Vision | lmms_eval | llava | ✔ |
+| Phi3-Vision | vlmeval | pile/llava | ✔ |
+| Llava-v1.5 | lmms_eval | pile/llava | - |
+| CogVLM2 | lmms_eval | pile/llava | ✔ |
@@ -140,3 +132,4 @@ For more details on quantization, inference, evaluation, and environment, see th
+
diff --git a/docs/Qwen2.5-14B-Instruct_sym.md b/docs/Qwen2.5-14B-Instruct_sym.md
new file mode 100644
index 000000000..6b6c3db99
--- /dev/null
+++ b/docs/Qwen2.5-14B-Instruct_sym.md
@@ -0,0 +1,209 @@
+## Model Details
+
+This model is an int4 model with group_size 128 and symmetric quantization of [Qwen/Qwen2.5-14B-Instruct](https://huggingface.co/Qwen/Qwen2.5-14B-Instruct) generated by [intel/auto-round](https://github.com/intel/auto-round). Load the model with `revision="f86a564"` to use AutoGPTQ format.
+
+## How To Use
+
+### INT4 Inference(CPU/HPU/CUDA)
+
+CPU requires auto-round version>0.3.1
+
+```python
+from auto_round import AutoRoundConfig ##must import for auto-round format
+from transformers import AutoModelForCausalLM,AutoTokenizer
+quantized_model_dir = "Intel/Qwen2.5-14B-Instruct-int4-inc"
+tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
+
+model = AutoModelForCausalLM.from_pretrained(
+ quantized_model_dir,
+ torch_dtype='auto',
+ device_map="auto",
+ ##revision="f86a564" ##AutoGPTQ format
+)
+
+##import habana_frameworks.torch.core as htcore ## uncommnet it for HPU
+##import habana_frameworks.torch.hpu as hthpu ## uncommnet it for HPU
+##model = model.to(torch.bfloat16).to("hpu") ## uncommnet it for HPU
+
+prompt = "There is a girl who likes adventure,"
+messages = [
+ {"role": "system", "content": "You are a helpful assistant."},
+ {"role": "user", "content": prompt}
+]
+
+text = tokenizer.apply_chat_template(
+ messages,
+ tokenize=False,
+ add_generation_prompt=True
+)
+model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
+
+generated_ids = model.generate(
+ model_inputs.input_ids,
+ max_new_tokens=200, ##change this to align with the official usage
+ do_sample=False ##change this to align with the official usage
+)
+generated_ids = [
+output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
+]
+
+response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
+print(response)
+
+prompt = "There is a girl who likes adventure,"
+##INT4:
+""" and she wants to go on a trip. She has 10 different types of snacks, and she can only carry 4 of them in her bag. How many different combinations of snacks can she choose from? To determine the number of different combinations of snacks the girl can choose from, we need to calculate the number of ways to choose 4 snacks out of 10. This is a classic combination problem where the order of selection does not matter.
+
+The formula for combinations is given by:
+\[
+\binom{n}{r} = \frac{n!}{r!(n-r)!}
+\]
+where \( n \) is the total number of items to choose from, \( r \) is the number of items to choose, and \( ! \) denotes factorial.
+
+In this problem, \( n = 10 \) and \( r = 4 \). Plugging these values into the formula, we get:
+\[
+\binom{10}{4}"""
+
+##BF16:
+""" and she has a hobby of collecting rocks. She wants to go on a trip to collect some unique rocks. She plans to visit three different locations: a mountain, a beach, and a desert. Each location has its own set of challenges and opportunities for rock collecting.
+
+1. The mountain is known for its rare mineral deposits, but the terrain is steep and rocky, making it difficult to navigate.
+2. The beach offers a variety of sedimentary rocks and fossils, but the tides can be unpredictable and dangerous.
+3. The desert provides an opportunity to find petrified wood and volcanic rocks, but the heat and lack of water can be challenging.
+
+The girl has a backpack that can carry up to 10 kilograms of rocks. She also has a map that shows the locations of specific types of rocks at each site. Her goal is to maximize the number of unique rock types she collects while staying within her weight limit.
+
+Given the following information:
+- Mountain: 5 unique rock types"""
+
+prompt = "9.11和9.8哪个数字大"
+#INT4:
+"""? 9.11 比 9.8 大。
+
+为了比较这两个数,我们可以从它们的小数部分开始:
+
+- 9.11 可以看作是 9 + 0.11
+- 9.8 可以看作是 9 + 0.8
+
+由于 0.11 小于 0.8,所以 9.11 小于 9.8。因此,9.8 比 9.11 大。
+
+总结:9.8 > 9.11。所以,9.8 是较大的数字。如果你的问题是问哪个数字较大,则答案是 9.8。如果问题是问哪个数字较小,则答案是 9.11。请确认你的问题需求。根据你的描述,9.8 是较大的数字。
+
+希望这能帮助你!如有其他问题,请随时提问。
+
+(注意:在"""
+
+##BF16:
+"""? 9.11 比 9.8 大。
+
+在比较两个小数时,我们从左到右逐位进行比较。首先比较整数部分,如果相同,则比较小数部分。对于9.11 和 9.8:
+
+- 整数部分都是9,相等。
+- 比较小数部分:0.11 和 0.8。
+
+由于0.11 < 0.8,所以9.11 < 9.8。
+
+因此,9.8 比 9.11 大。
+
+所以,正确的答案是:9.8 比 9.11 大。
+
+希望这能帮助你理解!如果你有其他问题,请随时告诉我。
+
+总结:9.8 > 9.11。
+
+希望这个解释清楚了你的疑问。如果有任何进一步的问题或需要更多帮助,请告诉我!
+
+再次确认:9"""
+
+
+prompt = "Once upon a time,"
+##INT4:
+""" there was a young man named John who had a passion for music. He loved playing the guitar and would spend hours every day practicing and perfecting his skills. However, he struggled to find an audience for his music and felt discouraged.
+"""
+
+##BF16:
+""" there was a young man named John who lived in a small village. He was an orphan and had to work hard to make ends meet. Despite his difficult circumstances, he was kind-hearted and always willing to help others. One day, a wise old man came to the village and saw John's kindness. The old man decided to test John's character by giving him a bag of gold coins and telling him to distribute it among the villagers. John was overjoyed at first but then realized that he could use the money for himself. However, he remembered the wise man's words and distributed the coins equally among the villagers. The wise man was pleased with John's actions and revealed himself as a fairy godfather. He granted John three wishes, but with a twist - each wish would come true only if John could prove that he deserved it. What are some possible wishes that John might make and how could he prove that he deserves them?
+John, being a kind-hearted individual, might consider wishes that"""
+
+
+prompt = "请简短介绍一下阿里巴巴公司"
+##INT4:
+"""阿里巴巴集团创立于1999年,是全球领先的电子商务及零售贸易平台。阿里巴巴集团的使命是让世界各地的企业都能平等地进行贸易。阿里巴巴集团旗下的业务包括淘宝、天猫、菜鸟网络、阿里云等。阿里巴巴集团致力于通过技术创新,为中小企业提供更便捷、高效的商业服务,推动数字经济的发展。阿里巴巴集团在全球范围内拥有数百万商家和消费者用户,已成为全球最大的零售贸易平台之一。阿里巴巴集团总部位于中国杭州,并在全球范围内设有多个办事处和研发中心。阿里巴巴集团的愿景是构建一个开放、协作、可持续发展的数字经济生态系统,为全球商业带来更多的机遇和价值。阿里巴巴集团在2014年上市,成为当时全球最大的IPO。阿里巴巴集团的创始人马云是中国著名的企业家和慈善家。阿里巴巴集团在社会责任方面也做出了积极贡献,包括支持教育、环保、扶贫等公益事业。阿里巴巴集团是一家具有高度社会责任感的企业。阿里巴巴集团的业务涵盖了电子商务、金融、物流
+"""
+
+##BF16:
+"""阿里巴巴集团创立于1999年,是全球领先的电子商务及零售平台,业务涵盖B2B、C2C、B2C等各个领域。阿里巴巴旗下拥有淘宝网、天猫、菜鸟网络、阿里云等知名子公司和品牌,致力于打造开放、协同、繁荣的商业生态系统,为全球中小企业提供一站式数字化转型服务。阿里巴巴在全球范围内拥有超过20万名员工,并在纽约证券交易所上市。阿里巴巴一直秉承“让天下没有难做的生意”的使命,不断创新和发展,成为全球领先的数字经济体之一。阿里巴巴还积极履行企业社会责任,关注环保、公益等领域,努力实现可持续发展。阿里巴巴已经成为中国互联网行业的领军企业之一,在全球范围内也具有广泛的影响力。阿里巴巴的发展历程充满了挑战与机遇,未来将继续引领数字经济的发展趋势,推动全球经济的繁荣与发展。阿里巴巴是一家总部位于中国杭州的跨国科技公司,主要业务包括电子商务、金融、物流、云计算等。阿里巴巴旗下的淘宝、天猫等电商平台已成为
+"""
+```
+
+### Evaluate the model
+
+pip3 install lm-eval==0.4.5
+
+```bash
+auto-round --model "Intel/Qwen2.5-14B-Instruct-int4-inc" --eval --eval_bs 16 --tasks leaderboard_ifeval,leaderboard_mmlu_pro,gsm8k,lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,cmmlu,ceval-valid
+```
+
+| Metric | BF16 | INT4 |
+| :----------------------------------------- | :----: | :----: |
+| Avg | 0.6947 | 0.6954 |
+| leaderboard_mmlu_pro 5 shots | 0.5375 | 0.5292 |
+| leaderboard_ifeval inst_level_strict_acc | 0.6331 | 0.6475 |
+| leaderboard_ifeval prompt_level_strict_acc | 0.5102 | 0.5287 |
+| mmlu | 0.7882 | 0.7809 |
+| cmmlu | 0.8377 | 0.8240 |
+| ceval-valid | 0.8351 | 0.8232 |
+| gsm8k 5 shots | 0.7900 | 0.8120 |
+| lambada_openai | 0.7283 | 0.7250 |
+| hellaswag | 0.6556 | 0.6508 |
+| winogrande | 0.7585 | 0.7672 |
+| piqa | 0.8166 | 0.8156 |
+| truthfulqa_mc1 | 0.5153 | 0.5202 |
+| openbookqa | 0.3640 | 0.3700 |
+| boolq | 0.8798 | 0.8810 |
+| arc_easy | 0.8582 | 0.8535 |
+| arc_challenge | 0.6049 | 0.5981 |
+
+
+
+### Generate the model
+
+Here is the sample command to generate the model.
+
+```bash
+auto-round \
+--model Qwen/Qwen2.5-14B-Instruct \
+--device 0 \
+--group_size 128 \
+--nsamples 512 \
+--bits 4 \
+--iter 1000 \
+--disable_eval \
+--model_dtype "fp16" \
+--format 'auto_gptq,auto_round' \
+--output_dir "./tmp_autoround"
+```
+
+## Ethical Considerations and Limitations
+
+The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
+
+Therefore, before deploying any applications of the model, developers should perform safety testing.
+
+## Caveats and Recommendations
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
+
+Here are a couple of useful links to learn more about Intel's AI software:
+
+- Intel Neural Compressor [link](https://github.com/intel/neural-compressor)
+
+## Disclaimer
+
+The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.
+
+## Cite
+
+@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, journal={arXiv preprint arXiv:2309.05516}, year={2023} }
+
+[arxiv](https://arxiv.org/abs/2309.05516) [github](https://github.com/intel/auto-round)
diff --git a/docs/Qwen2.5-32B-Instruct_sym.md b/docs/Qwen2.5-32B-Instruct_sym.md
new file mode 100644
index 000000000..7d7f24fb7
--- /dev/null
+++ b/docs/Qwen2.5-32B-Instruct_sym.md
@@ -0,0 +1,200 @@
+## Model Details
+
+This model is an int4 model with group_size 128 and symmetric quantization of [Qwen/Qwen2.5-32B-Instruct](https://huggingface.co/Qwen/Qwen2.5-32B-Instruct) generated by [intel/auto-round](https://github.com/intel/auto-round).
+
+## How To Use
+
+### INT4 Inference(CPU/HPU/CUDA)
+
+CPU/ CUDA requires auto-round version>0.3.1
+
+```python
+from auto_round import AutoRoundConfig ##must import for auto-round format
+from transformers import AutoModelForCausalLM,AutoTokenizer
+quantized_model_dir = "Intel/Qwen2.5-32B-Instruct-int4-inc"
+tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
+
+model = AutoModelForCausalLM.from_pretrained(
+ quantized_model_dir,
+ torch_dtype='auto',
+ device_map="auto",
+)
+
+##import habana_frameworks.torch.core as htcore ## uncommnet it for HPU
+##import habana_frameworks.torch.hpu as hthpu ## uncommnet it for HPU
+##model = model.to(torch.bfloat16).to("hpu") ## uncommnet it for HPU
+
+prompt = "There is a girl who likes adventure,"
+messages = [
+ {"role": "system", "content": "You are a helpful assistant."},
+ {"role": "user", "content": prompt}
+]
+
+text = tokenizer.apply_chat_template(
+ messages,
+ tokenize=False,
+ add_generation_prompt=True
+)
+model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
+
+generated_ids = model.generate(
+ model_inputs.input_ids,
+ max_new_tokens=200, ##change this to align with the official usage
+ do_sample=False ##change this to align with the official usage
+)
+generated_ids = [
+output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
+]
+
+response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
+print(response)
+
+prompt = "There is a girl who likes adventure,"
+##INT4:
+"""It sounds like you're starting to tell a story! Would you like me to help you continue it? Here's one possible continuation:
+
+There is a girl who loves adventure. Her name is Lily and she has always been drawn to the unknown. From a young age, she would spend hours exploring the woods behind her house, imagining herself as a brave explorer discovering new lands. As she grew older, her thirst for adventure only intensified. She began traveling the world, seeking out thrilling experiences and pushing herself out of her comfort zone at every turn.
+
+Is there anything specific you had in mind for this character or story? I'd be happy to help develop it further if you have any ideas!"""
+
+##BF16:
+"""That sounds exciting! What would you like to know or do regarding this girl who loves adventure? Perhaps you're looking for ideas on activities she might enjoy or ways to support her adventurous spirit. Let me know how I can assist you further!"""
+
+prompt = "9.11和9.8哪个数字大"
+#INT4:
+
+
+##BF16:
+"""要比较两个数字的大小,我们可以从左到右逐位进行比较。
+
+首先,我们看整数部分:
+- 9.11 的整数部分是 9。
+- 9.8 的整数部分也是 9。
+
+因为它们的整数部分相同,我们需要比较小数部分:
+- 9.11 的小数部分是 0.11。
+- 9.8 的小数部分是 0.8。
+
+接下来,我们比较小数部分的第一位:
+- 0.11 的第一位是 1。
+- 0.8 的第一位是 8。
+
+显然,8 比 1 大。因此,0.8 比 0.11 大。
+
+所以,9.8 比 9.11 大。"""
+
+
+prompt = "Once upon a time,"
+##INT4:
+
+
+##BF16:
+"""Once upon a time, in a land far, far away, there was a small village nestled between rolling hills and dense forests. The villagers lived simple lives, farming the land and tending to their livestock. They were a close-knit community, always ready to help one another in times of need.
+
+In the heart of this village stood an ancient oak tree, under which the elders would gather to share stories and make important decisions for the community. Among these tales were whispers of a hidden treasure, said to be guarded by a mythical creature deep within the forest.
+
+One day, a young girl named Elara overheard these whispers while fetching water from the well. Her curiosity piqued, she decided to embark on a quest to uncover the truth behind the legend. With nothing but her wits, a small backpack, and a map drawn by the village cartographer, Elara set off into the unknown.
+
+As she ventured deeper into the forest, she encountered various challenges and made unexpected friends along
+"""
+
+
+prompt = "请简短介绍一下阿里巴巴公司"
+##INT4:
+
+##BF16:
+"""阿里巴巴集团是一家中国跨国科技公司,成立于1999年,总部位于中国杭州。该公司以B2B电子商务平台起家,现已发展成为涵盖零售、金融、物流、云计算等多个领域的综合性企业集团。旗下拥有淘宝、天猫、阿里云等知名业务,是全球最大的电子商务和零售平台之一。阿里巴巴致力于通过技术创新和商业生态系统建设,推动数字经济的发展,并为消费者和企业提供优质的产品与服务。
+"""
+```
+
+### Evaluate the model
+
+pip3 install lm-eval==0.4.5
+
+```bash
+auto-round --model "Intel/Qwen2.5-32B-Instruct-int4-inc" --eval --eval_bs 16 --tasks leaderboard_ifeval,leaderboard_mmlu_pro,gsm8k,lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,cmmlu,ceval-valid
+```
+
+| Metric | BF16 | INT4 |
+| :----------------------------------------- | :----: | :----: |
+| Avg | 0.7120 | 0.7089 |
+| leaderboard_mmlu_pro 5 shots | 0.5917 | 0.5795 |
+| leaderboard_ifeval inst_level_strict_acc | 0.7314 | 0.7254 |
+| leaderboard_ifeval prompt_level_strict_acc | 0.6248 | 0.6285 |
+| mmlu | 0.8169 | 0.8148 |
+| cmmlu | 0.8673 | 0.8586 |
+| ceval-valid | 0.8811 | 0.8700 |
+| gsm8k 5 shots | 0.7680 | 0.8052 |
+| lambada_openai | 0.7522 | 0.7417 |
+| hellaswag | 0.6685 | 0.6643 |
+| winogrande | 0.7372 | 0.7324 |
+| piqa | 0.8085 | 0.8134 |
+| truthfulqa_mc1 | 0.4871 | 0.4749 |
+| openbookqa | 0.3580 | 0.3480 |
+| boolq | 0.8966 | 0.8841 |
+| arc_easy | 0.8237 | 0.8228 |
+| arc_challenge | 0.5785 | 0.5785 |
+
+
+
+### Generate the model
+
+Here is the sample command to generate the model.
+
+For symmetric quantization, we found overflow/NAN will occur for some backends, so better fallback some layers. auto_round requires version >0.4.1
+
+```bash
+auto-round \
+--model Qwen/Qwen2.5-32B-Instruct \
+--device 0 \
+--group_size 128 \
+--nsamples 512 \
+--bits 4 \
+--iter 1000 \
+--disable_eval \
+--fp_layers "model.layers.5.mlp.down_proj,model.layers.5.mlp.up_proj,model.layers.5.mlp.gate_proj" \
+--model_dtype "fp16" \
+--format 'auto_round' \
+--output_dir "./tmp_autoround"
+```
+
+Asym
+
+```bash
+auto-round \
+--model Qwen/Qwen2.5-32B-Instruct \
+--device 0 \
+--group_size 128 \
+--nsamples 512 \
+--bits 4 \
+--iter 1000 \
+--disable_eval \
+--asym
+--model_dtype "fp16" \
+--format 'auto_round' \
+--output_dir "./tmp_autoround"
+```
+
+## Ethical Considerations and Limitations
+
+The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
+
+Therefore, before deploying any applications of the model, developers should perform safety testing.
+
+## Caveats and Recommendations
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
+
+Here are a couple of useful links to learn more about Intel's AI software:
+
+- Intel Neural Compressor [link](https://github.com/intel/neural-compressor)
+
+## Disclaimer
+
+The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.
+
+## Cite
+
+@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, journal={arXiv preprint arXiv:2309.05516}, year={2023} }
+
+[arxiv](https://arxiv.org/abs/2309.05516) [github](https://github.com/intel/auto-round)
diff --git a/docs/Qwen2.5-72B-Instruct_sym.md b/docs/Qwen2.5-72B-Instruct_sym.md
new file mode 100644
index 000000000..30bca2040
--- /dev/null
+++ b/docs/Qwen2.5-72B-Instruct_sym.md
@@ -0,0 +1,199 @@
+## Model Details
+
+This model is an int4 model with group_size 128 and and symmetric quantization of [Qwen/Qwen2.5-72B-Instruct](https://huggingface.co/Qwen/Qwen2.5-72B-Instruct) generated by [intel/auto-round](https://github.com/intel/auto-round). Load the model with `revision="b7ea233"` to use AutoGPTQ format.
+## How To Use
+
+### INT4 Inference(CPU/HPU/CUDA)
+
+CPU requires auto-round version>0.3.1
+
+```python
+from auto_round import AutoRoundConfig ##must import for auto-round format
+from transformers import AutoModelForCausalLM,AutoTokenizer
+quantized_model_dir = "Intel/Qwen2.5-72B-Instruct-int4-inc"
+tokenizer = AutoTokenizer.from_pretrained(quantized_model_dir)
+
+model = AutoModelForCausalLM.from_pretrained(
+ quantized_model_dir,
+ torch_dtype='auto',
+ device_map="auto",
+ ##revision="b7ea233" ##AutoGPTQ format
+)
+
+##import habana_frameworks.torch.core as htcore ## uncommnet it for HPU
+##import habana_frameworks.torch.hpu as hthpu ## uncommnet it for HPU
+##model = model.to(torch.bfloat16).to("hpu") ## uncommnet it for HPU
+
+prompt = "There is a girl who likes adventure,"
+messages = [
+ {"role": "system", "content": "You are a helpful assistant."},
+ {"role": "user", "content": prompt}
+]
+
+text = tokenizer.apply_chat_template(
+ messages,
+ tokenize=False,
+ add_generation_prompt=True
+)
+model_inputs = tokenizer([text], return_tensors="pt").to(model.device)
+
+generated_ids = model.generate(
+ model_inputs.input_ids,
+ max_new_tokens=200, ##change this to align with the official usage
+ do_sample=False ##change this to align with the official usage
+)
+generated_ids = [
+output_ids[len(input_ids):] for input_ids, output_ids in zip(model_inputs.input_ids, generated_ids)
+]
+
+response = tokenizer.batch_decode(generated_ids, skip_special_tokens=True)[0]
+print(response)
+
+prompt = "There is a girl who likes adventure,"
+##INT4:
+"""That's great to hear! A love for adventure can lead to so many exciting experiences and personal growth. What kind of adventures does she enjoy? Is it exploring new places, trying out thrilling activities, or perhaps diving into new hobbies and challenges? Knowing more about her interests could help in suggesting fun ideas or planning future adventures.
+"""
+
+##BF16:
+"""That sounds like a great quality to have! A love for adventure can lead to exciting experiences and personal growth. What kind of adventures does she enjoy? Does she like exploring new places, trying new activities, or seeking out thrilling experiences? Knowing more about her interests can help suggest specific ideas or activities that might appeal to her adventurous spirit.
+"""
+
+prompt = "9.11和9.8哪个数字大"
+#INT4:
+"""要比较9.11和9.8的大小,可以按照以下步骤进行:
+
+1. **比较整数部分**:两个数字的整数部分都是9,所以需要进一步比较小数部分。
+2. **比较小数部分**:
+ - 9.11的小数部分是0.11
+ - 9.8的小数部分是0.8
+
+3. **比较0.11和0.8**:
+ - 0.11可以写成0.110
+ - 0.8可以写成0.800
+
+4. **逐位比较**:
+ - 第一位:1 < 8,所以0.110 < 0.800
+
+因此,9.11 < 9.8。
+
+结论:9.8比9.11大。"""
+
+##BF16:
+"""比较两个数字 9.11 和 9.8,可以按照以下步骤进行:
+
+1. **整数部分**:两个数字的整数部分都是 9,所以需要比较小数部分。
+2. **小数部分**:
+ - 9.11 的小数部分是 0.11
+ - 9.8 的小数部分是 0.8
+
+3. **比较小数部分**:
+ - 0.11 和 0.8 比较时,0.8 明显大于 0.11。
+
+因此,9.8 大于 9.11。"""
+
+
+prompt = "Once upon a time,"
+##INT4:
+"""Once upon a time, in a far-off land, there was a kingdom filled with wonder and magic. The kingdom was ruled by a wise and just king who loved his people dearly. In the heart of the kingdom stood a magnificent castle, surrounded by lush forests and rolling hills.
+
+The people of the kingdom lived happily, tending to their farms, crafting beautiful goods, and enjoying the simple pleasures of life. However, one day, a great darkness began to spread across the land. A wicked sorcerer had risen from the shadows, seeking to claim the throne for himself and plunge the kingdom into chaos.
+
+The king, knowing that he could not face this threat alone, called upon the bravest and most skilled heroes from all corners of the realm. Among them was a young knight named Sir Cedric, who had earned a reputation for his courage and unwavering sense of justice.
+
+Sir Cedric, along with a group of loyal companions, set out on a perilous journey to stop the sor"""
+
+##BF16:
+"""Once upon a time, in a land far, far away, there was a kingdom known for its beauty and prosperity. The kingdom was ruled by a wise and just king who loved his people dearly. In the heart of the kingdom stood a magnificent castle, surrounded by lush gardens and sparkling fountains.
+
+The king had a young daughter named Princess Elara, who was as kind and gentle as she was beautiful. She spent her days helping the poor and spreading joy throughout the kingdom. The people adored her, and she was beloved by all.
+
+One day, a great challenge arose. A dark forest on the outskirts of the kingdom began to grow wild and dangerous, threatening the safety of the villagers. The king called for a hero to tame the forest and protect his people. Many brave knights and warriors came forward, but none could succeed.
+
+Princess Elara, determined to help, decided to venture into the forest herself. Her father was hesitant, but he saw the determination in her eyes and knew"""
+
+
+prompt = "请简短介绍一下阿里巴巴公司"
+##INT4:
+"""阿里巴巴集团是一家总部位于中国杭州的全球领先的电子商务和科技公司。它成立于1999年,由马云和他的团队创立。阿里巴巴旗下拥有包括淘宝、天猫、阿里云等在内的多个知名业务平台,涵盖了在线零售、批发贸易、云计算、数字娱乐、金融服务等多个领域。
+
+阿里巴巴的愿景是让世界各地的企业都能够平等地进行贸易,通过技术创新推动数字经济的发展,为社会创造更多的价值。目前,阿里巴巴已经发展成为世界领先的互联网公司之一,业务遍布全球多个国家和地区,服务着数以亿计的用户和商家。"""
+
+##BF16:
+"""阿里巴巴集团是一家总部位于中国杭州的全球领先的电子商务和科技公司。成立于1999年,阿里巴巴最初是一个B2B在线市场,旨在连接中国制造商与全球买家。经过二十多年的发展,阿里巴巴已经发展成为涵盖电子商务、金融、物流、云计算等多个领域的综合性企业集团。
+
+阿里巴巴旗下拥有淘宝网、天猫、菜鸟网络、阿里云等知名品牌,为消费者提供购物、支付、娱乐等多元化服务,同时也为企业提供营销、销售、物流和技术支持等全方位解决方案。此外,阿里巴巴还积极投资和孵化创新项目,推动数字经济的发展。
+
+阿里巴巴始终秉持“让天下没有难做的生意”的使命,致力于通过技术创新促进全球经济的可持续发展。"""
+```
+
+### Evaluate the model
+
+pip3 install lm-eval==0.4.5
+
+```bash
+auto-round --model "Intel/Qwen2.5-72B-Instruct-int4-inc" --eval --eval_bs 16 --tasks leaderboard_ifeval,leaderboard_mmlu_pro,gsm8k,lambada_openai,hellaswag,piqa,winogrande,truthfulqa_mc1,openbookqa,boolq,arc_easy,arc_challenge,cmmlu,ceval-valid
+```
+
+| Metric | BF16 | INT4 |
+| :----------------------------------------- | :----: | :----: |
+| Avg | 0.7413 | 0.7448 |
+| leaderboard_mmlu_pro 5 shots | 0.5919 | 0.5864 |
+| leaderboard_ifeval inst_level_strict_acc | 0.7770 | 0.7866 |
+| leaderboard_ifeval prompt_level_strict_acc | 0.6858 | 0.6932 |
+| mmlu | 0.8334 | 0.8308 |
+| cmmlu | 0.8727 | 0.8673 |
+| ceval-valid | 0.8975 | 0.8960 |
+| gsm8k 5 shots | 0.9037 | 0.9098 |
+| lambada_openai | 0.7518 | 0.7563 |
+| hellaswag | 0.7031 | 0.7014 |
+| winogrande | 0.7601 | 0.7687 |
+| piqa | 0.8313 | 0.8232 |
+| truthfulqa_mc1 | 0.5239 | 0.5263 |
+| openbookqa | 0.3860 | 0.3820 |
+| boolq | 0.9049 | 0.9046 |
+| arc_easy | 0.8632 | 0.8611 |
+| arc_challenge | 0.6135 | 0.6237 |
+
+
+
+### Generate the model
+
+Here is the sample command to generate the model.
+
+```bash
+auto-round \
+--model Qwen/Qwen2.5-72B-Instruct \
+--device 0 \
+--group_size 128 \
+--nsamples 512 \
+--bits 4 \
+--iter 1000 \
+--disable_eval \
+--model_dtype "fp16" \
+--format 'auto_gptq,auto_round' \
+--output_dir "./tmp_autoround"
+```
+
+## Ethical Considerations and Limitations
+
+The model can produce factually incorrect output, and should not be relied on to produce factually accurate information. Because of the limitations of the pretrained model and the finetuning datasets, it is possible that this model could generate lewd, biased or otherwise offensive outputs.
+
+Therefore, before deploying any applications of the model, developers should perform safety testing.
+
+## Caveats and Recommendations
+
+Users (both direct and downstream) should be made aware of the risks, biases and limitations of the model.
+
+Here are a couple of useful links to learn more about Intel's AI software:
+
+- Intel Neural Compressor [link](https://github.com/intel/neural-compressor)
+
+## Disclaimer
+
+The license on this model does not constitute legal advice. We are not responsible for the actions of third parties who use this model. Please consult an attorney before using this model for commercial purposes.
+
+## Cite
+
+@article{cheng2023optimize, title={Optimize weight rounding via signed gradient descent for the quantization of llms}, author={Cheng, Wenhua and Zhang, Weiwei and Shen, Haihao and Cai, Yiyang and He, Xin and Lv, Kaokao and Liu, Yi}, journal={arXiv preprint arXiv:2309.05516}, year={2023} }
+
+[arxiv](https://arxiv.org/abs/2309.05516) [github](https://github.com/intel/auto-round)