PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
update readme (#531)
Wenhua Cheng87579ef03570
diff --git a/README.md b/README.mdindex 39e747949..3ae180ddc 100644--- a/README.md+++ b/README.md@@ -72,7 +72,7 @@ pip install auto-round-lib## Model Quantization-### Basic Usage (Gaudi/CPU/XPU/GPU)+### Command Line Usage (Gaudi/CPU/XPU/GPU)A user guide detailing the full list of supported arguments is provided by calling ```auto-round -h``` on the terminal.Set the format you want in `format` and@@ -161,7 +161,7 @@ from auto_round import AutoRoundbits, group_size, sym = 4, 128, Trueautoround = AutoRound(model, tokenizer, bits=bits, group_size=group_size, sym=sym)-## the best accuracy, 3X slower, low_gpu_mem_usage could save ~20G but ~30% slower+## the best accuracy, 4-5X slower, low_gpu_mem_usage could save ~20G but ~30% slower# autoround = AutoRound(model, tokenizer, nsamples=512, iters=1000, low_gpu_mem_usage=True, bits=bits, group_size=group_size, sym=sym)## 2-3X speedup, slight accuracy drop at W4G128@@ -334,8 +334,8 @@ print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))AutoRound automatically selects the best available backend based on the installed libraries and prompts the user toinstall additional libraries when a better backend is found. On CUDA, the default priority is Marlin > ExLLaMAV2 >-Triton, but the final choice depends on factors such as bits group_size packing format compatibility, etc. Please refer-to the following table for the details.+Triton, but the final choice depends on factors such as bits, group_size, packing format compatibility, etc. And the backend may not always be the most suitable for certain devices. Please refer+to the following table for the details and specify the backend you want.| Name | Devices | Bits | Dtypes | Priority | Packing format | Requirements ||--------------------------------------|---------|---------|-----------|----------|-----------------|-------------------------------|@@ -345,12 +345,13 @@ to the following table for the details.| exllamav2 or<br/>gptqmodel:exllamav2 | cuda | 4 | BF16/FP16 | 5 | gptq | gptqmodel || exllamav2 or<br/>gptq:exllamav2 | cuda | 4 | FP16 | 5 | gptq_zp+-1 | auto-gptq || gptq:cuda | cuda | 2,3,4,8 | FP16 | 0 | gptq_zp+-1 | auto-gptq |-| triton | cuda | 2,3,8 | BF16/FP16 | 1 | gptq/gptq_zp+-1 | auto-round |+| triton | cuda | 2,4,8 | BF16/FP16 | 1 | gptq/gptq_zp+-1 | auto-round || awq | cuda | 4 | FP16 | 5 | awq | auto-awq || hpu | hpu | 4 | BF16 | 0 | gptq/gptq_zp+-1 | auto-round |```pythonfrom transformers import AutoModelForCausalLM, AutoTokenizer+from auto_round import AutoRoundConfigquantized_model_path = "./tmp_autoround"quantization_config = AutoRoundConfig(backend="auto")diff --git a/docs/step_by_step.md b/docs/step_by_step.mdindex b2b60502a..2a5fcc5de 100644--- a/docs/step_by_step.md+++ b/docs/step_by_step.md@@ -3,7 +3,7 @@ Step-by-StepThis document presents step-by-step instructions for auto-round llm quantization.-# 1 Prerequisite+## 1 PrerequisiteInstall auto-round or install from source@@ -11,14 +11,18 @@ Install auto-round or install from sourcepip install auto-round```-## 2. Prepare Calibration Dataset+## 2 Prepare Calibration Dataset### Default DatasetThe [NeelNanda/pile-10k](https://huggingface.co/datasets/NeelNanda/pile-10k) in huggingface is adopted as the default-calibration data and will be downloaded automatically from the datasets Hub. To customize a dataset, please kindly-follow our dataset code.-See more about loading [huggingface dataset](https://huggingface.co/docs/datasets/main/en/quickstart)+calibration data and will be downloaded automatically from the datasets Hub. Other available datasets include:++- `swift/pile-val-backup` for addressing HF network issue+- `BAAI/CCI3-HQ` for Chinese+- `codeparrot/github-code-clean` for code+- `madao33/new-title-chinese` for Chinese+- `mbpp` for code### Customized Dataset@@ -45,53 +49,252 @@ See more about loading [huggingface dataset](https://huggingface.co/docs/datasettokens.append(token)return tokens~~~- **Dataset combination**:We support combination of different datasets and parametrization of calibration datasets by using "--dataset ./tmp.json:- concat,NeelNanda/pile-10k:split=train+val:num=256,mbpp:concat=True:num=128:apply_chat_template". Both local calibration file- and huggingface dataset are supported. Through parametrization, users could specify splits of a dataset by setting "- split=split1+split2".-- **Samples concatenation**: A concatenation option could enable users to merge calibration samples. '--dataset NeelNanda/pile-10k:concat=True'-- **Apply chat template**: '--dataset NeelNanda/pile-10k:apply_chat_template' would enable users to apply chat_template to calibration- data before tokenization and is widely used by instruct-models in generation. Please note that samples shorter than- args.seqlen will be dropped when concatenation option is not enabled.-- Please use ',' to split datasets, ':' to split parameters of a dataset and '+' to add values for one targeted parameter.--<br />+### Dataset operations++**Dataset combination**:We support combination of different datasets and parametrization of calibration datasets by+using `--dataset ./tmp.json,NeelNanda/pile-10k:num=256,mbpp:num=128`. Both local calibration file+and huggingface dataset are supported. You could specify splits of a dataset by setting `split=split1+split2`.++**Samples concatenation**: An optional setting allows users to concatenate calibration samples+using `--dataset NeelNanda/pile-10k:concat=True`.+All samples will be concatenated first, then split into chunks of seqlen length.++**Apply chat template**: Using `--dataset NeelNanda/pile-10k:apply_chat_template` enables application of a chat template+to the calibration data before tokenization. This is commonly used for instruct-style models during generation. To+customize the system prompt,+use:`--dataset 'NeelNanda/pile-10k:apply_chat_template:system_prompt="You are a helpful assistant."'`++Note: If the concatenation option is not enabled, samples shorter than args.seqlen will be dropped.++Please use ',' to split datasets, ':' to split parameters of a dataset and '+' to add values for one targeted parameter.++## 3 Quantization-## 2. Run Quantization+### Supported Quantization Configurations-- **Default Settings:**+AutoRound supports several quantization configurations:++- **Int8 Weight Only**+- **Int4 Weight Only**+- **Int3 Weight Only**+- **Int2 Weight Only**+- **Mixed bits Weight only**++### Hardware Compatibility++CPU, XPU, HPU,and CUDA for both quantization and inference.++### Command Line Usage+++- **AutoRound recipe:**++ This setting offers a better trade-off between accuracy and tuning cost, and is recommended in all scenarios.```bash- auto-round --model facebook/opt-125m --bits 4 --format "auto_round,auto_gptq" --disble_eval+ auto-round --model facebook/opt-125m --bits 4 --group_size 128 --format "auto_gptq,auto_awq,auto_round"+ ```++- **Best Settings:**++ This setting provides the best accuracy in most scenarios but is 4–5× slower than the standard AutoRound recipe. It is especially recommended for 2-bit quantization and is a good choice if sufficient resources are available.++- ```bash+ auto-round-best --model facebook/opt-125m --bits 4 --group_size 128 --format "auto_gptq,auto_awq,auto_round"```+- **Light Settings:**++ This setting offers the best speed (2 - 3X faster than AutoRound), but it may cause a significant accuracy drop for small models and 2-bit quantization. It is recommended for 4-bit settings and models larger than 3B++ ```bash+ auto-round-light --model facebook/opt-125m --bits 4 --group_size 128 --format "auto_gptq,auto_awq,auto_round"+ ```++### API usage+#### AutoRound API Usage+This setting offers a better trade-off between accuracy and tuning cost, and is recommended in all scenarios.++```python+from transformers import AutoModelForCausalLM, AutoTokenizer+from auto_round import AutoRound++model_name = "facebook/opt-125m"+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto")+tokenizer = AutoTokenizer.from_pretrained(model_name)+bits, group_size, sym = 4, 128, True+autoround = AutoRound(+ model,+ tokenizer,+ bits=bits,+ group_size=group_size,+ sym=sym,+ # enable_torch_compile=True,+)++output_dir = "./tmp_autoround"+# format= 'auto_round'(default), 'auto_gptq', 'auto_awq'+autoround.quantize_and_save(output_dir, format='auto_round')+```++#### mixed bits Usage+```python+from transformers import AutoModelForCausalLM, AutoTokenizer+from auto_round import AutoRound++model_name = "facebook/opt-125m"+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto")+tokenizer = AutoTokenizer.from_pretrained(model_name)+bits, group_size, sym = 4, 128, True+layer_config = {# Supports both full layer names and fuzzy (partial) matching+ "model.decoder.layers.6.self_attn.out_proj": {"bits": 8, "group_size": 32},+ "model.decoder.layers.*k_proj": {"bits": 2, "group_size": 32}+ }+autoround = AutoRound(+ model,+ tokenizer,+ bits=bits,+ group_size=group_size,+ sym=sym,+ layer_config=layer_config,+)++output_dir = "./tmp_autoround"+autoround.quantize_and_save(output_dir, format='auto_round')+```++#### AutoRoundBest recipe+This setting provides the best accuracy in most scenarios but is 4–5× slower than the standard AutoRound recipe. It is especially recommended for 2-bit quantization and is a good choice if sufficient resources are available.+```python+from transformers import AutoModelForCausalLM, AutoTokenizer+from auto_round import AutoRound++model_name = "facebook/opt-125m"+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto")+tokenizer = AutoTokenizer.from_pretrained(model_name)+bits, group_size, sym = 4, 128, True+autoround = AutoRound(+ model,+ tokenizer,+ bits=bits,+ group_size=group_size,+ sym=sym,+ nsamples=512,+ iters=1000,+ low_gpu_mem_usage=True+)++output_dir = "./tmp_autoround"+autoround.quantize_and_save(output_dir, format='auto_round')+```+#### AutoRoundLight recipe+This setting offers the best speed (2 - 3X faster than AutoRound), but it may cause a significant accuracy drop for small models and 2-bit quantization. It is recommended for 4-bit settings and models larger than 3B.++```python+from transformers import AutoModelForCausalLM, AutoTokenizer+from auto_round import AutoRound++model_name = "facebook/opt-125m"+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto")+tokenizer = AutoTokenizer.from_pretrained(model_name)+bits, group_size, sym = 4, 128, True+autoround = AutoRound(+ model,+ tokenizer,+ bits=bits,+ group_size=group_size,+ sym=sym,+ iters=50,+ lr=5e-3,+)++output_dir = "./tmp_autoround"+autoround.quantize_and_save(output_dir, format='auto_round')+```++### RTN mode+AutoRound also supports RTN (Round-To-Nearest) mode for fast, calibration-free baseline quantization. try setting iters=0 and use group_size=32 for better results.++```python+from transformers import AutoModelForCausalLM, AutoTokenizer+from auto_round import AutoRound++model_name = "facebook/opt-125m"+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto")+tokenizer = AutoTokenizer.from_pretrained(model_name)+bits, group_size, sym = 4, 32, True+autoround = AutoRound(+ model,+ tokenizer,+ bits=bits,+ group_size=group_size,+ sym=sym,+ iters=0,+)++output_dir = "./tmp_autoround"+autoround.quantize_and_save(output_dir, format='auto_round')+```++### GGUF format++This format is well-suited for CPU devices and is widely adopted by the community, **only q4_0 and q4_1 (W4G32) is supported in our repo**.++```python+from transformers import AutoModelForCausalLM, AutoTokenizer+from auto_round import AutoRound++model_name = "facebook/opt-125m"+model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto")+tokenizer = AutoTokenizer.from_pretrained(model_name)+bits, group_size, sym = 4, 32, True+autoround = AutoRound(+ model,+ tokenizer,+ bits=bits,+ group_size=group_size,+ sym=sym,+ iters=50,+ lr=5e-3,+)++output_dir = "./tmp_autoround"+autoround.quantize_and_save(output_dir, format='gguf:q4_0') # gguf:q4_1+```++### Adjust Hyperparameters+- **Reduced GPU Memory Usage:**- - enable low_gpu_mem_usage(more tuning cost)+ - enable `low_gpu_mem_usage`(more tuning cost)- - set "--train_bs 1 --gradient_accumulate_steps 8" (more tuning cost)+ - set `--bs 1 --gradient_accumulate_steps 8` (more tuning cost)- - reduce the train bs to 4(potential accuracy drop)+ - reduce the `bs` to 4(potential accuracy drop)- - reduce the seqlen to 512 (potential accuracy drop)+ - reduce the `seqlen` to 512 (potential accuracy drop)- or combine them-- **Reduced CPU Memory Usage (only available for .bin file currently):**+- **Reduced CPU Memory Usage :**++ - Trigger immediate packing: Packing will be triggered immediately when using the command-line interface or the+ quantize_and_save API, as long as only one export format is specified.- - set "--low_cpu_mem_mode 1" to use block-wise mode, load the weights from disk of each block when tuning and+ - (only available for .bin file currently) set "--low_cpu_mem_mode 1" to use block-wise mode, load the weights from+ disk of each block when tuning andrelease the memory of the block after tuning. (more tuning cost)- - set "--low_cpu_mem_mode 2" to use layer-wise mode, load the weights of each layer from disk when tuning, minimum- memory consumption and also slowest running speed.+ - (only available for .bin file currently) set "--low_cpu_mem_mode 2" to use layer-wise mode, load the weights of+ each layer from disk when tuning, minimum+ memory consumption and also the slowest running speed.- **Speedup the tuning:**+ - use `auto-round-light` configuration+- reduce the seqlen to 512(potential large accuracy drop for some scenarios)- reduce the train bs to 4(little accuracy drop. )@@ -101,62 +304,142 @@ See more about loading [huggingface dataset](https://huggingface.co/docs/dataset- **Enable quantized lm-head:**- Currently only support in AutoRound format inference for this config+ Currently only support in AutoRound format inference for this config```bashauto-round --model_name facebook/opt-125m --bits 4 --group_size 128 --quant_lm_head --format "auto_round"```-- **Enable marlin kernel:**-[//]: # ( - We support inference repacking for auto_round sym quantized models)+- **Utilize the AdamW Optimizer:**++ Include the flag `--adam`. Note that AdamW is less effective than sign gradient descent in many scenarios we tested.+-[//]: # ( ```python)+## 4 Inference-[//]: # ( from transformers import AutoModelForCausalLM, AutoTokenizer)+AutoRound automatically selects the best available backend based on the installed libraries and prompts the user to install additional libraries when a better backend is found.-[//]: # ( from auto_round import AutoRoundConfig)-[//]: # ( backend = "cuda_marlin" #supported in auto_round>0.3.1 and 'pip install -v gptqmodel --no-build-isolation'))+### CPU-[//]: # ( quantization_config = AutoRoundConfig(backend=backend))+Supports 2, 4, and 8 bits. We recommend using intel-extension-for-pytorch (IPEX) for 4 bits inference.-[//]: # ( quantized_model_path = "./tmp_autoround")+```python+from transformers import AutoModelForCausalLM, AutoTokenizer+from auto_round import AutoRoundConfig-[//]: # ( model = AutoModelForCausalLM.from_pretrained(quantized_model_path,)+model_name = "OPEA/Qwen2.5-1.5B-Instruct-int4-sym-inc"+model = AutoModelForCausalLM.from_pretrained(model_name, device_map="cpu", torch_dtype="auto")+tokenizer = AutoTokenizer.from_pretrained(model_name)+text = "There is a girl who likes adventure,"+inputs = tokenizer(text, return_tensors="pt").to(model.device)+print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))+```-[//]: # ( device_map=backend.split(':')[0], quantization_config=quantization_config))-[//]: # ( ```)- - To leverage auto-gptq marlin kernel, you need to install auto-gptq from source and export the model without sharding.+### XPU- ```bash- auto-round --model facebook/opt-125m --sym --bits 4 --group_size 128 --format "auto_gptq:marlin"- ```+Supports 4 bits only. We recommend using intel-extension-for-pytorch (IPEX) for inference.-- **Utilize the AdamW Optimizer:**+```python+from transformers import AutoModelForCausalLM, AutoTokenizer+from auto_round import AutoRoundConfig- Include the flag `--adam`. Note that AdamW is less effective than sign gradient descent in many scenarios we tested.+model_name = "OPEA/Qwen2.5-1.5B-Instruct-int4-sym-inc"+model = AutoModelForCausalLM.from_pretrained(model_name, device_map="xpu", torch_dtype="auto")+tokenizer = AutoTokenizer.from_pretrained(model_name)+text = "There is a girl who likes adventure,"+inputs = tokenizer(text, return_tensors="pt").to(model.device)+print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))+```+### CUDA-- **Code generation LLM:**+Supports 2, 3, 4, and 8 bits. We recommend using GPTQModel for 4 and 8 bits inference.- We utilized mbpp for calibration, but your own training dataset is highly recommended. Please note that samples with- seqlen < args.seqlen will be dropped in current version.+```python+from transformers import AutoModelForCausalLM, AutoTokenizer+from auto_round import AutoRoundConfig- ```bash- auto-round --model Salesforce/codegen25-7b-multi --bits 4 --dataset "mbpp" --seqlen 128 "- ```--## 4. Evaluation-### 4.1 Combine evaluation with tuning- - We leverage lm-eval-harnessing for the evaluation- ~~~bash- auto-round --model facebook/opt-125m --bits 4 --format "auto_round,auto_gptq" --tasks mmlu- ~~~- The last format will be used in evaluation if multiple formats have been exported.+model_name = "OPEA/Qwen2.5-1.5B-Instruct-int4-sym-inc"+model = AutoModelForCausalLM.from_pretrained(model_name, device_map="cuda", torch_dtype="auto")+tokenizer = AutoTokenizer.from_pretrained(model_name)+text = "There is a girl who likes adventure,"+inputs = tokenizer(text, return_tensors="pt").to(model.device)+print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))+```+++### HPU+docker image with Gaudi Software Stack is recommended. More details can be found+in [Gaudi Guide](https://docs.habana.ai/en/latest/).++```python+import habana_frameworks.torch.core as htcore+import habana_frameworks.torch.hpu as hthpu+from transformers import AutoModelForCausalLM,AutoTokenizer+from auto_round import AutoRoundConfig++model_name = "Intel/Qwen2-7B-int4-inc"+tokenizer = AutoTokenizer.from_pretrained(model_name)+model = AutoModelForCausalLM.from_pretrained(model_name).to('hpu').to(bfloat16)+text = "There is a girl who likes adventure,"+inputs = tokenizer(text, return_tensors="pt").to(model.device)+print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))+```+++### Specify Inference Backend++AutoRound automatically selects the backend for each layer based on compatibility. In general, the priority order is Marlin > ExLLaMAV2 > Triton, but the final choice depends on factors such as group size, bit width, packing format, hardware device, and other implementation details.++The backend may not always be the most suitable for certain devices.+You can specify your preferred backend such as "ipex" for CPU and XPU, "marlin/exllamav2/triton" for CUDA, according to your needs or hardware compatibility. Please note that additional corresponding libraries may be required.++```python+from transformers import AutoModelForCausalLM, AutoTokenizer, AutoRoundConfig++model_name = "OPEA/Qwen2.5-1.5B-Instruct-int4-sym-inc"+quantization_config = AutoRoundConfig(backend="ipex")+model = AutoModelForCausalLM.from_pretrained(model_name, device_map="cpu", quantization_config=quantization_config, torch_dtype="auto")+tokenizer = AutoTokenizer.from_pretrained(model_name)+text = "There is a girl who likes adventure,"+inputs = tokenizer(text, return_tensors="pt").to(model.device)+print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))+```+++### Convert GPTQ/AWQ to AutoRound++Most GPTQ/AWQ models can be converted to the AutoRound format for better compatibility and support with Intel devices. Please note that the quantization config will be changed if the model is serialized.++```python+from transformers import AutoModelForCausalLM, AutoTokenizer, AutoRoundConfig++model_name = "ybelkada/opt-125m-gptq-4bit"+quantization_config = AutoRoundConfig()+model = AutoModelForCausalLM.from_pretrained(model_name, device_map="cpu", quantization_config=quantization_config, torch_dtype="auto")+tokenizer = AutoTokenizer.from_pretrained(model_name)+text = "There is a girl who likes adventure,"+inputs = tokenizer(text, return_tensors="pt").to(model.device)+print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50, do_sample=False)[0]))+```+++## 5 Evaluation++### Combine evaluation with tuning++- We leverage lm-eval-harnessing for the evaluation.+If not explicitly specify '--task', the default value will be used (typically covering 10+ common tasks).+ ~~~bash+ auto-round --model facebook/opt-125m --bits 4 --format "auto_round,auto_gptq" --tasks mmlu+ ~~~+ The last format will be used in evaluation if multiple formats have been exported.++### Eval the Quantized model-### 4.2 Eval the Quantized model- AutoRound formatFor lm-eval-harness, you could just call~~~bash@@ -166,14 +449,15 @@ See more about loading [huggingface dataset](https://huggingface.co/docs/dataset~~~bashauto-round --model="your_model_path" --eval --device 0,1 --tasks lambada_openai --eval_bs 16~~~- For other evaluation framework, if the framework could support Huggingface models, typically it could support AutoRound format, only you need to do is import the following in the beginning of your code+ For other evaluation framework, if the framework could support Huggingface models, typically it could support+ AutoRound format, only you need to do is import the following in the beginning of your code~~~pythonfrom auto_round import AutoRoundConfig~~~-- AutoGPTQ/AutoAWQ format+- AutoGPTQ/AutoAWQ format- Please refer to their repo and check the evaluation framework's compatibility.+ Please refer to their repo and check the evaluation framework's compatibility.For lm-eval-harness, you could just call~~~bashlm_eval --model hf --model_args pretrained="your_model_path" --device cuda:0 --tasks lambada_openai --batch_size 16@@ -183,51 +467,8 @@ See more about loading [huggingface dataset](https://huggingface.co/docs/datasetCUDA_VISIBLE_DEVICES=0,1 lm_eval --model hf --model_args pretrained="your_model_path",parallelize=True --tasks lambada_openai --batch_size 16~~~-## Inference--### AutoRound format-**CPU**: **auto_round version >0.3.1**, pip install intel-extension-for-pytorch(much higher speed on Intel CPU) or pip-install intel-extension-for-transformers,--**HPU**: docker image with Gaudi Software Stack is recommended. More details can be found-in [Gaudi Guide](https://docs.habana.ai/en/latest/).--**CUDA**: no extra operations for sym quantization, for asym quantization, need to install auto-round from source+## 6 Known Issues-- The following code will automatically detect device, and typically some error message will remind you to install some extra libraries- ```python- from transformers import AutoModelForCausalLM, AutoTokenizer- from auto_round import AutoRoundConfig ## must import-- quantized_model_path = "./tmp_autoround"- device="cuda"- model = AutoModelForCausalLM.from_pretrained(quantized_model_path).to(device)- tokenizer = AutoTokenizer.from_pretrained(quantized_model_path)- text = "There is a girl who likes adventure,"- inputs = tokenizer(text, return_tensors="pt").to(model.device)- print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))- ```-- To specify device use a different backend-- ```python- from transformers import AutoModelForCausalLM, AutoTokenizer- from auto_round import AutoRoundConfig-- backend = "auto" ##cpu, hpu, cuda- quantization_config = AutoRoundConfig(- backend=backend- )- quantized_model_path = "./tmp_autoround"- model = AutoModelForCausalLM.from_pretrained(quantized_model_path,- device_map=backend.split(':')[0], quantization_config=quantization_config)- tokenizer = AutoTokenizer.from_pretrained(quantized_model_path)- text = "There is a girl who likes adventure,"- inputs = tokenizer(text, return_tensors="pt").to(model.device)- print(tokenizer.decode(model.generate(**inputs, max_new_tokens=50)[0]))- ```---## 6. Known Issues* Random quantization results in tuning some models-* ChatGlm-V1 is not supported\ No newline at end of file+* ChatGlm-V1 is not supporteddiff --git a/test/test_calib_dataset.py b/test/test_calib_dataset.pyindex dbf347a84..57990d9b1 100644--- a/test/test_calib_dataset.py+++ b/test/test_calib_dataset.py@@ -74,7 +74,7 @@ class TestLocalCalibDataset(unittest.TestCase):model_name = "Qwen/Qwen2.5-0.5B-Instruct"model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype="auto", trust_remote_code=True)tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)- dataset = "NeelNanda/pile-10k:apply_chat_template:system_prompt="""+ dataset = "NeelNanda/pile-10k:apply_chat_template:system_prompt=''"bits, group_size, sym = 4, 128, Trueautoround = AutoRound(model, tokenizer, bits=bits, group_size=group_size, sym=sym, iters=2, seqlen=128, dataset=dataset