PTQ-Level Efficiency, QAT-Level Accuracy | 训练后量化的效率,量化感知训练的精度
Update README.md
Haihao Shenbef3ebd2020a
diff --git a/README.md b/README.mdindex 396cc142a..863f49fea 100644--- a/README.md+++ b/README.md@@ -35,14 +35,14 @@ pip install auto-round```## Model quantization-### CPU/ Gaudi2/ GPU+### Gaudi2/ CPU/ GPU```pythonimport torchfrom transformers import AutoModelForCausalLM, AutoTokenizer-tuning_device = "cuda:0" ## or "cpu", "hpu"-dtype = "auto" if tuning_device != "hpu" else torch.bfloat16+device = "hpu" ## or "cpu", "cuda:0"+dtype = "auto" if device != "hpu" else torch.bfloat16model_name = "meta-llama/Llama-2-7b-hf"model = AutoModelForCausalLM.from_pretrained(model_name, torch_dtype=dtype, trust_remote_code=True)tokenizer = AutoTokenizer.from_pretrained(model_name, trust_remote_code=True)@@ -56,14 +56,12 @@ output_dir = "./tmp_autoround"autoround.save_quantized(output_dir)```--## Model inferencePlease run the quantization code first.-### Intel CPU+### CPU```python# Please save the quantized model in 'itrex' format first, then refer to the ITREX tutorial for more details on inference with the INT4 model.# (https://github.com/intel/intel-extension-for-transformers/tree/main/intel_extension_for_transformers/llm/runtime/neural_speed)