AirLLM 2.8T LLM inference with single 4GB GPU
Python chinese-llmchinese-nlpfinetunegenerative-aiinstruct-gptinstruction-setllamallmloraopen-modelsopen-sourceopen-source-modelsqlora
docs: fix QLoRA paper link
dajiaohuangc71fc476fec7
diff --git a/training/README_en.md b/training/README_en.mdindex 15ec4eb..94cc635 100644--- a/training/README_en.md+++ b/training/README_en.md@@ -43,7 +43,7 @@ Anima model is trained based on QLoRA's [33B guanaco](https://huggingface.co/timWe mainly use the Chinese training dataset put together by project [Chinese-Vicuna](https://github.com/Facico/Chinese-Vicuna): [guanaco_belle_merge_v1.0](https://huggingface.co/datasets/Chinese-Vicuna/guanaco_belle_merge_v1.0) in our finetune training work.* **Rationale**:-According to the conclusions in [QLoRA] (https://arxiv.org/abs/2305.14314)Appendix B.4 and Table 9's Grid Search: For QLoRA fine-tuning, a larger number of training samples is not necessarily better. 10,000 steps is a size with a relatively good ROI. Therefore, we want to choose a dataset with no less than 10,000 steps. The [Belle 10M](https://github.com/LianjiaTech/BELLE/blob/main/data/10M) dataset seems too big, and the data quality is unclear to us. Due to limited time, we will first choose guanaco_belle_merge_v1.0. Later, we will further test more datasets and the effects of data quality filtering in a more systematic way.+According to the conclusions in [QLoRA](https://arxiv.org/abs/2305.14314) Appendix B.4 and Table 9's Grid Search: For QLoRA fine-tuning, a larger number of training samples is not necessarily better. 10,000 steps is a size with a relatively good ROI. Therefore, we want to choose a dataset with no less than 10,000 steps. The [Belle 10M](https://github.com/LianjiaTech/BELLE/blob/main/data/10M) dataset seems too big, and the data quality is unclear to us. Due to limited time, we will first choose guanaco_belle_merge_v1.0. Later, we will further test more datasets and the effects of data quality filtering in a more systematic way.* **Acknowledgement**:Thanks [Chinese-Vicuna](https://github.com/Facico/Chinese-Vicuna)、[Belle](https://github.com/LianjiaTech/BELLE)、[GuanacoDataset](https://huggingface.co/datasets/JosephusCheung/GuanacoDataset) for their contributions to all the open datasets。