🧠 两小时从 0 训出你的 64M 大模型。Train your own 64M LLM from scratch in 2 hours.
update model
gongjye4ad822c402b
diff --git a/README.md b/README.mdindex 8ced660..18cd8c7 100644--- a/README.md+++ b/README.md@@ -152,7 +152,7 @@ streamlit run fast_inference.py* 0、环境安装```bash- pip install -r requirements.txt+ pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple```* 1、克隆项目代码```text@@ -702,10 +702,11 @@ minimind模型本身没有使用较大的数据集训练,也没有针对回答## 🤝[贡献者](https://github.com/jingyaogong/minimind/graphs/contributors)-[//]: # (<br/>)-[//]: # (<a href="https://github.com/jingyaogong/minimind/graphs/contributors">)-[//]: # ( <img src="https://contrib.rocks/image?repo=jingyaogong/minimind&v=3" />)-[//]: # (</a>)+<!--+<a href="https://github.com/jingyaogong/minimind/graphs/contributors">+ <img src="https://contrib.rocks/image?repo=jingyaogong/minimind&v3" />+</a>+--><a href="https://github.com/jingyaogong"><img src="https://avatars.githubusercontent.com/u/62287848" width="70px" height="70px"/></a> <a href="https://github.com/MuWinds"><img src="https://avatars.githubusercontent.com/u/93832089" width="70px" height="70px"/></a> diff --git a/README_en.md b/README_en.mdindex 81fdc00..a47b9e5 100644--- a/README_en.md+++ b/README_en.md@@ -779,20 +779,16 @@ your model with third-party UIs, such as fastgpt, OpenWebUI, etc.## 🤝[Contributors](https://github.com/jingyaogong/minimind/graphs/contributors)-[//]: # (<br/>)--[//]: # (<a href="https://github.com/jingyaogong/minimind/graphs/contributors">)--[//]: # ( <img src="https://contrib.rocks/image?repo=jingyaogong/minimind&v=3" />)+<!--+<a href="https://github.com/jingyaogong/minimind/graphs/contributors">+ <img src="https://contrib.rocks/image?repo=jingyaogong/minimind&v3" />+</a>+-->-[//]: # (</a>)+<a href="https://github.com/jingyaogong"><img src="https://avatars.githubusercontent.com/u/62287848" width="70px" height="70px"/></a> +<a href="https://github.com/MuWinds"><img src="https://avatars.githubusercontent.com/u/93832089" width="70px" height="70px"/></a> +<a href="https://github.com/chuanzhubin"><img src="https://avatars.githubusercontent.com/u/2813798" width="70px" height="70px"/></a> -<a href="https://github.com/jingyaogong"><img src="https://avatars.githubusercontent.com/u/62287848" width="70px" height="70px"/></a>- -<a href="https://github.com/MuWinds"><img src="https://avatars.githubusercontent.com/u/93832089" width="70px" height="70px"/></a>- -<a href="https://github.com/chuanzhubin"><img src="https://avatars.githubusercontent.com/u/2813798" width="70px" height="70px"/></a>- ## 😊Thanks fordiff --git a/model/LMConfig.py b/model/LMConfig.pyindex 5e958ba..bf0e4b9 100644--- a/model/LMConfig.py+++ b/model/LMConfig.py@@ -17,7 +17,7 @@ class LMConfig(PretrainedConfig):norm_eps: float = 1e-5,max_seq_len: int = 512,dropout: float = 0.0,- flash_attn: bool = False,+ flash_attn: bool = True,##################################################### Here are the specific configurations of MOE# When use_moe is false, the following is invaliddiff --git a/requirements.txt b/requirements.txtindex 516a4eb..5c5bedd 100644--- a/requirements.txt+++ b/requirements.txt@@ -1,6 +1,5 @@datasets==2.16.1datasketch==1.6.4-flash_attn==2.5.0Flask==3.0.3Flask_Cors==4.0.0jieba==0.42.1@@ -20,7 +19,7 @@ scikit_learn==1.5.1sentence_transformers==2.3.1simhash==2.1.2tiktoken==0.5.1-torch==2.1.2+cu121+torch==2.1.2transformers==4.44.0jinja2==3.1.2jsonlines==4.0.0