Skip to content

🧠 两小时从 0 训出你的 64M 大模型。Train your own 64M LLM from scratch in 2 hours.

update model

gongjye4ad822c402b
diff --git a/README.md b/README.md
index 8ced660..18cd8c7 100644
--- a/README.md
+++ b/README.md
@@ -152,7 +152,7 @@ streamlit run fast_inference.py
* 0、环境安装
```bash
- pip install -r requirements.txt
+ pip install -r requirements.txt -i https://pypi.tuna.tsinghua.edu.cn/simple
```
* 1、克隆项目代码
```text
@@ -702,10 +702,11 @@ minimind模型本身没有使用较大的数据集训练,也没有针对回答
## 🤝[贡献者](https://github.com/jingyaogong/minimind/graphs/contributors)
-[//]: # (<br/>)
-[//]: # (<a href="https://github.com/jingyaogong/minimind/graphs/contributors">)
-[//]: # ( <img src="https://contrib.rocks/image?repo=jingyaogong/minimind&v=3" />)
-[//]: # (</a>)
+<!--
+<a href="https://github.com/jingyaogong/minimind/graphs/contributors">
+ <img src="https://contrib.rocks/image?repo=jingyaogong/minimind&v3" />
+</a>
+-->
<a href="https://github.com/jingyaogong"><img src="https://avatars.githubusercontent.com/u/62287848" width="70px" height="70px"/></a>&nbsp;
<a href="https://github.com/MuWinds"><img src="https://avatars.githubusercontent.com/u/93832089" width="70px" height="70px"/></a>&nbsp;
diff --git a/README_en.md b/README_en.md
index 81fdc00..a47b9e5 100644
--- a/README_en.md
+++ b/README_en.md
@@ -779,20 +779,16 @@ your model with third-party UIs, such as fastgpt, OpenWebUI, etc.
## 🤝[Contributors](https://github.com/jingyaogong/minimind/graphs/contributors)
-[//]: # (<br/>)
-
-[//]: # (<a href="https://github.com/jingyaogong/minimind/graphs/contributors">)
-
-[//]: # ( <img src="https://contrib.rocks/image?repo=jingyaogong/minimind&v=3" />)
+<!--
+<a href="https://github.com/jingyaogong/minimind/graphs/contributors">
+ <img src="https://contrib.rocks/image?repo=jingyaogong/minimind&v3" />
+</a>
+-->
-[//]: # (</a>)
+<a href="https://github.com/jingyaogong"><img src="https://avatars.githubusercontent.com/u/62287848" width="70px" height="70px"/></a>&nbsp;
+<a href="https://github.com/MuWinds"><img src="https://avatars.githubusercontent.com/u/93832089" width="70px" height="70px"/></a>&nbsp;
+<a href="https://github.com/chuanzhubin"><img src="https://avatars.githubusercontent.com/u/2813798" width="70px" height="70px"/></a>&nbsp;
-<a href="https://github.com/jingyaogong"><img src="https://avatars.githubusercontent.com/u/62287848" width="70px" height="70px"/></a>
-&nbsp;
-<a href="https://github.com/MuWinds"><img src="https://avatars.githubusercontent.com/u/93832089" width="70px" height="70px"/></a>
-&nbsp;
-<a href="https://github.com/chuanzhubin"><img src="https://avatars.githubusercontent.com/u/2813798" width="70px" height="70px"/></a>
-&nbsp;
## 😊Thanks for
diff --git a/model/LMConfig.py b/model/LMConfig.py
index 5e958ba..bf0e4b9 100644
--- a/model/LMConfig.py
+++ b/model/LMConfig.py
@@ -17,7 +17,7 @@ class LMConfig(PretrainedConfig):
norm_eps: float = 1e-5,
max_seq_len: int = 512,
dropout: float = 0.0,
- flash_attn: bool = False,
+ flash_attn: bool = True,
####################################################
# Here are the specific configurations of MOE
# When use_moe is false, the following is invalid
diff --git a/requirements.txt b/requirements.txt
index 516a4eb..5c5bedd 100644
--- a/requirements.txt
+++ b/requirements.txt
@@ -1,6 +1,5 @@
datasets==2.16.1
datasketch==1.6.4
-flash_attn==2.5.0
Flask==3.0.3
Flask_Cors==4.0.0
jieba==0.42.1
@@ -20,7 +19,7 @@ scikit_learn==1.5.1
sentence_transformers==2.3.1
simhash==2.1.2
tiktoken==0.5.1
-torch==2.1.2+cu121
+torch==2.1.2
transformers==4.44.0
jinja2==3.1.2
jsonlines==4.0.0