Skip to content

A LLM trained only on data from certain time periods to reduce modern bias

Python

Update README.md

Hayk Grigorianfcf95b368998
diff --git a/README.md b/README.md
index d1f6a53..4b53aad 100644
--- a/README.md
+++ b/README.md
@@ -47,6 +47,9 @@ This should be doable on my own hardware, it's good too because I can hopefully
I will train this dataset today and it should take around 4-5 hours. Once it's done and I test it, I will give updates. Thank you again to everyone whos checking out my project, I've even had some people even giving me links to OCR resources so Thank you! I hope more people try this out and expirement with they're own datasets.
+### Training Update
+
+I started training on a 435MB (108 M tokens) corpus, it's going pretty smooth right now. Train loss dropped from 10.9 to 4.9 in the first 2800 iterations. I expect it'll take around 8 or 9 hours to complete. I'll post another update once it's done.
# Current Model Behavior & Limitations