Skip to content

A LLM trained only on data from certain time periods to reduce modern bias

Python

Update README.md

Hayk Grigorian883dfc659524
diff --git a/README.md b/README.md
index eb0f6d7..0bcf3a0 100644
--- a/README.md
+++ b/README.md
@@ -79,11 +79,6 @@ This should be doable on my own hardware, it's good too because I can hopefully
I will train this dataset today and it should take around 4-5 hours. Once it's done and I test it, I will give updates. Thank you again to everyone whos checking out my project, I've even had some people even giving me links to OCR resources so Thank you! I hope more people try this out and expirement with they're own datasets.
-## July 28th, 2025
-
-I've gone ahead and uploaded v0.5 to Hugging Face, [Check it out](https://huggingface.co/haykgrigorian/TimeCapsuleLLM) if youd like. You can now download my repo and run it locally. Unfortunately nanoGPT doesn't work natively with HuggingFace, so you'll have to download and run the model locally.
-
-Also I will begin curating data for my next training run, I believe I'll need maybe 5-10x more data to achieve reasoning capabilities.
### Training Update
@@ -93,6 +88,13 @@ I started training on a 435MB (108 M tokens) corpus, it's going pretty smooth ri
The training is done for the second model, it took my 4060 around 8 hours and 40 minutes (3,900 iters/hr) for 33,000 iters (5 epochs). Final train loss was 3.73. The outputs were suprisingly good it genuinely generates coherent 19th century style sentences now.
+## July 28th, 2025
+
+I've gone ahead and uploaded v0.5 to Hugging Face, [Check it out](https://huggingface.co/haykgrigorian/TimeCapsuleLLM) if youd like. You can now download my repo and run it locally. Unfortunately nanoGPT doesn't work natively with HuggingFace, so you'll have to download and run the model locally.
+
+Also I will begin curating data for my next training run, I believe I'll need maybe 5-10x more data to achieve reasoning capabilities.
+
+
# V0 Model Behavior & Limitations
Early prompts show the model responding with 1800's language and behavior. For example, I prompted it with "Who art Henry?" and it replied "I know that man, I have did not a black, the storm." and yeah that sentence makes no sense but the LLM is recognizing I'm asking about a person.