A LLM trained only on data from certain time periods to reduce modern bias
Python
Update README.md
Hayk Grigorian23af61596864
diff --git a/README.md b/README.mdindex b47f30f..c88d360 100644--- a/README.md+++ b/README.md@@ -142,6 +142,10 @@ You can train a different LLM if you want, but I used nanoGPT# FAQ+## What is Selective Temporal Training ?++Selective Temporal Training (STT) is a machine learning methodology where all training data is specifically curated to fall within a specific historical time period. It's done in order to model the language and knowledge of that era without influence from modern concepts. For example, the current model I have now (v0.5) is trained on data exclusively from 1800-1875, it's not fine tuned but trained from scratch resulting in output that reflects the linguistic style and historical context of that time period.+## Why not just use fine-tuning or LoRA?For this project I'm trying to create a language model that is unclouded from modern bias. If I fine-tune something like GPT-2, it's already pre-trained and that information won't go away. If I train from scratch the language model won't pretend to be old, it just will be. The Goal for this project right now is to create something can reason exclusively using knowledge from London books published between 1800 and 1850.