Skip to content

The best ChatGPT that $100 can buy.

Python

fix bug in chat_sft, the attention window must be preserved sigh

Andrej Karpathy8b4849d5480a
diff --git a/scripts/chat_sft.py b/scripts/chat_sft.py
index cad0d81..4c81f06 100644
--- a/scripts/chat_sft.py
+++ b/scripts/chat_sft.py
@@ -301,6 +301,7 @@ while True:
"n_head": model.config.n_head,
"n_kv_head": model.config.n_kv_head,
"n_embd": model.config.n_embd,
+ "window_pattern": model.config.window_pattern,
},
"user_config": user_config, # inputs to the training script
}