Skip to content

🧠 两小时从 0 训出你的 64M 大模型。Train your own 64M LLM from scratch in 2 hours.

Commits on master

[update] aux loss
jingyaogong committed 10 months ago
9d89857
[fix] experts unused
jingyaogong committed 10 months ago
c65335b
[fix] layers set 8
jingyaogong committed 10 months ago
bc8fd82
[fix] moe unused
jingyaogong committed 10 months ago
5dd4df7
[feat] get params
jingyaogong committed 10 months ago
9236260
[feat] get params
jingyaogong committed 10 months ago
288a1d7
[feat] update config
jingyaogong committed 10 months ago
eead953
[feat] update lr
jingyaogong committed 10 months ago
6242980
[feat] compatible tokenizer
jingyaogong committed 10 months ago
936d105
[feat] stream load data
jingyaogong committed 10 months ago
4a5c9f5
[feat] remove empty_cache
jingyaogong committed 10 months ago
7eae14f
[feat] explicit left padding
jingyaogong committed 11 months ago
11b962d
[fix] lora weight
jingyaogong committed 11 months ago
a9c56b2
3a18fdd
[feat] adjust seq length
jingyaogong committed 11 months ago
fe24501
[feat] update readme
jingyaogong committed 11 months ago
fa82707
[fix] dtype & lr
jingyaogong committed 11 months ago
5129f0e
Merge pull request #571 from dyhuachi/dyhuachi-patch-1
jingyaogong committed 11 months ago
aa7dc0f
[fix] reduce aux_loss_alpha
jingyaogong committed 11 months ago
ecd1ae1
[fix] cuda memory #559
jingyaogong committed 11 months ago
5e1447b
[feat] update yarn
jingyaogong committed 11 months ago
151fdf7
[feat] release memory
jingyaogong committed 11 months ago
6b86ea3
[fix] ppo mask
jingyaogong committed 1 year ago
d7f4f4e
[fix] model attn_mask
jingyaogong committed 1 year ago
f5374dc
[fix] update model
jingyaogong committed 1 year ago
a044578
ce93946
[fix] prompt length calculate
jingyaogong committed 1 year ago
9c98cab