🧠 两小时从 0 训出你的 64M 大模型。Train your own 64M LLM from scratch in 2 hours.
Commits on master
Merge pull request #868 from DaoyuanLi2816/fix/final-accumulation-checkpoint
6fff76f jingyaogong committed 1 month ago
Merge pull request #867 from DaoyuanLi2816/fix/lora-inference-paths
7da0f92 jingyaogong committed 1 month ago
fix: update residual gradients before saving final checkpoints
a6ffddc Daoyuan Li committed 1 month ago
Merge pull request #769 from qizwiz/fix/llm-response-unguarded
b00be3e jingyaogong committed 2 months ago
Merge pull request #821 from lycDragonSlayer/refactor/remove-redundant-update-policy
7f5ba40 jingyaogong committed 2 months ago
Merge pull request #829 from DaoyuanLi2816/fix/dpo-paired-think-format
949488f jingyaogong committed 2 months ago
Merge pull request #863 from Linxiushen/fix-fp16-attention-mask-nan
e41bf7c jingyaogong committed 2 months ago
Merge pull request #860 from Linxiushen/fix-tokenizer-pretrain-format
b330a4d jingyaogong committed 2 months ago
Merge pull request #865 from Linxiushen/fix-requirements-missing-fastapi
b956bad jingyaogong committed 2 months ago
Merge pull request #864 from Linxiushen/fix-nonstream-max-tokens
d698c06 jingyaogong committed 2 months ago
[fix] requirements.txt 缺 fastapi/uvicorn,按 README 跑 serve_openai_api.py 直接崩
984a2c3 Linxiushen committed 2 months ago
[fix] float16 下 attention mask 的 -1e9 溢出成 -inf,导致整条序列输出 NaN
ba10c4d Linxiushen committed 2 months ago
Merge pull request #857 from wbbeyourself/fix-ctrl-c-exception
e2ee878 jingyaogong committed 2 months ago
Merge pull request #859 from basil-k-aji-dev/fix/moe-router-gradient-topk1
1aae259 jingyaogong committed 2 months ago
[fix] keep the MoE router trainable at num_experts_per_tok=1
1c87643 basil-k-aji-dev committed 2 months ago
Merge pull request #855 from basil-k-aji-dev/perf/ddp-followups
7988bc8 jingyaogong committed 2 months ago
[perf] skip the per-forward RoPE buffer broadcast in the RL trainers
37ac206 basil-k-aji-dev committed 2 months ago
Merge pull request #847 from a0917-cell/fix/safe-math-eval
cdf0a93 jingyaogong committed 2 months ago
Merge pull request #854 from basil-k-aji-dev/fix/rl-ddp-grad-sync
c367572 jingyaogong committed 2 months ago
[fix] route RL trainer forward through DDP so gradients all-reduce
9c088dd basil-k-aji-dev committed 2 months ago