Skip to content

A straightforward method for training your LLM, from downloading data to generating text.

982 B
Raw Blame
"""
From-scratch post-training suite for ``train-llm-from-scratch``.

This package adds Supervised Fine-Tuning (SFT), reward modeling, PPO, DPO and
GRPO/RLVR on top of the repo's own custom :class:`~src.models.transformer.Transformer`.
Everything here is written in plain PyTorch in the same teaching style as the rest
of the repo -- no ``trl`` / ``peft`` / ``transformers`` for the algorithms.

Module map
----------
- ``chat_template``  : tiktoken r50k_base chat formatting + loss masking
- ``rollout``        : sampling + per-token log-prob utilities (the RL core)
- ``value_head``     : actor-critic wrapper for PPO
- ``reward_model``   : scalar reward head wrapper
- ``utils``          : frozen copies, checkpoint I/O, masked reductions, seeding
- ``distributed``    : optional pure-``torch.distributed`` DDP helpers
- ``sft`` / ``reward_train`` / ``dpo`` / ``ppo`` / ``grpo`` : the algorithms
- ``rewards``        : verifiable reward functions (GSM8K, arithmetic, format)
"""