A straightforward method for training your LLM, from downloading data to generating text.
Commits on main
test: run every training script end to end on generated data
1a2ac8a FareedKhan-dev committed 1 month ago
docs: contributing guide, issue and pull request templates, pre-commit hooks
f07be23 FareedKhan-dev committed 1 month ago
ci: lint, type-check and test on Linux, Windows and macOS
8c91fe8 FareedKhan-dev committed 1 month ago
feat(inference): speculative decoding and int8 weight quantization
2ebb64c FareedKhan-dev committed 1 month ago
test: check tensor shapes and types at runtime during the tests
52aa613 FareedKhan-dev committed 1 month ago
build: Python 3.10+, a uv dev group, and ruff / mypy / pytest settings
a58dc52 FareedKhan-dev committed 1 month ago
feat(scripts): benchmark the training speed of each preset on your machine
a97a7a7 FareedKhan-dev committed 1 month ago
feat(train): laptop presets and a modern-architecture switch for train_transformer.py
9721065 FareedKhan-dev committed 1 month ago
feat(generate): read size, architecture, tokenizer and window from the checkpoint
cdb2757 FareedKhan-dev committed 1 month ago
feat(presets): named model sizes from a laptop CPU to the original 3B
cd97a4d FareedKhan-dev committed 1 month ago
feat(tokenizer): byte-level BPE tokenizer written from scratch
6323899 FareedKhan-dev committed 1 month ago
fix(sft): apply grad_accum, which the config set but the trainer ignored
3f42da8 FareedKhan-dev committed 1 month ago
feat(pretrain): choose the optimizer and the learning-rate schedule
4d08b65 FareedKhan-dev committed 1 month ago
feat(models): build either architecture from one config
ad9aaeb FareedKhan-dev committed 1 month ago
feat(models): a modern decoder next to the classic Transformer
0cca130 FareedKhan-dev committed 1 month ago
feat(models): grouped-query and multi-head latent attention
0f9596d FareedKhan-dev committed 1 month ago
feat(models): Mixture-of-Experts layer with a load-balancing loss
63b4299 FareedKhan-dev committed 1 month ago
feat(models): RMSNorm, rotary embeddings, SwiGLU and a KV cache
70bc898 FareedKhan-dev committed 1 month ago
feat(config): typed, validated configs with clear errors
7554e51 FareedKhan-dev committed 1 month ago
fix(ui): read checkpoints, data and logs from the repo folders
170d4ac FareedKhan-dev committed 1 month ago