AirLLM 2.8T LLM inference with single 4GB GPU
Python chinese-llmchinese-nlpfinetunegenerative-aiinstruct-gptinstruction-setllamallmloraopen-modelsopen-sourceopen-source-modelsqlora
release: 3.1.0
Yu Li64a4e4fc3749
diff --git a/README.md b/README.mdindex 7585329..160cffd 100644--- a/README.md+++ b/README.md@@ -31,7 +31,7 @@* [Bloome — build & run AI agent teams in the cloud, zero setup](https://bloome.im/app?ref=G6BYnov0&utm_medium=github&utm_source=lyogavin-airllm-ivor-202606)## Updates-[2026/07] **Kimi K3 (2.8T)** support: the largest open-source model runs on a single card in **3.72GB** of VRAM, measured end to end on one RTX 6000 Ada. Per-expert streaming loads only the experts a token actually routes to. Note that K3 requires `flash-attn` — its own model code mandates it regardless of what you request — and therefore a CUDA 12 build of torch.+[2026/07] **Kimi K3 (2.8T)** support: the largest open-source model runs on a single card in **3.72GB** of VRAM, measured end to end on one RTX 6000 Ada. Per-expert streaming loads only the experts a token actually routes to. K3 brings three requirements of its own: `pip install compressed-tensors flash-attn` (its model code mandates flash attention regardless of what you request), a CUDA 12 build of torch, since no prebuilt flash-attn wheel exists for CUDA 13 yet, and `transformers` 4.56.x, as its remote code does not load on 5.x.[2026/06] **v3.0**: FP8 model support + the latest models. Run **DeepSeek-V3 (671B) on ~12GB** and **Qwen3-235B on ~3GB**, plus Qwen3, Llama 3.x/4, DeepSeek V2/V3, Phi-4, Gemma and more — all through a single `AutoModel`.diff --git a/air_llm/setup.py b/air_llm/setup.pyindex e1a8a62..3e08135 100644--- a/air_llm/setup.py+++ b/air_llm/setup.py@@ -15,11 +15,12 @@ for _readme in (os.path.join(here, "README.md"), os.path.join(here, os.pardir, "setuptools.setup(name="airllm",- version="3.0.1",+ version="3.1.0",author="Gavin Li",author_email="[email protected]",description="AirLLM runs 70B large language models on a single 4GB GPU without quantization, "- "distillation or pruning. 405B Llama 3.1 on 8GB, DeepSeek-V3 671B on ~12GB.",+ "distillation or pruning. 405B Llama 3.1 on 8GB, DeepSeek-V3 671B on ~12GB, "+ "Kimi K3 2.8T on under 4GB.",long_description=long_description,long_description_content_type="text/markdown",url="https://github.com/lyogavin/airllm",@@ -36,6 +37,8 @@ setuptools.setup('scipy','sentencepiece',# 'bitsandbytes' is optional (used only for --compression); we fall back gracefully when absent.+ # 'compressed-tensors' is optional too: only checkpoints stored in that format (Kimi K3's+ # MXFP4 weights) need it, and transformers raises a clear error naming it when it is missing.],classifiers=["Programming Language :: Python :: 3",