Skip to content

AirLLM 2.8T LLM inference with single 4GB GPU

release: 3.1.0

Yu Li64a4e4fc3749
diff --git a/README.md b/README.md
index 7585329..160cffd 100644
--- a/README.md
+++ b/README.md
@@ -31,7 +31,7 @@
* [Bloome — build & run AI agent teams in the cloud, zero setup](https://bloome.im/app?ref=G6BYnov0&utm_medium=github&utm_source=lyogavin-airllm-ivor-202606)
## Updates
-[2026/07] **Kimi K3 (2.8T)** support: the largest open-source model runs on a single card in **3.72GB** of VRAM, measured end to end on one RTX 6000 Ada. Per-expert streaming loads only the experts a token actually routes to. Note that K3 requires `flash-attn` — its own model code mandates it regardless of what you request — and therefore a CUDA 12 build of torch.
+[2026/07] **Kimi K3 (2.8T)** support: the largest open-source model runs on a single card in **3.72GB** of VRAM, measured end to end on one RTX 6000 Ada. Per-expert streaming loads only the experts a token actually routes to. K3 brings three requirements of its own: `pip install compressed-tensors flash-attn` (its model code mandates flash attention regardless of what you request), a CUDA 12 build of torch, since no prebuilt flash-attn wheel exists for CUDA 13 yet, and `transformers` 4.56.x, as its remote code does not load on 5.x.
[2026/06] **v3.0**: FP8 model support + the latest models. Run **DeepSeek-V3 (671B) on ~12GB** and **Qwen3-235B on ~3GB**, plus Qwen3, Llama 3.x/4, DeepSeek V2/V3, Phi-4, Gemma and more — all through a single `AutoModel`.
diff --git a/air_llm/setup.py b/air_llm/setup.py
index e1a8a62..3e08135 100644
--- a/air_llm/setup.py
+++ b/air_llm/setup.py
@@ -15,11 +15,12 @@ for _readme in (os.path.join(here, "README.md"), os.path.join(here, os.pardir, "
setuptools.setup(
name="airllm",
- version="3.0.1",
+ version="3.1.0",
author="Gavin Li",
author_email="[email protected]",
description="AirLLM runs 70B large language models on a single 4GB GPU without quantization, "
- "distillation or pruning. 405B Llama 3.1 on 8GB, DeepSeek-V3 671B on ~12GB.",
+ "distillation or pruning. 405B Llama 3.1 on 8GB, DeepSeek-V3 671B on ~12GB, "
+ "Kimi K3 2.8T on under 4GB.",
long_description=long_description,
long_description_content_type="text/markdown",
url="https://github.com/lyogavin/airllm",
@@ -36,6 +37,8 @@ setuptools.setup(
'scipy',
'sentencepiece',
# 'bitsandbytes' is optional (used only for --compression); we fall back gracefully when absent.
+ # 'compressed-tensors' is optional too: only checkpoints stored in that format (Kimi K3's
+ # MXFP4 weights) need it, and transformers raises a clear error naming it when it is missing.
],
classifiers=[
"Programming Language :: Python :: 3",