FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
Commits on main
feat(kernel): add an iq1_m mmq kernel for moe steps above 8 tokens (#835)
d2977fb Chen Cheng committed 21 days ago
fix(attention): size the decode KV tile to the shared-memory budget (#840)
c94535c festeraeb committed 21 days ago
refactor(layers): share one GatedDeltaNet op in layers/linear_attn (#841)
d043fb6 Xiaoze Fan committed 21 days ago
refactor(attention): move linear-attention metadata into a LinearAttnBackend (#837)
a2ce636 Xiaoze Fan committed 22 days ago
refactor: add engine and scheduler seams for multi-row decode steps (#833)
e328cf6 Xiaoze Fan committed 22 days ago
feat(scheduler): add radix_lookahead keys to the prefix cache (#834)
9221fd7 Xiaoze Fan committed 23 days ago
fix(server): recover Qwen tool calls after unclosed think (#825)
075cf57 Julien Pierre committed 23 days ago
feat(gguf): serve image input on minimax-m3 ggufs from a minimax_m3 mmproj (#829)
408e6ae Xiaoze Fan committed 23 days ago
feat(gguf): serve image input on muse-glimmer ggufs from a muse-glimmer mmproj (#830)
edab62c Xiaoze Fan committed 23 days ago
feat(gguf): serve image input on qwen3vl ggufs from a deepstack qwen3vl_merger mmproj (#827)
380693a Mukkesh Ganesh committed 23 days ago
fix(models): discover safetensors shards from the index (#819)
0bf7dbb Chris Qian committed 25 days ago
feat(gguf): serve image input on deepseek4 ggufs from a deepseek4v mmproj (#816)
271ff07 Mukkesh Ganesh committed 26 days ago
fix(e2e): respect cuda device visibility in aime memory probe (#821)
756108f Jesús Jódar Piernas committed 26 days ago
feat(flint): add nvfp4 modelopt and gguf support (#797)
2d93085 Jesús Jódar Piernas committed 26 days ago
feat: add --embed-weights host for a pinned host embedding table (#806)
fd863cc Xiaoze Fan committed 27 days ago
fix(kernel): convert fp32 to e4m3 in one rounding on sm_89 (#805)
6505b6a Chen Cheng committed 27 days ago
perf(attention): drop redundant activation copies in glm5 dsa prefill (#795)
0c50eb9 Mukkesh Ganesh committed 27 days ago
feat(gguf): serve image input on glm5-next ggufs from a glm5v mmproj (#810)
0b98e9d Mukkesh Ganesh committed 27 days ago
fix(rotary): preserve fp32 inputs in mrope torch fallbacks (#641)
507a8bf cherry77-cloud committed 27 days ago
feat(server): add /v1/cache/flush to drop the prefix cache when idle (#811)
e0a7e23 Xiaoze Fan committed 27 days ago
feat(gguf): read NVFP4 token_embd rows in the vendored get_rows (#801)
20d5d93 Manish Biswal committed 27 days ago
feat(gguf): serve qwen3vl and qwen3vlmoe gguf checkpoints text only (#807)
ed359ee Mukkesh Ganesh committed 27 days ago
perf(moe): fuse sigmoid + selection-bias routing into a triton kernel (#804)
19a60a5 Xiaoze Fan committed 27 days ago
perf(moe): skip inactive block-fp8 decode routes (#665)
7974c9c cherry77-cloud committed 27 days ago
fix(tests): pin host-dependent probes in nvfp4 kernel and graph capture tests (#799)
00fe83b Manish Biswal committed 27 days ago
feat(gguf): take image input on Qwen and Gemma-4 GGUF checkpoints from an mmproj (#803)
609191d Xiaoze Fan committed 27 days ago