FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
Commits on main
feat(gguf): serve DeepSeek-V4 Flash and Pro GGUF checkpoints on deepseek4 (#774)
546c5fe Mukkesh Ganesh committed 29 days ago
docs: add a guide to speeding up prefill with offloaded experts (#785)
b10c0c4 Xiaoze Fan committed 29 days ago
perf(moe): scan the hybrid eviction pool in chunks and add fetch rounding modes (#778)
161ffa5 Xiaoze Fan committed 29 days ago
feat(moe): fetch locked-layer experts through pinned staging (#742)
c5b5b72 Xiaoze Fan committed 1 month ago
feat(gguf): serve GLM-5.3-Flash GGUF checkpoints on glm5-next (#721)
cc56b6b Mukkesh Ganesh committed 1 month ago
fix(server): accept localhost authority in local file URIs (#758)
bb15d1d Your Open Source Tooling Specialist committed 1 month ago
feat(scripts): add checkpoint diagnosis and fix scripts (#762)
70550f8 Xiaoze Fan committed 1 month ago
fix(qwen4_exp): load gguf files with a quantized indexer (#760)
64b85c3 Xiaoze Fan committed 1 month ago
fix(qwen3_5_moe)!: apply the zero-centred norm weights' +1 in fp32 at runtime (#755)
05e7cb7 Xiaoze Fan committed 1 month ago
fix(qwen4_exp): read weights under the checkpoint's quant config (#759)
163d1de Xiaoze Fan committed 1 month ago
feat(gguf): add scripts that check which gguf files load (#756)
5596e6b Xiaoze Fan committed 1 month ago
perf(linear): run nvfp4 linears on nvfp4 activations on sm_120 (#737)
4d2e2e2 Xiaoze Fan committed 1 month ago
fix(muse_glimmer)!: keep the attention gate out of the qkv fusion (#741)
30dd715 Xiaoze Fan committed 1 month ago
perf(moe): prefill nvfp4 moe experts on nvfp4 activations on sm_120 (#736)
e4f69d9 Xiaoze Fan committed 1 month ago
perf(moe): pipeline the nvfp4 prefill scale loads on geforce sm_86 / sm_89 (#735)
fd410ee Xiaoze Fan committed 1 month ago
feat(moe): bench hybrid fetch fractions per moe layer kind (#734)
8f49e0a Xiaoze Fan committed 1 month ago
perf(moe): speed up the nvfp4 moe prefill gemm on rtx gpus (#733)
ac6a15d Xiaoze Fan committed 1 month ago
feat(gguf): apply the weight scales of nvfp4 gguf tensors (#732)
31c6ef4 Xiaoze Fan committed 1 month ago
feat(gguf): serve GLM-4.5, GLM-4.6 and GLM-4.7 GGUF checkpoints (#731)
6b8f200 Xiaoze Fan committed 1 month ago
perf(kernel): split triton decode attention by context length and sm count (#729)
11c1adb Xiaoze Fan committed 1 month ago
feat(cpu-moe): serve GGUF experts on the cpu and hybrid backends through ggml-cpu (#704)
c8559fd Xiaoze Fan committed 1 month ago
perf(kernel): rewrite the triton nvfp4 moe prefill gemm (#723)
03c2c9c Xiaoze Fan committed 1 month ago
feat(gguf): serve MiniMax-M2 and MiniMax-M3 GGUF checkpoints (#725)
4663b2b Xiaoze Fan committed 1 month ago
fix(gguf): set ignore_merges for llama 3 gguf tokenizers (#728)
e0dbb7d Xiaoze Fan committed 1 month ago
test(server): list the llama and qwen2 gguf archs in the parser auto-selection tables (#727)
312f324 Xiaoze Fan committed 1 month ago