FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
Commits on main
perf(kernel): read dsv41 indexer key rows as int32 words (#634)
799bec3 Xiaoze Fan committed 1 month ago
perf(kernel): avoid layout conversions in nvfp4 marlin decode (#631)
7644984 Xiaoze Fan committed 1 month ago
perf(moe): spin-wait the cpu moe pool between layers (#619)
1c183e7 Xiaoze Fan committed 1 month ago
perf(moe): Improved `fast_index_copy_multi_jit` for NVLink-C2C hosts (#610)
c1a504a SuperHotDogCat committed 1 month ago
fix(kvcache): clone inserted radix keys to release request buffers (#559)
555efd8 cherry77-cloud committed 1 month ago
fix(server): keep Anthropic inline system messages in place for prefix reuse (#608)
93b0a59 William Xinze Zheng committed 1 month ago
feat(moe): serve NVFP4 and MXFP4 experts fused through the expert banks (#601)
189c8ad Xiaoze Fan committed 1 month ago
fix(ple): fence disk staging against in-flight lookups (#588)
d3512b4 Xiaoze Fan committed 1 month ago
feat(rotary): size the rope table for YaRN and add --hf-overrides (#573)
235e201 YevheniiKotyrlo committed 1 month ago
feat(install): record the install root as FREETOKEN_HOME in environment.d (#576)
eba7120 Xiaoze Fan committed 1 month ago
perf(qsa): reduce block top-k candidates in levels past 16 splits (#571)
5031e3d YevheniiKotyrlo committed 1 month ago
fix(runtime): correct overlap token limits and cache safety (#546)
cee23bf Xiaoze Fan committed 1 month ago
fix(install): pin flashinfer packages to one version (#545)
71eb1f0 Xiaoze Fan committed 1 month ago
fix(models): preserve glm4 router correction bias in fp32 (#469)
cab110e cherry77-cloud committed 1 month ago
fix(engine): preserve greedy sampling in mixed batches (#471)
cc1f5c2 cherry77-cloud committed 2 months ago
fix(install): resolve sglang-kernel from PyPI, not the sglang index (#524)
a8eb986 Xiaoze Fan committed 2 months ago
fix(engine): raise WeightLoadError when the checkpoint cannot be read (#518)
6410c97 Xiaoze Fan committed 2 months ago
fix(checkpoint): serve and repair FTW checkpoints around the vision encoder (#486)
cade1a9 Xiaoze Fan committed 2 months ago
chore(docs): update wechat group qr code link to #482 (#485)
afd99cb Xiaoze Fan committed 2 months ago
feat(muse_glimmer): serve image input on Muse-Glimmer-30B (#481)
63d6471 Xiaoze Fan committed 2 months ago
fix(engine): resolve moe-strategy auto to fused on unified-memory GPUs (GB10) (#445)
bea8d06 Manish Biswal committed 2 months ago
feat(glm5_next): serve image input on GLM-5.3-Flash (#479)
3faef36 Xiaoze Fan committed 2 months ago