FreeToken brings datacenter-scale model serving to your desktop. Run massive models locally, fast and efficiently.
Commits on main
perf(attention): copy triton metadata without blocking the host (#726)
bc0e2fe Xiaoze Fan committed 1 month ago
feat(gguf): serve Llama and Mistral llama-arch GGUF checkpoints (#722)
21d9578 Xiaoze Fan committed 1 month ago
feat(moe): serve experts of several formats through per-format slot pools (#700)
e8c6bf1 Xiaoze Fan committed 1 month ago
feat(gguf): serve Qwen2, Qwen3 and Qwen3 MoE GGUF checkpoints (#707)
6c21dd5 Xiaoze Fan committed 1 month ago
fix(engine): upload cuda graphs at capture so a stream wait cannot block the first launch (#697)
9b68e00 Xiaoze Fan committed 1 month ago
feat(gguf): run experts whose gate_up and down mix float and quantized types (#702)
3381b6e Xiaoze Fan committed 1 month ago
feat(gguf): serve Qwen3.5 dense and MoE GGUF checkpoints (#701)
18448d2 Xiaoze Fan committed 1 month ago
fix(gguf): reject ggml weights past the 32-bit block offsets of MMQ / MMVQ (#699)
df8d450 Xiaoze Fan committed 1 month ago
fix(rocm): clamp QSA KV tile to the device LDS budget on RDNA (#587)
7d6490c Zhang Jason committed 1 month ago
feat(gguf): serve GGUF checkpoints on vendored ggml-cuda kernels (#690)
e5d3bd3 Xiaoze Fan committed 1 month ago
feat(moe): split the offload expert cache into slot pools (#689)
b416f85 Xiaoze Fan committed 1 month ago
perf(moe): split hybrid decode routes inside the ensure kernel (#685)
f02fbb1 Xiaoze Fan committed 1 month ago
refactor(cpu-moe): split the executor into weight formats and gpu sync backends (#670)
2010271 Xiaoze Fan committed 1 month ago
build(deps): move to flashlib 0.4.0 and drop the numpy<2.5 cap (#669)
9c635a1 Xiaoze Fan committed 1 month ago
feat(cpu-moe): serve block-fp8 experts on the cpu and hybrid backends (#662)
95ac86f Xiaoze Fan committed 1 month ago
fix(sampling): apply presence and frequency penalties (#527)
c818639 cherry77-cloud committed 1 month ago
build(deps): move to torch 2.14.1, triton 3.8 and sglang-kernel 0.4.9 (#630)
74a3877 Xiaoze Fan committed 1 month ago
fix(kernel): avoid nan in triton sampling softmax when a chunk starts with -inf (#654)
ee9789f Chen Cheng committed 1 month ago
perf(kernel): avoid the activation layout conversion in fp8 moe decode and read mxfp4 gemv weights as column pairs (#653)
2d8b9cc Xiaoze Fan committed 1 month ago
test(kernel): cover Triton activation and norm fallbacks (#652)
6c36af0 ZihaoMu committed 1 month ago
feat(deepseek_v4): serve image input on the V4-Flash-Vision release (#502)
23fc922 Mukkesh Ganesh committed 1 month ago
fix(kernel): correct jit tensor matcher checks and pdl arch guards (#638)
803a3b4 Xiaoze Fan committed 1 month ago
perf(kernel): skip masks and out-of-window tiles in triton extend attention (#632)
0781324 Xiaoze Fan committed 1 month ago
perf(kernel): keep 16-byte kv loads in dsv4 sparse attention and unroll gated pool (#633)
1c3aa0a Xiaoze Fan committed 1 month ago