Get up and running with Kimi, GLM, MiniMax, DeepSeek, gpt-oss, Qwen, Gemma and other models.
Commits on main
launch: disable claude code token countdown to preserve KV cache (#17918)
add1f92 Devon Rifkin committed 3 months ago
mlxrunner: keep prefill snapshots when a request is cancelled mid-prompt
c44575e Jesse Gross committed 3 months ago
mlxrunner: grow the prefix trie by whole child nodes so restore points survive resumed prefills
81f9a39 Jesse Gross committed 3 months ago
mlxrunner: page out generated tokens when close records them
30e2891 Jesse Gross committed 3 months ago
mlxrunner: clip prefill captures to their trie node's edge
b315b3e Jesse Gross committed 3 months ago
mlxrunner: settle the draft caches when a prefill is cancelled
c01eafa Jesse Gross committed 3 months ago
server: don't wedge chat and generate on a mid-stream parser error (#17883)
e0c95a5 Daniel Hiltgen committed 3 months ago
ci: plumb temporary MLX patch through to docker stages (#17874)
d1bd15c Daniel Hiltgen committed 3 months ago
Add a model metadata cache to reduce Ollama’s per-request overhead (#17752)
a5165c5 Gaurav Garg committed 3 months ago
renderers/qwen: tolerate non-leading system messages (#17757)
87abaa0 Daniel Hiltgen committed 3 months ago