LLM inference in C/C++
C++ ggml
Commits on master
scripts : use HF_TOKEN when downloading UI assets (#25280)
4871961 Adrien Gallouët committed 4 months ago
ggml : fix tensor-parallel + -ncmoe crash on MoE models (#25028)
4b2a0cd liminfei-amd committed 4 months ago
ggml: Update VMM Pool allocation ggml-cuda.cu - Turing P2P access fix (fixes #24489) (#24491)
7a63fde Vexxie committed 4 months ago
cuda : concat implementation for quantized types (#25303)
78d2f52 fairydreaming committed 4 months ago
llama : add guard for K/V rotation input when buffer is unallocated (#25215)
a410713 liminfei-amd committed 4 months ago
ui: add sync blocks so display/behavior settings can be set via --ui-config-file (#25132)
6658925 Pascal committed 4 months ago
ggml : fix broken CPU concat implementation for quantized types (#25247)
ef2d770 fairydreaming committed 4 months ago
chat: trim messages sent to StepFun parser (fixes long reasoning loops) (#25238)
2d97363 Piotr Wilkin (ilintar) committed 4 months ago
ui: strip path and weight extension from model id in single model mode (#25137)
f113e02 Pascal committed 4 months ago
cuda: enable topk-moe fusion for 288 experts (#25267)
75a48a9 Piotr Wilkin (ilintar) committed 4 months ago
ui: align persisted config with strict server schema and enable thinking by default (#25242)
067de93 Pascal committed 4 months ago
server + ui: ping silent SSE streams every 1s and kick only after 3s so slow prefill never drops healthy connections (#25241)
b5315e1 Pascal committed 4 months ago
ui: Add MCP Servers Opt-In for first time visitors (#25239)
9487528 Aleksander Grygier committed 4 months ago
Remove redundant CUDA copies after gated_delta_net. (#23940)
5a460de Gaurav Garg committed 4 months ago
vendor : update cpp-httplib to 0.49.0 (#25218)
c8ae9a7 Alessandro de Oliveira Faria (A.K.A.CABELO) committed 4 months ago
opencl: allow loading precompiled binary kernels from library (#23042)
4fc4ec5 lhez committed 4 months ago
hexagon: flash attention rework (optimizations, accuracy improvements, etc) (#25085)
13e6738 Max Krasnyansky committed 4 months ago
CUDA: consistent use of __restrict__ + PDL for FA (#25185)
b820cc8 Johannes Gäßler committed 4 months ago