LLM inference in C/C++
C++ ggml
Commits on master
metal : clamp K extent in tensor API mat-mat kernel for K not a multiple of 32 (#27450)
a302733 Georgi Gerganov committed 3 months ago
opencl: fix q6_K flat mul_mat for Adreno A6x/A7x GPUs with older E031 compilers (#26476)
6503355 Hongqiang Wang committed 3 months ago
vulkan: FA MMQ should use fp32 for Q quantization calculations (#27413)
78ec4c3 Jeff Bolz committed 3 months ago
metal : dequant kv cache only for large batches (#27438)
63b64a5 Georgi Gerganov committed 3 months ago
CI: Use LLVM's OpenMP over MSVC_DEBUG_non_redist on Windows (#26678)
bf0040e Oliver Simons committed 3 months ago
server: (router) lazy-load startup_models after main setup (#27424)
9855ad6 Xuan-Son Nguyen committed 3 months ago
server : fix --docker-repo being treated as router mode (#27416)
8a832e4 Aritro Bandyopadhyay committed 3 months ago
CUDA: adding switch points per HW and quant type to tune the mvq->MMQ decode crossover (#26079)
2b56210 Pranesh Gonegandla committed 3 months ago
common : gracefully fallback on unsupported regex patterns in JSON schema (#26939)
dc64a16 Aldehir Rojas committed 3 months ago
metal : dequantize quantized KV to F16 before flash attention (#27390)
70aff25 Georgi Gerganov committed 3 months ago
Revert "tensor-split meta backend fixes (#26502)" (#27433)
f20395d Georgi Gerganov committed 3 months ago
ggml: fix backend split scheduler race condition (#26040)
8497981 Ruben Ortlam committed 3 months ago
convert: fix get block count error for Nemotron 3 Ultra (#27101)
a3b1eff Rock Chen committed 3 months ago
ggml-cuda: provide static workspace for cuBLAS handles (#26574)
d9b6be0 Alexander Heisler committed 3 months ago
graph : create V as a view of K in the k_iswa build_attn (#27392)
929d47a Georgi Gerganov committed 3 months ago
spec : avoid binding reference to null pointer (#27404)
f466cfa Georgi Gerganov committed 3 months ago
opencl: make the MoE expert scatter deterministic (#26464)
9ee9fc0 Hongqiang Wang committed 3 months ago
hexagon: fix FA HMX queue ordering and pack the rescale D matrices (#27042)
990e3bf Yiwei Shao committed 3 months ago
opencl: port fused ssm_scan kernel (Mamba-2, d_state in {128, 256}) to GPU (#26439)
b062ba7 Hongqiang Wang committed 3 months ago
server: refactor sleep handling, allow access /metrics during sleep (#27376)
947fd9b Xuan-Son Nguyen committed 3 months ago
server : make models endpoints private when authentication is enabled (#26347)
ee0ea03 s0mecode committed 3 months ago
vulkan: add null checks in ggml_vk_queue_command_pools_cleanup (#27353)
b95502b Jetson Tan committed 3 months ago